[PATCH 6/6] memcg: fix performance of mem_cgroup_begin_update_page_stat()

From: KAMEZAWA Hiroyuki
Date: Sun Feb 05 2012 - 22:14:47 EST

mem_cgroup_begin_update_page_stat() should be very fast because
it's called very frequently. Now, it needs to look up page_cgroup
and its memcg....this is slow.

This patch adds a global variable to check "a memcg is moving or not".
By this, the caller doesn't need to visit page_cgroup and memcg.
This makes performance quicker.

Here is a test result. A test program makes page faults onto a file,
MAP_SHARED and makes each page's page_mapcount(page) > 1, and free
the range by madvise() and page fault again. This program causes
26214400 times of page fault onto a file(size was 1G.) and shows
shows the cost of mem_cgroup_begin_update_page_stat().
(please see 'sys' time and think error range is 0.2 - 0.4sec.)

Before the series of patch (linux-next Feb/06)
[kamezawa@bluextal test]$ time ./mmap 1G

real 0m23.534s
user 0m5.949s
sys 0m17.563s

After the patch for mem_cgroup_begin_update_page_stat()
[kamezawa@bluextal test]$ time ./mmap 1G

real 0m24.268s
user 0m6.059s
sys 0m18.187s

After this patch
[kamezawa@bluextal test]$ time ./mmap 1G

real 0m23.094s
user 0m6.045s
sys 0m17.030s

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@xxxxxxxxxxxxxx>
include/linux/memcontrol.h | 5 +++--
mm/memcontrol.c | 6 +++++-
2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 3f3ef33..3df9979 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -143,7 +143,7 @@ static inline bool mem_cgroup_disabled(void)

void __mem_cgroup_begin_update_page_stat(struct page *page,
bool *lock, unsigned long *flags);
+extern atomic_t memcg_moving;
static inline void mem_cgroup_begin_update_page_stat(struct page *page,
bool *lock, unsigned long *flags)
@@ -151,7 +151,8 @@ static inline void mem_cgroup_begin_update_page_stat(struct page *page,
*lock = false;
- return __mem_cgroup_begin_update_page_stat(page, lock, flags);
+ if (atomic_read(&memcg_moving))
+ return __mem_cgroup_begin_update_page_stat(page, lock, flags);

void __mem_cgroup_end_update_page_stat(struct page *page, bool *lock,
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index eedabee..322e381 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1273,16 +1273,20 @@ int mem_cgroup_swappiness(struct mem_cgroup *memcg)
* start move here.

+atomic_t memcg_moving __read_mostly;
static void mem_cgroup_start_move(struct mem_cgroup *memcg)
+ atomic_inc(&memcg_moving);

static void mem_cgroup_end_move(struct mem_cgroup *memcg)
- if (memcg)
+ if (memcg) {
+ atomic_dec(&memcg_moving);
+ }
* 2 routines for checking "mem" is under move_account() or not.

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/