Re: Variation in measure_migration_cost() with scheduler-cache-hot-autodetect.patch in -mm

From: Ingo Molnar
Date: Wed Jun 22 2005 - 03:34:44 EST

* Chen, Kenneth W <kenneth.w.chen@xxxxxxxxx> wrote:

> I'm consistently getting a smaller than expected cache migration cost
> as measured by Ingo's scheduler-cache-hot-autodetect.patch currently
> in -mm tree. In this patch, the memory used to calibrate migration
> cost is obtained by vmalloc call. Would it make sense to use
> __get_free_pages() instead? I did the following experiments on a
> variety of machines I have access to:
> migration cost migration cost
> with vmalloc mem with __get_free_pages
> 3.0GHz Xeon, 8MB cache 6.23 ms 6.32 ms
> 3.4GHz Xeon, 2MB cache 1.62 ms 2.00 ms
> 1.6GHz Itanium2, 9MB 9.2 ms 10.2 ms
> 1.4GHz Itanium2, 4MB 4.2 ms 4.4 ms
> Why the discrepancy? Possible cache coloring issue?

probably coloring effects, yes. Another reason could be that
touch_cache() touches 6 separate areas of memory, which combined with
the stack give a minimum of 7 hugepage TLBs. How many are there in these
Xeons? If there are say 4 of them then we could be trashing these TLB
entries. There are much more 4K TLBs. To reduce the number of TLBs
utilized, could you change touch_cache() to do something like:

unsigned long size = __size/sizeof(long), chunk1 = size/2;
unsigned long *cache = __cache;
int i;

for (i = 0; i < size/4; i += 8) {
switch (i % 4) {
case 0: cache[i]++;
case 1: cache[size-1-i]++;
case 2: cache[chunk1-i]++;
case 3: cache[chunk1+i]++;

does this change the migration-cost values? Btw., how did you determine
the value of the 'ideal' migration cost? Was this based on the database
benchmark measurements?

There are a couple of reasons vmalloc() is better than gfp(): 1) it has
no size limit in the measured range, and 2) it more accurately mimics
migration costs of userspace apps, which typically have most of their
cache-footprint in paged memory, not in hugepage memory.

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at