Re: [RFC PATCH 00/19] Cleanup and optimise the page allocator V2

[Ingo Molnar]

* Lin Ming <ming.m.lin@xxxxxxxxx> wrote:

> Thanks, I have used "perfstat -s" to collect cache misses 
> data.
> 
> 2.6.29-rc7-tip: tip/perfcounters/core (b5e8acf)
> 2.6.29-rc7-tip-mg2: v2 patches applied to tip/perfcounters/core
> 
> I collected 5 times netperf UDP-U-4k data with and without 
> mg-v2 patches applied to tip/perfcounters/core on a 4p 
> quad-core tigerton machine, as below "value" means UDP-U-4k 
> test result.
> 
> 2.6.29-rc7-tip
> ---------------
> value           cache misses    CPU migrations  cachemisses/migrations
> 5329.71          391094656       1710            228710
> 5641.59          239552767       2138            112045
> 5580.87          132474745       2172            60992
> 5547.19          86911457        2099            41406
> 5626.38          196751217       2050            95976
> 
> 2.6.29-rc7-tip-mg2
> -------------------
> value           cache misses    CPU migrations  cachemisses/migrations
> 4749.80          649929463       1132            574142
> 4327.06          484100170       1252            386661
> 4649.51          374201508       1489            251310
> 5655.82          405511551       1848            219432
> 5571.58          90222256        2159            41788
> 
> Lin Ming

Hm, these numbers look really interesting and give us insight 
into this workload. The workload is fluctuating but by measuring 
3 metrics at once instead of just one we see the following 
patterns:

 - Less CPU migrations means more cache misses and less 
   performance.

The lowest-score runs had the lowest CPU migrations count, 
coupled with a high amount of cachemisses.

This _probably_ means that in this workload migrations are 
desired: the sooner two related tasks migrate to the same CPU 
the better. If they stay separate (migration count is low) then 
they interact with each other from different CPUs, creating a 
lot of cachemisses and reducing performance.

You can reduce the migration barrier of the system by enabling 
CONFIG_SCHED_DEBUG=y and setting sched_migration_cost to zero:

   echo 0 > /proc/sys/kernel/sched_migration_cost

This will hurt other workloads - but if this improves the 
numbers then it proves that what this particular workload wants 
is easy migrations.

Now the question is, why does the mg2 patchset reduce the number 
of migrations? It might not be an inherent property of the mg2 
patches: maybe just unlucky timings push the workload across 
sched_migration_cost.

Setting sched_migration_cost to either zero or to a very high 
value and repeating the test will eliminate this source of noise 
and will tell us about other properties of the mg2 patchset.

There might be other effects i'm missing. For example what kind 
of UDP transport is used - localhost networking? That means that 
sender and receiver really wants to be coupled strongly and what 
controls this workload is whether such a 'pair' of tasks can 
properly migrate to the same CPU.

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/