Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing

[Huang, Ying]

Bharata B Rao <bharata@xxxxxxx> writes:

> On 17-Feb-23 11:33 AM, Huang, Ying wrote:
>> Bharata B Rao <bharata@xxxxxxx> writes:
>> 
>>> On 14-Feb-23 10:25 AM, Bharata B Rao wrote:
>>>> On 13-Feb-23 12:00 PM, Huang, Ying wrote:
>>>>>> I have a microbenchmark where two sets of threads bound to two 
>>>>>> NUMA nodes access the two different halves of memory which is
>>>>>> initially allocated on the 1st node.
>>>>>>
>>>>>> On a two node Zen4 system, with 64 threads in each set accessing
>>>>>> 8G of memory each from the initial allocation of 16G, I see that
>>>>>> IBS driven NUMA balancing (i,e., this patchset) takes 50% less time
>>>>>> to complete a fixed number of memory accesses. This could well
>>>>>> be the best case and real workloads/benchmarks may not get this much
>>>>>> uplift, but it does show the potential gain to be had.
>>>>>
>>>>> Can you find a way to show the overhead of the original implementation
>>>>> and your method?  Then we can compare between them?  Because you think
>>>>> the improvement comes from the reduced overhead.
>>>>
>>>> Sure, will measure the overhead.
>>>
>>> I used ftrace function_graph tracer to measure the amount of time (in us)
>>> spent in fault handling and task_work handling in both the methods when
>>> the above mentioned benchmark was running.
>>>
>>> 			Default		IBS
>>> Fault handling		29879668.71	1226770.84
>>> Task work handling	24878.894	10635593.82
>>> Sched switch handling			78159.846
>>>
>>> Total			29904547.6	11940524.51
>> 
>> Thanks!  You have shown the large overhead difference between the
>> original method and your method.  Can you show the number of the pages
>> migrated too?  I think the overhead / page can be a good overhead
>> indicator too.
>> 
>> Can it be translated to the performance improvement?  Per my
>> understanding, the total overhead is small compared with total run time.
>
> I captured some of the numbers that you wanted for two different runs.
> The first case shows the data for a short run (less number of memory access
> iterations) and the second one is for a long run (more number of iterations)
>
> Short-run
> =========
> Time taken or overhead (us) for fault, task_work and sched_switch
> handling
>
> 			Default		IBS
> Fault handling		29017953.99	1196828.67
> Task work handling	10354.40	10356778.53
> Sched switch handling			56572.21
> Total overhead		29028308.39	11610179.41
>
> Benchmark score(us)	194050290	53963650
> numa_pages_migrated	2097256		662755
> Overhead / page		13.84		17.51

>From above, the overhead/page is similar.

> Pages migrated per sec	72248.64	57083.95
>
> Default
> -------
> 			Total		Min	Max		Avg
> do_numa_page		29017953.99	0.1	307.63		15.97
> task_numa_work		10354.40	2.86	4573.60		175.50
> Total			29028308.39
>
> IBS
> ---
> 			Total		Min	Max		Avg
> ibs_overflow_handler	1196828.67	0.15	100.28		1.26
> task_ibs_access_work	10356778.53	0.21	10504.14	28.42
> hw_access_sched_in	56572.21	0.15	16.94		1.45
> Total			11610179.41
>
>
> Long-run
> ========
> Time taken or overhead (us) for fault, task_work and sched_switch
> handling
> 			Default		IBS
> Fault handling		27437756.73	901406.37
> Task work handling	1741.66		4902935.32
> Sched switch handling			100590.33
> Total overhead		27439498.38	5904932.02
>
> Benchmark score(us)	306786210.0	153422489.0
> numa_pages_migrated	2097218		1746099
> Overhead / page		13.08		3.38

But from this, the overhead/page is quite different.

One possibility is that there's more "local" hint page faults in the
original implementation, we can check "numa_hint_faults" and
"numa_hint_faults_local" in /proc/vmstat for that.

If

  numa_hint_faults_local / numa_hint_faults

is similar.  For each page migrated, the number of hint page fault is
similar, and the run time for each hint page fault handler is similar
too.  Or I made some mistake in analysis?

> Pages migrated per sec	6836.08		11380.98
>
> Default
> -------
> 			Total		Min	Max		Avg
> do_numa_page		27437756.73	0.08	363.475		15.03
> task_numa_work		1741.66		3.294	1200.71		42.48
> Total			27439498.38
>
> IBS
> ---
> 			Total		Min	Max		Avg
> ibs_overflow_handler	901406.37	0.15	95.51		1.06
> task_ibs_access_work	4902935.32	0.22	11013.68	9.64
> hw_access_sched_in	100590.33	0.14	91.97		1.52
> Total			5904932.02

Thank you very much for detailed data.  Can you provide some analysis
for your data?

Best Regards,
Huang, Ying