Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
From: Bharata B Rao
Date: Mon Feb 13 2023 - 23:55:42 EST
On 13-Feb-23 12:00 PM, Huang, Ying wrote:
>> I have a microbenchmark where two sets of threads bound to two
>> NUMA nodes access the two different halves of memory which is
>> initially allocated on the 1st node.
>>
>> On a two node Zen4 system, with 64 threads in each set accessing
>> 8G of memory each from the initial allocation of 16G, I see that
>> IBS driven NUMA balancing (i,e., this patchset) takes 50% less time
>> to complete a fixed number of memory accesses. This could well
>> be the best case and real workloads/benchmarks may not get this much
>> uplift, but it does show the potential gain to be had.
>
> Can you find a way to show the overhead of the original implementation
> and your method? Then we can compare between them? Because you think
> the improvement comes from the reduced overhead.
Sure, will measure the overhead.
>
> I also have interest in the pages migration throughput per second during
> the test, because I suspect your method can migrate pages faster.
I have some data on pages migrated over time for the benchmark I mentioned
above.
Pages migrated vs Time(s)
2500000 +---------------------------------------------------------------+
| + + + + + + + |
| Default ******* |
| IBS ####### |
| |
| ****************************|
| * |
2000000 |-+ * +-|
| * |
| ** |
P | * ## |
a | *### |
g | **# |
e 1500000 |-+ *## +-|
s | ## |
| # |
m | # |
i | *# |
g | *# |
r | ## |
a 1000000 |-+ # +-|
t | # |
e | #* |
d | #* |
| # * |
| # * |
500000 |-+ # * +-|
| # * |
| # * |
| # * |
| ## * |
| # * |
| # + * + + + + + + |
0 +---------------------------------------------------------------+
0 20 40 60 80 100 120 140 160
Time (s)
So acting upon the relevant accesses early enough seem to result in
pages migrating faster in the beginning.
Here is the actual data in case the above ascii graph gets jumbled up:
numa_pages_migrated vs time in seconds
======================================
Time Default IBS
---------------------------
5 2639 511
10 2639 17724
15 2699 134632
20 2699 253485
25 2699 386296
30 159805 524651
35 450678 667622
40 741762 811603
45 971848 950691
50 1108475 1084537
55 1246229 1215265
60 1385920 1336521
65 1508354 1446950
70 1624068 1544890
75 1739311 1629162
80 1854639 1700068
85 1979906 1759025
90 2099857 <end>
95 2099857
100 2099857
105 2099859
110 2099859
115 2099859
120 2099859
125 2099859
130 2099859
135 2099859
140 2099859
145 2099859
150 2099859
155 2099859
160 2099859
Regards,
Bharata.