Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing

[Bharata B Rao]

On 2/8/2023 11:42 PM, Dave Hansen wrote:
> On 2/8/23 10:03, Peter Zijlstra wrote:
>>> - Hardware provided access information could be very useful for driving
>>>   hot page promotion in tiered memory systems. Need to check if this
>>>   requires different tuning/heuristics apart from what NUMA balancing
>>>   already does.
>> I think Huang Ying looked at that from the Intel POV and I think the
>> conclusion was that it doesn't really work out. What you need is
>> frequency information, but the PMU doesn't really give you that. You
>> need to process a *ton* of PMU data in-kernel.
> 
> Yeah, there were two big problems.
> 
> First, IIRC, Intel PEBS at the time only gave guest virtual addresses in
> the PEBS records.  They had to be translated back to host addresses to
> be usable.  That was extra expensive.

Just to be clear, I am using IBS in host only and it can give both virtual
and physical address.

> 
> Second, it *did* take a lot of processing to turn raw memory accesses
> into actionable frequency data.  That meant that we started in a hole
> performance-wise and had to make *REALLY* good decisions about page
> migration to make up for it.

I touched upon the frequency aspect in reply to Peter, but please let
me know if I am missing something.

> 
> The performance data here don't look awful, but they don't seem to add
> up to a clear win either.  I'm having a hard time imagining who would
> turn this on and how widely it would get used in practice.

I am hopeful with more appropriate tuning of NUMA balancing logic to work
with hardware-provided access info (as against scan based NUMA hint faults),
we should be able to see a clear win. At least theoretically we wouldn't
have the overheads of address space scanning and hint faults handling.

Thanks for your inputs.

Regards,
Bharata.