Re: RFC: Memory Tiering Kernel Interfaces (v2)

[Alistair Popple]

Wei Xu <weixugc@xxxxxxxxxx> writes:

> On Tue, May 24, 2022 at 6:27 AM Aneesh Kumar K.V
> <aneesh.kumar@xxxxxxxxxxxxx> wrote:
>>
>> Wei Xu <weixugc@xxxxxxxxxx> writes:
>>
>> > On Wed, May 18, 2022 at 5:00 AM Jonathan Cameron
>> > <Jonathan.Cameron@xxxxxxxxxx> wrote:
>> >>
>> >> On Wed, 18 May 2022 00:09:48 -0700
>> >> Wei Xu <weixugc@xxxxxxxxxx> wrote:
>>
>> ...
>>
>> > Nice :)
>> >>
>> >> Initially I thought this was over complicated when compared to just leaving space, but
>> >> after a chat with Hesham just now you have us both convinced that this is an elegant solution.
>> >>
>> >> Few corners probably need fleshing out:
>> >> *  Use of an allocator for new tiers. Flat number at startup, or new one on write of unique
>> >>    value to set_memtier perhaps?  Also whether to allow drivers to allocate (I think
>> >>    we should).
>> >> *  Multiple tiers with same rank.  My assumption is from demotion path point of view you
>> >>    fuse them (treat them as if they were a single tier), but keep them expressed
>> >>    separately in the sysfs interface so that the rank can be changed independently.
>> >> *  Some guidance on what values make sense for given rank default that might be set by
>> >>    a driver. If we have multiple GPU vendors, and someone mixes them in a system we
>> >>    probably don't want the default values they use to result in demotion between them.
>> >>    This might well be a guidance DOC or appropriate set of #define
>> >
>> > All of these are good ideas, though I am afraid that these can make
>> > tier management too complex for what it's worth.
>> >
>> > How about an alternative tier numbering scheme that uses major.minor
>> > device IDs?  For simplicity, we can just start with 3 major tiers.
>> > New tiers can be inserted in-between using minor tier IDs.
>>
>>
>> What drives the creation of a new memory tier here?  Jonathan was
>> suggesting we could do something similar to writing to set_memtier for
>> creating a new memory tier.
>>
>> $ echo "memtier128" > sys/devices/system/node/node1/set_memtier
>>
>> But I am wondering whether we should implement that now. If we keep
>> "rank" concept and detach tier index (memtier0 is the memory tier with
>> index 0) separate from rank, I assume we have enough flexibility for a
>> future extension that will allow us to create a memory tier from userspace
>> and assigning it a rank value that helps the device to be placed before or
>> after DRAM in demotion order.
>>
>> ie, For now we will only have memtier0, memtier1, memtier2. We won't add
>> dynamic creation of memory tiers and the above memory tiers will have
>> rank value 0, 1, 2 according with demotion order 0 -> 1 -> 2.
>
> Great. So the consensus is to go with the "rank" approach.  The above
> sounds good to me as a starting point.

The rank approach seems good to me too.

 - Alistair

>> -aneesh