Re: [PATCH 00/26] DCD: Add support for Dynamic Capacity Devices (DCD)

[Jonathan Cameron]

On Wed, 1 May 2024 16:49:24 -0700
Ira Weiny <ira.weiny@xxxxxxxxx> wrote:

> Jonathan Cameron wrote:
> >   
> > > 
> > > Fan Ni's latest v5 of Qemu DCD was used for testing.[2]  
> > Hi Ira, Navneet.  
> > > 
> > > Remaining work:
> > > 
> > > 	1) Integrate the QoS work from Dave Jiang
> > > 	2) Interleave support  
> > 
> > 
> > More flag.  This one I think is potentially important and don't
> > see any handling in here.  
> 
> Nope I admit I missed the spec requirement.
> 
> > 
> > Whilst an FM could in theory be careful to avoid sending a
> > sparse set of extents, if the device is managing the memory range
> > (which is possible all it supports) and the FM issues an Initiate Dynamic
> > Capacity Add with Free (again may be all device supports) then we
> > can't stop the device issuing a bunch of sparse extents.
> > 
> > Now it won't be broken as such without this, but every time we
> > accept the first extent that will implicitly reject the rest.
> > That will look very ugly to an FM which has to poke potentially many
> > times to successfully allocate memory to a host.  
> 
> This helps me to see see why the more bit is useful.
> 
> > 
> > I also don't think it will be that hard to support, but maybe I'm
> > missing something?   
> 
> Just a bunch of code and refactoring busy work.  ;-)  It's not rocket
> science but does fundamentally change the arch again.
> 
> > 
> > My first thought is it's just a loop in cxl_handle_dcd_add_extent()
> > over a list of extents passed in then slightly more complex response
> > generation.  
> 
> Not exactly 'just a loop'.  No matter how I work this out there is the
> possibility that some extents get surfaced and then the kernel tries to
> remove them because it should not have.

Lets consider why it might need to back out.
1) Device sends an invalid set of extents - so maybe one in a later message
   overlaps with an already allocated extent.   Device bug, handling can
   be extremely inelegant - up to crashing the kernel.  Worst that happens
   due to race is probably a poison storm / machine check fun?  Not our
   responsibility to deal with something that broken (in my view!) Best effort
   only.

2) Host can't handle the extent for some reason and didn't know that until
   later - can just reject the ones it can't handle. 

> 
> To be most safe the cxl core is going to have to make 2 round trips to the
> cxl region layer for each extent.  The first determines if the extent is
> valid and creates the extent as much as possible.  The second actually
> surfaces the extents.  However, if the surface fails then you might not
> get the extents back.  So now we are in an invalid state.  :-/  WARN and
> continue I guess?!??!

Yes. Orchestrator can decide how to handle - probably reboot server in as
gentle a fashion as possible.


> 
> I think the safest way to handle this is add a new kernel notify event
> called 'extent create' which stops short of surfacing the extent.  [I'm
> not 100% sure how this is going to affect interleave.]
> 
> I think the safest logic for add is something like:
> 
> 	cxl_handle_dcd_add_event()
> 		add_extent(squirl_list, extent);
> 
> 		if (more bit) /* wait for more */
> 			return;
> 
> 		/* Create extents to hedge the bets against failure */
> 		for_each(squirl_list)
> 			if (notify 'extent create' != ok)
> 				send_response(fail);
> 				return;
> 		
> 		for_each(squirl_list)
> 			if (notify 'surface' != ok)
> 				/*
> 				 * If the more bit was set, some extents
> 				 * have been surfaced and now need to be
> 				 * removed...
> 				 *
> 				 * Try to remove them and hope...
> 				 */

If we failed to surface them all another option is just tell the device
that.   Responds with the extents that successfully surfaced and reject
all others (or all after the one that failed?)  So for the lower layers
send the device a response that says "thanks but I only took these ones"
and for the upper layers pretend "I was only offered these ones"

> 				WARN_ON('surface extents failed');
> 				for_each(squirl_list)
> 					notify 'remove without response'
> 				send_response(fail);
> 				return;
> 
> 		send_response(squirl_list, accept);
> 
> The logic for remove is not changed AFAICS because the device must allow
> for memory to be released at any time so the host is free to release each
> of the extents individually despite the 'more' bit???

Yes, but only after it accepted them - which needs to be done in one go.
So you can't just send releases before that (the device will return an
error and keep them in the pending list I think...)

> 
> > 
> > I don't want this to block getting initial DCD support in but it
> > will be a bit ugly if we quickly support the more flag and then end
> > up with just one kernel that an FM has to be careful with...  
> 
> I'm not sure which is worse.  Given your use case above it seems like the
> more bit may be more important for 'dumb' devices which want to add
> extents in blocks before responding to the FM.  Thus complicating the FM.
> 
> It seems 'smarter' devices which could figure this out (not requiring the
> more bit) are the ones which will be developed later.  So it seems the use
> case time line is the opposite of what we need right now.

Once we hit shareable capacity (which the smarter devices will use) then
this become the dominant approach to non contiguous allocations because
you can't add extents with a given tag in multiple goes.

So I'd expect the more flag to be more common not less over time.
> 
> For that reason I'm inclined to try and get this in.
> 

Great - but I'd not worry too much about bad effects if you get invalid
lists from the device.  If the only option is shout and panic, then fine
though I'd imagine we can do slightly better than that, so maybe warn
extensively and don't let the region be used.

Jonathan

> Ira
>