Re: [RFC PATCH rdma-next 08/10] RDMA/rxe: Implement flush execution in responder side
From: Jason Gunthorpe
Date: Wed Jan 05 2022 - 19:35:37 EST
On Thu, Dec 30, 2021 at 09:32:06PM -0500, Tom Talpey wrote:
> The global visibility is oriented toward supporting distributed
> shared memory workloads, or publish/subscribe on high scale systems.
> It's mainly about ensuring that all devices and CPUs see the data,
> something ordinary RDMA Write does not guarantee. This often means
> flushing write pipelines, possibly followed by invalidating caches.
Isn't that what that new ATOMIC_WRITE does? Why do I need to flush if
ATOMIC WRITE was specified as a release? All I need to do is acquire
on the ATOMIC_WRITE site and I'm good?
So what does FLUSH do here, and how does a CPU 'acquire' against this
kind of flush? The flush doesn't imply any ordering right, so how is
it useful on its own?
The write to persistance aspect I understand, but this notion of
global viability seems peculiar.
> Well, higher level wrappers may signal errors, for example if they're
> not available or unreliable, and you will need to handle them. Also,
> they may block. Is that ok in this context?
I'm not sure we have any other tools here beyond a release barrier
like wmb() or the special pmem cache flush. Neither are blocking or
can fail.
In pmem systems storage failure is handled via the memory failure
stuff asynchronously.
Jason