> We are very interested in these new operations and are moving in theWe specifically asked for the masked operations.
> direction of tightly integrating RDMA along with atomics (if
> available) into Oracle. We plan on testing some early prototypes of
> the these in the few months.
And you need the ConnectX-only masked atomics? Or do the standard IB
atomic operations work for you? Of course using atomics at all means
that things don't work on iWARP.
> Send with invalidate is an exact match for our current RDS V3 rdmaThe model we implement is based on "use once" keys - we issue the key to the rdma server and want to toss it as soon as the rdma is complete. Today, we explicitly free the key after the rdma completes and we get a message from the rdma server - saying rdma is complete. If the key is auto invalidated by the recv'ing HCA then we do not need to do it in the driver... which also meanswe do not need to issue the sync tpts to force the HCA to be update its cache.
> driver - and should be more efficient than the current background
> syncing of the tpt to ensure keys are invalidated.
How does send with invalidate interact with the current IB FMR stuff?
Seems that you would run into trouble keeping the state of the FMR
straight if the remote side is invalidating them.
Also I would think that send-with-invalidate would be much moreThis is the one piece we do not know - our plans are to test this and see where the trade offs are. We will keep the current design / implementation to run over NICs that do not support send-with-invalidate.
expensive than the current FMR method of batching up the invalidates,
since you don't get to amortize the cost of syncing up all the internal
HCA state.
- R.--