Re: Phyr Starter

From: Christoph Hellwig
Date: Thu Jan 20 2022 - 08:56:11 EST


On Mon, Jan 10, 2022 at 08:41:26PM -0400, Jason Gunthorpe wrote:
> > Finally, it may be possible to stop using scatterlist to describe the
> > input to the DMA-mapping operation. We may be able to get struct
> > scatterlist down to just dma_address and dma_length, with chaining
> > handled through an enclosing struct.
>
> Can you talk about this some more? IMHO one of the key properties of
> the scatterlist is that it can hold huge amounts of pages without
> having to do any kind of special allocation due to the chaining.
>
> The same will be true of the phyr idea right?

No special allocations as in no vmalloc? The chaining still has to
allocate memory using a mempool.

Anyway, to explain my idea which is very similar but not identical to
the one willy has:

- on the input side to dma mapping the bio_vecs (or phyrs) are chained
as bios or whatever the containing structure is. These already exist
and have infrastructure at least in the block layer
- on the output side I plan for two options:

1) we have a sane IOMMU and everyting will be coalesced into a
single dma_range. This requires setting the block layer
merge boundary to match the IOMMU page size, but that is
a very good thing to do anyway.
2) we have no IOMMU (or a weird one) and get one output dma_range
per input bio_vec. We'd eithe have to support chaining or use
vmalloc or huge numbers of entries.

> If you limit to that scenario then we can be more optimal because
> things like byte granular offsets and size in the interior pages don't
> need to exist. Every interior chunk is always aligned to its order and
> we only need to record the order.

The block layer does not small offsets. Direct I/O can often be
512 byte aligned, and some other passthrough commands can have even
smaller alignment, although I don't think we ever go below 4-byte
alignment anywhere in the block layer.

> IMHO storage density here is quite important, we end up having to keep
> this stuff around for a long time.

If we play these tricks it won't be general purpose enough to get rid
of the existing scatterlist usage.