RE: [PATCH 1/1] swiotlb: Track and report io_tlb_used high water mark in debugfs

[Michael Kelley (LINUX)]

From: Petr Tesarik <petrtesarik@xxxxxxxxxxxxxxx> Sent: Wednesday, March 22, 2023 7:33 AM
> 
> On 2/27/2023 5:41 PM, Michael Kelley wrote:
> > swiotlb currently reports the total number of slabs and the instantaneous
> > in-use slabs in debugfs. But with increased usage of swiotlb for all I/O
> > in Confidential Computing (coco) VMs, it has become difficult to know
> > how much memory to allocate for swiotlb bounce buffers, either via the
> > automatic algorithm in the kernel or by specifying a value on the
> > kernel boot line. The current automatic algorithm generously allocates
> > swiotlb bounce buffer memory, and may be wasting significant memory in
> > many use cases.
> 
> I like the idea. Tracking and exposing the high watermark is definitely a step in the right
> direction. I would also appreciate an indicator of fragmentation, but that can be
> implemented later.
> 
> However, you apparently have a scenario where swiotlb is used intensely. Are you able
> to measure swiotlb performance? If yes, can you suggest a suitable method? I'm asking
> for my work on making swiotlb more flexible (able to grow and shrink).
> 

Yes, in the CoCo VM scenario, all DMA transfers must use bounce buffers
because the hardware doesn't support I/O devices doing DMA to encrypted
memory.  Our measurements have primarily been of disk I/O bandwidth
and latency using 'fio'.  We compare the measurements in a normal VM
vs. a CoCo VM, with the delta mostly being the overhead of doing the
bounce buffering.  The delta includes the cost of swiotlb allocate/free,
plus the data copying.  Admittedly, this is a more macro-level measurement,
but it is directly relevant because it's ultimately the impact that users of
CoCo VMs see.  We have not done micro-level measurements of just the
swiotlb allocate/free functions.

Making the swiotlb able to grow and shrink would likely be very desirable
for CoCo VMs.  I'll look forward to seeing how that comes out. :-)

Michael