Re: [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore

From: Ming Lei

Date: Wed Nov 18 2026 - 21:16:19 EST


On Wed, Nov 18, 2026 at 08:02:33PM -0500, Qiliang Yuan wrote:
> On Tue, Nov 17, 2026 at 07:40:18PM -0600, Ming Lei wrote:
> > Can you share your setup? It is supposed that you run fio from VM,
> > otherwise you can just compare ublk with scsi backed with tcmu.
> >
> > But if you run fio from VM, I believe you can setup virtio-scsi backed by
> > ublk block device directly, which should cover your requirement, or maybe
> > I miss something?
>
> Everything runs in one 32 CPU guest. For tcm_loop and iSCSI (16
> sessions over 127.0.0.1), fio runs in that guest on the initiator
> disks. For vhost-scsi, fio runs in a nested 16 vCPU KVM guest on the
> vhost-scsi disk, with the target and the backstore server in the outer
> guest. In the ublk runs, kublk serves the file as /dev/ublkbN and
> target_core_iblock exports that through the same fabric.
>
> You are not missing anything. This backstore has block semantics: the
> kernel emulates the SCSI commands, PR and ALUA included, exactly as for
> iblock, so ublk behind target_core_iblock offers the same function on
> every fabric, and for a VM a virtio-scsi or vhost-scsi disk on top of
> the ublk device covers it as well. What the series adds over that is
> only performance, and outside tcm_loop the numbers I sent do not show a
> consistent gain.

But in your last reply, not see the setup(virtio-scsi over ublk) and
perf number/compare with your scsi:target optimization.

>
> [...]
>
> > My question is actually why there is regression wrt. 4k randread:
> >
> > workload TCM/USER (MB/s) uring (MB/s) change
> > 4k randread, 1 job 426 (409-463) 369 (366-384) -13%
> >
> > Maybe you can focus on single job perf only, which is easy to compare and
> > investigate.
>
> It comes from the scheduler moving fio between CPUs. The pwrite() I
> pointed to in my last mail is not it: doing that copy inside
> io_uring_enter() instead makes no difference.
>
> tcm_loop queues the LIO submission and completion work on the CPU that
> submitted the command, so with one job, fio's CPU always has bound
> kworkers and ksoftirqd runnable next to fio. The load balancer cannot
> move those and moves fio instead. This backstore gives it more chances:
> the completion work on fio's CPU handles 14 commands per run here
> against 24 with TCM/USER, so per 1000 I/Os the completion kworker and
> ksoftirqd run about 57 times against 35, fio is preempted as often, and
> it is moved two to three times as often. With balancing on, fio's own
> CPU time per I/O doubles here, from 3.1 to 6.3 us, against 3.4 to 4.9 us
> with TCM/USER, spread over the whole submission path rather than in one
> function.

That is one disadvantage for running IO from kernel space, and it is also one
reason why ioutgt ( https://github.com/io-target/ioutgt ) performs much better
than nvmet, especially for nvmet/tcp, not mention ioutgt doesn't implement
io_uring rxzc yet, and there are also other network optimizations available,
all can be applied flexibly.

>
> Turning load balancing off for fio's CPUs and nothing else reverses the
> result. Single job, 4k randread at iodepth 32 over tcm_loop, 6 s per run,
> five alternating runs, median (min-max). fio runs in a cpuset of CPUs
> 4-31 and the server threads are pinned to CPUs 0-3; the cpuset is either
> a normal member (balanced) or an isolated partition (not balanced):
>
> fio CPUs 4-31, backstore bandwidth (MB/s) change
> balanced, TCM/USER 439 (398-462)
> balanced, uring 379 (371-388) -13.7%
> not balanced, TCM/USER 662 (544-710)
> not balanced, uring 760 (575-798) +14.8%
>
> Booting the guest with idle=poll removes the gap as well: with fio free
> to move, five runs give a median of 612 MB/s with TCM/USER and 627 MB/s
> here.
>
> That points at what each switch costs in a guest. An idle vCPU executes
> HLT, which exits to the host and puts the vCPU thread to sleep; waking
> it for a kworker, ksoftirqd or a migrated fio takes an IPI, a host
> wakeup and a VM entry, and the host may run the thread on another
> physical CPU, so the guest code then runs with cold caches. With
> idle=poll the vCPUs spin instead, stay on their physical CPUs and take
> the IPI without an exit. This backstore makes more of those switches
> per I/O, so it pays more of that cost, and fio's slowdown spread over
> the whole submission path looks like cold caches.
>
> What is measured is that idle=poll removes the gap and that the gap
> follows the switches and migrations. Which part of the cost is the exit
> and wakeup latency and which part is the vCPU thread landing on another
> physical CPU is inferred: I cannot trace the host side here. Waking an
> idle CPU is much cheaper on bare metal, so I expect the gap to be
> smaller there, but I have not measured it. idle=poll is only a
> diagnostic, it burns a host CPU per idle vCPU.
>
> [...]
>
> > When userspace target becomes more efficient, I'd suggest you to think
> > about why these applications can't be improved or refactored for the
> > potential (bigger)perf requirement.
>
> Agreed. Since this backstore only takes block commands, an application
> that can move to it can just as well move to ublk, and one that needs
> SCSI semantics in userspace is not served by it either.

Why? SCSI semantics can be served in userspace, just like existing
virtio-scsi user backend.

Such as, uring-scsi may be supported in guest IO emulation application(qemu),
or one uring-virtqueue bridge, the later may be done in both userspace or
vhost_scsi, I guess.

Thanks,
Ming