Re: vdpa legacy guest support (was Re: [PATCH] vdpa/mlx5: set_features should allow reset to zero)
From: Si-Wei Liu
Date: Tue Dec 14 2021 - 20:05:59 EST
On 12/13/2021 9:06 PM, Michael S. Tsirkin wrote:
On Mon, Dec 13, 2021 at 05:59:45PM -0800, Si-Wei Liu wrote:
On 12/12/2021 1:26 AM, Michael S. Tsirkin wrote:
On Fri, Dec 10, 2021 at 05:44:15PM -0800, Si-Wei Liu wrote:
Sorry for reviving this ancient thread. I was kinda lost for the conclusion
it ended up with. I have the following questions,
1. legacy guest support: from the past conversations it doesn't seem the
support will be completely dropped from the table, is my understanding
correct? Actually we're interested in supporting virtio v0.95 guest for x86,
which is backed by the spec at
https://urldefense.com/v3/__https://ozlabs.org/*rusty/virtio-spec/virtio-0.9.5.pdf__;fg!!ACWV5N9M2RV99hQ!dTKmzJwwRsFM7BtSuTDu1cNly5n4XCotH0WYmidzGqHSXt40i7ZU43UcNg7GYxZg$ . Though I'm not sure
if there's request/need to support wilder legacy virtio versions earlier
beyond.
I personally feel it's less work to add in kernel than try to
work around it in userspace. Jason feels differently.
Maybe post the patches and this will prove to Jason it's not
too terrible?
I suppose if the vdpa vendor does support 0.95 in the datapath and ring
layout level and is limited to x86 only, there should be easy way out.
Note a subtle difference: what matters is that guest, not host is x86.
Matters for emulators which might reorder memory accesses.
I guess this enforcement belongs in QEMU then?
Right, I mean to get started, the initial guest driver support and the
corresponding QEMU support for transitional vdpa backend can be limited
to x86 guest/host only. Since the config space is emulated in QEMU, I
suppose it's not hard to enforce in QEMU. QEMU can drive GET_LEGACY,
GET_ENDIAN et al ioctls in advance to get the capability from the
individual vendor driver. For that, we need another negotiation protocol
similar to vhost_user's protocol_features between the vdpa kernel and
QEMU, way before the guest driver is ever probed and its feature
negotiation kicks in. Not sure we need a GET_MEMORY_ORDER ioctl call
from the device, but we can assume weak ordering for legacy at this
point (x86 only)?
I
checked with Eli and other Mellanox/NVDIA folks for hardware/firmware level
0.95 support, it seems all the ingredient had been there already dated back
to the DPDK days. The only major thing limiting is in the vDPA software that
the current vdpa core has the assumption around VIRTIO_F_ACCESS_PLATFORM for
a few DMA setup ops, which is virtio 1.0 only.
2. suppose some form of legacy guest support needs to be there, how do we
deal with the bogus assumption below in vdpa_get_config() in the short term?
It looks one of the intuitive fix is to move the vdpa_set_features call out
of vdpa_get_config() to vdpa_set_config().
/*
* Config accesses aren't supposed to trigger before features are
set.
* If it does happen we assume a legacy guest.
*/
if (!vdev->features_valid)
vdpa_set_features(vdev, 0);
ops->get_config(vdev, offset, buf, len);
I can post a patch to fix 2) if there's consensus already reached.
Thanks,
-Siwei
I'm not sure how important it is to change that.
In any case it only affects transitional devices, right?
Legacy only should not care ...
Yes I'd like to distinguish legacy driver (suppose it is 0.95) against the
modern one in a transitional device model rather than being legacy only.
That way a v0.95 and v1.0 supporting vdpa parent can support both types of
guests without having to reconfigure. Or are you suggesting limit to legacy
only at the time of vdpa creation would simplify the implementation a lot?
Thanks,
-Siwei
I don't know for sure. Take a look at the work Halil was doing
to try and support transitional devices with BE guests.
Hmmm, we can have those endianness ioctls defined but the initial QEMU
implementation can be started to support x86 guest/host with little
endian and weak memory ordering first. The real trick is to detect
legacy guest - I am not sure if it's feasible to shift all the legacy
detection work to QEMU, or the kernel has to be part of the detection
(e.g. the kick before DRIVER_OK thing we have to duplicate the tracking
effort in QEMU) as well. Let me take a further look and get back.
Meanwhile, I'll check internally to see if a legacy only model would
work. Thanks.
Thanks,
-Siwei
On 3/2/2021 2:53 AM, Jason Wang wrote:
On 2021/3/2 5:47 下午, Michael S. Tsirkin wrote:
On Mon, Mar 01, 2021 at 11:56:50AM +0800, Jason Wang wrote:
On 2021/3/1 5:34 上午, Michael S. Tsirkin wrote:
On Wed, Feb 24, 2021 at 10:24:41AM -0800, Si-Wei Liu wrote:
Detecting it isn't enough though, we will need a new ioctl to notify
the kernel that it's a legacy guest. Ugh :(
Well, although I think adding an ioctl is doable, may I
know what the use
case there will be for kernel to leverage such info
directly? Is there a
case QEMU can't do with dedicate ioctls later if there's indeed
differentiation (legacy v.s. modern) needed?
BTW a good API could be
#define VHOST_SET_ENDIAN _IOW(VHOST_VIRTIO, ?, int)
#define VHOST_GET_ENDIAN _IOW(VHOST_VIRTIO, ?, int)
we did it per vring but maybe that was a mistake ...
Actually, I wonder whether it's good time to just not support
legacy driver
for vDPA. Consider:
1) It's definition is no-normative
2) A lot of budren of codes
So qemu can still present the legacy device since the config
space or other
stuffs that is presented by vhost-vDPA is not expected to be
accessed by
guest directly. Qemu can do the endian conversion when necessary
in this
case?
Thanks
Overall I would be fine with this approach but we need to avoid breaking
working userspace, qemu releases with vdpa support are out there and
seem to work for people. Any changes need to take that into account
and document compatibility concerns.
Agree, let me check.
I note that any hardware
implementation is already broken for legacy except on platforms with
strong ordering which might be helpful in reducing the scope.
Yes.
Thanks