Re: [RFC 1/2] virtio_net: timeout control virtqueue commands
From: Eugenio Perez Martin
Date: Fri Nov 20 2026 - 05:58:57 EST
On Mon, Mar 16, 2026 at 10:26 AM Eugenio Perez Martin
<eperezma@xxxxxxxxxx> wrote:
>
> On Fri, Mar 13, 2026 at 1:06 PM Michael S. Tsirkin <mst@xxxxxxxxxx> wrote:
> >
> > On Fri, Mar 13, 2026 at 12:32:18PM +0100, Eugenio Perez Martin wrote:
> > > On Fri, Mar 13, 2026 at 12:31 PM Eugenio Perez Martin
> > > <eperezma@xxxxxxxxxx> wrote:
> > > >
> > > > On Sun, Nov 9, 2025 at 10:47 PM Michael S. Tsirkin <mst@xxxxxxxxxx> wrote:
> > > > >
> > > > > On Wed, Nov 05, 2025 at 10:02:48AM +0100, Eugenio Perez Martin wrote:
> > > > > > On Tue, Oct 28, 2025 at 3:57 PM Eugenio Perez Martin
> > > > > > <eperezma@xxxxxxxxxx> wrote:
> > > > > > >
> > > > > > > On Tue, Oct 28, 2025 at 3:42 PM Michael S. Tsirkin <mst@xxxxxxxxxx> wrote:
> > > > > > > >
> > > > > > > > On Tue, Oct 28, 2025 at 03:37:09PM +0100, Eugenio Perez Martin wrote:
> > > > > > > > > On Tue, Oct 28, 2025 at 3:10 PM Michael S. Tsirkin <mst@xxxxxxxxxx> wrote:
> > > > > > > > > >
> > > > > > > > > > On Wed, Oct 22, 2025 at 02:55:18PM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > On Wed, Oct 22, 2025 at 1:43 PM Michael S. Tsirkin <mst@xxxxxxxxxx> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Oct 22, 2025 at 12:50:53PM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > > Let me switch to MQ as I think it illustrates the point better.
> > > > > > > > > > > > >
> > > > > > > > > > > > > IIUC the workflow:
> > > > > > > > > > > > > a) virtio-net sends MQ_VQ_PAIRS_SET 2 to the device
> > > > > > > > > > > > > b) VDUSE CVQ sends ok to the virtio-net driver
> > > > > > > > > > > > > c) VDUSE CVQ sends the command to the VDUSE device
> > > > > > > > > > > > > d) Now the virtio-net driver sends virtio-net sends MQ_VQ_PAIRS_SET 1
> > > > > > > > > > > > > e) VDUSE CVQ sends ok to the virtio-net driver
> > > > > > > > > > > > >
> > > > > > > > > > > > > The device didn't process the MQ_VQ_PAIRS_SET 1 command at this point,
> > > > > > > > > > > > > so it potentially uses the second rx queue. But, by the standard:
> > > > > > > > > > > > >
> > > > > > > > > > > > > The device MUST NOT queue packets on receive queues greater than
> > > > > > > > > > > > > virtqueue_pairs once it has placed the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET
> > > > > > > > > > > > > command in a used buffer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So the driver does not expect rx buffers on that queue at all. From
> > > > > > > > > > > > > the driver's POV, the device is invalid, and it could mark it as
> > > > > > > > > > > > > broken.
> > > > > > > > > > > >
> > > > > > > > > > > > ok intresting. Note that if userspace processes vqs it should process
> > > > > > > > > > > > cvq too. I don't know what to do in this case yet, I'm going on
> > > > > > > > > > > > vacation, let me ponder this a bit.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Sure.
> > > > > > > > > >
> > > > > > > > > > So let me ask you this, how are you going to handle device reset?
> > > > > > > > > > Same issue, it seems to me.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Well my proposal is to mark it as broken so it needs to be reset
> > > > > > > > > manually.
> > > > > > > >
> > > > > > > >
> > > > > > > > Heh but guest assumes after reset device does not poke at guest
> > > > > > > > memory, and will free up and reuse that memory.
> > > > > > > > If userspace still pokes at it -> plus plus ungood.
> > > > > > > >
> > > > > > >
> > > > > > > I don't get this part. Once the device is reset, the device should not
> > > > > > > poke at guest memory (unless it is malicious or similar). Why would it
> > > > > > > do it?
> > > > > >
> > > > > > Friendly ping.
> > > > >
> > > > >
> > > > >
> > > > > OK I thought about it a bunch. A lot of net drivers actually
> > > > > just queue ethtool commands and finish them asynchronously.
> > > > > Thinkably virtio could expose an API on whether it is safe to
> > > > > wait for buffers to be used. virtio-net would then either
> > > > > send commands directly or do the asynchronous thing.
> > > > > Hmm?
> > > > >
> > > >
> > > > I think it's the best solution too, yes. The series [1] implements for
> > > > set_rx_mode. We can do it for all the CVQ commands and then whitelist
> > > > them in VDUSE code. Am I missing something? Does anyone foresee any
> > > > command that will not be possible to convert?
> > > >
> > >
> > > Adding the [1] link...
> > >
> > > [1] https://lore.kernel.org/netdev/20260112181626.20117-1-viswanathiyyappan@xxxxxxxxx/
> >
> > what's the status of that one?
> >
>
> Still under review.
>
> Aleksandr Loktionov <aleksandr.loktionov@xxxxxxxxx> added the
> Reviewed-by tag to most of patches but he spotted a sync issue in the
> realtek 8139cp driver.
>
> https://lore.kernel.org/lkml/IA3PR11MB8986856F0C1BC5A02F29B32EE540A@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/
Ok, that one is merged now but it still does not solve the rtnl issue
:(. It just moves the driver call to set rx mode to a place where it
can sleep while holding the rtnl.
In the case of the mac address change it actually takes the lock way
earlier, as soon as rtnl_newlink. I'm trying to remove the rtnl when
the CVQ command is issued, but that complicates the scenario quickly.
The plan is to get the callers of
net/core/dev.c:netif_set_mac_address() not have the rtnl taken, only
the device one. And then some polished version of this:
@@ -10223,16 +10223,21 @@ int netif_set_mac_address(struct net_device
*dev, struct sockaddr_storage *ss,
return -EINVAL;
if (!netif_device_present(dev))
return -ENODEV;
+ rtnl_lock();
err = netif_pre_changeaddr_notify(dev, ss->__data, extack);
if (err)
return err;
if (memcmp(dev->dev_addr, ss->__data, dev->addr_len)) {
+ rtnl_unlock();
err = ops->ndo_set_mac_address(dev, ss);
if (err)
return err;
+ rtnl_lock();
}
dev->addr_assign_type = NET_ADDR_SET;
call_netdevice_notifiers(NETDEV_CHANGEADDR, dev);
+ rtnl_unlock();
add_device_randomness(dev->dev_addr, dev->addr_len);
return 0;
}
However syncrhronizing the NETDEV_PRE_CHANGEADDR notifications to the
actual CVQ command seems hard, as new notifiers can be added in the
middle. It checks if any listener objects to the MAC address change.
As far as I can see, only Nvidia's spectrum driver can deny that MAC
address change if there are too many MAC addresses, so maybe we can
just adapt it, but I might be missing something [1].
If my guess is right, setting the MAc address out of rtnl and run the
PRE_CHANGEADDR notifiers again with rtnl taken could be a step
forward. If the notifiers fail (the Nvidia device denies the address
change), we can trigger a rollback of the MAC address outside the
rtnl.
Ccing netdev and Stanislav who added the set_rx_mode_async to check if
this is the right direction or if there are efforts / attempt to make
this happen.
Rx mode, the number of enabled virtqueues, etc., have similar
problems, but maybe we can start with the MAC address.
[1]
$ git grep -n 'NETDEV_PRE_CHANGEADDR'
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9921:
case NETDEV_PRE_CHANGEADDR:
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9949:
case NETDEV_PRE_CHANGEADDR:
drivers/net/ipvlan/ipvlan_main.c:867: case NETDEV_PRE_CHANGEADDR:
include/linux/netdevice.h:3255: NETDEV_PRE_CHANGEADDR,
/* notify before the address change */
net/bridge/br.c:76: case NETDEV_PRE_CHANGEADDR:
net/core/dev.c:10192: * netif_pre_changeaddr_notify() - Call
NETDEV_PRE_CHANGEADDR.
net/core/dev.c:10209: rc =
call_netdevice_notifiers_info(NETDEV_PRE_CHANGEADDR, &info.info);
net/core/lock_debug.c:35: case NETDEV_PRE_CHANGEADDR:
$ git grep -n -A9 'case NETDEV_PRE_CHANGEADDR' net/bridge/br.c
net/bridge/br.c:76: case NETDEV_PRE_CHANGEADDR:
net/bridge/br.c:77: if (br->dev->addr_assign_type ==
NET_ADDR_SET)
net/bridge/br.c:78: break;
net/bridge/br.c:79: prechaddr_info = ptr;
net/bridge/br.c:80: err = netif_pre_changeaddr_notify(br->dev,
net/bridge/br.c:81:
prechaddr_info->dev_addr,
net/bridge/br.c:82: extack);
net/bridge/br.c:83: if (err)
net/bridge/br.c:84: return notifier_from_errno(err);
net/bridge/br.c:85: break;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9736:static int
mlxsw_sp_router_port_pre_changeaddr_event(struct mlxsw_sp_rif *rif,
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9737:
struct netdev_notifier_pre_changeaddr_info *info)
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9738:{
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9739:
struct mlxsw_sp *mlxsw_sp = rif->mlxsw_sp;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9740:
struct mlxsw_sp_rif_mac_profile *profile;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9741:
struct netlink_ext_ack *extack;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9742: u8
max_rif_mac_profiles;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9743: u64 occ;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9744:
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9745:
extack = netdev_notifier_info_to_extack(&info->info);
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9746:
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9747:
profile = mlxsw_sp_rif_mac_profile_find(mlxsw_sp, info->dev_addr);
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9748: if (profile)
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9749:
return 0;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9750:
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9751:
max_rif_mac_profiles = mlxsw_sp->router->max_rif_mac_profile;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9752: occ
= mlxsw_sp_rif_mac_profiles_occ_get(mlxsw_sp);
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9753: if
(occ < max_rif_mac_profiles)
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9754:
return 0;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9755:
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9756: if
(!mlxsw_sp_rif_mac_profile_is_shared(rif))
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9757:
return 0;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9758:
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9759:
NL_SET_ERR_MSG_MOD(extack, "Exceeded number of supported router
interface MA
C profiles");
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9760:
return -ENOBUFS;
drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c:9761:}