Re: [PATCH net] net: core: orphan frags before queuing to slow qdisc

From: Eric Dumazet
Date: Fri Jan 17 2014 - 09:28:31 EST

On Fri, 2014-01-17 at 17:42 +0800, Jason Wang wrote:
> Many qdiscs can queue a packet for a long time, this will lead an issue
> with zerocopy skb. It means the frags will not be orphaned in an expected
> short time, this breaks the assumption that virtio-net will transmit the
> packet in time.
> So if guest packets were queued through such kind of qdisc and hit the
> limitation of the max pending packets for virtio/vhost. All packets that
> go to another destination from guest will also be blocked.
> A case for reproducing the issue:
> - Boot two VMs and connect them to the same bridge kvmbr.
> - Setup tbf with a very low rate/burst on eth0 which is a port of kvmbr.
> - Let VM1 send lots of packets thorugh eth0
> - After a while, VM1 is unable to send any packets out since the number of
> pending packets (queued to tbf) were exceeds the limitation of vhost/virito

So whats the problem ? If the limit is low, you cannot sent packets.

Solution : increase the limit, or tell the vm to lower its rate.

Oh wait, are you bitten because you did some prior skb_orphan() to allow
the vm to send unlimited number of skbs ???

> Solve this issue by orphaning the frags before queuing it to a slow qdisc (the
> one without TCQ_F_CAN_BYPASS).

Why orphaning the frags only solves the problem ? A skb without zerocopy
frags should also be blocked for a while.

Seriously, lets admit this zero copy stuff is utterly broken.

TCQ_F_CAN_BYPASS is not enough. Some NIC have separate queues with
strict priorities.

It seems to me that you are pushing to use FIFO (the only qdisc setting
TCQ_F_CAN_BYPASS), by adding yet another test in fast path (I do not
know how we can still call it a fast path), while we already have smart
qdisc to avoid the inherent HOL and unfairness problems of FIFO.

> Cc: Michael S. Tsirkin <mst@xxxxxxxxxx>
> Signed-off-by: Jason Wang <jasowang@xxxxxxxxxx>
> ---
> net/core/dev.c | 7 +++++++
> 1 file changed, 7 insertions(+)
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 0ce469e..1209774 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -2700,6 +2700,12 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
> contended = qdisc_is_running(q);
> if (unlikely(contended))
> spin_lock(&q->busylock);
> + if (!(q->flags & TCQ_F_CAN_BYPASS) &&
> + unlikely(skb_orphan_frags(skb, GFP_ATOMIC))) {
> + kfree_skb(skb);
> + rc = NET_XMIT_DROP;
> + goto out;
> + }

Are you aware that copying stuff takes time ?

If yes, why is it done after taking the busylock spinlock ?

> spin_lock(root_lock);
> if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) {
> @@ -2739,6 +2745,7 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
> }
> }
> spin_unlock(root_lock);
> +out:
> if (unlikely(contended))
> spin_unlock(&q->busylock);
> return rc;

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at