Re: [PATCH v3 0/2] kvm: level irqfd and new eoifd

From: Alex Williamson
Date: Mon Jul 16 2012 - 10:03:50 EST

On Sun, 2012-07-15 at 11:33 +0300, Avi Kivity wrote:
> On 07/12/2012 07:19 PM, Alex Williamson wrote:
> > On Thu, 2012-07-12 at 12:35 +0300, Avi Kivity wrote:
> >> On 07/11/2012 10:57 PM, Alex Williamson wrote:
> >> >>
> >> >> > We still have classic KVM device assignment to provide fast-path INTx.
> >> >> > But if we want to replace it midterm, I think it's necessary for VFIO to
> >> >> > be able to provide such a path as well.
> >> >>
> >> >> I would like VFIO to have no regressions vs. kvm device assignment,
> >> >> except perhaps in uncommon corner cases. So I agree.
> >> >
> >> > I ran a few TCP_RR netperf tests forcing a 1Gb tg3 nic to use INTx.
> >> > Without irqchip support vfio gets a bit more than 60% of KVM device
> >> > assignment. That's a little bit of an unfair comparison since it's more
> >> > than just the I/O path. With the proposed interfaces here, enabling
> >> > irqchip, vfio is within 10% of KVM device assignment for INTx. For MSI,
> >> > I can actually make vfio come out more than 30% better than KVM device
> >> > assignment if I send the eventfd from the hard irq handler. Using a
> >> > threaded handler as the code currently does, vfio is still behind KVM.
> >> > It's hard to beat a direct call chain.
> >>
> >> We can have a direct call chain with vfio too, using a custom eventfd
> >> poll function, no? Assuming we set up a fast path for unicast msi.
> >
> > You'll have to help me out a little, eventfd_signal walks the wait_queue
> > and calls each function. On the injection path that includes
> > irqfd_wakeup.
> This is what I meant, except I forgot that we already do direct path for
> MSI.

Ok, vfio now does it for the unmask irqfd-line interface too. Except
when we re-inject from eoifd we have to do the eventfd_signal from a
work queue as we can't have nested eventfd_signals. We probably need to
do some benchmarks to see if that re-injection path saves us anything vs
letting hardware fire again.

> > For an MSI that seems to already provide direct
> > injection.
> Ugh, even for a broadcast MSI into 254 vcpu guests. That's going to be
> one slow interrupt.
> > For level we'll schedule_work, so that explains the overhead
> > in that path, but it's not too dissimilar to a a threaded irq. vfio
> > does something very similar, so there's a schedule_work both on inject
> > and on eoi. I'll have to check whether anything prevents the unmask
> > from the wait_queue function in vfio, that could be a significant chunk
> > of the gap. Where's the custom poll function come into play? Thanks,
> So I don't understand where the gap comes from. The number of context
> switches for kvm and vfio is the same, as long as both use MSI (and
> either both use threaded irq or both don't).

Right, we're not exactly apples to apples yet. Using threaded
interrupts and work queue injection, vfio is a little slower. There's
an extra work queue in that path vs kvm though. Using non-threaded
interrupts and direct injection, vfio is faster. Once kvm moves to
non-threaded interrupt handling, I expect we'll be pretty similar. My
benchmarks are just rough estimates at this point as I'm both trying to
work out lockdep and get some ballpark performance comparison. Thanks,


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at