Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (perf+ regression IRQs)
From: Jesper Dangaard Brouer
Date: Wed Nov 19 2008 - 17:58:50 EST
On Thu, 13 Nov 2008, David Miller wrote:
From: Jesper Dangaard Brouer <jdb@xxxxxxx>
Date: Thu, 13 Nov 2008 11:29:31 +0100
Although I'm not happy about the new perf numbers, as I now on a SMP
system only can route approx 290 kpps, remember I could route 319 kpps
using a single CPU nosmp kernel.
That unfortunately (can be) the cost of SMP :-/
[Regression]
Well that was not the real cause of the performance loss. Because on
kernel 2.6.27 I get really good performance (900-1200kpps) compared to
2.6.28 (git net-2.6).
The cause of this problem (tracked down together with Robert Olsson) is
that on 2.6.28 I have a lot less IRQs available. It seems max 34 IRQs.
Due the reduced number of IRQs the NIU driver cannot get enough IRQs to
the interfaces, and starts to use "IO-APIC" based IRQs.
On kernel 2.6.28:
My eth2 is using 10 IRQs all "PCI-MSI-edge".
BUT my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared
with the usb driver...
Think thats must be my performance problem on 2.6.28.
With multi-flow tests, Robert Olsson is getting 4.2 mpps rates with
NIU and pktgen. That's what this card is designed for, good
multi-flow workload performance, rather than striving for maximum
single-flow performance.
[Packet performance]
Yes, I know, I do use pktgen and multi-flows (rand dest IP+port).
For the two drivers NIU and Suns NXGE, my packet per sec performance is
now, on 2.6.27 (with backported NIU fixes).
With NIU driver I can route 900 kpps.
With NXGE driver (and enqueue=NULL hack) I can route 1200 kpps.
Actually I think I can go higher, because I'm limited by my packet rate
generator. I use pktgen (with rand dst IP+port) and can only generate 1200
kpps.
(I have actually ordered some new hardware, so I can get a faster pktgen
machine and perhaps test it as a router too. Also ordered the hardware
because I want to test PCI-express v.2.0. I have a prototype 12-port
gigabit NIC (from hotlava systems) that support PCIe v.2.0 and has 6x
82575 chips (4RX/4TX queues))
Hilsen
Jesper Brouer
--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/