RE: e1000 softirq load balancing

[Madhukar.Mythri]

First make-sure whether you have one dedicated interrupt-line for each
network card.
You got 100% CPU utilization with NAPI Disabled or Enabled ?

With NAPI enabled, and assign each interrupt-line to each CPU. And check
whether the interrupts are distributed as per  you assigned, by seeing
"cat /proc/interrupts" and by pumping traffic.
And also check how the interrupts balanced between CPU's.

As per my observation, when NAPI enabled, the interrupt-line cannot
assign to each CPU. In NAPI-mode for whichever the CPU  got the first
interrupt, that CPU only carried out in poll-mode for rest of the loaded
packets(H/W buffers). 

So, with NAPI enabled, if interrupts are not distributed on assigned
CPU's and one CPU got loaded more, then try by Disabling NAPI, and use
'irqbalance' utility to load-balance the interrupts between all CPU's.
(http://www.irqbalance.org/).

Regards,
Madhukar.

-----Original Message-----
From: Don Porter [mailto:porterde@xxxxxxxxxxxxx] 
Sent: Wednesday, October 15, 2008 6:23 PM
To: Mythri, Madhukar [NETPWR/EMBED/HYDE]
Cc: linux-net@xxxxxxxxxxxxxxx
Subject: Re: e1000 softirq load balancing

I believe I have 4 interrupt lines, but I will have to double-check.

I have successfully used /proc/irq/####/smp_affinity to assign the
interrupts to 4 different CPUs.  The problem is in the softirq portion
of the interrupt handling, where CPU usage indicates that they are all
being funneled back to a single CPU. 

Does that make sense?  I feel like one ought to be able to have 4
softirq daemons servicing incoming packets, not just one.

Thanks,
Don

Madhukar.Mythri@xxxxxxxxxxx wrote:
> You are saying that, 4 Intel  82571EB Gb NICs (2 pci cards x 2
> NICs/chip) using the e1000 driver.
> So, do you have 4 interrupt lines or only 2-lines?
>
> Based on this, if you have 4 interrupt lines, then you can assign each

> interrupt-line to each CPU-core, with NAPI enabled(for good 
> performance).
>
> Regards,
> Madhukar.
> -----Original Message-----
> From: linux-net-owner@xxxxxxxxxxxxxxx
> [mailto:linux-net-owner@xxxxxxxxxxxxxxx] On Behalf Of Don Porter
> Sent: Wednesday, October 15, 2008 12:36 AM
> To: linux-net@xxxxxxxxxxxxxxx
> Subject: e1000 softirq load balancing
>
> Hi,
>
> Background:
>
> I have a 16 core x86_64 machine (4 chips x 4 cores/chip) that has 4 
> Intel 82571EB Gb NICs (2 pci cards x 2 NICs/chip) using the e1000 
> driver.
>
> I have a simple client/server micro-benchmark that pounds a server on 
> each NIC with requests to measure peak throughput.  I am running 
> Ubuntu 8.04.1, kernel version 2.6.24.
>
> Problem:
>
> What I am observing is that a single ksoftirqd thread is becoming a 
> bottleneck for the system.
> More specifically, one cpu runs ksoftirqd at 100% cpu utilization, 
> while
> 4 cpus each run their servers at about 25%.  I carefully used
> sched_setaffinity() to map server threads to cpus and 
> /proc/irq/<device>/smp_affinity to map hardware interrupts to cpus 
> such that there should be exactly 1 cpu per server thread and 1 cpu 
> for servicing hardware interrupts per device.
>
> I can observe (via /proc/interrupts) that the interrupts are being 
> distributed properly, but despite this I only see 1 or 2 ksoftirqd 
> running, and the server daemons bottlenecked behind them.  (This is 
> with NAPI disabled.  With NAPI enabled, I can't get even 2 ksoftirqd 
> threads to run).  I have tried varous permutations such as assigning 
> each hardware interrupt to a different physical chip.
>
> Desired Result:
>
> It seems to me that with 4 independent NICs and plenty of CPUs to 
> spare, I ought to be able to assign one softirq daemon to each NIC 
> rather than funnelling all of the traffic through 1 or 2.
>
> Any advice on this issue is greatly appreciated.
>
> Best regards,
> Don Porter
> --
> To unsubscribe from this list: send the line "unsubscribe linux-net" 
> in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html