Re: irq load balancing

From: Chris Snook
Date: Thu Sep 13 2007 - 17:30:57 EST


Venkat Subbiah wrote:
Since most network devices have a single status register for both
receiver and transmit (and errors and the like), which needs a lock to
protect access, you will likely end up with serious thrashing of moving
the lock between cpus.
Any ways to measure the trashing of locks?

Since most network devices have a single status register for both
receiver and transmit (and errors and the like)
These register accesses will be mostly within the irq handler which I
plan on keeping on the same processor. The network driver is actually
tg3. Will looks closely into the driver.

Why are you trying to do this, anyway? This is a classic example of fairness hurting both performance and efficiency. Unbalanced distribution of a single IRQ gives superior performance. There are cases when this is a worthwhile tradeoff, but the network stack is not one of them. In the HPC world, people generally want to squeeze maximum performance out of CPU/cache/RAM so they just accept the imbalance because it performs better than balancing it, and irqbalance can keep things fair over longer intervals if that's important. In the realtime world, people generally bind everything they can to one or two CPUs, and bind their realtime applications to the remaining ones to minimize contention.

Distributing your network interrupts in a round-robin fashion will make your computer do exactly one thing faster: heat up the room.

-- Chris
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/