the direct cause of 'Reentering the interrupt handler!',
'Tx request while isr active' and 'Interrupted while
interrupts are masked!' messages and interface stoppage
was that mysteriously the 8390 irq handler was executing
on CPU#0, while we were busy transmitting on CPU#1. We
have synchronize_irq() to catch such cases, the 8390.c
code is indeed correct at first sight:
/* Mask interrupts from the ethercard. */
outb_p(0x00, e8390_base + EN0_IMR);
synchronize_irq();
first, there is a 8390-specific problem with this code:
it has to be executed in cli() mode, since the 8390 has
a 'current command page' state, which is by convention
PAGE0, but can be changed to PAGE1 by an interrupt
handler. This assumption works correctly on UP, but not
on SMP, if we have an IRQ handler executing on CPU#0,
the above code might have no effect. But this window is
very small and fixing it did not prevent those messages.
the second, much more subtle problem is that if this
code is executed on a non-external-IRQ-accepting CPU
(currently all IRQs go to CPU#0), it does not seem
to be guaranteed that the external IO-APIC stops
emitting interrupts. I have measured the time window, and
it's around ~9 usecs. As far as i could experiment with
this on a 2-CPU box, this ill-behavior happens only if
we do the above code from the second CPU. This is
probably a matter of timing, the receiving CPU has much
less paralellizm with an IRQ handler than a different CPU.
[see the patch for more speculation and timing-info]
the first solution was to add a 10 usecs delay to
synchronize_irq(), then i tried to find a wait-less
solution, but no luck ... eg. i've tried to
disable_irq()/enable_irq() the IO-APIC (which under Linux
is in 8259-emulation mode), but no effect. Another
solution would be to issue an IPI to all CPUs, but this
has higher overhead than the 10 usecs wait (and it scales
linearly with the number of CPUs ...), i think.
thinking about it, this problem should affect _all_
synchronize_irq() users, maybe this explains some of
the Adaptec+SMP 2.1 anomalies as well? If anyone has
a better solution, and/or a good explanation why
Intel has done the IO-APIC this way, please speak up ;)
-- mingo
--- linux/arch/i386/kernel/irq.c.orig Tue Dec 23 01:43:33 1997
+++ linux/arch/i386/kernel/irq.c Thu Dec 25 01:08:16 1997
@@ -35,6 +35,7 @@
#include <asm/bitops.h>
#include <asm/smp.h>
#include <asm/pgtable.h>
+#include <asm/delay.h>
#include "irq.h"
@@ -412,19 +413,39 @@
* are no interrupts that are executing on another
* CPU we need to call this function.
*
+ * We have to give pending interrupts a chance to
+ * arrive (ie. let them get until hard_irq_enter()),
+ * even if they are arriving to another CPU.
+ *
* On UP this is a no-op.
*/
void synchronize_irq(void)
{
- int cpu = smp_processor_id();
- int local_count = local_irq_count[cpu];
-
- /* Do we need to wait? */
- if (local_count != atomic_read(&global_irq_count)) {
- /* The stupid way to do this */
- cli();
- sti();
- }
+ /*
+ * Yes it's ugly, since we cannot know in advance what is
+ * pending on _another_ CPU. (without doing an IPI that is)
+ */
+ cli();
+
+ udelay(10); /*
+ * if a card has just issued an IRQ shortly before, we
+ * have to wait about this amount of time worst case,
+ * to make sure the IRQ gets propagated from the PIC
+ * to the CPU properly.
+ *
+ * - ~8 usecs INTA handling overhead
+ * [we cannot stop an already accepted irq from
+ * doing it's INTA cycle, i think]
+ * - ~1 usecs until the CPU executes
+ * hard_irq_enter()
+ */
+ sti(); /*
+ * here we let them in (they are polling
+ * on &global_irq currently, or are pending
+ * on this local CPU)
+ */
+ cli(); /* we make sure they are finished */
+ sti();
}
static inline void get_irqlock(int cpu, unsigned long where)
--- linux/drivers/net/8390.c.orig Mon Dec 22 23:21:56 1997
+++ linux/drivers/net/8390.c Thu Dec 25 02:10:02 1997
@@ -184,8 +184,10 @@
length = skb->len;
/* Mask interrupts from the ethercard. */
+ cli();
outb_p(0x00, e8390_base + EN0_IMR);
- synchronize_irq();
+ synchronize_irq(); /* does implicit sti() */
+
if (dev->interrupt) {
printk("%s: Tx request while isr active.\n",dev->name);
outb_p(ENISR_ALL, e8390_base + EN0_IMR);