Re: MMIO and gcc re-ordering issue

From: Roland Dreier
Date: Tue May 27 2008 - 17:29:58 EST


> > Writes are posted yes, but not reordered arbitrarily. If I have code like:
> >
> > spin_lock(&mmio_lock);
> > writel(val1, reg1);
> > writel(val2, reg2);
> > spin_unlock(&mmio_lock);
> >
> > then I have a reasonable expectation that if two CPUs run this at the
> > same time, their writes to reg1/reg2 won't be interleaved with each
> > other (because the whole section is inside a spinlock). And Altix
> > violates that expectation.
>
> Does that necessarily follow?
>
> If you've got a large system with multiple pci bridges, could you end
> up with posted writes coming from different cpus taking a different
> amount of time to propagate to a device and thus colliding?

Not on x86. And a given PCI device can only be reached from a single
host bridge, so I don't see how it can happen. But on SGI altix
systems, there is a routed fabric between the CPU and the PCI bus, so
the reordering can happen there. Hence mmiowb() and the endless supply
of driver bugs that it causes.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/