Memory vs. MMIO ordering, again

From: Robert Hancock
Date: Wed Dec 24 2008 - 13:07:23 EST


A question came up recently about memory ordering in libata. Essentially we have the classic case (ata_bmdma_setup in drivers/ata/libata-sff.c) where we do some writes to memory (specifically the PRD table) and want to make sure the device will see those writes before we poke it to go and use that memory. Right now this is using an mb() (which seems like overkill, even in the worst case it seems like only a wmb() should be needed).

I just finished digging through the 2006 LKML discussion "Opinion on ordering of writel vs. stores to RAM" which ironically failed to clearly answer the question asked in its subject.

Documentation/memory-barriers.txt does not answer this question either. The only thing it contains that seems to pertain to this question is the following:

"Memory mapped I/O usually takes place through memory locations that are part of a window in the CPU's memory space that has different properties assigned than the usual RAM directed window.

Amongst these properties is usually the fact that such accesses bypass the caching entirely and go directly to the device buses. This means MMIO accesses may, in effect, overtake accesses to cached memory that were emitted earlier. A memory barrier isn't sufficient in such a case, but rather the cache must be flushed between the cached memory write and the MMIO access if the two are in any way dependent."

This seems like BS to me.. Flush the cache? How is a driver supposed to know how to do that? Furthermore, why should it need to worry about this kind of detail? This seems wrong or at least a low-level detail that normal code should not have to be concerned with.

From what I can vaguely infer from that 2006 discussion it seems like powerpc was going to be fixed so that writel, etc. would provide the expected ordering with respect to memory writes, however I don't know if this is actually the case. The documentation that driver writers would rely on should be updated to be explicit on this question...

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/