> On Sun, Aug 15, 1999, Gerard Roudier <groudier@club-internet.fr> wrote:
>
> >In order to make things right, the driver writers/maintainers must be
> >carefull about the order of operations each time this is required.
> >For that to be achieved, all the parts that may buffer data, and so
> >reorder operations, must be considered. The C compiler, the CPU, the host
> >bridge, inter-bus bridges and the PCI device are the parts I am talking
> >about.
>
> Yes, but this can quickly become a real nightmare. Event on PCI, ordering
> rules can change from one platform to another, from one PCI chipset to
> another, etc...
The PCI specs are fine. You just have to read them several dozen of times
in order to understand most of them. :)
And indeed, real PCI chipsets most of the time donnot follow exactly the
official PCI ordering rules. Probably, designers of bridges haven't had
time to read the specs times enough. ;)
> There are lots of drivers who don't really need the very best
> performances out of the bridge path, possibly because most actual data
> transfer is done in DMA and the only few i/os are only necessary to setup
> the DMA descriptors, or simply because the device is a really simple one.
Facing posted writes and ordering rules that occur in a system, nothing is
simple.
> In all those cases, handling all the different ordering rules of all the
> different platforms and busses becomes a challenge that would prevent
> lots of people from even trying to write a driver ;-)
Why?
Writing and maintaining drivers, having to solve weird problems is great
fun. And you have to learn lots of things if you want to have some chance
to understand what actually happens.
People that really like IOs or some kind of IOs will work on drivers,
in my opinion.
Even for CPU only concern when dealing with SMP, you also have to be
careful about consistency issues. A device driver that deals with
a sophisticated device is just kind of AMP (A=asymetric) and you only
have a weak handle on ordering; this is the funniest thing.
> That's why I like the idea of having simple, CPU-ordered readl/writel
> (driver writer still need to take care of write posting done by the
> various bridges, as it has always been the case, and as it was, I think,
> never correctly handled in most drivers).
Existing bridges intentionnal misdesigns (compared to official
specifications) can easily be categorized. Perhaps there are not too many
ways to make things wrong ;). It also appears that misbehaviours due to
Errata (unintentionnal) are often due to similar mistakes, probably since
engineers have failed the same way facing the same difficulties.
This let me think that it should be possible to support existing stuff
without needing too much help from E.T. :)
> P.S. A question: As far as PCI write posting is concerned, is it
> _guaranteed_ that a read to the same bus range will flush any outstanding
> writes to the same range or is it simply a common practice of PCI bridges ?
If a read in one direction does not flush posted writes to the same
direction then nothing reliable can be achieved.
The ideal situation is what the PCI spec says about transaction ordering.
Basically (from memory):
- Strong ordering in each direction.
- A read must complete at its destination after all posted writes to the
same direction have completed and must complete at the originator after
all posted writes in the opposite direction have completed.
- The design of the whole PCI system must ensure that no deadlock or
livelock may occur.
- The interrupt mechanism must not be used to synchronize transactions.
- Etc...
But the reality is quite different:
Some system-PCI bridges may:
- Reorder write to memory.
- Only flush on a read posted write transactions that go in the
direction of this read transaction (donnot flush PWs in the opposite
direction)
- Only ensure that everything is flushed prior to the interrupt delivery.
- Not snoop DMA accesses.
- Etc...
There is no nightmare here, but just too much fun. ;-)
Gérard.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/