On Mon, 14 Feb 2000, Zachary Amsden wrote:
>
> If you had a memory architecture where each sibling has local memory, with
> it's other siblings local mappable into it's address space, it could probe by
> doing memory detection for each possible sibling.
>
> Initial kernel boot could copy it's image to local memory at each sibling,
> fill in a "start up data area", and use whatever hardware mechanism to call
> the main entry point for the initial processor at each sibling. The start up
> data area would have any info needed to boot, i.e. maybe a node number and
> hardware list. After this, the "initial" kernel would set up it's own start
> up data area, and call it's main kernel entry point.
>
> So this would really be bootstrapping code, and kernel entry and setup
> wouldn't really be affected. This would be pretty clean, but I'm assuming a
> specialized architecture that helps you quite a bit here.
>
Unless you guys want to cough up some hardware to start this diddy, I
think we have as special as intel smp gets. HAR!
So you are thinking that the bootstrap code should analyze the system as
compared to having the first sibling do it? Ok...
> To do it on generic SMP, you could partition system memory into N segments,
> copy the kernel image to each segment, and call the particular entry point on
> each sibling. (Each sibling would have a different physical memory entry
> point and would need the ability to boot from any specified physical memory
> address).
>
> If you can only boot from a fixed physical address, you need to play more
> games. You have all siblings but the master re-map their portion of physical
> memory to "local" (a fixed address), wait for the masters command (when it
> finishes probing hardware are dividing memory). Then they can all jump to
> their local entry point, and they all have a uniform memory structure.
>
> Of course with generic SMP, you have a pretty major problem trying to share a
> peripheral bus between isolated kernels, unless you divy up hardware between
> siblings. When you did this, you put more intelligence behind interrupt
> routing. However, each sibling would need it's own disk/network, unless your
> sibling communication protocol provides sharing of those resources.
>
That would have to happen to make this worthwhile. It (IMVHO) this only
makes sense to do this if it can be implemented on fairly common
hardware. You guys over at SGI build awesome stuff, but the average guy
can't afford the SMP stuff available (without re-mortgageing the
house). The protocol will have to allow for sharing so that the less well
endowed machines can use this.
After pouring over what I could find of the mp spec for intel('cause thats
what I have to work with) I discovered that both FreeBSD and Linux use the
"Virtual-wire" mode for the APIC under smp. This means that only the BSP
handles irq's! Seems like that is done to keep the code down, but I am not
an OS engineer. I could find little about the "symmetric-mode" that is
also available and only a description that states that 'each processor can
handle ANY interrupt." This sound like it is more useful in our
circumstance? I think it could lead to the need to 'pass' interrupts from
processor to processor, but allow for all processors to use the same
hardware more effectivly.
> You wouldn't need a communication protocol as heavy as TCP, I would expect
> data loss/corruption to be dealt with in hardware for whatever communication
> method you are using (likely shared memory).
Not sure... help on this one? We may not have that option on the Intel
stuff, I wonder how the ALPHA's do it...
>
> The only part of the kernel that needs to be aware of the different sibling
> physical mappings is the communication layer itself, so that wouldn't pose to
> great of a problem to the kernel.
>
Some kind of master table that is copied from sibling to sibling at
initial booting and gives the hard memory layout of each sibling. In proc?
> Some of the things you might want to do with the communication protocol:
>
> cross-sibling UNIX domain sockets/shm
> FS sharing
> Hardware arbitration/sharing
> borrow/return free pages
>
Now were cooking with gas.
Matt Berglund
Unix is best described as an old, sturdy tree.
Well built, always growing, and has stood the test of time.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Tue Feb 15 2000 - 21:00:28 EST