Re: Process Migration on Linux - Impossible?

kwrohrer@enteract.com
Thu, 2 Oct 1997 01:33:29 -0500 (CDT)

And lo, jleu@doit.wisc.edu saith unto me:
> Our discussions led us to a idea of network memmory AND a network process
> space (a lot like a Beowulf). Our ideas take this one step further, why
> not make the network address space part of the virtual memmory system?
> The idea I'm getting at is that if a machine decided it is to busy to continue
> running a given process it will ship the active page off to another machine
> (with the other necessary process structures) This machine would then run
> in the current page until it needed another page, at which time it would
> request that page from the originating machine as if it was coming from swap.
This network memory space, if it is also a source of shared as well as private
memory, can neatly solve the difficulties of moving a process with open
shared memory...so long as at most one machine can have a valid copy of
any given page at any given point in time. Relaxing this last restriction,
however, brings down all the horrors of fully general shared memory upon
you.

> The only problem that we really see with this is that you could also slip into
> the line of thought of "Why not use this other machines memmory as swap space?"
> if you do that then you start running in to the situation where a machine has
> started to swap out a process to machine A, then it decides it wants to
> ship the process off to a machien B, and now some pages are spread across
> mutilple machines, which would be hard to account for. In otherwords we
> havn't figured out how to handle the page accounting when differnt machines
> hold differnt pages.
Actually, I'd heard that some research showed that a remote machine's RAM
(or many remote machines'?) as the paging device performs better than using
a disk. As for paging to A, then migrating to B, the situation is no more
difficult than some pages living on /dev/sda3 while others live in /dev/sdb2
and others live in /tmp/extraswap. If you implement this network memory
space just like virtual memory, and use it just like swap space (except
now the "disk" can ask you to page something out), nobody really has to care
which node originally owned what page. Swapping a page out to a remote machine
would allow the remote machine to pick the "address" within whatever swap space
it provided; you don't want a global address space unless you're damn sure you
need it. And, of course, when you migrate a process, all but the pages you
prefault over would be changed from local-style physical locations to netswap-
style physical locations. Care is required when speccing how to clean up after
failures, but a good spec for that should allow you to retain a node's netswap
contents across a controlled shutdown/reboot...

Paging over NFS can suck; paging over a good, robust network protocol could
be really neat. Up until every node on the network starts Photoshopping plus-
sized clothing patterns in 30-bit color at 2400 dpi because each thinks "there's
so much memory available!", anyway. This actually sounds like a doable and
useful project (netswap, not the clothing patterns), but current research
should be checked before anyone actually tries it.

Keith