Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creationwith a specified id)

[Serge E. Hallyn]

 Quoting Oren Laadan (orenl@xxxxxxxxxxxxxxx):
>
>
> Serge E. Hallyn wrote:
>> Quoting Dave Hansen (dave@xxxxxxxxxxxxxxxxxx):
>>> On Wed, 2008-07-09 at 18:58 -0700, Eric W. Biederman wrote:
>>>> In the worst case today we can restore a checkpoint by replaying all of
>>>> the user space actions that took us to get there.  That is a tedious
>>>> and slow approach.
>>> Yes, tedious and slow, *and* minimally invasive in the kernel.  Once we
>>> have a tedious and slow process, we'll have some really good points when
>>> we try to push the next set of patches to make it less slow and tedious.
>>> We'll be able to describe an _actual_ set of problems to our fellow
>>> kernel hackers.
>>>
>>> So, the checkpoint-as-a-corefile idea sounds good to me, but it
>>> definitely leaves a lot of questions about exactly how we'll need to do
>>> the restore.
>>
>> Talking with Dave over irc, I kind of liked the idea of creating a new
>> fs/binfmt_cr.c that executes a checkpoint-as-a-coredump file.
>>
>> One thing I do not like about the checkpoint-as-coredump is that it begs
>> us to dump all memory out into the file.  Our plan/hope was to save
>> ourselves from writing out most memory by:
>>
>> 	1. associating a separate swapfile with each container
>> 	2. doing a swapfile snapshot at each checkpoint
>> 	3. dumping the pte entries (/proc/self/)
>>
>> If we do checkpoint-as-a-coredump, then we need userspace to coordinate
>> a kernel-generated coredump with a user-generated (?) swapfile snapshot.
>> But I guess we figure that out later.
>
> I'm not sure how this approach integrates with (a) live migration (and
> the iterative process of sending over memory modified since previous
> iteration), and (b) incremental checkpoint (where except for the first
> snapshot, additional snapshots only save what changed since the previous
> one).

Oh, well I was seeing them as pretty orthogonal actually.  The reason is
that my checkpoint-as-a-coredump file would NOT include memory contents.
I'm still hoping that we can lock a container into its own nfs-mounted
swapfile, and take a snapshot of that at checkpoint.  Let the snapshot
solution take care of the incremental snapshot.

Attached are one kernel and one cryo patch, purely for a toy
implementation.  It doesn't even *quite* work yet.  Here are the notes
I wrote when I put it down wednesday afternoon:

===============================================================

The kernel patch impelments (only for x86_32) sys_checkpoint.  You
can create a checkpoint with the following test program:

#include <unistd.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <errno.h>

int main()
{
        int ret;

        ret = syscall(327, "output.kckpt");
        printf("ret was %d\n", ret);
        if (ret == -1)
                perror("checkpoint");
        return 0;
}

Make output.kckpt executable and run it.  Works fine.

With the cryo patch, you can do:

SHELL 1:
./tests/sleep
SHELL 2:
./cr -p <sleeppid> -f o1.bin

and you'll see <sleeppid.kckpt> created.  Make that
executable and run it, and you'll see it runs as though
it were the original sleep program.

On the other hand, two problems with using cryo:

        1. the checkpointed app segfaults

        2. if you restart with "cr -r -f o1.bin" it
           fails somewhere.

===============================================================

Matt has already pointed out some coding errors so *really* it's
just to show one way we could integrate a binary format handler
for partial restart with more userspace control.  I'd be too
embarassed to send it out, but figure I should send it out before
the mini-summit.

-serge