Re: [PATCH 1/2 (repost)] mm: serialize OOM kill operations

[Dave Peterson]

On Thursday 27 April 2006 14:09, Andrew Morton wrote:
> Paul Jackson <pj@xxxxxxx> wrote:
> > Adding a 'oom_notify' bitfield after the existing 'dumpable'
> > bitfield in mm_struct would save that slot:
> >
> >         unsigned dumpable:2;
> > 	unsigned oom_notify:1;
>
> Note that these will occupy the same machine word.  So they'll need
> locking.  (Good luck trying to demonstrate the race though!)

Taking a quick look at the code, I think a race could happen like this:
Task A is executing a system call handler such as sys_setgid(),
sys_setuid(), etc. that assigns a value to the 'dumpable' field (i.e.
current->mm->dumpable = suid_dumpable; ).  At the same time the OOM
killer decides to shoot task A.  The following sequence of events could
take place:

    1.  Task A reads the machine word containing 'dumpable' into a
        register and modifies the value in the register so that
        'dumpable' is set to the desired value.

    2.  Task B, which is executing the OOM killer code, has decided to
        shoot task A.  B reads the machine word containing 'dumpable'
        into a register, sets the bit corresponding to 'oom_notify',
        and writes the machine word back to memory.

    3.  Task A writes its machine word back to memory, which wipes out
        the setting of the 'oom_notify' bit by task B.  Now the OOM
        kill operation will never terminate, and all future OOM kill
        operations will be prevented.

I guess it would be kind of nice if we had some sort of primitive for
atomically modifying multiple bits within a machine word.
Oh well :-/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/