Re: Odd ENOMEM being returned in 3.8-rcX

From: Eric W. Biederman
Date: Fri Feb 08 2013 - 15:36:39 EST


Josh Boyer <jwboyer@xxxxxxxxxx> writes:

> On Fri, Feb 08, 2013 at 01:19:49PM -0500, Josh Boyer wrote:
>> On Thu, Feb 07, 2013 at 07:35:01PM -0500, Josh Boyer wrote:
>> > On Thu, Feb 07, 2013 at 02:15:02PM -0800, Andrew Morton wrote:
>> > > On Thu, 7 Feb 2013 16:57:42 -0500
>> > > Josh Boyer <jwboyer@xxxxxxxxxx> wrote:
>> > >
>> > > > Hi All,
>> > > >
>> > > > We've hit a weird error in Fedora using the 3.8-rcX kernels. It seems
>> > > > the mock tool is getting back ENOMEM when doing very simple things that
>> > > > normally just work. The 3.7 kernels on the same userspace work just
>> > > > fine. It seems just running 'mock init -v' is enough to cause the
>> > > > failure.
>> > >
>> > > I assume you're not seeing the "page allocation failure" message and
>> > > backtrace. This means that either
>> >
>> > Right. If I disable our debug options, I see no backtraces at all and
>> > the python app still gets ENOMEM returned. (See below for those
>> > interested).
>> >
>> > > a) it's a __GFP_NOWARN callsite. This is rare. Or
>> > >
>> > > b) it's actually a different error but someone went and overwrote a
>> > > callee's return value with -ENOMEM. We do this a lot and it sucks.
>> >
>> > We do it in copy_io :\.
>> >
>> > > > At first glance it seems copy_io is failing (possibly because
>> > > > get_task_io_context fails), and then the above fallout is printed. The
>> > > > warning seems fairly valid, but I don't think that is the root of the
>> > > > problem.
>> > >
>> > > yes, get_task_io_context() might be the place. Tried adding a few
>> > > error-path printks in there to see what's happening?
>> >
>> > Yeah, that's my next step. I guess I know what I'll be doing tomorrow.
>> >
>> > > I can't see anything around there which leaves interrupts disabled
>> > > though. It's quite likely that there's some code with is forgetting to
>> > > reenable interrupts on a rarely-tested error path, and that ENOMEM is
>> > > tickling the bug.
>> >
>> > Right, agreed. As I said, I think that is mostly a secondary issue.
>> > Hopefully it will be easy to fix once we figure out why we're getting
>> > the ENOMEM error.
>> >
>> > Python backtrace below. Seems to be failing on forking a umount command
>> > after init'ing the chroot. I can put the full output somewhere if
>> > people are interested.
>>
>> OK. I've bisected this down to:
>>
>> 50804fe3737ca6a5942fdc2057a18a8141d00141 is the first bad commit
>> commit 50804fe3737ca6a5942fdc2057a18a8141d00141
>> Author: Eric W. Biederman <ebiederm@xxxxxxxxxxxx>
>> Date: Tue Mar 2 15:41:50 2010 -0800
>>
>> pidns: Support unsharing the pid namespace.
>>
>>
>> I haven't really gotten much farther than that yet, but the bisect was
>> pretty straight forward. Eric, is there anything specific I can gather
>> or do to help figure out why that is causing mock to get such a weird
>> error? I can provide the bisect log if you'd like.
>
> I took a look at what mock was doing and it was mostly very simple
> stuff. The two exceptions were that it was calling unshare, then doing
> some file checks and I/O, and then calling fork to exec off some helper
> things. Up until the point it fails, the forks work and the children go
> do whatever it is they were supposed to do. I've CC'd Clark Williams
> just in case people have questions on mock itself, but I'm not sure that
> will be needed.

Our emails crossed paths. You have just confirmed my suspicion about
what was going wrong.

The practical question is why mock is calling unshare(CLONE_NEWPID)
because it clearly seems not to understand how to unshare the pid
namespace and use it that way.

Except for forgeting to reenable irqs in the failure path of alloc_pid
the behavior is exactly correct and is how the pid namespace is designed
to behave in the case of unshare.


> which is consistent with what mock is seeing. If I comment out the call
> to unshare, it seems to always work. It seems to consistently fail with
> ENOMEM after the first 3-5 forked children, but it varies within that
> range.

If you add a waitpid or space out your forks you will see that it always
fails after your first child in the pid namespace has exited.

We don't allow children in a pid namespace after fork has exited.

Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/