Re: FW: Re: [dm-devel] [BUG 3.12.rc4] Oops: unable to handle kernelpaging request during shutdown

From: Greg KH
Date: Wed Oct 30 2013 - 13:18:54 EST


On Wed, Oct 30, 2013 at 12:37:04PM -0400, Mikulas Patocka wrote:
>
>
> On Mon, 28 Oct 2013, Alasdair G Kergon wrote:
>
> > On Fri, Oct 25, 2013 at 11:48 AM, Linus Torvalds
> > <torvalds@xxxxxxxxxxxxxxxxxxxx> wrote:
> > >
> > > Yes, but nobody has actually been able to trigger it with those. It's
> > > pretty rare, and the debug options are so expensive that they aren't
> > > reasonable to enable generally...
> > >
> > > So we need to try to figure out how to trigger it, or narrow things
> > > down some way..
> >
> > Ok, still trying to figure this out, and I do have another bug as a
> > result. I don't think this one is really the fundamental one either
> > that caused my crash during "yum upgrade", nor necessarily Knut's
> > problem during shutdown, but I'll keep looking.
> >
> > And who knows.. Maybe this *does* explain Knut's issue.
> >
> > Appended is a warning I get with DEBUG_TIMER_OBJECTS. Seems to be a
> > device-mapper issue. Alasdair, Neil, comments? It looks like
> > dm_destroy() is freeing an delayed_work entry that is still active...
> >
> > I don't know exactly which field in the 'struct mapped_device' has
> > that delayed-work thing, but I assume it's the kobject.. Somebody who
> > knows this code better, please take a look!
> >
> > Linus
>
> No field in device mapper has 'struct delayed_work' in it --- except
> struct kobject:
> #ifdef CONFIG_DEBUG_KOBJECT_RELEASE
> struct delayed_work release;
> #endif
>
> - so this is kobject manipulation bug. Dm calls dm_sysfs_exit, which calls
> kobject_put. Dm then frees the structure that contained the kobject with
> kfree and it triggers this warning.
>
> Documentation/kobject.txt says that kobjects shouldn't be used this way -
> that the structure should be freed from the release method. However, using
> the API the correct way is impossible.
>
> Unloading of a device driver is supposed to work like this:
> 1) you call the unload routine
> 2) it calls kobject_put (but the kobject may still be referenced by other
> kernel code release method will be called when the references are dropped)
> 3) you don't free the driver structure
> 4) you exit the unload routing
>
> ...
>
> 5) the other references to kobject are dropped
> 6) release method is called, it calls kfree on the driver routine
> 7) the release method exits
>
> Now, the problem is that between steps 4) and 5), someone may unload the
> module and trigger the crash. It is impossible to protect against it.
>
> Another problem is that between steps 4) and 5), the driver is essentially
> dead, but it must still respond to attr_show and attr_store methods on the
> kobect - it is possible to handle it correctly, but it is not easy to test
> - there is a possibility of a lot of bugs in drivers.
>
>
> I suggest that you implement a function kobject_put_free, that decrements
> the kobject reference count and waits until others stop using the kobject
> and the reference count drops to zero. Then, you change drivers to use
> kobject_put_free instead of kobject_put in their unload routine - that
> will fix this sort of module unload races.

The "module unload" issue is rare, thankfully, but yes, this type of
function will be showing up in 3.13-rc1 through the btrfs tree as it
needs that functionality, so feel free to use it to resolve this issue
if you need it.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/