Re: [syzbot] [net?] INFO: task hung in register_nexthop_notifier (3)

[Antoine Tenart]

Quoting Eric Dumazet (2024-03-18 15:46:37)
> On Mon, Mar 18, 2024 at 12:26 PM syzbot
> <syzbot+99b8125966713aa4b0c3@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
> >
> > INFO: task syz-executor.3:6975 blocked for more than 143 seconds.
> >       Not tainted 6.8.0-rc7-syzkaller-02500-g76839e2f1fde #0
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > task:syz-executor.3  state:D stack:20920 pid:6975  tgid:6975  ppid:1      flags:0x00004006
> > Call Trace:
> >  <TASK>
> >  context_switch kernel/sched/core.c:5400 [inline]
> >  __schedule+0x17d1/0x49f0 kernel/sched/core.c:6727
> >  __schedule_loop kernel/sched/core.c:6802 [inline]
> >  schedule+0x149/0x260 kernel/sched/core.c:6817
> >  schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6874
> >  __mutex_lock_common kernel/locking/mutex.c:684 [inline]
> >  __mutex_lock+0x6a3/0xd70 kernel/locking/mutex.c:752
> >  register_nexthop_notifier+0x84/0x290 net/ipv4/nexthop.c:3863
> >  nsim_fib_create+0x8a6/0xa70 drivers/net/netdevsim/fib.c:1587
> >  nsim_drv_probe+0x747/0xb80 drivers/net/netdevsim/dev.c:1582
> >  really_probe+0x29e/0xc50 drivers/base/dd.c:658
> >  __driver_probe_device+0x1a2/0x3e0 drivers/base/dd.c:800
> >  driver_probe_device+0x50/0x430 drivers/base/dd.c:830
> >  __device_attach_driver+0x2d6/0x530 drivers/base/dd.c:958
> >  bus_for_each_drv+0x24e/0x2e0 drivers/base/bus.c:457
> >  __device_attach+0x333/0x520 drivers/base/dd.c:1030
> >  bus_probe_device+0x189/0x260 drivers/base/bus.c:532
> >  device_add+0x8ff/0xca0 drivers/base/core.c:3639
> >  nsim_bus_dev_new drivers/net/netdevsim/bus.c:442 [inline]
> >  new_device_store+0x3f2/0x890 drivers/net/netdevsim/bus.c:173
> >  kernfs_fop_write_iter+0x3a4/0x500 fs/kernfs/file.c:334
> 
> So we have a sysfs handler ultimately calling register_nexthop_notifier() or any
> other network control path requiring RTNL.
> 
> Note that we have rtnl_trylock() for a reason...

Mentioning the below in case that gives some ideas; feel free to
disregard.

When I looked at similar issues a while ago the rtnl deadlock actually
happened with the kernfs_node refcount; haven't looked at this one in
details though. The mutex in there was just preventing concurrent
writers.

> Or maybe the reason is wrong, if we could change kernfs_fop_write_iter()
> to no longer hold a mutex...

At the time I found a way to safely drop the refcount of those
kernfs_node which then allowed to call rtnl_lock from sysfs handlers,
https://lore.kernel.org/all/20231018154804.420823-1-atenart@xxxxxxxxxx/T/

Note that this relied on how net device are unregistered (calling
device_del under rtnl and later waiting for refs on the netdev to drop
outside of the lock; and a few other things), so extra modifications
would be needed to generalize the approach. Also it's a tradeoff between
fixing those deadlocks without rtnl_trylock and maintaining a quite
complex logic...

Antoine