Re: fasync race in fs/fcntl.c
From: Al Viro
Date: Sat Mar 02 2013 - 14:49:28 EST
On Sat, Mar 02, 2013 at 03:00:28AM -0800, Russ Dill wrote:
> I'm seeing a race in fs/fcntl.c. I'm not sure exactly how the race is
> occurring, but the following is my best guess. A kernel log is
> attached.
[snip the analysis - it's a different lock anyway]
The traces below are essentially sys_execve() getting to get_random_bytes(),
to kill_fasync(), to send_sigio(), which spins on tasklist_lock.
Could you rebuild it with lockdep enabled and try to reproduce that?
I very much doubt that this execve() is a part of deadlock - it's
getting caught on one, but it shouldn't be holding any locks that
nest inside tasklist_lock at that point, so even it hadn't been there,
the process holding tasklist_lock probably wouldn't have progressed any
further...
> I exercise this by running a UML instance which uses /dev/random.
> After a day or so, I'll eventually get a crash or a hang. I captured
> this with netconsole.
>
>
> [172635.399438] ------------[ cut here ]------------
> [172635.399449] WARNING: at
> /build/buildd/linux-3.8.0/kernel/watchdog.c:246
> watchdog_overflow_callback+0x9c/0xd0()
> [172635.399451] Hardware name: VPCSE190X
> [172635.399454] Watchdog detected hard LOCKUP on cpu 0
> [172635.399456] Modules linked in: ftdi_sio netconsole(F) configfs(F)
> ext2(F) nls_iso8859_1(F) usb_storage(F) pl2303 usbserial snd
> _usb_audio snd_usbmidi_lib parport_pc(F) ppdev(F) lp(F) parport(F)
> bnep rfcomm bluetooth binfmt_misc(F) dm_crypt(F) uvcvideo arc4(
> F) snd_hda_codec_hdmi iwldvm snd_hda_codec_realtek snd_hda_intel
> videobuf2_vmalloc videobuf2_memops mac80211 videobuf2_core snd_hd
> a_codec videodev snd_hwdep(F) coretemp snd_pcm(F) snd_page_alloc(F)
> joydev(F) iwlwifi[172635.399521] ------------[ cut here ]------------
> [172635.399525] WARNING: at
> /build/buildd/linux-3.8.0/net/core/skbuff.c:573
> skb_release_head_state+0xed/0x100()
> [172635.399525] Hardware name: VPCSE190X
> [172635.399526] Modules linked in: ftdi_sio netconsole(F) configfs(F)
> ext2(F) nls_iso8859_1(F) usb_storage(F) pl2303 usbserial snd_usb_audio
> snd_usbmidi_lib parport_pc(F) ppdev(F) lp(F) parport(F) bnep rfcomm
> bluetooth binfmt_misc(F) dm_crypt(F) uvcvideo arc4(F)
> snd_hda_codec_hdmi iwldvm snd_hda_codec_realtek snd_hda_intel
> videobuf2_vmalloc videobuf2_memops mac80211 videobuf2_core
> snd_hda_codec videodev snd_hwdep(F) coretemp snd_pcm(F)
> snd_page_alloc(F) joydev(F) iwlwifi kvm_intel snd_seq_midi(F)
> snd_seq_midi_event(F) snd_rawmidi(F) kvm cfg80211 dm_multipath(F)
> snd_seq(F) snd_seq_device(F) tpm_infineon snd_timer(F) psmouse(F)
> snd(F) soundcore(F) scsi_dh serio_raw(F) sony_laptop rtsx_pci_ms mei
> microcode(F) memstick mac_hid pcspkr lpc_ich tpm_tis firewire_sbp2
> firewire_core crc_itu_t(F) btrfs(F) zlib_deflate(F) libcrc32c(F)
> hid_generic usbhid hid rtsx_pci_sdmmc ghash_clmulni_intel(F)
> aesni_intel(F) aes_x86_64(F) xts(F) lrw(F) gf128mul(F) ablk_helper(F)
> cryptd(F) radeon ahci(F) libahci(F) r8169 i915 ttm i2c_algo_bit
> drm_kms_helper video(F) drm rtsx_pci
> [172635.399572] Pid: 12575, comm: flock Tainted: GF
> 3.8.0-7-generic #15-Ubuntu
> [172635.399572] Call Trace:
> [172635.399573] <NMI> [<ffffffff810587cf>] warn_slowpath_common+0x7f/0xc0
> [172635.399577] [<ffffffff8105882a>] warn_slowpath_null+0x1a/0x20
> [172635.399579] [<ffffffff815b51bd>] skb_release_head_state+0xed/0x100
> [172635.399580] [<ffffffff815b4fb2>] __kfree_skb+0x12/0xa0
> [172635.399582] [<ffffffff815b551c>] consume_skb+0x2c/0x80
> [172635.399586] [<ffffffffa014cf47>] rtl8169_poll+0x4b7/0x6d0 [r8169]
> [172635.399590] [<ffffffff815d96c2>] netpoll_poll_dev+0x162/0x580
> [172635.399593] [<ffffffff815b666b>] ? __alloc_skb+0x8b/0x2a0
> [172635.399595] [<ffffffff815d9c6c>] netpoll_send_skb_on_dev+0x18c/0x3a0
> [172635.399598] [<ffffffff815da198>] netpoll_send_udp+0x278/0x2a0
> [172635.399600] [<ffffffffa07c4967>] write_msg+0xc7/0x110 [netconsole]
> [172635.399603] [<ffffffff810594f1>]
> call_console_drivers.constprop.13+0x91/0x100
> [172635.399605] [<ffffffff81059e4b>] console_unlock+0x2db/0x420
> [172635.399606] [<ffffffff8105a7dd>] vprintk_emit+0x1fd/0x4e0
> [172635.399608] [<ffffffff810e8ffc>] ? watchdog_overflow_callback+0x9c/0xd0
> [172635.399610] [<ffffffff816b78c1>] printk+0x67/0x69
> [172635.399613] [<ffffffff810c0423>] print_modules+0xa3/0xd0
> [172635.399617] [<ffffffff810587ca>] warn_slowpath_common+0x7a/0xc0
> [172635.399618] [<ffffffff810588cc>] warn_slowpath_fmt+0x4c/0x50
> [172635.399620] [<ffffffff8109170d>] ? sched_clock_cpu+0xbd/0x110
> [172635.399623] [<ffffffff810e8f60>] ? touch_nmi_watchdog+0x80/0x80
> [172635.399625] [<ffffffff810e8ffc>] watchdog_overflow_callback+0x9c/0xd0
> [172635.399627] [<ffffffff8112609d>] __perf_event_overflow+0x9d/0x230
> [172635.399630] [<ffffffff81025277>] ? x86_perf_event_set_period+0xd7/0x160
> [172635.399632] [<ffffffff81126cb4>] perf_event_overflow+0x14/0x20
> [172635.399634] [<ffffffff8102ad8b>] intel_pmu_handle_irq+0x1ab/0x330
> [172635.399637] [<ffffffff816c5b9d>] perf_event_nmi_handler+0x1d/0x20
> [172635.399640] [<ffffffff816c5351>] nmi_handle.isra.0+0x51/0x80
> [172635.399641] [<ffffffff816c5460>] do_nmi+0xe0/0x360
> [172635.399643] [<ffffffff816c4981>] end_repeat_nmi+0x1e/0x2e
> [172635.399645] [<ffffffff813536d9>] ? __read_lock_failed+0x9/0x20
> [172635.399648] [<ffffffff813536d9>] ? __read_lock_failed+0x9/0x20
> [172635.399649] [<ffffffff813536d9>] ? __read_lock_failed+0x9/0x20
> [172635.399651] <<EOE>> [<ffffffff816c3e13>] _raw_read_lock+0x13/0x20
> [172635.399654] [<ffffffff811a4532>] send_sigio+0x52/0xf0
> [172635.399656] [<ffffffff811a4631>] kill_fasync+0x61/0x90
> [172635.399658] [<ffffffff8143ad83>] account+0x113/0x1d0
> [172635.399661] [<ffffffff816c4618>] ? page_fault+0x28/0x30
> [172635.399663] [<ffffffff8143b335>] extract_entropy+0x65/0x140
> [172635.399664] [<ffffffff8143b650>] get_random_bytes+0x20/0x30
> [172635.399666] [<ffffffff816bb2a9>] create_elf_tables+0xaa/0x614
> [172635.399669] [<ffffffff811e8a84>] load_elf_binary+0xae4/0xe00
> [172635.399672] [<ffffffff811e7fa0>] ? load_elf_library+0x240/0x240
> [172635.399674] [<ffffffff8119999e>] search_binary_handler+0x19e/0x340
> [172635.399677] [<ffffffff8119ad15>] do_execve_common.isra.22+0x3c5/0x470
> [172635.399679] [<ffffffff8119add8>] do_execve+0x18/0x20
> [172635.399680] [<ffffffff8119b07d>] sys_execve+0x3d/0x60
> [172635.399682] [<ffffffff816ccb49>] stub_execve+0x69/0xc0
> [172635.399685] ---[ end trace 5ccb38c703860d58 ]---
> [172635.399687] ------------[ cut here ]------------
> [172635.399688] WARNING: at
> /build/buildd/linux-3.8.0/net/core/skbuff.c:573
> skb_release_head_state+0xed/0x100()
> [172635.399689] Hardware name: VPCSE190X
> [172635.399689] Modules linked in: ftdi_sio netconsole(F) configfs(F)
> ext2(F) nls_iso8859_1(F) usb_storage(F) pl2303 usbserial snd_usb_audio
> snd_usbmidi_lib parport_pc(F) ppdev(F) lp(F) parport(F) bnep rfcomm
> bluetooth binfmt_misc(F) dm_crypt(F) uvcvideo arc4(F)
> snd_hda_codec_hdmi iwldvm snd_hda_codec_realtek snd_hda_intel
> videobuf2_vmalloc videobuf2_memops mac80211 videobuf2_core
> snd_hda_codec videodev snd_hwdep(F) coretemp snd_pcm(F)
> snd_page_alloc(F) joydev(F) iwlwifi kvm_intel snd_seq_midi(F)
> snd_seq_midi_event(F) snd_rawmidi(F) kvm cfg80211 dm_multipath(F)
> snd_seq(F) snd_seq_device(F) tpm_infineon snd_timer(F) psmouse(F)
> snd(F) soundcore(F) scsi_dh serio_raw(F) sony_laptop rtsx_pci_ms mei
> microcode(F) memstick mac_hid pcspkr lpc_ich tpm_tis firewire_sbp2
> firewire_core crc_itu_t(F) btrfs(F) zlib_deflate(F) libcrc32c(F)
> hid_generic usbhid hid rtsx_pci_sdmmc ghash_clmulni_intel(F)
> aesni_intel(F) aes_x86_64(F) xts(F) lrw(F) gf128mul(F) ablk_helper(F)
> cryptd(F) radeon ahci(F) libahci(F) r8169 i915 ttm i2c_algo_bit
> drm_kms_helper video(F) drm rtsx_pci
> [172635.399725] Pid: 12575, comm: flock Tainted: GF W
> 3.8.0-7-generic #15-Ubuntu
> [172635.399726] Call Trace:
> [172635.399726] <NMI> [<ffffffff810587cf>] warn_slowpath_common+0x7f/0xc0
> [172635.399728] [<ffffffff8105882a>] warn_slowpath_null+0x1a/0x20
> [172635.399729] [<ffffffff815b51bd>] skb_release_head_state+0xed/0x100
> [172635.399731] [<ffffffff815b4fb2>] __kfree_skb+0x12/0xa0
> [172635.399732] [<ffffffff815b551c>] consume_skb+0x2c/0x80
> [172635.399735] [<ffffffffa014cf47>] rtl8169_poll+0x4b7/0x6d0 [r8169]
> [172635.399738] [<ffffffff815d96c2>] netpoll_poll_dev+0x162/0x580
> [172635.399740] [<ffffffff815b666b>] ? __alloc_skb+0x8b/0x2a0
> [172635.399742] [<ffffffff815d9c6c>] netpoll_send_skb_on_dev+0x18c/0x3a0
> [172635.399744] [<ffffffff815da198>] netpoll_send_udp+0x278/0x2a0
> [172635.399746] [<ffffffffa07c4967>] write_msg+0xc7/0x110 [netconsole]
> [172635.399749] [<ffffffff810594f1>]
> call_console_drivers.constprop.13+0x91/0x100
> [172635.399750] [<ffffffff81059e4b>] console_unlock+0x2db/0x420
> [172635.399752] [<ffffffff8105a7dd>] vprintk_emit+0x1fd/0x4e0
> [172635.399754] [<ffffffff810e8ffc>] ? watchdog_overflow_callback+0x9c/0xd0
> [172635.399756] [<ffffffff816b78c1>] printk+0x67/0x69
> [172635.399757] [<ffffffff810c0423>] print_modules+0xa3/0xd0
> [172635.399760] [<ffffffff810587ca>] warn_slowpath_common+0x7a/0xc0
> [172635.399761] [<ffffffff810588cc>] warn_slowpath_fmt+0x4c/0x50
> [172635.399763] [<ffffffff8109170d>] ? sched_clock_cpu+0xbd/0x110
> [172635.399765] [<ffffffff810e8f60>] ? touch_nmi_watchdog+0x80/0x80
> [172635.399766] [<ffffffff816c4981>] end_repeat_nmi+0x1e/0x2e
> [<ffffffff811a4631>] kill_fasync+0x61/0x90
> [172635.399793] [<ffffffff8143ad83>] account+0x113/0x1d0
> [172635.399794] [<ffffffff816c4618>] ? page_fault+0x28/0x30
> [172635.399796] [172635.399800] [<ffffffff816bb2a9>]
> create_elf_tables+0xaa/0x614
> [172635.399802] [<ffffffff811e8a84>] load_elf_binary+0xae4/0xe00
> [172635.399804] [<ffffffff811e7fa0>] ? load_elf_library+0x240/0x240
> [<ffffffff8119ad15>] do_execve_common.isra.22+0x3c5/0x470
> [172635.399810] [<ffffffff8119add8>] do_execve+0x18/0x20
> [172635.399812] [<ffffffff8119b07d>] sys_execve+0x3d/0x60
> [172635.399813] netconsole(F) configfs(F) ext2(F) bnep rfcomm
> bluetooth binfmt_misc(F) dm_crypt(F) uvcvideo arc4(F)
> snd_hda_codec_hdmi iwldvm snd_hwdep(F) coretemp snd_pcm(F)
> snd_page_alloc(F) joydev(F) iwlwifi kvm_intel snd_seq_midi(F) scsi_dh
> serio_raw(F) sony_laptop rtsx_pci_ms mei aes_x86_64(F) xts(F) lrw(F)
> gf128mul(F) ablk_helper(F) i2c_algo_bit drm_kms_helper[172635.399858]
> [<ffffffff815b51bd>] skb_release_head_state+0xed/0x100
> [172635.399860] [<ffffffff815b4fb2>] __kfree_skb+0x12/0xa0
> [172635.399861] [<ffffffff815b551c>] consume_skb+0x2c/0x80
> [<ffffffff815b666b>] ? __alloc_skb+0x8b/0x2a0
> [172635.399870] [<ffffffff815d9c6c>] netpoll_send_skb_on_dev+0x18c/0x3a0
> [<ffffffff810e8ffc>] ? watchdog_overflow_callback+0x9c/0xd0
> [172635.399884] [<ffffffff816b78c1>] printk+0x67/0x69
> [<ffffffff810588cc>] warn_slowpath_fmt+0x4c/0x50
> [172635.399891] [<ffffffff8109170d>] ? sched_clock_cpu+0xbd/0x110
> [172635.399893] [<ffffffff810e8f60>] ? touch_nmi_watchdog+0x80/0x80
> [172635.399894] [<ffffffff810e8ffc>] watchdog_overflow_callback+0x9c/0xd0
> [172635.399896] [172635.399903] [172635.399907] [172635.399912]
> [<ffffffff813536d9>] ? __read_lock_failed+0x9/0x20
> [172635.399914] [<ffffffff813536d9>] ? __read_lock_failed+0x9/0x20
> [172635.399916] <<EOE>> [<ffffffff816c3e13>] _raw_read_lock+0x13/0x20
> [172635.399918] [<ffffffff811a4532>] send_sigio+0x52/0xf0
> [<ffffffff816bb2a9>] create_elf_tables+0xaa/0x614
> [172635.399929] [172635.399939] [<ffffffff8119b07d>] sys_execve+0x3d/0x60
> [172635.399941] [<ffffffff816ccb49>] stub_execve+0x69/0xc0
> [172635.399943] ---[ end trace 5ccb38c703860d5a ]---
> [172635.399944] ------------[ cut here ]------------
> nls_iso8859_1(F) usb_storage(F) parport_pc(F) ppdev(F) bluetooth
> binfmt_misc(F) snd_hda_intel videobuf2_vmalloc videobuf2_memops
> mac80211 videobuf2_core snd_hda_codec videodev snd_hwdep(F) snd_pcm(F)
> snd_page_alloc(F) joydev(F) iwlwifi kvm_intel snd_seq_midi(F)
> snd_seq_midi_event(F) dm_multipath(F) snd_seq(F) snd_seq_device(F)
> tpm_infineon snd_timer(F)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/