HARD LOCKUP: Strange hard lock up on spin_lock(&sighand->siglock);
From: Nikolay Borisov
Date: Mon Jul 27 2015 - 08:54:18 EST
Hello,
I have a machine running kernel 3.13.3, in fact I have 3 such machines
which are connected via corosync in a cluster. On one of the machines
I observed the following lock-up:
Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------
Jul 26 06:01:14 shrek kernel: WARNING: CPU: 6 PID: 30236 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0()
Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 6
Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log
lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca
Jul 26 06:01:14 shrek kernel: CPU: 6 PID: 30236 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc06bb8 ffffffff815a6c8e 00000000000000f5
Jul 26 06:01:14 shrek kernel: ffff88087fc06c08 ffff88087fc06bf8 ffffffff81044202 ffff88087fc06bd8
Jul 26 06:01:14 shrek kernel: ffff880869940000 0000000000000000 ffff88087fc06d48 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <NMI> [<ffffffff815a6c8e>] dump_stack+0x49/0x5b
Jul 26 06:01:14 shrek kernel: [<ffffffff81044202>] warn_slowpath_common+0x82/0xb0
Jul 26 06:01:14 shrek kernel: [<ffffffff810442d1>] warn_slowpath_fmt+0x41/0x50
Jul 26 06:01:14 shrek kernel: [<ffffffff810b7898>] watchdog_overflow_callback+0x98/0xc0
Jul 26 06:01:14 shrek kernel: [<ffffffff810c00c8>] __perf_event_overflow+0x98/0x240
Jul 26 06:01:14 shrek kernel: [<ffffffff810c07a4>] perf_event_overflow+0x14/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8101c14c>] intel_pmu_handle_irq+0x1cc/0x3b0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff8131e770>] ? apei_read+0x60/0x90
Jul 26 06:01:14 shrek kernel: [<ffffffff810160c4>] perf_event_nmi_handler+0x34/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff81006e46>] nmi_handle+0x76/0x110
Jul 26 06:01:14 shrek kernel: [<ffffffff81006fdd>] do_nmi+0xfd/0x360
Jul 26 06:01:14 shrek kernel: [<ffffffff815abc2a>] end_repeat_nmi+0x1e/0x32
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: <<EOE>> [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84d ]---
Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------
Jul 26 06:01:14 shrek kernel: WARNING: CPU: 9 PID: 30215 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0()
Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 9
Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log
lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca
Jul 26 06:01:14 shrek kernel: CPU: 9 PID: 30215 Comm: pkill Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc66bb8 ffffffff815a6c8e 00000000000000f5
Jul 26 06:01:14 shrek kernel: ffff88087fc66c08 ffff88087fc66bf8 ffffffff81044202 ffff88087fc66b10
Jul 26 06:01:14 shrek kernel: ffff880869990000 0000000000000000 ffff88087fc66d48 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <NMI> [<ffffffff815a6c8e>] dump_stack+0x49/0x5b
Jul 26 06:01:14 shrek kernel: [<ffffffff81044202>] warn_slowpath_common+0x82/0xb0
Jul 26 06:01:14 shrek kernel: [<ffffffff810442d1>] warn_slowpath_fmt+0x41/0x50
Jul 26 06:01:14 shrek kernel: [<ffffffff810b7898>] watchdog_overflow_callback+0x98/0xc0
Jul 26 06:01:14 shrek kernel: [<ffffffff810c00c8>] __perf_event_overflow+0x98/0x240
Jul 26 06:01:14 shrek kernel: [<ffffffff8102d460>] ? hw_nmi_get_sample_period+0x20/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810c07a4>] perf_event_overflow+0x14/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8101c14c>] intel_pmu_handle_irq+0x1cc/0x3b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8131e770>] ? apei_read+0x60/0x90
Jul 26 06:01:14 shrek kernel: [<ffffffff810160c4>] perf_event_nmi_handler+0x34/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff81006e46>] nmi_handle+0x76/0x110
Jul 26 06:01:14 shrek kernel: [<ffffffff81006fdd>] do_nmi+0xfd/0x360
Jul 26 06:01:14 shrek kernel: [<ffffffff815abc2a>] end_repeat_nmi+0x1e/0x32
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15f>] ? _raw_spin_lock+0x1f/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15f>] ? _raw_spin_lock+0x1f/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15f>] ? _raw_spin_lock+0x1f/0x30
Jul 26 06:01:14 shrek kernel: <<EOE>> [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119ecca>] proc_pid_status+0x3ea/0x7c0
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84e ]---
Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------
Jul 26 06:01:14 shrek kernel: WARNING: CPU: 10 PID: 30390 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0()
Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 10
Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log
lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca
Jul 26 06:01:14 shrek kernel: CPU: 10 PID: 30390 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc86bb8 ffffffff815a6c8e 00000000000000f5
Jul 26 06:01:14 shrek kernel: ffff88087fc86c08 ffff88087fc86bf8 ffffffff81044202 ffff88087fc86bd8
Jul 26 06:01:14 shrek kernel: ffff8808699b0000 0000000000000000 ffff88087fc86d48 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <NMI> [<ffffffff815a6c8e>] dump_stack+0x49/0x5b
Jul 26 06:01:14 shrek kernel: [<ffffffff81044202>] warn_slowpath_common+0x82/0xb0
Jul 26 06:01:14 shrek kernel: [<ffffffff810442d1>] warn_slowpath_fmt+0x41/0x50
Jul 26 06:01:14 shrek kernel: [<ffffffff810b7898>] watchdog_overflow_callback+0x98/0xc0
Jul 26 06:01:14 shrek kernel: [<ffffffff810c00c8>] __perf_event_overflow+0x98/0x240
Jul 26 06:01:14 shrek kernel: [<ffffffff810c07a4>] perf_event_overflow+0x14/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8101c14c>] intel_pmu_handle_irq+0x1cc/0x3b0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff8131e770>] ? apei_read+0x60/0x90
Jul 26 06:01:14 shrek kernel: [<ffffffff810160c4>] perf_event_nmi_handler+0x34/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff81006e46>] nmi_handle+0x76/0x110
Jul 26 06:01:14 shrek kernel: [<ffffffff81006fdd>] do_nmi+0xfd/0x360
Jul 26 06:01:14 shrek kernel: [<ffffffff815abc2a>] end_repeat_nmi+0x1e/0x32
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: <<EOE>> [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84f ]---
Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------
Jul 26 06:01:14 shrek kernel: WARNING: CPU: 2 PID: 30144 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0()
Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 2
Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log
lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca
Jul 26 06:01:14 shrek kernel: CPU: 2 PID: 30144 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88046fc46bb8 ffffffff815a6c8e 00000000000000f5
Jul 26 06:01:14 shrek kernel: ffff88046fc46c08 ffff88046fc46bf8 ffffffff81044202 ffff880867bd2510
Jul 26 06:01:14 shrek kernel: ffff880469de0000 0000000000000000 ffff88046fc46d48 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <NMI> [<ffffffff815a6c8e>] dump_stack+0x49/0x5b
Jul 26 06:01:14 shrek kernel: [<ffffffff81044202>] warn_slowpath_common+0x82/0xb0
Jul 26 06:01:14 shrek kernel: [<ffffffff810442d1>] warn_slowpath_fmt+0x41/0x50
Jul 26 06:01:14 shrek kernel: [<ffffffff810b7898>] watchdog_overflow_callback+0x98/0xc0
Jul 26 06:01:14 shrek kernel: [<ffffffff810c00c8>] __perf_event_overflow+0x98/0x240
Jul 26 06:01:14 shrek kernel: [<ffffffff8102d460>] ? hw_nmi_get_sample_period+0x20/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810c07a4>] perf_event_overflow+0x14/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8101c14c>] intel_pmu_handle_irq+0x1cc/0x3b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8131e770>] ? apei_read+0x60/0x90
Jul 26 06:01:14 shrek kernel: [<ffffffff810160c4>] perf_event_nmi_handler+0x34/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff81006e46>] nmi_handle+0x76/0x110
Jul 26 06:01:14 shrek kernel: [<ffffffff81006fdd>] do_nmi+0xfd/0x360
Jul 26 06:01:14 shrek kernel: [<ffffffff815abc2a>] end_repeat_nmi+0x1e/0x32
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: <<EOE>> [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed850 ]---
Jul 26 06:01:14 shrek kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 8 18} (detected by 12, t=14712 jiffies, g=115985071, c=115985070, q=186073)
Jul 26 06:01:14 shrek kernel: sending NMI to all CPUs:
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 1
Jul 26 06:01:14 shrek kernel: CPU: 1 PID: 30726 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff88013d551260 ti: ffff88013d5517b8 task.ti: ffff88013d5517b8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880300eb5b08 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 000000000000000a RSI: ffff880300eb5c30 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff880300eb5b08 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff880300eb5c30 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 0000034a18fe5700(0000) GS:ffff88046fc20000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000000019bfc18 CR3: 00000000015ba000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880300eb5b38 ffffffff81053474 ffff880300ff2e10 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880300eb5c98 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 0
Jul 26 06:01:14 shrek kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffffffff81a0a920 ti: ffffffff81a0ae78 task.ti: ffffffff81a0ae78
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffffffff81a01ec8 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffffffff81a0ae80 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fc0c200
Jul 26 06:01:14 shrek kernel: RBP: ffffffff81a01ec8 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81a0ae80
Jul 26 06:01:14 shrek kernel: R13: ffffffff81a0ae80 R14: ffffffff81afa5c8 R15: ffffffff81a0ae80
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fc00000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 000002dd81723ae0 CR3: 00000000015b8000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffffffff81a01ed8 ffffffff8100c6b6 ffffffff81a01f48 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000286 ffffffff81a0ae78 ffffffff81a0ae80 ffffffff81a0ae78
Jul 26 06:01:14 shrek kernel: ffffffff81885200 77e46a1d0cffc667 00000000000937e0 ffffffff81c940c0
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8159ea42>] rest_init+0x72/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff81c11069>] start_kernel+0x3f5/0x3fc
Jul 26 06:01:14 shrek kernel: [<ffffffff81c10aa9>] ? repair_env_string+0x62/0x62
Jul 26 06:01:14 shrek kernel: [<ffffffff815a5eac>] ? memblock_reserve+0x46/0x4b
Jul 26 06:01:14 shrek kernel: [<ffffffff81c10120>] ? early_idt_handlers+0x120/0x120
Jul 26 06:01:14 shrek kernel: [<ffffffff81c104ed>] x86_64_start_reservations+0x2a/0x2c
Jul 26 06:01:14 shrek kernel: [<ffffffff81c10643>] x86_64_start_kernel+0x154/0x15f
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 2
Jul 26 06:01:14 shrek kernel: CPU: 2 PID: 30144 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880867824980 ti: ffff880867824ed8 task.ti: ffff880867824ed8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15f>] [<ffffffff815ab15f>] ffffffff815ab15f
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88063e911bc8 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000004 RSI: ffff88063e911cf0 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff88063e911bc8 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff88063e911cf0 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 000002d271e24700(0000) GS:ffff88046fc40000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000000010da108 CR3: 00000000015bc000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff88063e911bf8 ffffffff81053474 ffff88037e541ce8 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff88063e911d58 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 3
Jul 26 06:01:14 shrek kernel: CPU: 3 PID: 1270 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff8806883ad2b0 ti: ffff8806883ad808 task.ti: ffff8806883ad808
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15a>] [<ffffffff815ab15a>] ffffffff815ab15a
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8805b3375b48 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 000000000000000b RSI: ffff8805b3375c70 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8805b3375b48 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8805b3375c70 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 0000029bd8f09700(0000) GS:ffff88046fc60000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000003ca0498 CR3: 00000000015be000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8805b3375b78 ffffffff81053474 ffff88010261a258 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8805b3375cd8 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: b6 03 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 <f3> 90 0f b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 4
Jul 26 06:01:14 shrek kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d70930 ti: ffff880469d70e88 task.ti: ffff880469d70e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d79e88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d70e90 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fc8c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d79e88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d70e90
Jul 26 06:01:14 shrek kernel: R13: ffff880469d70e90 R14: ffffffff81afa5c8 R15: ffff880469d70e90
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fc80000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 000002e423d730a0 CR3: 00000000015c0000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d79e98 ffffffff8100c6b6 ffff880469d79f08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000004 ffff880469d70e88 ffff880469d70e90 ffff880469d70e88
Jul 26 06:01:14 shrek kernel: 000000000043fd9f b6726a4c2c676c66 ffff880469d79ef8 0000000000000004
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 5
Jul 26 06:01:14 shrek kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d71260 ti: ffff880469d717b8 task.ti: ffff880469d717b8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d7be88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d717c0 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fcac200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d7be88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d717c0
Jul 26 06:01:14 shrek kernel: R13: ffff880469d717c0 R14: ffffffff81afa5c8 R15: ffff880469d717c0
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fca0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015c2000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d7be98 ffffffff8100c6b6 ffff880469d7bf08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000005 ffff880469d717b8 ffff880469d717c0 ffff880469d717b8
Jul 26 06:01:14 shrek kernel: 0000000000000206 a8d39d71d1e5d619 ffff880469d7bef8 0000000000000005
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 6
Jul 26 06:01:14 shrek kernel: INFO: NMI handler (ffffffff8102d460) took too long to run: 45.273 msecs
Jul 26 06:01:14 shrek kernel: CPU: 6 PID: 30236 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff8806a1f78930 ti: ffff8806a1f78e88 task.ti: ffff8806a1f78e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880801077be8 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000008 RSI: ffff880801077d10 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff880801077be8 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff880801077d10 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 000002fb353c6700(0000) GS:ffff88087fc00000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000000021c82a8 CR3: 00000000015c4000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880801077c18 ffffffff81053474 ffff88012c217b58 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880801077d78 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: INFO: NMI handler (ffffffff8102d460) took too long to run: 52.833 msecs
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 7
Jul 26 06:01:14 shrek kernel: CPU: 7 PID: 29973 Comm: ps Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880867821b90 ti: ffff8808678220e8 task.ti: ffff8808678220e8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807ed2fdb08 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000002 RSI: ffff8807ed2fdc30 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8807ed2fdb08 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8807ed2fdc30 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 00000344c1e647a0(0000) GS:ffff88087fc20000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000344c1e011dc CR3: 00000000015c6000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8807ed2fdb38 ffffffff81053474 ffff880122b384b0 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8807ed2fdc98 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 8
Jul 26 06:01:14 shrek kernel: CPU: 8 PID: 29979 Comm: sg-queue Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880622796e40 ti: ffff880622797398 task.ti: ffff880622797398
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab0bd>] [<ffffffff815ab0bd>] ffffffff815ab0bd
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88067e6a1e68 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: ffff880622796e40
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: ffff88067e6a1eb8 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff88067e6a1e68 R08: ffff88067e6a1eb8 R09: 000000000000751a
Jul 26 06:01:14 shrek kernel: R10: 0000000000000008 R11: 0000000000000246 R12: ffff88067e6a1e98
Jul 26 06:01:14 shrek kernel: R13: ffff88067e6a1eb8 R14: 0000000002d7c680 R15: 0000000002d7ccb0
Jul 26 06:01:14 shrek kernel: FS: 0000037e10229700(0000) GS:ffff88087fc40000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000037e0f8978d0 CR3: 00000000015c8000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff88067e6a1e88 ffffffff81053620 0000000000000000 0000000000000002
Jul 26 06:01:14 shrek kernel: ffff88067e6a1ea8 ffffffff8105369a 0000000000000000 0000000002d7ccb0
Jul 26 06:01:14 shrek kernel: ffff88067e6a1ef8 ffffffff81056fcb 0000000000000000 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053620>] __set_current_blocked+0x30/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff8105369a>] sigprocmask+0x4a/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff81056fcb>] SyS_rt_sigprocmask+0xbb/0xd0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab6bd>] ? retint_swapgs+0xe/0x11
Jul 26 06:01:14 shrek kernel: Code: 48 83 c4 08 b8 01 00 00 00 5b c9 c3 66 90 55 48 89 e5 fa b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 9
Jul 26 06:01:14 shrek kernel: CPU: 9 PID: 30215 Comm: pkill Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff88072c811260 ti: ffff88072c8117b8 task.ti: ffff88072c8117b8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807ed29fbc8 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000007 RSI: ffff8807ed29fd00 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8807ed29fbc8 R08: 0000000000000001 R09: 000000000000fffd
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8807ed29fd00 R14: ffff880622797420 R15: 0000000000000000
Jul 26 06:01:14 shrek kernel: FS: 000002e88ef8a700(0000) GS:ffff88087fc60000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 000000000459c008 CR3: 00000000015ca000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8807ed29fbf8 ffffffff81053474 ffff88010a1a6640 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff8807ed29fd58 ffffffff8119ecca
Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119ecca>] proc_pid_status+0x3ea/0x7c0
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 10
Jul 26 06:01:14 shrek kernel: CPU: 10 PID: 30390 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff8804b4c60930 ti: ffff8804b4c60e88 task.ti: ffff8804b4c60e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15f>] [<ffffffff815ab15f>] ffffffff815ab15f
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880594293b08 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000009 RSI: ffff880594293c30 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff880594293b08 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff880594293c30 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 0000031905858700(0000) GS:ffff88087fc80000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000000009fc3c8 CR3: 00000000015cc000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880594293b38 ffffffff81053474 ffff880101077450 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880594293c98 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 11
Jul 26 06:01:14 shrek kernel: CPU: 11 PID: 30213 Comm: pkill Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff8804a95b1260 ti: ffff8804a95b17b8 task.ti: ffff8804a95b17b8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15f>] [<ffffffff815ab15f>] ffffffff815ab15f
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880696bf3ba8 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000006 RSI: ffff880696bf3ce0 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff880696bf3ba8 R08: 0000000000000001 R09: 000000000000fffd
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff880696bf3ce0 R14: ffff880622797420 R15: 0000000000000000
Jul 26 06:01:14 shrek kernel: FS: 000002bb0925b700(0000) GS:ffff88087fca0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000000be3018 CR3: 00000000015ce000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880696bf3bd8 ffffffff81053474 ffff88007993a3e8 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff880696bf3d38 ffffffff8119ecca
Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119ecca>] proc_pid_status+0x3ea/0x7c0
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 12
Jul 26 06:01:14 shrek kernel: CPU: 12 PID: 0 Comm: swapper/12 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d752b0 ti: ffff880469d75808 task.ti: ffff880469d75808
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff812ab2c4>] [<ffffffff812ab2c4>] ffffffff812ab2c4
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88046fcc3d38 EFLAGS: 00000046
Jul 26 06:01:14 shrek kernel: RAX: 00000000a7bd4e81 RBX: 0000000000000000 RCX: 00000000a7bd4e5d
Jul 26 06:01:14 shrek kernel: RDX: 000000000000000c RSI: 000000000000000c RDI: 0000000000033452
Jul 26 06:01:14 shrek kernel: RBP: ffff88046fcc3d38 R08: ffffffff81afa5c8 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000001848 R11: 000000002b300043 R12: 0000000000001000
Jul 26 06:01:14 shrek kernel: R13: ffffffff81afa5c8 R14: 0000000000000400 R15: 0000000000000096
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fcc0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d0000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff88046fcc3d48 ffffffff812ab289 ffff88046fcc3d68 ffffffff8102ca62
Jul 26 06:01:14 shrek kernel: 0000000000000002 000000000000a024 ffff88046fcc3db8 ffffffff8102d428
Jul 26 06:01:14 shrek kernel: ffffffff00000003 000000000000000d ffff88046fcc3d98 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <IRQ>
Jul 26 06:01:14 shrek kernel: [<ffffffff812ab289>] __const_udelay+0x29/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff8102ca62>] native_safe_apic_wait_icr_idle+0x22/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff8102d428>] default_send_IPI_mask_sequence_phys+0xb8/0xd0
Jul 26 06:01:14 shrek kernel: [<ffffffff810306c2>] physflat_send_IPI_all+0x12/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8102d537>] arch_trigger_all_cpu_backtrace+0x47/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff81091e87>] rcu_check_callbacks+0x617/0x640
Jul 26 06:01:14 shrek kernel: [<ffffffff810515c3>] update_process_times+0x43/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8109c2a1>] tick_sched_timer+0x51/0x130
Jul 26 06:01:14 shrek kernel: [<ffffffff8106980f>] __run_hrtimer+0x5f/0x130
Jul 26 06:01:14 shrek kernel: [<ffffffff81069cc6>] hrtimer_interrupt+0x106/0x250
Jul 26 06:01:14 shrek kernel: [<ffffffff8102c1f6>] local_apic_timer_interrupt+0x36/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff8102cade>] smp_apic_timer_interrupt+0x3e/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff815acba3>] apic_timer_interrupt+0x83/0x90
Jul 26 06:01:14 shrek kernel: <EOI>
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c456>] ? default_idle+0x6/0x10
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: a0 00 00 0f 1f 00 0f ae e8 0f 31 89 c1 eb 16 0f 1f 84 00 00 00 00 00 f3 90 65 8b 14 25 20 a0 00 00 39 d6 75 12 0f 1f 00 0f ae e8 <0f> 31 89 c2 29 ca 39 fa 72 e2 c9 c3 29 c1 89 d6 01 cf 0f 1f 00
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 13
Jul 26 06:01:14 shrek kernel: CPU: 13 PID: 0 Comm: swapper/13 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d75be0 ti: ffff880469d76138 task.ti: ffff880469d76138
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8be88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d76140 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fcec200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8be88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d76140
Jul 26 06:01:14 shrek kernel: R13: ffff880469d76140 R14: ffffffff81afa5c8 R15: ffff880469d76140
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d2000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d8be98 ffffffff8100c6b6 ffff880469d8bf08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 000000000000000d ffff880469d76138 ffff880469d76140 ffff880469d76138
Jul 26 06:01:14 shrek kernel: 0043fd9fa00c6708 5e3321e5c10994cc ffff880469d8bef8 000000000000000d
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 14
Jul 26 06:01:14 shrek kernel: CPU: 14 PID: 0 Comm: swapper/14 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d76510 ti: ffff880469d76a68 task.ti: ffff880469d76a68
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8de88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d76a70 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd0c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8de88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d76a70
Jul 26 06:01:14 shrek kernel: R13: ffff880469d76a70 R14: ffffffff81afa5c8 R15: ffff880469d76a70
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd00000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000008b70820 CR3: 00000000015d4000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d8de98 ffffffff8100c6b6 ffff880469d8df08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 000000000000000e ffff880469d76a68 ffff880469d76a70 ffff880469d76a68
Jul 26 06:01:14 shrek kernel: 0043fd9fa1d85012 617db263e885420d ffff880469d8def8 000000000000000e
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 15
Jul 26 06:01:14 shrek kernel: CPU: 15 PID: 0 Comm: swapper/15 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d76e40 ti: ffff880469d77398 task.ti: ffff880469d77398
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8fe88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d773a0 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd2c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8fe88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d773a0
Jul 26 06:01:14 shrek kernel: R13: ffff880469d773a0 R14: ffffffff81afa5c8 R15: ffff880469d773a0
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd20000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d6000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d8fe98 ffffffff8100c6b6 ffff880469d8ff08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 000000000000000f ffff880469d77398 ffff880469d773a0 ffff880469d77398
Jul 26 06:01:14 shrek kernel: 0043fd9fa3a15351 a19db38607e2294d ffff880469d8fef8 000000000000000f
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 16
Jul 26 06:01:14 shrek kernel: CPU: 16 PID: 0 Comm: swapper/16 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d90000 ti: ffff880469d90558 task.ti: ffff880469d90558
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d99e88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d90560 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd4c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d99e88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d90560
Jul 26 06:01:14 shrek kernel: R13: ffff880469d90560 R14: ffffffff81afa5c8 R15: ffff880469d90560
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd40000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000001758000 CR3: 00000000015d8000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d99e98 ffffffff8100c6b6 ffff880469d99f08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000010 ffff880469d90558 ffff880469d90560 ffff880469d90558
Jul 26 06:01:14 shrek kernel: 0043fd9fa56a1bdf 2e70ed886688fb1c ffff880469d99ef8 0000000000000010
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 17
Jul 26 06:01:14 shrek kernel: CPU: 17 PID: 0 Comm: swapper/17 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d90930 ti: ffff880469d90e88 task.ti: ffff880469d90e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d9be88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d90e90 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd6c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d9be88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d90e90
Jul 26 06:01:14 shrek kernel: R13: ffff880469d90e90 R14: ffffffff81afa5c8 R15: ffff880469d90e90
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd60000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000034fa60bf398 CR3: 00000000015da000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d9be98 ffffffff8100c6b6 ffff880469d9bf08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000011 ffff880469d90e88 ffff880469d90e90 ffff880469d90e88
Jul 26 06:01:14 shrek kernel: ffff880469d9beb2 05735dbd652beb34 ffff880469d9bef8 0000000000000011
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 18
Jul 26 06:01:14 shrek kernel: CPU: 18 PID: 29972 Comm: ps Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880622792df0 ti: ffff880622793348 task.ti: ffff880622793348
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8806db9e3b38 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000001 RSI: ffff8806db9e3c60 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8806db9e3b38 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8806db9e3c60 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 000002969e6097a0(0000) GS:ffff88087fcc0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 000002969e5e01dc CR3: 00000000015dc000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8806db9e3b68 ffffffff81053474 ffff880129ded2c0 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8806db9e3cc8 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 19
Jul 26 06:01:14 shrek kernel: CPU: 19 PID: 0 Comm: swapper/19 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d91b90 ti: ffff880469d920e8 task.ti: ffff880469d920e8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d9fe88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d920f0 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fcec200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d9fe88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d920f0
Jul 26 06:01:14 shrek kernel: R13: ffff880469d920f0 R14: ffffffff81afa5c8 R15: ffff880469d920f0
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fce0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000001816cf8 CR3: 00000000015de000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d9fe98 ffffffff8100c6b6 ffff880469d9ff08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000013 ffff880469d920e8 ffff880469d920f0 ffff880469d920e8
Jul 26 06:01:14 shrek kernel: 0043fd9faac3c713 e2b091c5f11a3308 ffff880469d9fef8 0000000000000013
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 20
Jul 26 06:01:14 shrek kernel: CPU: 20 PID: 0 Comm: swapper/20 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d924c0 ti: ffff880469d92a18 task.ti: ffff880469d92a18
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469da1e88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d92a20 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fd0c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469da1e88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d92a20
Jul 26 06:01:14 shrek kernel: R13: ffff880469d92a20 R14: ffffffff81afa5c8 R15: ffff880469d92a20
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015e0000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469da1e98 ffffffff8100c6b6 ffff880469da1f08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000014 ffff880469d92a18 ffff880469d92a20 ffff880469d92a18
Jul 26 06:01:14 shrek kernel: 0043fd9fac8b8e6d 0fa0aa77f886bbaa ffff880469da1ef8 0000000000000014
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 21
Jul 26 06:01:14 shrek kernel: CPU: 21 PID: 30214 Comm: pkill Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880867a2ee40 ti: ffff880867a2f398 task.ti: ffff880867a2f398
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8804b4dc9b68 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000005 RSI: ffff8804b4dc9ca0 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8804b4dc9b68 R08: 0000000000000001 R09: 000000000000fffd
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8804b4dc9ca0 R14: ffff880622797420 R15: 0000000000000000
Jul 26 06:01:14 shrek kernel: FS: 0000032a2ff54700(0000) GS:ffff88087fd20000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000001319008 CR3: 00000000015e2000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8804b4dc9b98 ffffffff81053474 ffff8800794e9c20 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff8804b4dc9cf8 ffffffff8119ecca
Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119ecca>] proc_pid_status+0x3ea/0x7c0
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 22
Jul 26 06:01:14 shrek kernel: CPU: 22 PID: 30017 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880490800930 ti: ffff880490800e88 task.ti: ffff880490800e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15f>] [<ffffffff815ab15f>] ffffffff815ab15f
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807f1d03b08 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000003 RSI: ffff8807f1d03c30 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8807f1d03b08 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8807f1d03c30 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 0000025ccaa4c700(0000) GS:ffff88087fd40000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000003f2a4f8 CR3: 00000000015e4000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8807f1d03b38 ffffffff81053474 ffff88013bb36190 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8807f1d03c98 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 23
Jul 26 06:01:14 shrek kernel: CPU: 23 PID: 0 Comm: swapper/23 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d94050 ti: ffff880469d945a8 task.ti: ffff880469d945a8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469da7e88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d945b0 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fd6c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469da7e88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d945b0
Jul 26 06:01:14 shrek kernel: R13: ffff880469d945b0 R14: ffffffff81afa5c8 R15: ffff880469d945b0
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fd60000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015e6000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469da7e98 ffffffff8100c6b6 ffff880469da7f08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000017 ffff880469d945a8 ffff880469d945b0 ffff880469d945a8
Jul 26 06:01:14 shrek kernel: 0043fd9fb1e28232 a8f790adc213c71a ffff880469da7ef8 0000000000000017
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
My investigation showed up that the CPUs are in the following state:
CPUS 1, 2, 3, 6, 7, 8, 9, 10, 11, 18, 21, 22 are locked on the spinlock
inside __lock_task_sighand, in a rcu_read_lock() section. Whereas
CPUS: 4, 5, 13, 14, 15, 16, 17, 19, 20, 23 are staying idle.
>From the logs I gather that 2 different subsystems detected that:
the NMI watchdog and the rcu stall detector (cpu's 8 and 18).
While looking at the code and googling around I discovered the
following commit from PeterZ: https://lkml.org/lkml/2008/11/13/157
Based on that I think could be happening is that the sighand itself is
being freed while we are in the grace period inside __lock_task_sighand
but the slab page itself is not freed as per the semantics of
SLAB_DESTROY_BY_RCU. I looked up the source of this function in the
latest kernels and saw that Oleg had put a comment clarifying the
semantics but I'm still not convinced that it is safe. What if
we are trying to lock the spinlock before this particular slab is
initialised with sighand_ctor? Looking at the code and reading
Peter's explanation of SLAB_RCU I cannot identify which portion
of __lock_task_sighand implements the following logic:
if (!try_get_ref(obj)) // might fail for free objects
goto again;
Also what's more peculiar was the fact that when "Shrek" experienced
this issue, resources were migrated to the 2nd machine and it
also experienced the same behavior - CPUs locked on the sighand
spinlock.
That's the first time I've seen this particular issue. And I'm not
really able to come up with a way where a task's sighand could be
corrupted in a such a way as to prevent acquiring the lock.
Regards,
Nikolay
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/