Re: [2.6.30-rc1] RCU detected CPU 1 stall
From: Tetsuo Handa
Date: Sun Apr 12 2009 - 20:51:20 EST
Hello.
Paul E. McKenney wrote:
> Is this reproducible?
Not always, but it is reproducible.
Al Viro wrote:
> I'd really love to see results of repeated alt-sysrq-p/alt-sysrq-l, just
> to see where was it actually spinning.
Below is sysrq message.
Maybe something related to khelper's current->mm == NULL warning problem.
Full log is at http://I-love.SAKURA.ne.jp/tmp/dmesg-2.6.30-rc1-200904130930.txt .
[ 47.412519] SysRq : Show Regs
[ 47.413986]
[ 47.414584] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform
[ 47.415804] EIP: 0060:[<c0379c3d>] EFLAGS: 00000293 CPU: 0
[ 47.415804] EIP is at __get_user_4+0x11/0x17
[ 47.415804] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000
[ 47.415804] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10
[ 47.415804] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 47.415804] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0
[ 47.415804] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 47.415804] DR6: ffff0ff0 DR7: 00000400
[ 47.415804] Call Trace:
[ 47.415804] [<c0225f4e>] ? count+0x3e/0xb0
[ 47.415804] [<c0228581>] do_execve+0x621/0x890
[ 47.415804] [<c022bd8b>] ? getname+0x6b/0xa0
[ 47.415804] [<c010237e>] sys_execve+0x5e/0xb0
[ 47.415804] [<c0103d19>] syscall_call+0x7/0xb
[ 47.415804] [<c010aee4>] ? kernel_execve+0x24/0x30
[ 47.415804] [<c0172b6f>] ? ____call_usermodehelper+0xff/0x170
[ 47.415804] [<c0172a70>] ? ____call_usermodehelper+0x0/0x170
[ 47.415804] [<c0104707>] ? kernel_thread_helper+0x7/0x10
[ 52.551469] SysRq : Show backtrace of all active CPUs
[ 52.551803] CPU0:
[ 52.551803]
[ 52.551803] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform
[ 52.551803] EIP: 0060:[<c0379c3d>] EFLAGS: 00000293 CPU: 0
[ 52.551803] EIP is at __get_user_4+0x11/0x17
[ 52.551803] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000
[ 52.551803] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10
[ 52.551803] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 52.551803] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0
[ 52.551803] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 52.551803] DR6: ffff0ff0 DR7: 00000400
[ 52.551803] Call Trace:
[ 52.551803] [<c0225f4e>] ? count+0x3e/0xb0
[ 52.551803] [<c0228581>] do_execve+0x621/0x890
[ 52.551803] [<c022bd8b>] ? getname+0x6b/0xa0
[ 52.551803] [<c010237e>] sys_execve+0x5e/0xb0
[ 52.551803] [<c0103d19>] syscall_call+0x7/0xb
[ 52.551803] [<c010aee4>] ? kernel_execve+0x24/0x30
[ 52.551803] [<c0172b6f>] ? ____call_usermodehelper+0xff/0x170
[ 52.551803] [<c0172a70>] ? ____call_usermodehelper+0x0/0x170
[ 52.551803] [<c0104707>] ? kernel_thread_helper+0x7/0x10
[ 60.956967] SysRq : Show Regs
[ 60.958362]
[ 60.958896] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform
[ 60.959802] EIP: 0060:[<c0379c3d>] EFLAGS: 00000293 CPU: 0
[ 60.959802] EIP is at __get_user_4+0x11/0x17
[ 60.959802] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000
[ 60.959802] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10
[ 60.959802] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 60.959802] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0
[ 60.959802] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 60.959802] DR6: ffff0ff0 DR7: 00000400
[ 60.959802] Call Trace:
[ 60.959802] [<c0225f4e>] ? count+0x3e/0xb0
[ 60.959802] [<c0228581>] do_execve+0x621/0x890
[ 60.959802] [<c022bd8b>] ? getname+0x6b/0xa0
[ 60.959802] [<c010237e>] sys_execve+0x5e/0xb0
[ 60.959802] [<c0103d19>] syscall_call+0x7/0xb
[ 60.959802] [<c010aee4>] ? kernel_execve+0x24/0x30
[ 60.959802] [<c0172b6f>] ? ____call_usermodehelper+0xff/0x170
[ 60.959802] [<c0172a70>] ? ____call_usermodehelper+0x0/0x170
[ 60.959802] [<c0104707>] ? kernel_thread_helper+0x7/0x10
[ 69.014489] SysRq : Show backtrace of all active CPUs
[ 69.015802] CPU0:
[ 69.015802]
[ 69.015802] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform
[ 69.015802] EIP: 0060:[<c0379c3d>] EFLAGS: 00000293 CPU: 0
[ 69.015802] EIP is at __get_user_4+0x11/0x17
[ 69.015802] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000
[ 69.015802] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10
[ 69.015802] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 69.015802] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0
[ 69.015802] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 69.015802] DR6: ffff0ff0 DR7: 00000400
[ 69.015802] Call Trace:
[ 69.015802] [<c0225f4e>] ? count+0x3e/0xb0
[ 69.015802] [<c0228581>] do_execve+0x621/0x890
[ 69.015802] [<c022bd8b>] ? getname+0x6b/0xa0
[ 69.015802] [<c010237e>] sys_execve+0x5e/0xb0
[ 69.015802] [<c0103d19>] syscall_call+0x7/0xb
[ 69.015802] [<c010aee4>] ? kernel_execve+0x24/0x30
[ 69.015802] [<c0172b6f>] ? ____call_usermodehelper+0xff/0x170
[ 69.015802] [<c0172a70>] ? ____call_usermodehelper+0x0/0x170
[ 69.015802] [<c0104707>] ? kernel_thread_helper+0x7/0x10
[ 73.695801] INFO: RCU detected CPU 0 stall (t=4294910558/10000 jiffies)
[ 73.695801] Pid: 3655, comm: khelper Tainted: G W 2.6.30-rc1 #1
[ 73.695801] Call Trace:
[ 73.695801] [<c015949d>] ? printk+0x1d/0x30
[ 73.695801] [<c01b38b5>] print_cpu_stall+0x45/0xa0
[ 73.695801] [<c01b39f8>] check_cpu_stall+0xe8/0x190
[ 73.695801] [<c01ad59d>] ? get_timestamp+0xd/0x20
[ 73.695801] [<c01b43ab>] __rcu_pending+0x1b/0x160
[ 73.695801] [<c01b451e>] rcu_pending+0x2e/0x70
[ 73.695801] [<c01692ab>] update_process_times+0x3b/0x80
[ 73.695801] [<c018bcf0>] tick_periodic+0x40/0x90
[ 73.695801] [<c018bd5e>] tick_handle_periodic+0x1e/0xa0
[ 73.695801] [<c01045e0>] ? apic_timer_interrupt+0x28/0x34
[ 73.695801] [<c012215f>] local_apic_timer_interrupt+0x6f/0x80
[ 73.695801] [<c0161184>] ? irq_enter+0x14/0x60
[ 73.695801] [<c05e7d83>] smp_apic_timer_interrupt+0x33/0x42
[ 73.695801] [<c01045e7>] apic_timer_interrupt+0x2f/0x34
[ 73.695801] [<c0379c3d>] ? __get_user_4+0x11/0x17
[ 73.695801] [<c0225f4e>] ? count+0x3e/0xb0
[ 73.695801] [<c0228581>] do_execve+0x621/0x890
[ 73.695801] [<c022bd8b>] ? getname+0x6b/0xa0
[ 73.695801] [<c010237e>] sys_execve+0x5e/0xb0
[ 73.695801] [<c0103d19>] syscall_call+0x7/0xb
[ 73.695801] [<c010aee4>] ? kernel_execve+0x24/0x30
[ 73.695801] [<c0172b6f>] ? ____call_usermodehelper+0xff/0x170
[ 73.695801] [<c0172a70>] ? ____call_usermodehelper+0x0/0x170
[ 73.695801] [<c0104707>] ? kernel_thread_helper+0x7/0x10
[ 79.029887] SysRq : Show Regs
[ 79.030975]
[ 79.031578] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform
[ 79.031800] EIP: 0060:[<c0379c3d>] EFLAGS: 00000293 CPU: 0
[ 79.031800] EIP is at __get_user_4+0x11/0x17
[ 79.031800] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000
[ 79.031800] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10
[ 79.031800] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 79.031800] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0
[ 79.031800] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 79.031800] DR6: ffff0ff0 DR7: 00000400
[ 79.031800] Call Trace:
[ 79.031800] [<c0225f4e>] ? count+0x3e/0xb0
[ 79.031800] [<c0228581>] do_execve+0x621/0x890
[ 79.031800] [<c022bd8b>] ? getname+0x6b/0xa0
[ 79.031800] [<c010237e>] sys_execve+0x5e/0xb0
[ 79.031800] [<c0103d19>] syscall_call+0x7/0xb
[ 79.031800] [<c010aee4>] ? kernel_execve+0x24/0x30
[ 79.031800] [<c0172b6f>] ? ____call_usermodehelper+0xff/0x170
[ 79.031800] [<c0172a70>] ? ____call_usermodehelper+0x0/0x170
[ 79.031800] [<c0104707>] ? kernel_thread_helper+0x7/0x10
[ 87.925907] SysRq : Show backtrace of all active CPUs
[ 87.927799] CPU0:
[ 87.927799]
[ 87.927799] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform
[ 87.927799] EIP: 0060:[<c0379c3d>] EFLAGS: 00000293 CPU: 0
[ 87.927799] EIP is at __get_user_4+0x11/0x17
[ 87.927799] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000
[ 87.927799] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10
[ 87.927799] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 87.927799] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0
[ 87.927799] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 87.927799] DR6: ffff0ff0 DR7: 00000400
[ 87.927799] Call Trace:
[ 87.927799] [<c0225f4e>] ? count+0x3e/0xb0
[ 87.927799] [<c0228581>] do_execve+0x621/0x890
[ 87.927799] [<c022bd8b>] ? getname+0x6b/0xa0
[ 87.927799] [<c010237e>] sys_execve+0x5e/0xb0
[ 87.927799] [<c0103d19>] syscall_call+0x7/0xb
[ 87.927799] [<c010aee4>] ? kernel_execve+0x24/0x30
[ 87.927799] [<c0172b6f>] ? ____call_usermodehelper+0xff/0x170
[ 87.927799] [<c0172a70>] ? ____call_usermodehelper+0x0/0x170
[ 87.927799] [<c0104707>] ? kernel_thread_helper+0x7/0x10
[ 98.785592] BUG: soft lockup - CPU#1 stuck for 61s! [khelper:3657]
[ 98.785592] Modules linked in: ipv6 pcnet32 crc32
[ 98.785592] irq event stamp: 0
[ 98.785592] hardirqs last enabled at (0): [<(null)>] (null)
[ 98.785592] hardirqs last disabled at (0): [<c0155bcb>] copy_process+0x2ab/0x11a0
[ 98.785592] softirqs last enabled at (0): [<c0155bcb>] copy_process+0x2ab/0x11a0
[ 98.785592] softirqs last disabled at (0): [<(null)>] (null)
[ 98.785592]
Regards.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/