Re: kvm: GPF in kvm_lapic_latched_init

From: Dmitry Vyukov
Date: Wed Jun 22 2016 - 04:29:28 EST

Next message: Jirka Hladky: "Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel"
Previous message: Alexandre Courbot: "[PATCH 1/6] regulator: core: Allow simultaneous use of enable op and GPIO"
Next in thread: Paolo Bonzini: "Re: kvm: GPF in kvm_lapic_latched_init"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Fri, Jan 15, 2016 at 6:12 PM, Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
> On Fri, Jan 8, 2016 at 7:42 PM, Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
>> Hello,
>>
>> The following program triggers GPF in kvm_lapic_latched_init if run in
>> a parallel loop:
>> https://gist.githubusercontent.com/dvyukov/524b398f379440b21115/raw/9627095f57a72501fb51bf7565471d31732beeee/gistfile1.txt
>>
>> kasan: GPF could be caused by NULL-ptr deref or user memory
>> accessgeneral protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN
>> Modules linked in:
>> CPU: 3 PID: 14426 Comm: a.out Not tainted 4.4.0-rc8+ #217
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> task: ffff880061099780 ti: ffff880062e30000 task.ti: ffff880062e30000
>> RIP: 0010:[<ffffffff81057171>] [<ffffffff81057171>]
>> kvm_arch_vcpu_ioctl+0xa31/0x2ef0
>> RSP: 0018:ffff880062e37900 EFLAGS: 00010206
>> RAX: dffffc0000000000 RBX: 1ffff1000c5c6f25 RCX: 1ffff1000c41b7cb
>> RDX: 000000000000001e RSI: 000000008040ae9f RDI: 00000000000000f0
>> RBP: ffff880062e37c10 R08: 0000000000000000 R09: 0000000000000000
>> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
>> R13: 0000000000000000 R14: ffff880062e37be8 R15: 0000000000000000
>> FS: 00007f4aa815f700(0000) GS:ffff88006d700000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> CR2: 00007f4aa795de78 CR3: 00000000613c2000 CR4: 00000000000026e0
>> Stack:
>> 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> 0000000020006fe4 0000000041b58ab3 ffffffff86e2e588 ffffffff81056740
>> 0000000000000001 ffff880061099f60 0000000000000498 ffff880061099f68
>> Call Trace:
>> [<ffffffff8101cb52>] kvm_vcpu_ioctl+0x1e2/0xd00
>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:2526
>> [< inline >] vfs_ioctl fs/ioctl.c:43
>> [<ffffffff817b36b1>] do_vfs_ioctl+0x681/0xe40 fs/ioctl.c:607
>> [< inline >] SYSC_ioctl fs/ioctl.c:622
>> [<ffffffff817b3eff>] SyS_ioctl+0x8f/0xc0 fs/ioctl.c:613
>> [<ffffffff85e745b6>] entry_SYSCALL_64_fastpath+0x16/0x7a
>> arch/x86/entry/entry_64.S:185
>> Code: 85 2d 20 00 00 4d 8b a4 24 60 03 00 00 e8 c8 8b 50 00 49 8d bc
>> 24 f0 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
>> 3c 02 00 0f 85 f3 1f 00 00 4d 8b a4 24 f0 00 00 00 41 83 e4
>> RIP [< inline >] constant_test_bit ./arch/x86/include/asm/bitops.h:311
>> RIP [< inline >] kvm_lapic_latched_init arch/x86/kvm/lapic.h:164
>> RIP [< inline >] kvm_vcpu_ioctl_x86_get_vcpu_events
>> arch/x86/kvm/x86.c:2936
>> RIP [<ffffffff81057171>] kvm_arch_vcpu_ioctl+0xa31/0x2ef0
>> arch/x86/kvm/x86.c:3347
>> RSP <ffff880062e37900>
>> ---[ end trace 16449377928e034b ]---
>>
>>
>> or:
>>
>> kasan: GPF could be caused by NULL-ptr deref or user memory
>> accessgeneral protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN
>> Modules linked in:
>> CPU: 0 PID: 9555 Comm: syz-executor Not tainted 4.4.0-rc8+ #217
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> task: ffff88006301de00 ti: ffff880062568000 task.ti: ffff880062568000
>> RIP: 0010:[<ffffffff810cf5ab>] [<ffffffff810cf5ab>]
>> wait_lapic_expire+0x6b/0x560
>> RSP: 0018:ffff88006256fa48 EFLAGS: 00010006
>> RAX: dffffc0000000000 RBX: 0000000000000000 RCX: ffff88006301e5c8
>> RDX: 0000000000000011 RSI: 0000000000000000 RDI: ffff880033590360
>> RBP: ffff88006256fa88 R08: 0000000000000001 R09: 0000000000000002
>> R10: 0000000000000001 R11: 0000000000000001 R12: ffff880033590000
>> R13: ffff880033590030 R14: 0000000000000088 R15: ffff88003359002c
>> FS: 00007f4809354700(0000) GS:ffff88003ec00000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 00007f4808b53000 CR3: 0000000033f3f000 CR4: 00000000000026f0
>> Stack:
>> ffff88006256fa70 0000000000000082 0000000000000003 ffff88006301de00
>> ffff880033590030 ffff880033590030 ffff880033590000 ffff88003359002c
>> ffff88006256fc10 ffffffff8106a1dc ffffffff8106a75b 0000000000013210
>> Call Trace:
>> [< inline >] vcpu_enter_guest arch/x86/kvm/x86.c:6523
>> [< inline >] vcpu_run arch/x86/kvm/x86.c:6660
>> [<ffffffff8106a1dc>] kvm_arch_vcpu_ioctl_run+0x25ec/0x5820
>> arch/x86/kvm/x86.c:6818
>> [<ffffffff8101cf61>] kvm_vcpu_ioctl+0x5f1/0xd00
>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:2375
>> [< inline >] vfs_ioctl fs/ioctl.c:43
>> [<ffffffff817b36b1>] do_vfs_ioctl+0x681/0xe40 fs/ioctl.c:607
>> [< inline >] SYSC_ioctl fs/ioctl.c:622
>> [<ffffffff817b3eff>] SyS_ioctl+0x8f/0xc0 fs/ioctl.c:613
>> [<ffffffff85e745b6>] entry_SYSCALL_64_fastpath+0x16/0x7a
>> arch/x86/entry/entry_64.S:185
>> Code: 60 03 00 00 0f 1f 44 00 00 e8 92 07 49 00 4c 8d b3 88 00 00 00
>> e8 86 07 49 00 4c 89 f2 48 b8 00 00 00 00 00 fc ff df 48 c1 ea 03 <80>
>> 3c 02 00 0f 85 d8 04 00 00 4c 8b ab 88 00 00 00 4d 85 ed 75
>> RIP [<ffffffff810cf5ab>] wait_lapic_expire+0x6b/0x560 arch/x86/kvm/lapic.c:1245
>> RSP <ffff88006256fa48>
>> ---[ end trace 560c2b85e36670bc ]---
>>
>> or:
>>
>> kasan: GPF could be caused by NULL-ptr deref or user memory
>> accessgeneral protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN
>> Modules linked in:
>> CPU: 3 PID: 11264 Comm: syz-executor Not tainted 4.4.0-rc8+ #217
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> task: ffff880064d55e00 ti: ffff880064dc0000 task.ti: ffff880064dc0000
>> RIP: 0010:[<ffffffff810d138d>] [<ffffffff810d138d>]
>> apic_has_pending_timer+0x7d/0x210
>> RSP: 0018:ffff880064dc7a60 EFLAGS: 00010206
>> RAX: dffffc0000000000 RBX: 0000000000000000 RCX: 0000000000000004
>> RDX: 0000000000000017 RSI: 0000000000000000 RDI: 00000000000000b8
>> RBP: ffff880064dc7a70 R08: 0000000000000002 R09: 0000000000000001
>> R10: ffff880064d55e00 R11: ffff880063528220 R12: ffff880063250030
>> R13: ffff880063250030 R14: ffff880063250000 R15: 0000000000000000
>> FS: 00007fb05f305700(0000) GS:ffff88006d700000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 00000000006d7760 CR3: 0000000065ae9000 CR4: 00000000000026e0
>> Stack:
>> ffff880063250000 ffff880063250030 ffff880064dc7a88 ffffffff810c7af5
>> ffffffff86fee5c0 ffff880064dc7c10 ffffffff810685d4 ffffffff8106a75b
>> 0000000000013210 ffff880065a35000 1ffff1000c9b8f59 ffff880064dc0008
>> Call Trace:
>> [<ffffffff810c7af5>] kvm_cpu_has_pending_timer+0x15/0x20 arch/x86/kvm/irq.c:36
>> [< inline >] vcpu_run arch/x86/kvm/x86.c:6669
>> [<ffffffff810685d4>] kvm_arch_vcpu_ioctl_run+0x9e4/0x5820
>> arch/x86/kvm/x86.c:6818
>> [<ffffffff8101cf61>] kvm_vcpu_ioctl+0x5f1/0xd00
>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:2375
>> [< inline >] vfs_ioctl fs/ioctl.c:43
>> [<ffffffff817b36b1>] do_vfs_ioctl+0x681/0xe40 fs/ioctl.c:607
>> [< inline >] SYSC_ioctl fs/ioctl.c:622
>> [<ffffffff817b3eff>] SyS_ioctl+0x8f/0xc0 fs/ioctl.c:613
>> [<ffffffff85e745b6>] entry_SYSCALL_64_fastpath+0x16/0x7a
>> arch/x86/entry/entry_64.S:185
>> Code: ba e9 48 00 0f 1f 44 00 00 e8 b0 e9 48 00 e8 ab e9 48 00 48 8d
>> bb b8 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
>> 3c 02 00 0f 85 46 01 00 00 4c 8b a3 b8 00 00 00 48 b8 00 00
>> RIP [< inline >] arch_static_branch
>> ./arch/x86/include/asm/jump_label.h:21
>> RIP [< inline >] static_key_false include/linux/jump_label.h:133
>> RIP [< inline >] kvm_apic_hw_enabled arch/x86/kvm/lapic.h:117
>> RIP [< inline >] apic_enabled arch/x86/kvm/lapic.c:121
>> RIP [<ffffffff810d138d>] apic_has_pending_timer+0x7d/0x210
>> arch/x86/kvm/lapic.c:1731
>> RSP <ffff880064dc7a60>
>> ---[ end trace fe9c10b88e48c946 ]---
>>
>>
>> All crashes suggest that apic is NULL.
>>
>> On commit b06f3a168cdcd80026276898fd1fee443ef25743 (Jan 6).

Ping. Just hit it again on 67016f6cdfd079e632bbc49e33178b2d558c120a (Jun 20):

general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN
Modules linked in:
CPU: 1 PID: 12583 Comm: syz-executor Not tainted 4.7.0-rc4+ #2
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
task: ffff8800355d46c0 ti: ffff880034198000 task.ti: ffff880034198000
RIP: 0010:[<ffffffff811260ba>] [<ffffffff811260ba>]
wait_lapic_expire+0x6a/0x5e0
RSP: 0018:ffff88003419fa40 EFLAGS: 00010002
RAX: dffffc0000000000 RBX: ffff880033cf0040 RCX: ffffc900020e4000
RDX: 0000000000000010 RSI: 1ffff10006833001 RDI: ffff880033cf03a0
RBP: ffff88003419fa78 R08: 0000000000000001 R09: 0000000000000002
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
R13: ffff880033cf0070 R14: 0000000000000080 R15: ffff880033cf0040
FS: 00007fd1bdd14700(0000) GS:ffff88003ed00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd1bdcf3000 CR3: 0000000061716000 CR4: 00000000000026e0
Stack:
0000000000000086 0000000000000003 ffff880033cf0070 00000000ffffffff
ffff880033cf0070 ffff880033cf006c ffff880033cf0040 ffff88003419fc10
ffffffff810bee70 ffffffff810bf48f ffffffff81478fa0 ffff8800355d46c0
Call Trace:
[< inline >] vcpu_enter_guest arch/x86/kvm/x86.c:6638
[< inline >] vcpu_run arch/x86/kvm/x86.c:6777
[<ffffffff810bee70>] kvm_arch_vcpu_ioctl_run+0x2490/0x5f60
arch/x86/kvm/x86.c:6935
[<ffffffff8106841e>] kvm_vcpu_ioctl+0x61e/0xe70
arch/x86/kvm/../../../virt/kvm/kvm_main.c:2457
[< inline >] vfs_ioctl fs/ioctl.c:43
[<ffffffff818510bc>] do_vfs_ioctl+0x18c/0xff0 fs/ioctl.c:674
[< inline >] SYSC_ioctl fs/ioctl.c:689
[<ffffffff81851faf>] SyS_ioctl+0x8f/0xc0 fs/ioctl.c:680
[<ffffffff86a97000>] entry_SYSCALL_64_fastpath+0x23/0xc1
arch/x86/entry/entry_64.S:207
Code: 03 00 00 0f 1f 44 00 00 e8 54 04 47 00 4d 8d b4 24 80 00 00 00
e8 47 04 47 00 4c 89 f2 48 b8 00 00 00 00 00 fc ff df 48 c1 ea 03 <80>
3c 02 00 0f 85 58 05 00 00 4d 8b ac 24 80 00 00 00 4d 85 ed
RIP [<ffffffff811260ba>] wait_lapic_expire+0x6a/0x5e0 arch/x86/kvm/lapic.c:1300
RSP <ffff88003419fa40>
---[ end trace dc1986f78b27e282 ]---

Next message: Jirka Hladky: "Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel"
Previous message: Alexandre Courbot: "[PATCH 1/6] regulator: core: Allow simultaneous use of enable op and GPIO"
Next in thread: Paolo Bonzini: "Re: kvm: GPF in kvm_lapic_latched_init"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]