Debugging General Protection Faultâs

From: Muni Sekhar
Date: Tue Aug 18 2015 - 10:33:55 EST


[ Please keep me in CC as I'm not subscribed to the list]

Hello,

I have a doubt regarding debugging general protection faultâs.
I am running the driver tests on Intel(R) Core(TM)2 Duo CPU.
During the tests I see system hangs after continuous occurrence of
general protection faultâs.

First fault occurred on CPU: 0 , but it is not related to our driver,
looks like it is in kernel stack.
Second gpf fault and third Oops fault related to our own driver. Rest
other looks to be in kernel stack.

I would like to know, is the first fault triggered other faults?
Is all the faults needs to be fixed or just the first fault?

Full stack trace is attached.

[009298.685954] general protection fault: 0000 [#1] SMP
[009298.725436] general protection fault: 0000 [#2] SMP
[009298.866588] Oops: 0002 [#3] SMP
[009300.134033] general protection fault: 0000 [#4] SMP


Regards,
Sekhar
[009298.685954] general protection fault: 0000 [#1] SMP
[009298.686376] Modules linked in: ptrb(OE) t_mux(OE) tty_hif(OE) h_core(OE) os_abstract(OE) snd_usb_audio snd_usbmidi_lib snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore gpio(OE) uart(OE) deb_spi(OE) w1(OE) gpio_ich coretemp kvm_intel kvm i915 lpc_ich video drm_kms_helper pci(OE) drm i2c_algo_bit shpchp mac_hid lp parport pata_sil680 pata_acpi e1000e ptp pps_core [last unloaded: os_abstract]
[009298.688008] CPU: 0 PID: 21693 Comm: python Tainted: G IOE 3.16.0-30-generic #40~14.04.1-Ubuntu
[009298.688008] Hardware name: RadiSys Corporation CEGM45 /CEGM45T2-SL9-0 , BIOS 08.00.44 09/14/2011
[009298.688008] task: ffff880076ede5e0 ti: ffff880076fc8000 task.ti: ffff880076fc8000
[009298.688008] RIP: 0010:[<ffffffff811b6995>] [<ffffffff811b6995>] __kmalloc+0x95/0x230
[009298.688008] RSP: 0018:ffff880076fcbcb0 EFLAGS: 00010286
[009298.688008] RAX: 0000000000000000 RBX: ffff880004e129c0 RCX: 0000000002bee000
[009298.688008] RDX: 0000000002bedfff RSI: 0000000000000000 RDI: 0000000000000008
[009298.688008] RBP: ffff880076fcbce0 R08: 0000000000016240 R09: ffffffff8124dce4
[009298.688008] R10: ffff88007a801a00 R11: 0000000011111111 R12: 00000000000080d0
[009298.688008] R13: 7b8bdd2b4a3e4480 R14: 0000000000000042 R15: ffff88007a801a00
[009298.688008] FS: 00007ff1ee414740(0000) GS:ffff88007d200000(0000) knlGS:0000000000000000
[009298.688008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[009298.688008] CR2: 00007ff1ea0a3c90 CR3: 00000000781b6000 CR4: 00000000000407f0
[009298.688008] Stack:
[009298.688008] ffffffff8124dce4 ffff880004e129c0 ffff880075c42a80 ffff8800420001b8
[009298.688008] 0000000029091136 00000000f08c7f95 ffff880076fcbd18 ffffffff8124dce4
[009298.688008] ffff880004e129c0 ffff880076fcbdc0 ffff8800201548a0 0000000000000000
[009298.688008] Call Trace:
[009298.688008] [<ffffffff8124dce4>] ? ext4_htree_store_dirent+0x34/0x120
[009298.688008] [<ffffffff8124dce4>] ext4_htree_store_dirent+0x34/0x120
[009298.688008] [<ffffffff8125d659>] htree_dirblock_to_tree+0x169/0x190
[009298.688008] [<ffffffff8125e8e6>] ext4_htree_fill_tree+0xc6/0x270
[009298.688008] [<ffffffff8118e73c>] ? handle_mm_fault+0x7fc/0x10b0
[009298.688008] [<ffffffff811b7176>] ? kmem_cache_alloc_trace+0x1c6/0x1f0
[009298.688008] [<ffffffff8124cf3a>] ? free_rb_tree_fname+0x1a/0x90
[009298.688008] [<ffffffff8124d505>] ext4_readdir+0x185/0x910
[009298.688008] [<ffffffff811e6cf3>] iterate_dir+0xa3/0x130
[009298.688008] [<ffffffff811e3382>] ? final_putname+0x22/0x50
[009298.688008] [<ffffffff811e718a>] SyS_getdents+0x8a/0x110
[009298.688008] [<ffffffff811e6e50>] ? fillonedir+0xd0/0xd0
[009298.688008] [<ffffffff8176aced>] system_call_fastpath+0x1a/0x1f
[009298.688008] Code: cd 00 00 49 8b 50 08 4d 8b 28 49 8b 40 10 4d 85 ed 0f 84 26 01 00 00 48 85 c0 0f 84 1d 01 00 00 49 63 42 20 48 8d 4a 01 4d 8b 02 <49> 8b 5c 05 00 4c 89 e8 65 49 0f c7 08 0f 94 c0 84 c0 74 b8 49
[009298.688008] RIP [<ffffffff811b6995>] __kmalloc+0x95/0x230
[009298.688008] RSP <ffff880076fcbcb0>
[009298.724864] ---[ end trace cc195a9ec5b7c913 ]---
[009298.725436] general protection fault: 0000 [#2] SMP
[009298.725455] Modules linked in: ptrb(OE) t_mux(OE) tty_hif(OE) h_core(OE) os_abstract(OE) snd_usb_audio snd_usbmidi_lib snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore gpio(OE) uart(OE) deb_spi(OE) w1(OE) gpio_ich coretemp kvm_intel kvm i915 lpc_ich video drm_kms_helper pci(OE) drm i2c_algo_bit shpchp mac_hid lp parport pata_sil680 pata_acpi e1000e ptp pps_core [last unloaded: os_abstract]
[009298.725457] CPU: 1 PID: 26531 Comm: rcv_buffer_stal Tainted: G D IOE 3.16.0-30-generic #40~14.04.1-Ubuntu
[009298.725459] Hardware name: RadiSys Corporation CEGM45 /CEGM45T2-SL9-0 , BIOS 08.00.44 09/14/2011
[009298.725460] task: ffff880078074750 ti: ffff880077d44000 task.ti: ffff880077d44000
[009298.725468] RIP: 0010:[<ffffffffc04560b4>] [<ffffffffc04560b4>] stale_timer_poll_thread+0x74/0x1d0 [ptrb]
[009298.725469] RSP: 0018:ffff880077d47e80 EFLAGS: 00010202
[009298.725470] RAX: 0000000000000000 RBX: ffff88007849ca00 RCX: 000000000000a70e
[009298.725471] RDX: 0000000000000000 RSI: 0000000000000286 RDI: 0000000000000286
[009298.725472] RBP: ffff880077d47ec8 R08: 0000000000000286 R09: 000000000000000f
[009298.725473] R10: 00000000002b5d0b R11: 0000000000000000 R12: ffff88007849caf8
[009298.725474] R13: 0000000000000001 R14: ffff88007849caf8 R15: 4b50e63de8c365e6
[009298.725476] FS: 0000000000000000(0000) GS:ffff88007d300000(0000) knlGS:0000000000000000
[009298.725477] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[009298.725478] CR2: 00007f13ac005138 CR3: 0000000078817000 CR4: 00000000000407e0
[009298.725479] Stack:
[009298.725481] ffff880078074750 ffff88007849cae8 0000000055ceb94b 00000000000533aa
[009298.725483] ffff88007914cd40 ffff88007849ca00 ffffffffc0456040 0000000000000000
[009298.725485] 0000000000000000 ffff880077d47f48 ffffffff810912c2 0000000000000000
[009298.725486] Call Trace:
[009298.725490] [<ffffffffc0456040>] ? ptrb_delete+0x40/0x40 [ptrb]
[009298.725496] [<ffffffff810912c2>] kthread+0xd2/0xf0
[009298.725499] [<ffffffff810911f0>] ? kthread_create_on_node+0x1c0/0x1c0
[009298.725501] [<ffffffff8176ac3c>] ret_from_fork+0x7c/0xb0
[009298.725503] [<ffffffff810911f0>] ? kthread_create_on_node+0x1c0/0x1c0
[009298.725522] Code: 48 8b 40 08 48 8b 40 10 a8 04 0f 85 3b 01 00 00 bf 01 00 00 00 e8 ed 5c c2 c0 4c 8b bb f8 00 00 00 4d 39 fc 74 c9 4d 85 ff 74 c4 <41> 8b 4f 50 41 bd 01 00 00 00 85 c9 0f 8e c5 00 00 00 66 2e 0f
[009298.725525] RIP [<ffffffffc04560b4>] stale_timer_poll_thread+0x74/0x1d0 [ptrb]
[009298.725526] RSP <ffff880077d47e80>
[009298.725528] ---[ end trace cc195a9ec5b7c914 ]---
[009298.862657] BUG: unable to handle kernel NULL pointer dereference at (null)
[009298.864262] IP: [<ffffffffc0457261>] ptrb_close+0x71/0x150 [ptrb]
[009298.865850] PGD 0
[009298.866588] Oops: 0002 [#3] SMP
[009298.866588] Modules linked in: ptrb(OE) t_mux(OE) tty_hif(OE) h_core(OE) os_abstract(OE) snd_usb_audio snd_usbmidi_lib snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore gpio(OE) uart(OE) deb_spi(OE) w1(OE) gpio_ich coretemp kvm_intel kvm i915 lpc_ich video drm_kms_helper pci(OE) drm i2c_algo_bit shpchp mac_hid lp parport pata_sil680 pata_acpi e1000e ptp pps_core [last unloaded: os_abstract]
[009298.866588] CPU: 1 PID: 21693 Comm: python Tainted: G D IOE 3.16.0-30-generic #40~14.04.1-Ubuntu
[009298.866588] Hardware name: RadiSys Corporation CEGM45 /CEGM45T2-SL9-0 , BIOS 08.00.44 09/14/2011
[009298.866588] task: ffff880076ede5e0 ti: ffff880076fc8000 task.ti: ffff880076fc8000
[009298.866588] RIP: 0010:[<ffffffffc0457261>] [<ffffffffc0457261>] ptrb_close+0x71/0x150 [ptrb]
[009298.866588] RSP: 0018:ffff880076fcba38 EFLAGS: 00010293
[009298.866588] RAX: 0000000000000001 RBX: ffff88007849ca00 RCX: 000000018040003b
[009298.866588] RDX: 0000000000000000 RSI: ffffea0001e11500 RDI: 0000000040000000
[009298.866588] RBP: ffff880076fcba70 R08: ffff880078454b00 R09: 000000018040003b
[009298.866588] R10: ffffea0001e11500 R11: ffffffffc0457cef R12: ffff88007849cb38
[009298.866588] R13: ffff880077ecb900 R14: ffff88007849ca70 R15: ffff8800775ac900
[009298.866588] FS: 00007ff1ee414740(0000) GS:ffff88007d300000(0000) knlGS:0000000000000000
[009298.866588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[009298.866588] CR2: 0000000000000000 CR3: 00000000792f6000 CR4: 00000000000407e0
[009298.866588] Stack:
[009298.866588] 0000000000000000 0000000000000000 0000000000000000 ffff880077ecb900
[009298.866588] 0000000000000008 ffff8800791902f8 ffff880075c7c2a0 ffff880076fcbab8
[009298.866588] ffffffff811d5684 ffff8800791902f8 ffff880077ecb910 ffff880076068800
[009298.866588] Call Trace:
[009298.866588] [<ffffffff811d5684>] __fput+0xe4/0x220
[009298.866588] [<ffffffff811d580e>] ____fput+0xe/0x10
[009298.866588] [<ffffffff8108e134>] task_work_run+0xc4/0xe0
[009298.866588] [<ffffffff81070258>] do_exit+0x2b8/0xa60
[009298.866588] [<ffffffff81016aa9>] oops_end+0xa9/0x150
[009298.866588] [<ffffffff81016e5b>] die+0x4b/0x70
[009298.866588] [<ffffffff810139a6>] do_general_protection+0x126/0x1b0
[009298.866588] [<ffffffff8176ccc8>] general_protection+0x28/0x30
[009298.866588] [<ffffffff8124dce4>] ? ext4_htree_store_dirent+0x34/0x120
[009298.866588] [<ffffffff811b6995>] ? __kmalloc+0x95/0x230
[009298.866588] [<ffffffff8124dce4>] ? ext4_htree_store_dirent+0x34/0x120
[009298.866588] [<ffffffff8124dce4>] ext4_htree_store_dirent+0x34/0x120
[009298.866588] [<ffffffff8125d659>] htree_dirblock_to_tree+0x169/0x190
[009298.866588] [<ffffffff8125e8e6>] ext4_htree_fill_tree+0xc6/0x270
[009298.866588] [<ffffffff8118e73c>] ? handle_mm_fault+0x7fc/0x10b0
[009298.866588] [<ffffffff811b7176>] ? kmem_cache_alloc_trace+0x1c6/0x1f0
[009298.866588] [<ffffffff8124cf3a>] ? free_rb_tree_fname+0x1a/0x90
[009298.866588] [<ffffffff8124d505>] ext4_readdir+0x185/0x910
[009298.866588] [<ffffffff811e6cf3>] iterate_dir+0xa3/0x130
[009298.866588] [<ffffffff811e3382>] ? final_putname+0x22/0x50
[009298.866588] [<ffffffff811e718a>] SyS_getdents+0x8a/0x110
[009298.866588] [<ffffffff811e6e50>] ? fillonedir+0xd0/0xd0
[009298.866588] [<ffffffff8176aced>] system_call_fastpath+0x1a/0x1f
[009298.866588] Code: ff 01 48 8d 55 d4 4c 89 ee 48 8d 78 28 e8 08 10 00 00 49 8b 74 24 68 4c 89 ef e8 fb 09 00 00 83 f8 0f 77 0c 49 8b 54 24 d8 89 c0 <f0> 48 0f b3 02 4d 39 6c 24 e8 74 42 0f 1f 00 4c 89 f7 e8 e8 1b
[009298.866588] RIP [<ffffffffc0457261>] ptrb_close+0x71/0x150 [ptrb]
[009298.866588] RSP <ffff880076fcba38>
[009298.866588] CR2: 0000000000000000
[009298.866588] ---[ end trace cc195a9ec5b7c915 ]---
[009298.866588] Fixing recursive fault but reboot is needed!
[009300.134033] general protection fault: 0000 [#4] SMP
[009300.135219] Modules linked in: ptrb(OE) t_mux(OE) tty_hif(OE) h_core(OE) os_abstract(OE) snd_usb_audio snd_usbmidi_lib snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore gpio(OE) uart(OE) deb_spi(OE) w1(OE) gpio_ich coretemp kvm_intel kvm i915 lpc_ich video drm_kms_helper pci(OE) drm i2c_algo_bit shpchp mac_hid lp parport pata_sil680 pata_acpi e1000e ptp pps_core [last unloaded: os_abstract]
[009300.136008] CPU: 0 PID: 27180 Comm: check-disk Tainted: G D IOE 3.16.0-30-generic #40~14.04.1-Ubuntu
[009300.136008] Hardware name: RadiSys Corporation CEGM45 /CEGM45T2-SL9-0 , BIOS 08.00.44 09/14/2011
[009300.136008] task: ffff88007859a8c0 ti: ffff880078c30000 task.ti: ffff880078c30000
[009300.136008] RIP: 0010:[<ffffffff811b6995>] [<ffffffff811b6995>] __kmalloc+0x95/0x230
[009300.136008] RSP: 0018:ffff880078c33cb0 EFLAGS: 00010286
[009300.136008] RAX: 0000000000000000 RBX: ffff8800366a14e0 RCX: 0000000002bee000
[009300.136008] RDX: 0000000002bedfff RSI: 0000000000000000 RDI: 0000000000000009
[009300.136008] RBP: ffff880078c33ce0 R08: 0000000000016240 R09: ffffffff8124dce4
[009300.136008] R10: ffff88007a801a00 R11: 0000000018181818 R12: 00000000000080d0
[009300.136008] R13: 7b8bdd2b4a3e4480 R14: 0000000000000049 R15: ffff88007a801a00
[009300.136008] FS: 00007f50c2ad3740(0000) GS:ffff88007d200000(0000) knlGS:0000000000000000
[009300.136008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[009300.136008] CR2: 000000000273a6c8 CR3: 0000000079102000 CR4: 00000000000407f0
[009300.136008] Stack:
[009300.136008] ffffffff8124dce4 ffff8800366a14e0 ffff880076e02780 ffff88006bc77018
[009300.136008] 00000000df8c8534 000000000c3bbb6a ffff880078c33d18 ffffffff8124dce4
[009300.136008] ffff8800366a14e0 ffff880078c33dc0 ffff8800365ab7e0 0000000000000000
[009300.136008] Call Trace:
[009300.136008] [<ffffffff8124dce4>] ? ext4_htree_store_dirent+0x34/0x120
[009300.136008] [<ffffffff8124dce4>] ext4_htree_store_dirent+0x34/0x120
[009300.136008] [<ffffffff8125d659>] htree_dirblock_to_tree+0x169/0x190
[009300.136008] [<ffffffff8125e8e6>] ext4_htree_fill_tree+0xc6/0x270
[009300.136008] [<ffffffff8118e983>] ? handle_mm_fault+0xa43/0x10b0
[009300.136008] [<ffffffff811e3629>] ? putname+0x29/0x40
[009300.136008] [<ffffffff811b7176>] ? kmem_cache_alloc_trace+0x1c6/0x1f0
[009300.136008] [<ffffffff8124cf3a>] ? free_rb_tree_fname+0x1a/0x90
[009300.136008] [<ffffffff8124d505>] ext4_readdir+0x185/0x910
[009300.136008] [<ffffffff811e6cf3>] iterate_dir+0xa3/0x130
[009300.136008] [<ffffffff811e3382>] ? final_putname+0x22/0x50
[009300.136008] [<ffffffff811e718a>] SyS_getdents+0x8a/0x110
[009300.136008] [<ffffffff811e6e50>] ? fillonedir+0xd0/0xd0
[009300.136008] [<ffffffff8176cd28>] ? page_fault+0x28/0x30
[009300.136008] [<ffffffff8176aced>] system_call_fastpath+0x1a/0x1f
[009300.136008] Code: cd 00 00 49 8b 50 08 4d 8b 28 49 8b 40 10 4d 85 ed 0f 84 26 01 00 00 48 85 c0 0f 84 1d 01 00 00 49 63 42 20 48 8d 4a 01 4d 8b 02 <49> 8b 5c 05 00 4c 89 e8 65 49 0f c7 08 0f 94 c0 84 c0 74 b8 49
[009300.136008] RIP [<ffffffff811b6995>] __kmalloc+0x95/0x230
[009300.136008] RSP <ffff880078c33cb0>
[009300.193154] ---[ end trace cc195a9ec5b7c916 ]--