panic handling futex syscall
From: Steven Hartland
Date: Thu Nov 14 2019 - 17:36:08 EST
We're seeing Oops panics running a binary, which all seem to point to
sys_futex as the cause.
We're have quite a few versions of this most haven't resulted in a full
trace due to double faults and most won't even decode with 7.2.3 version
of crash included in ubuntu LTS 16.04 only with 7.2.7 built from source;
however we have one which has so here are the details:
[54978.681991] unable to execute userspace code (SMEP?) (uid: 2000)
[54978.688191] BUG: unable to handle kernel paging request at
ffffffff93eb1990
[54978.697039] IP: _copy_from_user+0x0/0x70
[54978.701689] PGD 51b20e067 P4D 51b20e067 PUD 51b20f063 PMD 51a4000e1
[54978.708788] Oops: 0010 [#1] SMP PTI
[54978.712484] Modules linked in: binfmt_misc ipmi_devintf
ipmi_msghandler ip6table_filter ip6_tables iptable_filter ip_tables
x_tables input_leds pvpanic serio_raw ib_iser rdma_cm iw_cm ib_cm
ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4
btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel
aes_x86_64 crypto_simd glue_helper cryptd psmouse virtio_net
[54978.783832] CPU: 6 PID: 25729 Comm: PoolThread 6 Not tainted
4.15.0-1041-gcp #43-Ubuntu
[54978.800464] Hardware name: Google Google Compute Engine/Google
Compute Engine, BIOS Google 01/01/2011
[54978.809908] RIP: 0010:_copy_from_user+0x0/0x70
[54978.814470] RSP: 0018:ffffad564a203eb0 EFLAGS: 00010246
[54978.819837] RAX: 0000000000000009 RBX: 0000000000000009 RCX:
00007f1c0672fcf8
[54978.827090] RDX: 0000000000000010 RSI: 00007f1c0672fcf8 RDI:
ffffad564a203ee0
[54978.834344] RBP: ffffad564a203f20 R08: 00007f1c03979d20 R09:
00000000ffffffff
[54978.841616] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000189
[54978.848960] R13: 00007f1c0672fcf8 R14: 00007f1c03979d4c R15:
00000000000022a1
[54978.856219] FS:Â 00007f1c06730700(0000) GS:ffff9126bfd80000(0000)
knlGS:0000000000000000
[54978.864432] CS:Â 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[54978.870304] CR2: ffffffff93eb1990 CR3: 0000000799a1c004 CR4:
00000000003606e0
[54978.877563] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[54978.885368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[54978.892636] Call Trace:
[54978.895211]Â ? SyS_futex+0xbb/0x180
[54978.898825]Â do_syscall_64+0x7b/0x150
[54978.903310]Â entry_SYSCALL_64_after_hwframe+0x42/0xb7
[54978.908478] RIP: 0033:0x7f1c13fa6709
[54978.912170] RSP: 002b:00007f1c0672fc70 EFLAGS: 00000202 ORIG_RAX:
00000000000000ca
[54978.919886] RAX: ffffffffffffffda RBX: 00007f1c03979d48 RCX:
00007f1c13fa6709
[54978.927139] RDX: 00000000000022a1 RSI: 0000000000000189 RDI:
00007f1c03979d4c
[54978.934407] RBP: 0000000000000001 R08: 00007f1c03979d20 R09:
00000000ffffffff
[54978.941674] R10: 00007f1c0672fcf8 R11: 0000000000000202 R12:
00000000000022a1
[54978.948930] R13: 00007f1c0672fcf8 R14: ffffffffffffff92 R15:
000000005dc96200
[54978.956185] Code: d2 60 4f 00 74 15 4c 89 e7 e8 cd 1b 50 00 0f b6 45
ef 48 83 c4 08 5b 41 5c 5d c3 48 83 c4 08 b8 01 00 00 00 5b 41 5c 5d c3
90 90 <55> 65 48 8b 04 25 00 5c 01 00 48 89 e5 41 55 41 54 53 48 89 d3
[54978.975188] RIP: _copy_from_user+0x0/0x70 RSP: ffffad564a203eb0
[54978.981214] CR2: ffffffff93eb1990
ÂÂÂÂÂ KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-1041-gcp
ÂÂÂ DUMPFILE: /var/crash/201911111331/dump.201911111331Â [PARTIAL DUMP]
ÂÂÂÂÂÂÂ CPUS: 8
ÂÂÂÂÂÂÂ DATE: Mon Nov 11 13:30:54 2019
ÂÂÂÂÂ UPTIME: 03:52:51
LOAD AVERAGE: 1.52, 0.57, 0.37
ÂÂÂÂÂÂ TASKS: 402
ÂÂÂ NODENAME: p-gce-XXX
ÂÂÂÂ RELEASE: 4.15.0-1041-gcp
ÂÂÂÂ VERSION: #43-Ubuntu SMP Wed Aug 21 09:04:51 UTC 2019
ÂÂÂÂ MACHINE: x86_64Â (2000 Mhz)
ÂÂÂÂÂ MEMORY: 30 GB
ÂÂÂÂÂÂ PANIC: "BUG: unable to handle kernel paging request at
ffffffff93eb1990"
ÂÂÂÂÂÂÂÂ PID: 25729
ÂÂÂÂ COMMAND: "PoolThread 6"
ÂÂÂÂÂÂÂ TASK: ffff912678c98000Â [THREAD_INFO: ffff912678c98000]
ÂÂÂÂÂÂÂÂ CPU: 6
ÂÂÂÂÂÂ STATE: TASK_RUNNING (PANIC)
crash> dis -l ffffffff93eb1990
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/lib/usercopy.c: 8
0xffffffff93eb1990 <_copy_from_user>:ÂÂ pushÂÂ %rbp
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/arch/x86/include/asm/current.h: 15
0xffffffff93eb1991 <_copy_from_user+1>: mov %gs:0x15c00,%rax
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/lib/usercopy.c: 8
0xffffffff93eb199a <_copy_from_user+10>:ÂÂÂÂÂÂÂ mov %rsp,%rbp
0xffffffff93eb199d <_copy_from_user+13>:ÂÂÂÂÂÂÂ pushÂÂ %r13
0xffffffff93eb199f <_copy_from_user+15>:ÂÂÂÂÂÂÂ pushÂÂ %r12
0xffffffff93eb19a1 <_copy_from_user+17>:ÂÂÂÂÂÂÂ pushÂÂ %rbx
0xffffffff93eb19a2 <_copy_from_user+18>:ÂÂÂÂÂÂÂ mov %rdx,%rbx
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/lib/usercopy.c: 11
0xffffffff93eb19a5 <_copy_from_user+21>:ÂÂÂÂÂÂÂ mov 0x1358(%rax),%rdx
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/arch/x86/include/asm/uaccess.h: 61
0xffffffff93eb19ac <_copy_from_user+28>:ÂÂÂÂÂÂÂ mov %rsi,%rax
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/lib/usercopy.c: 8
0xffffffff93eb19af <_copy_from_user+31>:ÂÂÂÂÂÂÂ mov %rdi,%r12
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/arch/x86/include/asm/uaccess.h: 61
0xffffffff93eb19b2 <_copy_from_user+34>:ÂÂÂÂÂÂÂ mov %rbx,%r13
0xffffffff93eb19b5 <_copy_from_user+37>:ÂÂÂÂÂÂÂ add %rbx,%rax
0xffffffff93eb19b8 <_copy_from_user+40>:ÂÂÂÂÂÂÂ jb 0xffffffff93eb19d8
<_copy_from_user+72>
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/lib/usercopy.c: 11
0xffffffff93eb19ba <_copy_from_user+42>:ÂÂÂÂÂÂÂ cmp %rax,%rdx
0xffffffff93eb19bd <_copy_from_user+45>:ÂÂÂÂÂÂÂ jb 0xffffffff93eb19c9
<_copy_from_user+57>
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/arch/x86/include/asm/uaccess_64.h:
37
0xffffffff93eb19bf <_copy_from_user+47>:ÂÂÂÂÂÂÂ mov %ebx,%edx
0xffffffff93eb19c1 <_copy_from_user+49>:ÂÂÂÂÂÂÂ callq 0xffffffff943a38e0
<copy_user_enhanced_fast_string>
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/arch/x86/include/asm/uaccess_64.h:
46
0xffffffff93eb19c6 <_copy_from_user+54>:ÂÂÂÂÂÂÂ mov %eax,%r13d
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/lib/usercopy.c: 15
0xffffffff93eb19c9 <_copy_from_user+57>:ÂÂÂÂÂÂÂ test %r13,%r13
0xffffffff93eb19cc <_copy_from_user+60>:ÂÂÂÂÂÂÂ jne 0xffffffff93eb19d8
<_copy_from_user+72>
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/lib/usercopy.c: 18
0xffffffff93eb19ce <_copy_from_user+62>:ÂÂÂÂÂÂÂ popÂÂÂ %rbx
0xffffffff93eb19cf <_copy_from_user+63>:ÂÂÂÂÂÂÂ mov %r13,%rax
0xffffffff93eb19d2 <_copy_from_user+66>:ÂÂÂÂÂÂÂ popÂÂÂ %r12
0xffffffff93eb19d4 <_copy_from_user+68>:ÂÂÂÂÂÂÂ popÂÂÂ %r13
0xffffffff93eb19d6 <_copy_from_user+70>:ÂÂÂÂÂÂÂ popÂÂÂ %rbp
0xffffffff93eb19d7 <_copy_from_user+71>:ÂÂÂÂÂÂÂ retq
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/include/linux/string.h: 332
0xffffffff93eb19d8 <_copy_from_user+72>:ÂÂÂÂÂÂÂ sub %r13,%rbx
0xffffffff93eb19db <_copy_from_user+75>:ÂÂÂÂÂÂÂ mov %r13,%rdx
0xffffffff93eb19de <_copy_from_user+78>:ÂÂÂÂÂÂÂ xor %esi,%esi
0xffffffff93eb19e0 <_copy_from_user+80>:ÂÂÂÂÂÂÂ lea (%r12,%rbx,1),%rdi
0xffffffff93eb19e4 <_copy_from_user+84>:ÂÂÂÂÂÂÂ callq 0xffffffff943a60e0
<__memset>
/build/linux-gcp-lp4Fx0/linux-gcp-4.15.0/lib/usercopy.c: 18
0xffffffff93eb19e9 <_copy_from_user+89>:ÂÂÂÂÂÂÂ mov %r13,%rax
0xffffffff93eb19ec <_copy_from_user+92>:ÂÂÂÂÂÂÂ popÂÂÂ %rbx
0xffffffff93eb19ed <_copy_from_user+93>:ÂÂÂÂÂÂÂ popÂÂÂ %r12
0xffffffff93eb19ef <_copy_from_user+95>:ÂÂÂÂÂÂÂ popÂÂÂ %r13
0xffffffff93eb19f1 <_copy_from_user+97>:ÂÂÂÂÂÂÂ popÂÂÂ %rbp
0xffffffff93eb19f2 <_copy_from_user+98>:ÂÂÂÂÂÂÂ retq
0xffffffff93eb19f3 <_copy_from_user+99>:ÂÂÂÂÂÂÂ noplÂÂ (%rax)
0xffffffff93eb19f6 <_copy_from_user+102>:ÂÂÂÂÂÂ nopw %cs:0x0(%rax,%rax,1)
Other relevant info:
Linux version 4.15.0-1041-gcp (buildd@lgw01-amd64-018) (gcc version
5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)) #43-Ubuntu SMP Wed Aug
21 09:04:51 UTC 2019
Linux p-gce-XXX 4.15.0-1041-gcp #43-Ubuntu SMP Wed Aug 21 09:04:51 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux
GNU CÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 5.4.0
GNU MakeÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 4.1
BinutilsÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 2.26.1
Util-linuxÂÂÂÂÂÂÂÂÂÂÂÂÂ 2.27.1
MountÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 2.27.1
Module-init-toolsÂÂÂÂÂÂ 22
E2fsprogsÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1.42.13
XfsprogsÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 4.3.0
Linux C LibraryÂÂÂÂÂÂÂÂ 2.23
Dynamic linker (ldd)ÂÂÂ 2.23
Linux C++ LibraryÂÂÂÂÂÂ 6.0.21
ProcpsÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 3.3.10
Net-toolsÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1.60
KbdÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1.15.5
Console-toolsÂÂÂÂÂÂÂÂÂÂ 1.15.5
Sh-utilsÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 8.25
UdevÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 229
Modules LoadedÂÂÂÂÂÂÂÂÂ aesni_intel aes_x86_64 async_memcpy async_pq
async_raid6_recov async_tx async_xor autofs4 binfmt_misc btrfs
crc32_pclmul crct10dif_pclmul cryptd crypto_simd ghash_clmulni_intel
glue_helper ib_cm ib_core ib_iser input_leds ip6table_filter ip6_tables
ipmi_devintf ipmi_msghandler iptable_filter ip_tables iscsi_tcp iw_cm
libiscsi libiscsi_tcp linear multipath pcbc psmouse pvpanic raid0 raid10
raid1 raid456 raid6_pq rdma_cm scsi_transport_iscsi serio_raw virtio_net
xor x_tables zstd_compress
ÂÂÂ Regards
ÂÂÂ Steve