2.0.31 problems (in get_empty_inode?, "unable to handle kernel paging request")

Peter Woit (woit@shire.math.columbia.edu)
Thu, 30 Oct 1997 11:09:44 -0500 (EST)

We've been having periodic crashes of our main linux server
due to some sort of consistent kernel problem that is still
there in 2.0.31. Crashes typically occur when "find" is
being run by updatedb as a cron job, but other pieces of
code (rpc.nfsd, ps) seem to also manage to hit this problem.
I'm including
the latest kernel crash debugging output as well as various
info about the the system configuration (the only thing
that may not appear there is that we are running netatalk 1.3.3),
and some earlier crashes also. I'd greatly
appreciate any advice from the experts about what is causing
this and what to try to do to get around it.

A related question: Can one get Linux to automatically reboot
on this kind of kernel crash? It would be a lot easier to live
with if the machine would reboot instead of just staying hung.

Peter Woit
woit@math.columbia.edu

Oct 30 07:40:04 cpw kernel: Unable to handle kernel paging request at
virtual address d6f6ff44
Oct 30 07:40:04 cpw kernel: current->tss.cr3 = 0521c000, |r3 = 0521c000
Oct 30 07:40:04 cpw kernel: *pde = 00000000
Oct 30 07:40:04 cpw kernel: Oops: 0000
Oct 30 07:40:04 cpw kernel: CPU: 1
Oct 30 07:40:04 cpw kernel: EIP: 0010:[get_empty_inode+68/352]
Oct 30 07:40:04 cpw kernel: EFLAGS: 00010202
Oct 30 07:40:04 cpw kernel: eax: 00000b00 ebx: 078b6dc8 ecx: 16f6fec4
edx: 00000006
Oct 30 07:40:04 cpw kernel: esi: 000001b3 edi: 00000001 ebp: 001f8e48
esp: 0521dee0
Oct 30 07:40:04 cpw kernel: ds: 0018 es: 0018 fs: 002b gs: 002b
ss: 0018
Oct 30 07:40:04 cpw kernel: Process find (pid: 2192, process nr: 80,
stackpage=0521d000)
Oct 30 07:40:04 cpw kernel: Stack: 00000000 001e5668 00000000 00129e09
031d5dc8 0521df74 00000007 03d7e000
Oct 30 07:40:04 cpw kernel: 00164525 00164558 001f8e48 0001e62e
00000001 031d5dc8 0521df74 0521df74
Oct 30 07:40:04 cpw kernel: 00000007 031d5dc8 0001e62e 0013109a
031d5dc8 03d7e000 00000007 0521df74
Oct 30 07:40:04 cpw kernel: Call Trace: [__iget+97/516]
[ext2_lookup+129/368] [ext2_lookup+180/368] [lookup+238/264]
[_namei+91/228] [lnamei+48/72] [sys_newlstat+41/88]
Oct 30 07:40:04 cpw kernel: [system_call+258/320]
Oct 30 07:40:04 cpw kernel: Code: 66 83 b9 80 00 00 00 00 75 26 ba e7 03
00 00 8a 81 84 00 00

Linux cpw 2.0.31 #2 Wed Oct 29 20:44:01 EST 1997 i686
Kernel modules found
Gnu C 2.7.2.3
Binutils linux-2.8.1.0.1
Linux C Library 5.4.38
Dynamic Linker (ld.so) 1.9.6
Linux C++ Library 27.2.8
Procps 1.2
Mount 2.7f
Net-tools (1997-09-21)
Kbd 0.94
Sh-utils 1.12

Oct 30 10:18:33 cpw kernel: Intel MultiProcessor Specification v1.1
Oct 30 10:18:33 cpw kernel: Virtual Wire compatibility mode.
Oct 30 10:18:33 cpw kernel: OEM ID: OEM00000 Product ID: PROD00000000 APIC
at: 0xFEE00000
Oct 30 10:18:33 cpw kernel: Processor #1 Pentium(tm) Pro APIC version 17
Oct 30 10:18:33 cpw kernel: Processor #0 Pentium(tm) Pro APIC version 17
Oct 30 10:18:33 cpw kernel: I/O APIC #2 Version 17 at 0xFEC00000.
Oct 30 10:18:33 cpw kernel: Processors: 2
Oct 30 10:18:33 cpw kernel: Console: 16 point font, 400 scans
Oct 30 10:18:33 cpw kernel: Console: colour VGA+ 80x25, 1 virtual console
(max 63)
Oct 30 10:18:33 cpw kernel: pcibios_init : BIOS32 Service Directory
structure at 0x000fad00
Oct 30 10:18:33 cpw kernel: pcibios_init : BIOS32 Service Directory entry
at 0xfb180
Oct 30 10:18:33 cpw kernel: pcibios_init : PCI BIOS revision 2.10 entry at
0xfb1b0
Oct 30 10:18:33 cpw kernel: Probing PCI hardware.
Oct 30 10:18:33 cpw kernel: Calibrating delay loop.. ok - 199.07 BogoMIPS
Oct 30 10:18:33 cpw kernel: Memory: 127824k/131072k available (760k kernel
code, 384k reserved, 2104k data)
Oct 30 10:18:33 cpw kernel: Swansea University Computer Society TCP/IP for
NET3.034
Oct 30 10:18:33 cpw kernel: IP Protocols: IGMP, ICMP, UDP, TCP
Oct 30 10:18:33 cpw kernel: Checking 386/387 coupling... Ok, fpu using
exception 16 error reporting.
Oct 30 10:18:33 cpw kernel: Checking 'hlt' instruction... Ok.
Oct 30 10:18:33 cpw kernel: Linux version 2.0.31 (root@cpw) (gcc version
2.7.2.3) #2 Wed Oct 29 20:44:01 EST 1997
Oct 30 10:18:33 cpw kernel: Booting processor 0 stack 00002000:
Calibrating delay loop.. ok - 199.07 BogoMIPS
Oct 30 10:18:33 cpw kernel: Total of 2 processors activated (398.13
BogoMIPS).
Oct 30 10:18:33 cpw kernel: Starting kswapd v 1.4.2.2
Oct 30 10:18:33 cpw kernel: EATA (Extended Attachment) driver version:
2.59b
Oct 30 10:18:36 cpw lpd[64]: /dev/lp: No such file or directory
Oct 30 10:18:33 cpw kernel: developed in co-operation with DPT
Oct 30 10:18:33 cpw kernel: (c) 1993-96 Michael Neuffer,
mike@i-Connect.Net
Oct 30 10:18:33 cpw kernel: Registered HBAs:
Oct 30 10:18:33 cpw kernel: HBA no. Boardtype Revis EATA Bus BaseIO
IRQ DMA Ch ID Pr QS S/G IS
Oct 30 10:18:33 cpw kernel: scsi1 : PM3334UW v07L.0 2.0c PCI 0x8210
10 BMST 2 7 N 64 252 Y
Oct 30 10:18:33 cpw kernel: scsi0 : BusLogic BT-958
Oct 30 10:18:33 cpw kernel: scsi1 : EATA (Extended Attachment) HBA driver
Oct 30 10:18:33 cpw kernel: scsi : 2 hosts.
Oct 30 10:18:33 cpw kernel: Vendor: Quantum Model: XP34300W
Rev: L912
Oct 30 10:18:33 cpw kernel: Type: Direct-Access
ANSI SCSI revision: 02
Oct 30 10:18:33 cpw kernel: Detected scsi disk sda at scsi0, channel 0, id
0, lun 0
Oct 30 10:18:33 cpw kernel: Vendor: TEAC Model: CD-ROM CD-56S
Rev: 1.0D
Oct 30 10:18:33 cpw kernel: Type: CD-ROM
ANSI SCSI revision: 02
Oct 30 10:18:33 cpw kernel: Detected scsi CD-ROM sr0 at scsi0, channel 0,
id 5, lun 0
Oct 30 10:18:33 cpw kernel: Vendor: DPT Model: RAID-5
Rev: 07L0
Oct 30 10:18:33 cpw kernel: Type: Direct-Access
ANSI SCSI revision: 02
Oct 30 10:18:33 cpw kernel: Detected scsi disk sdb at scsi1, channel 0, id
11, lun 0
Oct 30 10:18:33 cpw kernel: scsi : detected 1 SCSI cdrom 2 SCSI disks
total.
Oct 30 10:18:33 cpw kernel: SCSI device sda: hdwr sector= 512 bytes.
Sectors= 8399520 [4101 MB] [4.1 GB]
Oct 30 10:18:33 cpw kernel: SCSI device sdb: hdwr sector= 512 bytes.
Sectors= 26659968 [13017 MB] [13.0 GB]
Oct 30 10:18:33 cpw kernel: Found Intel i82557 PCI Speedo at I/O 0x8100,
IRQ 9.
Oct 30 10:18:33 cpw kernel: PCI latency timer (CFLT) is 0x20.
Oct 30 10:18:33 cpw kernel: eepro100.c:v0.36 10/20/97 Donald Becker
linux-eepro100@cesdis.gsfc.nasa.gov
Oct 30 10:18:33 cpw kernel: eepro100.c:v0.36 10/20/97 Donald Becker
linux-eepro100@cesdis.gsfc.nasa.gov
Oct 30 10:18:33 cpw kernel: Partition check:
Oct 30 10:18:33 cpw kernel: sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 >
Oct 30 10:18:33 cpw kernel: sdb: sdb1 sdb2
Oct 30 10:18:33 cpw kernel: VFS: Mounted root (ext2 filesystem) readonly.
Oct 30 10:18:33 cpw kernel: Adding Swap: 128516k swap-space (priority -1)
Oct 30 10:18:33 cpw kernel: Adding Swap: 120452k swap-space (priority -2)
Oct 30 10:18:33 cpw kernel: Adding Swap: 120452k swap-space (priority -3)

Older crashes

Jun 14 14:53:01 cpw kernel: Unable to handle kernel paging request at
virtual address d63d4764
Jun 14 14:53:01 cpw kernel: current->tss.cr3 = 07a8f000, |r3 = 07a8f000
Jun 14 14:53:01 cpw kernel: *pde = 00000000
Jun 14 14:53:01 cpw kernel: Oops: 0000
Jun 14 14:53:01 cpw kernel: CPU: 1
Jun 14 14:53:01 cpw kernel: EIP: 0010:[get_empty_inode+68/352]
Jun 14 14:53:01 cpw kernel: EFLAGS: 00010202
Jun 14 14:53:01 cpw kernel: eax: 00000c00 ebx: 04aaa4ec ecx: 163d46e4
edx: 00000002
Jun 14 14:53:01 cpw kernel: esi: 000002ec edi: 00000001 ebp: 0020d2f4
esp: 07a91ee0
Jun 14 14:53:01 cpw kernel: ds: 0018 es: 0018 fs: 002b gs: 002b
ss: 0018
Jun 14 14:53:01 cpw kernel: Process rpc.nfsd (pid: 114, process nr: 44,
stackpage=07a91000)
Jun 14 14:53:01 cpw kernel: Stack: 00000000 001f9530 00000000 00128a69
03fd82f4 02c9da98 00000018 06dcc01f
Jun 14 14:53:01 cpw kernel: 02c9da98 00161cbf 0020d2f4 000578e7
00000001 03fd82f4 07a91f74 07a91f74
Jun 14 14:53:01 cpw kernel: 00000018 00ba64b4 000578e7 0012fb7e
03fd82f4 06dcc01f 00000018 07a91f74
Jun 14 14:53:01 cpw kernel: Call Trace: [__iget+97/516]
[ext2_lookup+343/368] [lookup+238/264] [_namei+91/196]
[lnamei+48/72] [sys_newlstat+41/88] [system_call+207/272]
Jun 14 14:53:01 cpw kernel: Code: 66 83 b9 80 00 00 00 00 75 26 ba e7 03
00 00 8a 81 84 00 00

Jul 22 07:40:11 cpw kernel: Unable to handle kernel paging request at
virtual address d3f0095c
Jul 22 07:40:11 cpw kernel: current->tss.cr3 = 077aa000, |r3 = 077aa000
Jul 22 07:40:11 cpw kernel: *pde = 00000000
Jul 22 07:40:11 cpw kernel: Oops: 0000
Jul 22 07:40:11 cpw kernel: CPU: 1
Jul 22 07:40:11 cpw kernel: EIP: 0010:[get_empty_inode+68/352]
Jul 22 07:40:11 cpw kernel: EFLAGS: 00010216
Jul 22 07:40:11 cpw kernel: eax: 00000c00 ebx: 050e2dc8 ecx: 13f008dc
edx: 0000000d
Jul 22 07:40:11 cpw kernel: esi: 0000015f edi: 00000001 ebp: 0020d160
esp: 077c0ee0
Jul 22 07:40:11 cpw kernel: ds: 0018 es: 0018 fs: 002b gs: 002b
ss: 0018
Jul 22 07:40:11 cpw kernel: Process find (pid: 3900, process nr: 61,
stackpage=077c0000)
Jul 22 07:40:11 cpw kernel: Stack: 00000000 001f9318 00000000 00128a69
0050d2f4 077c0f74 00000008 013e7000
Jul 22 07:40:11 cpw kernel: 00161be9 00161c1c 0020d160 0002b69e
00000001 0050d2f4 077c0f74 077c0f74
Jul 22 07:40:11 cpw kernel: 00000008 0050d2f4 0002b69e 0012fb7e
0050d2f4 013e7000 00000008 077c0f74
Jul 22 07:40:11 cpw kernel: Call Trace: [__iget+97/516]
[ext2_lookup+129/368] [ext2_lookup+180/368] [lookup+238
/264] [_namei+91/196] [lnamei+48/72] [sys_newlstat+41/88]
Jul 22 07:40:11 cpw kernel: [system_call+207/272]
Jul 22 07:40:11 cpw kernel: Code: 66 83 b9 80 00 00 00 00 75 26 ba e7 03
00 00 8a 81 84 00 00

Aug 23 07:40:06 cpw kernel: Unable to handle kernel paging request at
virtual address d47b656c
Aug 23 07:40:06 cpw kernel: current->tss.cr3 = 03589000, |r3 = 03589000
Aug 23 07:40:06 cpw kernel: *pde = 00000000
Aug 23 07:40:06 cpw kernel: Oops: 0000
Aug 23 07:40:06 cpw kernel: CPU: 1
Aug 23 07:40:06 cpw kernel: EIP: 0010:[get_empty_inode+68/352]
Aug 23 07:40:06 cpw kernel: EFLAGS: 00010202
Aug 23 07:40:06 cpw kernel: eax: 00000c00 ebx: 010664ec ecx: 147b64ec
edx: 00000001
Aug 23 07:40:06 cpw kernel: esi: 0000033d edi: 00000001 ebp: 0020cfcc
esp: 02ca1ee0
Aug 23 07:40:06 cpw kernel: ds: 0018 es: 0018 fs: 002b gs: 002b
ss: 0018
Aug 23 07:40:06 cpw kernel: Process find (pid: 2253, process nr: 64,
stackpage=02ca1000)
Aug 23 07:40:06 cpw kernel: Stack: 00000000 001f8f88 00000000 00128a69
02e839d8 02ca1f74 00000009 03579000
Aug 23 07:40:06 cpw kernel: 00161be9 00161c1c 0020cfcc 0001f62a
00000001 02e839d8 02ca1f74 02ca1f74
Aug 23 07:40:06 cpw kernel: 00000009 02e839d8 0001f62a 0012fb7e
02e839d8 03579000 00000009 02ca1f74
Aug 23 07:40:06 cpw kernel: Call Trace: [__iget+97/516]
[ext2_lookup+129/368] [ext2_lookup+180/368] [lookup+238
/264] [_namei+91/196] [lnamei+48/72] [sys_newlstat+41/88]
Aug 23 07:40:06 cpw kernel: [system_call+207/272]
Aug 23 07:40:06 cpw kernel: Code: 66 83 b9 80 00 00 00 00 75 26 ba e7 03
00 00 8a 81 84 00 00

Aug 25 02:31:52 cpw kernel: Unable to handle kernel paging request at
virtual address d29bedc8
Aug 25 02:31:52 cpw kernel: current->tss.cr3 = 00c5a000, |r3 = 00c5a000
Aug 25 02:31:52 cpw kernel: *pde = 00000000
Aug 25 02:31:52 cpw kernel: Oops: 0000
Aug 25 02:31:52 cpw kernel: CPU: 1
Aug 25 02:31:52 cpw kernel: EIP: 0010:[__iget+60/516]
Aug 25 02:31:52 cpw kernel: EFLAGS: 00010202
Aug 25 02:31:52 cpw kernel: eax: 00000807 ebx: 129bedc8 ecx: 000678fa
edx: 000678fa
Aug 25 02:31:52 cpw kernel: esi: 001f9618 edi: 00000000 ebp: 0020d2f4
esp: 00cceee8
Aug 25 02:31:52 cpw kernel: ds: 0018 es: 0018 fs: 002b gs: 002b
ss: 0018
Aug 25 02:31:52 cpw kernel: Process tkdesksh (pid: 1508, process nr: 59,
stackpage=00cce000)
Aug 25 02:31:52 cpw kernel: Stack: 050694ec 00ccef6c 00000002 07d6b006
00161be9 00161c1c 0020d2f4 000678fa
Aug 25 02:31:52 cpw kernel: 00000001 050694ec 00ccef6c 00ccef6c
00000002 050694ec 000678fa 0012fb7e
Aug 25 02:31:52 cpw kernel: 050694ec 07d6b006 00000002 00ccef6c
00000000 00ccef6c 00ccefb4 000004c2
Aug 25 02:31:52 cpw kernel: Call Trace: [ext2_lookup+129/368]
[ext2_lookup+180/368] [lookup+238/264] [_namei+91
/196] [namei+48/72] [sys_access+195/292] [system_call+207/272]
Aug 25 02:31:52 cpw kernel: Code: 66 39 03 75 0d 8b 4c 24 1c 39 4b 04 0f
84 fa 00 00 00 8b 5b

Sep 8 07:48:12 cpw kernel: Unable to handle kernel paging request at
virtual address d549256c
Sep 8 07:48:12 cpw kernel: current->tss.cr3 = 012b7000, |r3 = 012b7000
Sep 8 07:48:12 cpw kernel: *pde = 00000000
Sep 8 07:48:12 cpw kernel: Oops: 0000
Sep 8 07:48:12 cpw kernel: CPU: 1
Sep 8 07:48:12 cpw kernel: EIP: 0010:[get_empty_inode+68/352]
Sep 8 07:48:12 cpw kernel: EFLAGS: 00010202
Sep 8 07:48:12 cpw kernel: eax: 00000c00 ebx: 031a13f0 ecx: 154924ec
edx: 00000006
Sep 8 07:48:12 cpw kernel: esi: 000002f2 edi: 00000001 ebp: 0020d2f4
esp: 01338ee0
Sep 8 07:48:12 cpw kernel: ds: 0018 es: 0018 fs: 002b gs: 002b
ss: 0018
Sep 8 07:48:12 cpw kernel: Process find (pid: 13197, process nr: 58,
stackpage=01338000)
Sep 8 07:48:12 cpw kernel: Stack: 00000000 001f9260 00000000 00128a69
05af80fc 038b1118 00000018 030c9000
Sep 8 07:48:12 cpw kernel: 038b1118 00161cbf 0020d2f4 0002f081
00000001 05af80fc 01338f74 01338f74
Sep 8 07:48:12 cpw kernel: 00000018 038b0d14 0002f081 0012fb7e
05af80fc 030c9000 00000018 01338f74
Sep 8 07:48:12 cpw kernel: Call Trace: [__iget+97/516]
[ext2_lookup+343/368] [lookup+238/264] [_namei+91/196]
[lnamei+48/72] [sys_newlstat+41/88] [system_call+207/272]
Sep 8 07:48:12 cpw kernel: Code: 66 83 b9 80 00 00 00 00 75 26 ba e7 03
00 00 8a 81 84 00 00

Sep 15 20:03:25 cpw kernel: Unable to handle kernel paging request at
virtual address d120ee48
Sep 15 20:03:25 cpw kernel: current->tss.cr3 = 00a91000, |r3 = 00a91000
Sep 15 20:03:25 cpw kernel: *pde = 00000000
Sep 15 20:03:25 cpw kernel: Oops: 0000
Sep 15 20:03:25 cpw kernel: CPU: 1
Sep 15 20:03:25 cpw kernel: EIP: 0010:[get_empty_inode+68/352]
Sep 15 20:03:25 cpw kernel: EFLAGS: 00010206
Sep 15 20:03:25 cpw kernel: eax: 00000c00 ebx: 05aecbd0 ecx: 1120edc8
edx: 00000001
Sep 15 20:03:25 cpw kernel: esi: 00000228 edi: 00000001 ebp: 00212ee0
esp: 00a9cee0
Sep 15 20:03:25 cpw kernel: ds: 0018 es: 0018 fs: 002b gs: 002b
ss: 0018
Sep 15 20:03:25 cpw kernel: Process rpc.nfsd (pid: 98, process nr: 20,
stackpage=00a9c000)
Sep 15 20:03:25 cpw kernel: Stack: 00000000 001fef68 00000000 00129cb9
00d428dc 00a9cf74 0000000e 041f901a
Sep 15 20:03:25 cpw kernel: 001641a9 001641dc 00212ee0 000578bd
00000001 00d428dc 00a9cf74 00a9cf74
Sep 15 20:03:25 cpw kernel: 0000000e 00d428dc 000578bd 00130e5a
00d428dc 041f901a 0000000e 00a9cf74
Sep 15 20:03:25 cpw kernel: Call Trace: [__iget+97/516]
[ext2_lookup+129/368] [ext2_lookup+180/368] [lookup+238
/264] [_namei+91/228] [lnamei+48/72] [sys_newlstat+41/88]
Sep 15 20:03:25 cpw kernel: [system_call+255/320]
Sep 15 20:03:25 cpw kernel: Code: 66 83 b9 80 00 00 00 00 75 26 ba e7 03
00 00 8a 81 84 00 00

Sep 16 10:44:16 cpw kernel: Unable to handle kernel paging request at
virtual address d1b89e48
Sep 16 10:44:16 cpw kernel: current->tss.cr3 = 00b16000, |r3 = 00b16000
Sep 16 10:44:16 cpw kernel: *pde = 00000000
Sep 16 10:44:16 cpw kernel: Oops: 0000
Sep 16 10:44:16 cpw kernel: CPU: 1
Sep 16 10:44:16 cpw kernel: EIP: 0010:[get_empty_inode+68/352]
Sep 16 10:44:16 cpw kernel: EFLAGS: 00010206
Sep 16 10:44:16 cpw kernel: eax: 00000c00 ebx: 028cfbd0 ecx: 11b89dc8
edx: 00000004
Sep 16 10:44:16 cpw kernel: esi: 00000453 edi: 00000004 ebp: 00212ee0
esp: 00b1dee0
Sep 16 10:44:16 cpw kernel: ds: 0018 es: 0018 fs: 002b gs: 002b
ss: 0018
Sep 16 10:44:16 cpw kernel: Process rpc.nfsd (pid: 134, process nr: 20,
stackpage=00b1d000)
Sep 16 10:44:16 cpw kernel: Stack: 00000000 001ff2b0 00000000 00129cb9
023343f0 00b1df74 0000000d 066fe01e
Sep 16 10:44:16 cpw kernel: 001641a9 001641dc 00212ee0 00083124
00000001 023343f0 00b1df74 00b1df74
Sep 16 10:44:16 cpw kernel: 0000000d 023343f0 00083124 00130e5a
023343f0 066fe01e 0000000d 00b1df74
Sep 16 10:44:16 cpw kernel: Call Trace: [__iget+97/516]
[ext2_lookup+129/368] [ext2_lookup+180/368] [lookup+238
/264] [_namei+91/228] [lnamei+48/72] [sys_newlstat+41/88]
Sep 16 10:44:16 cpw kernel: [system_call+255/320]
Sep 16 10:44:16 cpw kernel: Code: 66 83 b9 80 00 00 00 00 75 26 ba e7 03
00 00 8a 81 84 00 00