Hello!
I finally managed to attach serial console to my server.
Today I came to it and found that server crashed in the night.
Looking at the logs I found that oopses produced
logfile of 2825 lines!
Here I include first two decoded. Second one repeating several times, anyway
( I run squid with AIO threads)
the far you look - the more strange results you see.
E.g. there is oops where instead of process name multiline part of
one of my logs printed.
Anyway Box is Dual P3-450, 256M RAM.
Linux ccssu 2.2.15pre4 #3 SMP Tue Jan 25 16:46:10 EET 2000 i686 unknown
This one is patched with raid-012345-... patch.
Disk subsystem is mixed SCSI Adaptec AIC-7895 Ultra SCSI host adapter
(2 SCSI disks) and one big IDE drive.
There is also Moxa 8 port serial board. I use usual serial driver for it
(setup with setserial), not new moxa driver, that went in at 2.2.14 time.
BTW, it seems that I hit this problem second time, already. First time
was with 2.2.14 near a week ago.
Full oops-dump (not decoded) with appropriate System.map is
at http://crimea.edu/~green/oops/oops.tgz, for anybody who interested.
If any other info is needed
Options used: -V (default)
-o /lib/modules/2.2.15pre4/ (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-m /boot/System.map-2.2.15pre4 (specified)
-c 1 (default)
Unable to handle kernel NULL pointer dereference at virtual address 00000070
current->tss.cr3 = 01e03000, %cr3 = 01e03000
*pde = 0e79a067
Oops: 0002
CPU: 0
EIP: 0010:[<c01acf2d>]
EFLAGS: 00010282
eax: 00000070 ebx: 00000039 ecx: 00000019 edx: 00000000
esi: cf607e49 edi: 0000000c ebp: cf607c00 esp: c31fbe54
ds: 0018 es: 0018 ss: 0018
Process squid (pid: 10521, process nr: 50, stackpage=c31fb000)
Stack: c803b000 00000000 00000000 c8c3e1a0 cf607f2f 0000000c ffff39b0 c01acc6c
cf607c00 cf607c00 c803b000 00000000 c31fbee0 c01ac883 cf607c00 c803b000
c023dac4 c01a3ccb c803b000 00000000 c01a3c92 c0005d80 00000001 c011b82d
Call Trace: [<c01acc6c>] [<c01ac883>] [<c01a3ccb>] [<c01a3c92>] [<c011b82d>] [<c0113805>] [<c01134bc>]
[<c01130b4>] [<c0133388>] [<c0133751>] [<c010ef51>] [<c0109304>] [<c010002b>]
Code: f0 ff 4a 70 0f 94 c0 84 c0 74 09 52 e8 22 8d fa ff 83 c4 04
>>EIP: c01acf2d <ppp_async_encode+241/288>
Trace: c01acc6c <ppp_tty_push+100/180>
Trace: c01ac883 <ppp_tty_wakeup+5f/74>
Trace: c01a3ccb <do_softint+2f/44>
Trace: c01a3c92 <do_serial_bh+56/60>
Trace: c011b82d <do_bottom_half+85/a8>
Trace: c0113805 <schedule+2ed/3b0>
Trace: c01134bc <schedule_timeout+6c/8c>
Trace: c01130b4 <process_timeout+0/10>
Code: c01acf2d <ppp_async_encode+241/288> 00000000 <_EIP>: <===
Code: c01acf2d <ppp_async_encode+241/288> 0: f0 ff 4a 70 lock decl 0x70(%edx) <===
Code: c01acf31 <ppp_async_encode+245/288> 4: 0f 94 c0 sete %al
Code: c01acf34 <ppp_async_encode+248/288> 7: 84 c0 testb %al,%al
Code: c01acf36 <ppp_async_encode+24a/288> 9: 74 09 je c01acf41 <ppp_async_encode+255/288>
Code: c01acf38 <ppp_async_encode+24c/288> b: 52 pushl %edx
Code: c01acf39 <ppp_async_encode+24d/288> c: e8 22 8d fa ff call c0155c60 <__kfree_skb+0/ac>
Code: c01acf3e <ppp_async_encode+252/288> 11: 83 c4 04 addl $0x4,%esp
Aiee, killing interrupt handler
Scheduling in interrupt
Unable to handle kernel NULL pointer dereference at virtual address 00000000
current->tss.cr3 = 00101000, %cr3 = 00101000
*pde = 00000000
Oops: 0002
CPU: 0
EIP: 0010:[<c01138b2>]
EFLAGS: 00010282
eax: 00000018 ebx: c31fa000 ecx: 000002fd edx: cf4fe000
esi: c31fa000 edi: cdd34000 ebp: c31fbdb8 esp: c31fbda0
ds: 0018 es: 0018 ss: 0018
Process squid (pid: 10521, process nr: 50, stackpage=c31fb000)
Stack: c31fa000 cdd34000 c011a3b0 c31fa000 00000000 00000246 00000000 c011a7c5
c31fbe18 c31fa000 c31fbe18 00000000 c0109892 c31fa000 c0109897 0000000b
00000000 c011127e c01d87b5 c31fbe18 00000002 c31fa000 cf607e49 0000000c
Call Trace: [<c011a3b0>] [<c011a7c5>] [<c0109892>] [<c0109897>] [<c011127e>] [<c01d87b5>] [<c010943d>]
[<c01acf2d>] [<c01acc6c>] [<c01ac883>] [<c01a3ccb>] [<c01a3c92>] [<c011b82d>] [<c0113805>] [<c01134bc>]
[<c01130b4>] [<c0133388>] [<c0133751>] [<c010ef51>] [<c0109304>] [<c010002b>]
Code: c7 05 00 00 00 00 00 00 00 00 8d 65 e4 5b 5e 5f 89 ec 5d c3
>>EIP: c01138b2 <schedule+39a/3b0>
Trace: c011a3b0 <exit_notify+150/29c>
Trace: c011a7c5 <do_exit+2c9/2d0>
Trace: c0109892 <die+4e/54>
Trace: c0109897 <die+53/54>
Trace: c011127e <do_page_fault+2fa/3d0>
Trace: c01d87b5 <lk_lockmsg+124b/1306>
Trace: c010943d <error_code+2d/40>
Trace: c01acf2d <ppp_async_encode+241/288>
Trace: c01130b4 <process_timeout+0/10>
Code: c01138b2 <schedule+39a/3b0> 00000000 <_EIP>: <===
Code: c01138b2 <schedule+39a/3b0> 0: c7 05 00 00 00 movl $0x0,0x0 <===
Code: c01138b7 <schedule+39f/3b0> 5: 00 00 00 00 00
Code: c01138bc <schedule+3a4/3b0> a: 8d 65 e4 leal 0xffffffe4(%ebp),%esp
Code: c01138bf <schedule+3a7/3b0> d: 5b popl %ebx
Code: c01138c0 <schedule+3a8/3b0> e: 5e popl %esi
Code: c01138c1 <schedule+3a9/3b0> f: 5f popl %edi
Code: c01138c2 <schedule+3aa/3b0> 10: 89 ec movl %ebp,%esp
Code: c01138c4 <schedule+3ac/3b0> 12: 5d popl %ebp
Code: c01138c5 <schedule+3ad/3b0> 13: c3 ret
Aiee, killing interrupt handler
Scheduling in interrupt
768 warnings issued. Results may not be reliable.
Boot sequence is:
Linux version 2.2.15pre4 (root@ccssu) (gcc version 2.7.2.3) #3 SMP Tue Jan 25 16:46:10 EET 2000
Intel MultiProcessor Specification v1.1
Virtual Wire compatibility mode.
OEM ID: OEM00000 Product ID: PROD00000000 APIC at: 0xFEE00000
Processor #0 Pentium(tm) Pro APIC version 17
Processor #1 Pentium(tm) Pro APIC version 17
I/O APIC #2 Version 17 at 0xFEC00000.
Processors: 2
mapped APIC to ffffe000 (fee00000)
mapped IOAPIC to ffffd000 (fec00000)
Detected 451030504 Hz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 448.92 BogoMIPS
Memory: 257836k/262144k available (996k kernel code, 420k reserved, 2840k data, 52k init)
Dentry hash table entries: 32768 (order 6, 256k)
Buffer cache hash table entries: 262144 (order 8, 1024k)
Page cache hash table entries: 65536 (order 6, 256k)
VFS: Diskquotas version dquot_6.4.0 initialized
Checking 386/387 coupling... OK, FPU using exception 16 error reporting.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
mtrr: v1.35a (19990819) Richard Gooch (rgooch@atnf.csiro.au)
per-CPU timeslice cutoff: 100.00 usecs.
CPU0: Intel Pentium III (Katmai) stepping 03
calibrating APIC timer ...
..... CPU clock speed is 451.0418 MHz.
..... system bus clock speed is 100.2312 MHz.
Booting processor 1 eip 2000
Calibrating delay loop... 450.56 BogoMIPS
OK.
CPU1: Intel Pentium III (Katmai) stepping 03
Total of 2 processors activated (899.48 BogoMIPS).
enabling symmetric IO mode... ...done.
ENABLING IO-APIC IRQs
init IO_APIC IRQs
IO-APIC (apicid-pin) 2-0, 2-20, 2-21, 2-22, 2-23 not connected.
number of MP IRQ sources: 23.
number of IO-APIC #2 registers: 24.
testing the IO APIC.......................
IO APIC #2......
.... register #00: 02000000
....... : physical APIC id: 02
.... register #01: 00170011
....... : max redirection entries: 0017
....... : IO APIC version: 0011
.... register #02: 00000000
....... : arbitration: 00
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
00 000 00 1 0 0 0 0 0 0 00
01 000 00 0 0 0 0 0 1 1 59
02 0FF 0F 0 0 0 0 0 1 1 51
03 000 00 0 0 0 0 0 1 1 61
04 000 00 0 0 0 0 0 1 1 69
05 000 00 0 0 0 0 0 1 1 71
06 000 00 0 0 0 0 0 1 1 79
07 000 00 0 0 0 0 0 1 1 81
08 000 00 0 0 0 0 0 1 1 89
09 000 00 0 0 0 0 0 1 1 91
0a 000 00 0 0 0 0 0 1 1 99
0b 000 00 0 0 0 0 0 1 1 A1
0c 000 00 0 0 0 0 0 1 1 A9
0d 000 00 1 0 0 0 0 0 0 00
0e 000 00 0 0 0 0 0 1 1 B1
0f 000 00 0 0 0 0 0 1 1 B9
10 0FF 0F 1 1 0 1 0 1 1 C1
11 0FF 0F 1 1 0 1 0 1 1 C9
12 0FF 0F 1 1 0 1 0 1 1 D1
13 0FF 0F 1 1 0 1 0 1 1 D9
14 000 00 1 0 0 0 0 0 0 00
15 000 00 1 0 0 0 0 0 0 00
16 000 00 1 0 0 0 0 0 0 00
17 000 00 1 0 0 0 0 0 0 00
.................................... done.
checking TSC synchronization across CPUs: passed.
mtrr: your CPUs had inconsistent fixed MTRR settings
mtrr: probably your BIOS does not setup all CPUs
PCI: PCI BIOS revision 2.10 entry at 0xfb370
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI->APIC IRQ transform: (B0,I8,P0) -> 16
PCI->APIC IRQ transform: (B0,I9,P0) -> 17
PCI->APIC IRQ transform: (B0,I10,P0) -> 18
PCI->APIC IRQ transform: (B0,I11,P0) -> 19
PCI->APIC IRQ transform: (B0,I11,P1) -> 16
PCI->APIC IRQ transform: (B0,I12,P0) -> 16
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Unix domain sockets 1.0 for Linux NET4.0.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
TCP: Hash tables configured (ehash 262144 bhash 65536)
NET4: Linux IPX 0.38 for NET4.0
IPX Portions Copyright (c) 1995 Caldera, Inc.
Initializing RT netlink socket
Starting kswapd v 1.5
Detected PS/2 Mouse Port.
Serial driver version 4.27 with MANY_PORTS MULTIPORT SHARE_IRQ enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
Real Time Clock Driver v1.09
PIIX4: IDE controller on PCI bus 00 dev 39
PIIX4: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:pio, hdb:DMA
ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:pio
hdb: QUANTUM FIREBALLlct08 17, ATA DISK drive
hdc: CD-ROM CDU4011, ATAPI CDROM drive
hdd: IOMEGA ZIP 100 ATAPI Floppy, ATAPI FLOPPY drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hdb: QUANTUM FIREBALLlct08 17, 16555MB w/418kB Cache, CHS=20510/81/32, UDMA
md driver 0.90.0 MAX_MD_DEVS=256, MAX_REAL=12
raid1 personality registered
(scsi0) <Adaptec AIC-7895 Ultra SCSI host adapter> found at PCI 11/0
(scsi0) Wide Channel A, SCSI ID=0, 32/255 SCBs
(scsi0) Warning - detected auto-termination
(scsi0) Please verify driver detected settings are correct.
(scsi0) If not, then please properly set the device termination
(scsi0) in the Adaptec SCSI BIOS by hitting CTRL-A when prompted
(scsi0) during machine bootup.
(scsi0) Cables present (Int-50 NO, Int-68 YES, Ext-68 NO)
(scsi0) Downloading sequencer code... 374 instructions downloaded
(scsi1) <Adaptec AIC-7895 Ultra SCSI host adapter> found at PCI 11/1
(scsi1) Wide Channel B, SCSI ID=0, 32/255 SCBs
(scsi1) Warning - detected auto-termination
(scsi1) Please verify driver detected settings are correct.
(scsi1) If not, then please properly set the device termination
(scsi1) in the Adaptec SCSI BIOS by hitting CTRL-A when prompted
(scsi1) during machine bootup.
(scsi1) Cables present (Int-50 NO, Int-68 NO, Ext-68 NO)
(scsi1) Downloading sequencer code... 374 instructions downloaded
scsi0 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.23/3.2.4
<Adaptec AIC-7895 Ultra SCSI host adapter>
scsi1 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.23/3.2.4
<Adaptec AIC-7895 Ultra SCSI host adapter>
scsi : 2 hosts.
(scsi0:0:5:0) Synchronous at 40.0 Mbyte/sec, offset 8.
Vendor: IBM Model: DNES-309170W Rev: SA30
Type: Direct-Access ANSI SCSI revision: 03
Detected scsi disk sda at scsi0, channel 0, id 5, lun 0
(scsi0:0:6:0) Synchronous at 40.0 Mbyte/sec, offset 8.
Vendor: IBM Model: DNES-309170W Rev: SA30
Type: Direct-Access ANSI SCSI revision: 03
Detected scsi disk sdb at scsi0, channel 0, id 6, lun 0
scsi : detected 2 SCSI disks total.
SCSI device sda: hdwr sector= 512 bytes. Sectors= 17916240 [8748 MB] [8.7 GB]
SCSI device sdb: hdwr sector= 512 bytes. Sectors= 17916240 [8748 MB] [8.7 GB]
PPP: version 2.3.7 (demand dialling)
TCP compression code copyright 1989 Regents of the University of California
PPP line discipline registered.
ne2k-pci.c:vpre-1.00e 5/27/99 D. Becker/P. Gortmaker http://cesdis.gsfc.nasa.gov/linux/drivers/ne2k-pci.html
ne2k-pci.c: PCI NE2000 clone 'RealTek RTL-8029' at I/O 0xd400, IRQ 16.
eth0: RealTek RTL-8029 found at 0xd400, IRQ 16, 00:00:01:00:A2:35.
ne2k-pci.c: PCI NE2000 clone 'RealTek RTL-8029' at I/O 0xd800, IRQ 17.
eth1: RealTek RTL-8029 found at 0xd800, IRQ 17, 00:60:52:07:86:F0.
ne2k-pci.c: PCI NE2000 clone 'RealTek RTL-8029' at I/O 0xdc00, IRQ 18.
eth2: RealTek RTL-8029 found at 0xdc00, IRQ 18, 00:00:01:00:91:31.
rtl8139.c:v1.07 5/6/99 Donald Becker http://cesdis.gsfc.nasa.gov/linux/drivers/rtl8139.html
eth3: SMC1211TX EZCard 10/100 (RealTek RTL8139) at 0xe800, IRQ 16, 00:e0:29:33:f1:97.
Partition check:
sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 >
sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 sdb6 sdb7 sdb8 >
md.c: sizeof(mdp_super_t) = 4096
hdb: [PTBL] [3309/255/63] hdb1 hdb2
autodetecting RAID arrays
(read) sda6's sb offset: 1574272 [events: 0000005d]
(read) sdb6's sb offset: 1574272 [events: 0000005d]
autorun ...
considering sdb6 ...
adding sdb6 ...
adding sda6 ...
created md0
bind<sda6,1>
bind<sdb6,2>
running: <sdb6><sda6>
now!
sdb6's event counter: 0000005d
sda6's event counter: 0000005d
md: md0: raid array is not clean -- starting background reconstruction
md0: max total readahead window set to 128k
md0: 1 data-disks, max readahead per data-disk: 128k
raid1: device sdb6 operational as mirror 1
raid1: device sda6 operational as mirror 0
raid1: raid set md0 not clean; reconstructing mirrors
raid1: raid set md0 active with 2 out of 2 mirrors
md: updating md0 RAID superblock on device
sdb6 [events: 0000005e](write) sdb6's sb offset: 1574272
md: syncing RAID array md0
md: minimum _guaranteed_ reconstruction speed: 100 KB/sec.
md: using maximum available idle IO bandwith for reconstruction.
md: using 128k window.
sda6 [events: 0000005e](write) sda6's sb offset: 1574272
.
... autorun DONE.
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 52k freed
Adding Swap: 104384k swap-space (priority 1)
Adding Swap: 104384k swap-space (priority 1)
md0: blocksize changed
nr_blocks changed to 32 (blocksize 4096, j 6880, max_blocks 393568)
registered device ppp0
hdc: ATAPI 40X CD-ROM drive, 120kB Cache
Bye,
Oleg
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Mon Feb 07 2000 - 21:00:09 EST