I am trying to determine the cause of a double Oops caused during
stress testing on new server.
I suspect flaky hardware, but I am not very versed in in diagnosing
hardware errors from Oops'es.
The hardware: Athlon-700, 768MB RAM, one 36Gig SCSI-U2W disk on
a Tekram DC-390U2W (sym53c895) controller. LVD/SE terminator in place
(can't tell if its active or passive).
Kernel: Stock 2.2.14, compiled to the system in case.
The stress loops:
. In one VT: "while [ $? == 0 ]; do make clean && make -j8 bzImage; done"
(on the 2.2.14 kernel tree).
. In another VT: "while [ $? == 0 ]; do bonnie -d . -s 1024 -m lvd; done"
(top was running in a third VT).
I had stressed the system the night before with just a 'make -j4 bzImage'
loop, without problems.
Funnily enough, the Oopses hit 'top' and 'kflushd', not bonnie or the
make loop. $DEITY only knows why (and maybe Linus ... :-)
The Oopses passed through ksymoops (see below for source code pinpoints):
ksymoops 2.3.3 on i686 2.2.14. Options used
-V (specified)
-k /proc/ksyms (default)
-L (specified)
-O (specified)
-m /boot/System.map-2.2.14 (specified)
Unable to handle kernel paging request at virtual address 43429ca8
current->tss.cr3 = 1f90d000, %cr3 = 1f90d000
*pde = 00000000
Oops: 0000
CPU: 0
EIP: 0010:[<c01309e2>]
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010a83
eax: 43429c40 ebx: 0004000a ecx: c0219c90 edx: 43429c40
esi: effd1c00 edi: effd1c00 ebp: d31ded80 esp: d24cbea8
ds: 0018 es: 0018 ss: 0018
Process top (pid: 1840, process nr: 62, stackpage=d24cb000)
Stack: c0130ce0 effd1c00 0004000a c0219c90 0004000a c01f34e4 effd1c00 00040002
c01403fa effd1c00 0004000a c01f34e4 00040000 c01cc535 c0140a73 effd1c00
0004000a c01f34e4 fffffff4 d31ded80 d6a88dd0 d31dee80 d6a88dd0 ffffffea
Call Trace: [<c0130ce0>] [<c01403fa>] [<c01cc535>] [<c0140a73>] [<c012afe7>] [<c012b1d0>] [<c012b339>]
[<c0123cee>] [<c0123f58>] [<c0109094>]
Code: 39 70 68 75 0d 39 58 18 75 08 ff 40 1c 5b 5e c3 89 f6 8b 12
>>EIP; c01309e2 <find_inode+16/34> <=====
Trace; c0130ce0 <iget+30/6c>
Trace; c01403fa <proc_get_inode+3e/f0>
Trace; c01cc535 <cprt+1bb3/7ac8>
Trace; c0140a73 <proc_lookup+93/c4>
Trace; c012afe7 <real_lookup+4f/a4>
Trace; c012b1d0 <lookup_dentry+10c/1ac>
Trace; c012b339 <open_namei+6d/2ec>
Trace; c0123cee <filp_open+46/f8>
Trace; c0123f58 <sys_open+38/94>
Trace; c0109094 <system_call+34/38>
Code; c01309e2 <find_inode+16/34>
00000000 <_EIP>:
Code; c01309e2 <find_inode+16/34> <=====
0: 39 70 68 cmpl %esi,0x68(%eax) <=====
Code; c01309e5 <find_inode+19/34>
3: 75 0d jne 12 <_EIP+0x12> c01309f4 <find_inode+28/34>
Code; c01309e7 <find_inode+1b/34>
5: 39 58 18 cmpl %ebx,0x18(%eax)
Code; c01309ea <find_inode+1e/34>
8: 75 08 jne 12 <_EIP+0x12> c01309f4 <find_inode+28/34>
Code; c01309ec <find_inode+20/34>
a: ff 40 1c incl 0x1c(%eax)
Code; c01309ef <find_inode+23/34>
d: 5b popl %ebx
Code; c01309f0 <find_inode+24/34>
e: 5e popl %esi
Code; c01309f1 <find_inode+25/34>
f: c3 ret
Code; c01309f2 <find_inode+26/34>
10: 89 f6 movl %esi,%esi
Code; c01309f4 <find_inode+28/34>
12: 8b 12 movl (%edx),%edx
Unable to handle kernel paging request at virtual address 000064b4
current->tss.cr3 = 00101000, %cr3 = 00101000
*pde = 00000000
Oops: 0002
CPU: 0
EIP: 0010:[<c012566f>]
EFLAGS: 00010202
eax: 00006480 ebx: 00000000 ecx: eafbfc00 edx: efc6b8f8
esi: eafbfc00 edi: 00000754 ebp: 00000000 esp: effe9fc4
ds: 0018 es: 0018 ss: 0018
Process kflushd (pid: 2, process nr: 2, stackpage=effe900)
Stack: eafbfc00 00000001 eafbfba0 c0126e56 eafbfc00 00000f00 efffbfcc c0106000
000000a0 00000003 eafbfc00 c0107b2b 00000000 00000f00 c0203fd8
Call Trace: [<c0126e56>] [<c0106000>] [<c0107b2b>]
Code: 89 50 34 c7 01 00 00 00 00 89 02 c7 41 34 00 00 00 00 ff 0d
>>EIP; c012566f <remove_from_queues+a3/130> <=====
Trace; c0126e56 <bdflush+d2/220>
Trace; c0106000 <get_options+0/74>
Trace; c0107b2b <kernel_thread+23/30>
Code; c012566f <remove_from_queues+a3/130>
00000000 <_EIP>:
Code; c012566f <remove_from_queues+a3/130> <=====
0: 89 50 34 movl %edx,0x34(%eax) <=====
Code; c0125672 <remove_from_queues+a6/130>
3: c7 01 00 00 00 movl $0x0,(%ecx)
Code; c0125677 <remove_from_queues+ab/130>
8: 00
Code; c0125678 <remove_from_queues+ac/130>
9: 89 02 movl %eax,(%edx)
Code; c012567a <remove_from_queues+ae/130>
b: c7 41 34 00 00 movl $0x0,0x34(%ecx)
Code; c012567f <remove_from_queues+b3/130>
10: 00 00
Code; c0125681 <remove_from_queues+b5/130>
12: ff 0d 00 00 00 decl 0x0
Code; c0125686 <remove_from_queues+ba/130>
17: 00
The first Oops arises in fs/inode.c:
/*
* Called with the inode lock held.
*/
static struct inode * find_inode(struct super_block * sb, unsigned long ino, struct list_head *head)
{
struct list_head *tmp;
struct inode * inode;
tmp = head;
for (;;) {
tmp = tmp->next;
inode = NULL;
if (tmp == head)
break;
inode = list_entry(tmp, struct inode, i_hash);
if (inode->i_sb != sb) /* <=== OOPS OCCURS HERE */
continue;
if (inode->i_ino != ino)
continue;
inode->i_count++;
break;
}
return inode;
}
The second in fs/buffer.c (inlined into remove_from_queues):
static inline void remove_from_hash_queue(struct buffer_head * bh)
{
struct buffer_head **pprev = bh->b_pprev;
if (pprev) {
struct buffer_head * next = bh->b_next;
if (next) {
next->b_pprev = pprev; /* <=== OOPS OCCURS HERE */
bh->b_next = NULL;
}
*pprev = next;
bh->b_pprev = NULL;
nr_hashed_buffers--;
}
}
In both cases, the pointer deref is the direct cause (cf the asm).
I've been trying to identify whether it's a parity error causing the problem,
or it is SCSI related.
The DIMMs are non-parity (I know I know, but I didn't order the system, OK?),
so this is high on my list of possible villains. OTOH, the memory blocks
are from a reputedly high-quality vendor (Memory Card Technology), so I'm
reluctant to make a blanket statement about bad DIMMs.
I'd be very happy for suggestions as to the cause of these oopses.
I can try to reproduce them if need be (the box isn't getting released
for 'production' before this has been resolved).
TIA,
Kai
-- Kai Harrekilde-Petersen Exbit Technology A/S- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Thu Mar 23 2000 - 21:00:20 EST