Hello,
last week our server wasn't responding anymore and we had to reboot it
using alt sysreq (emergency sync, umount, reboot). When the machine came
up again we found the log file contained the attached oops. It was a
null pointer dereference in prune_dcache (the back link of a list
element was zero when prune_dcache tried to delete it from the list).
Since we use NFS it may be possible that we see the same bug described
in "NFS client stuck in NFS_free_dentries (2.2.15)", but I'm not
sure. The server crashed several times during the last week without
leaving a trace in the syslogs (and our logging over the serial line
wasn't working so we don't have the oopses). If I can provide any
additional information feel free to ask me.
Jean
PS: the oops processed by ksymoops
ksymoops 2.3.4 on i686 2.2.14. Options used
-v /home/jw5/tmp//vmlinux (specified)
-K (specified)
-L (specified)
-O (specified)
-m /home/jw5/tmp//System.map-2.2.14 (specified)
Oops: 0002
CPU: 3
EIP: 0010:[<c0130f54>]
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010202
eax: c02138ac ebx: c804e840 ecx: c804efa0 edx: 00000000
esi: 00000001 edi: c804eea0 ebp: 00000001 esp: cddd5e94
ds: 0018 es: 0018 ss: 0018
Process save (pid: 4872, process nr: 112, stackpage=cddd5000)
Stack: 00000a05 cfcfd200 00000a05 c013216e fffff8fc 00000345 c02138d4 005a78cd
c024755c cfcfd200 c2f30380 c1d6c880 cddd5ec4 cddd5ec4 c01321fc 00000a05
c02138d4 005a78cd c024755c cfcfd200 cfcfd200 c01325a0 c1ff1760 c01325b1
Call Trace: [try_to_free_inodes+306/384] [grow_inodes+32/464] [get_new_inode+176/300] [get_new_inode+193/300] [iget+120/132] [ext2_lookup+92/144] [real_lookup+80/168] [lookup_dentry+268/428] [__namei+41/92] [sys_newlstat+46/148] [system_call+52/56]
Code: 89 02 89 1b 89 5b 04 8d 7b e0 83 7b e0 00 75 d0 8b 53 fc 8b
>>EIP; c0130f54 <prune_dcache+30/110> <=====
Code; c0130f54 <prune_dcache+30/110>
00000000 <_EIP>:
Code; c0130f54 <prune_dcache+30/110> <=====
0: 89 02 mov %eax,(%edx) <=====
Code; c0130f56 <prune_dcache+32/110>
2: 89 1b mov %ebx,(%ebx)
Code; c0130f58 <prune_dcache+34/110>
4: 89 5b 04 mov %ebx,0x4(%ebx)
Code; c0130f5b <prune_dcache+37/110>
7: 8d 7b e0 lea 0xffffffe0(%ebx),%edi
Code; c0130f5e <prune_dcache+3a/110>
a: 83 7b e0 00 cmpl $0x0,0xffffffe0(%ebx)
Code; c0130f62 <prune_dcache+3e/110>
e: 75 d0 jne ffffffe0 <_EIP+0xffffffe0> c0130f34 <prune_dcache+10/110>
Code; c0130f64 <prune_dcache+40/110>
10: 8b 53 fc mov 0xfffffffc(%ebx),%edx
Code; c0130f67 <prune_dcache+43/110>
13: 8b 00 mov (%eax),%eax
*******************************************************************************
The disassembled prune_dcache:
prune_dcache():
1a8: 83 ec 04 sub $0x4,%esp
1ab: 55 push %ebp
1ac: 57 push %edi
1ad: 56 push %esi
1ae: 53 push %ebx
1af: 8b 4c 24 1c mov 0x1c(%esp,1),%ecx
1b3: 89 4c 24 10 mov %ecx,0x10(%esp,1)
1b7: 90 nop
1b8: 8b 1d 04 00 00 00 mov 0x4,%ebx
1be: 81 fb 00 00 00 00 cmp $0x0,%ebx
1c4: 0f 84 db 00 00 00 je 2a5 <prune_dcache+0xfd>
1ca: ff 0d 04 00 00 00 decl 0x4
1d0: 8b 53 04 mov 0x4(%ebx),%edx
1d3: 8b 03 mov (%ebx),%eax
1d5: 89 50 04 mov %edx,0x4(%eax)
1d8: 89 02 mov %eax,(%edx) <===========
1da: 89 1b mov %ebx,(%ebx)
1dc: 89 5b 04 mov %ebx,0x4(%ebx)
1df: 8d 7b e0 lea 0xffffffe0(%ebx),%edi
********************************************************************************
The source:
int prune_dcache(int d_nr, int i_nr)
{
int __i_nr = i_nr;
for (;;) {
struct dentry *dentry;
struct list_head *tmp = dentry_unused.prev;
if (tmp == &dentry_unused)
break;
dentry_stat.nr_unused--;
list_del(tmp);
*******************************************************************************
Source mixed with assembly:
list_del(tmp) == __list_del(tmp->prev, tmp->next)
static __inline__ void __list_del(struct list_head * prev,
struct list_head * next)
{
%edx = tmp->prev
%eax = tmp->next
1d0: 8b 53 04 mov 0x4(%ebx),%edx
1d3: 8b 03 mov (%ebx),%eax
/home/jw5/tmp/linux22/include/linux/list.h:61
next->prev = prev;
1d5: 89 50 04 mov %edx,0x4(%eax)
/home/jw5/tmp/linux22/include/linux/list.h:62
prev->next = next;
1d8: 89 02 mov %eax,(%edx) <===========
So it looks like the backlink to the previous list element was zero.
********************************************************************************
Our configuration:
CONFIG_EXPERIMENTAL=y
CONFIG_M686=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_1GB=y
CONFIG_SMP=y
CONFIG_NET=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_QUIRKS=y
CONFIG_PCI_OLD_PROC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_SYSVIPC=y
CONFIG_SYSCTL=y
CONFIG_BINFMT_AOUT=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=y
CONFIG_BLK_DEV_FD=y
CONFIG_PARIDE_PARPORT=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_ALIAS=y
CONFIG_SKB_LARGE=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_BUSLOGIC=y
CONFIG_SCSI_GDTH=y
CONFIG_NETDEVICES=y
CONFIG_NET_ETHERNET=y
CONFIG_NET_EISA=y
CONFIG_DEC_ELCP=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_SERIAL=y
CONFIG_SERIAL_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_UNIX98_PTY_COUNT=512
CONFIG_MOUSE=y
CONFIG_PSMOUSE=y
CONFIG_RTC=y
CONFIG_AUTOFS_FS=y
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_MINIX_FS=y
CONFIG_PROC_FS=y
CONFIG_DEVPTS_FS=y
CONFIG_EXT2_FS=y
CONFIG_NFS_FS=y
CONFIG_NFSD=y
CONFIG_SUNRPC=y
CONFIG_LOCKD=y
CONFIG_NLS=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_15=y
CONFIG_VGA_CONSOLE=y
CONFIG_MAGIC_SYSRQ=y
Jean
-- I get up each morning, gather my wits. Pick up the paper, read the obits. if I'm not there I know I'm not dead. So I eat a good breakfast and go back to bed. Peete Seeger- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Tue May 23 2000 - 21:00:21 EST