Suggestions with hard lockup on 4 systems, have oops report
From: Brian McEntire
Date: Fri Jul 16 2004 - 10:03:56 EST
Thank you for taking time from your busy days to read this. You all
(kernel maintainers) rock! :)
I have four Linux hosts, with identical hardware and OSs, that exhibit a
very tough to troubleshoot hang/freeze. About once every two weeks (and
infrequently, up to a couple of times a day) these systems lock up. I
cannot ping them, cannot toggle caps lock or num lock, nor get any mouse
movement. Even the Magic SysRQ key, enabled just while troubleshooting
this issue, does not respond. (I have tested to make sure it works when
the systems are not frozen.) The screen can be blank or frozen with a
GNOME desktop visible.
When it locks, it does not write an OOPS to the screen, nor to
/var/log/messages, nor to the remote log host although syslog.conf is set
up and does log to them under normal conditions. I connected a serial
terminal and was able to capture an oops report there and have run it
though ksymoops on the system where it was captured. That is attached (run
through ksymoops) to this e-mail.
This has proven impossible (so far) to replicate on demand. I've tried
looping kernel compiles to stress test the CPUs, and replicated that test
with the source on an NFS mounted file system to stress the network sub
system but I can't force the hang to happen. I used memtest86, the RAM
checks out okay. I don't think it could be a hardware issue since it
affects all 4 identical systems. I remade the swap partition with -c to
check and mark bad blocks. These did not fix the hang.
The systems are:
Dual Xeon 2.4GHz processors
2 GB RAM
2 GB swap
Ethernet controller: PCI device 14e4:16a7 (BROADCOM Corporation) (rev 2)
Dual channel SCSI storage controller: PCI device 1000:0030 (Symbios
Logic Inc. (formerly NCR)) (rev 7)
VGA compatible controller: nVidia Corporation NV11
Matrox Graphics, Inc. MGA G400 AGP
The OS specifics:
RH 7.2 with latest patches except running kernel 2.4.9-31enterprise for
CM reasons (at one point, I tried the latest available RH 7.2 kernel but
it did not improve stability so I went back.)
bcm5700-7.1.22-1
nvidia ?? (no RPM listed, didn't know where to find the version.)
I have been logging /proc/meminfo and `uptime` into a file in /tmp every
minute. Load isn't usually above 1 when the systems lock. 0-10 people are
on the system. Sometimes it happens during work hours, other times over
night or over the weekend when no one is running any interactive commands.
* HighFree and especially LowFree usually approach zero just before the
hang. Also, although swap is enabled (`free` shows it and kswapd is
running,) SwapUsed never goes above 0 for the duration of the uptime.
Any ideas? I understand this is an old kernel and BCM5700 is a proprietary
driver module so there may be little anyone can offer. But, just in case
the swap is an issue, or the bug stems from something other than just the
network module, I wanted to send a report in and see if anyone has ideas
or fixes.
Thank you so much!
Brian McEntire
[ksymoops output attached.]
Also, tried Linus's trick to 'disassemble the "Code:" part' ... this
doesn't mean anything to me but maybe its a big clue to someone else:
(gdb) disassemble str
Dump of assembler code for function str:
0x8049384 <str>: movl $0x0,(%ebx)
0x804938a <str+6>: mov 0xffffffec(%ebp),%ecx
0x804938d <str+9>: mov %ebx,0x4(%esp,1)
0x8049391 <str+13>: mov %ecx,(%esp,1)
0x8049394 <str+16>: call 0x804fd15
0x8049399 <str+21>: add %al,(%eax)
0x804939b <str+23>: add %al,(%eax)
End of assembler dump.
ksymoops 2.4.1 on i686 2.4.9-31enterprise. Options used
-v /boot/vmlinuz-2.4.9-31enterprise (specified)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.9-31enterprise/ (default)
-m /boot/System.map-2.4.9-31enterprise (default)
Error (pclose_local): read_nm_symbols pclose failed 0x100
Warning (read_vmlinux): no kernel symbols in vmlinux, is /boot/vmlinuz-2.4.9-31enterprise a valid vmlinux file?
Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
Error (expand_objects): cannot stat(/lib/mptscsih.o) for mptscsih
Error (expand_objects): cannot stat(/lib/mptbase.o) for mptbase
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
Warning (compare_maps): ksyms_base symbol GPLONLY_IO_APIC_get_PCI_irq_vector not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol GPLONLY_pci_hp_change_slot_info not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol GPLONLY_pci_hp_deregister not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol GPLONLY_pci_hp_register not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol __brlock_array , ksyms_base says c02f33a0, System.map says c02f2160. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol _ctype , ksyms_base says c02f328c, System.map says c02f204c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol abi_defhandler_coff , ksyms_base says c02dca0c, System.map says c02db7cc. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol abi_defhandler_elf , ksyms_base says c02dca10, System.map says c02db7d0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol abi_defhandler_lcall7 , ksyms_base says c02dca14, System.map says c02db7d4. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol abi_defhandler_libcso , ksyms_base says c02dca18, System.map says c02db7d8. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol arp_broken_ops , ksyms_base says c02f1040, System.map says c02efe00. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol arp_tbl , ksyms_base says c02f1060, System.map says c02efe20. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol blk_buffers_wait , ksyms_base says c02e894c, System.map says c02e770c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol boot_cpu_data , ksyms_base says c02db500, System.map says c02da2c0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol br_fdb_get_hook , ksyms_base says c02f2d04, System.map says c02f1ac4. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol br_fdb_put_hook , ksyms_base says c02f2d08, System.map says c02f1ac8. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol br_handle_frame_hook , ksyms_base says c02ef148, System.map says c02edf08. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol buffermem_pages , ksyms_base says c02df51c, System.map says c02de2dc. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol cap_bset , ksyms_base says c02dd9ac, System.map says c02dc76c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol color_table , ksyms_base says c02e2b7c, System.map says c02e193c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol dcache_lock , ksyms_base says c02dfa78, System.map says c02de838. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol def_blk_fops , ksyms_base says c02df640, System.map says c02de400. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol default_blu , ksyms_base says c02e2c0c, System.map says c02e19cc. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol default_grn , ksyms_base says c02e2bcc, System.map says c02e198c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol default_red , ksyms_base says c02e2b8c, System.map says c02e194c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol dev_base , ksyms_base says c02eb73c, System.map says c02ea4fc. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol dev_base_lock , ksyms_base says c02eb740, System.map says c02ea500. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol dma_spin_lock , ksyms_base says c02dc8c0, System.map says c02db680. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol fbcon_cfb16 , ksyms_base says c02ee740, System.map says c02ed500. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol fbcon_cfb24 , ksyms_base says c02ee780, System.map says c02ed540. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol fbcon_cfb32 , ksyms_base says c02ee7c0, System.map says c02ed580. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol fbcon_cfb8 , ksyms_base says c02ee6e0, System.map says c02ed4a0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol fbcon_dummy , ksyms_base says c02ee520, System.map says c02ed2e0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol file_lock_list , ksyms_base says c02dfa68, System.map says c02de828. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol files_lock , ksyms_base says c02df49c, System.map says c02de25c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol files_stat , ksyms_base says c02df480, System.map says c02de240. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol fs_overflowgid , ksyms_base says c02dda2c, System.map says c02dc7ec. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol fs_overflowuid , ksyms_base says c02dda28, System.map says c02dc7e8. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol generic_ro_fops , ksyms_base says c02df340, System.map says c02de100. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol global_irq_holder , ksyms_base says c02dad40, System.map says c02d9b00. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol hotplug_path , ksyms_base says c02ddb80, System.map says c02dc940. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol ide_fops , ksyms_base says c02eb9a8, System.map says c02ea768. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol if_port_text , ksyms_base says c02ef108, System.map says c02edec8. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol inet_dgram_ops , ksyms_base says c02f1a00, System.map says c02f07c0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol inet_family_ops , ksyms_base says c02f1a44, System.map says c02f0804. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol inet_stream_ops , ksyms_base says c02f19a0, System.map says c02f0760. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol inetdev_lock , ksyms_base says c02f1498, System.map says c02f0258. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol init_mm , ksyms_base says c02da5a0, System.map says c02d9360. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol io_request_lock , ksyms_base says c02e8948, System.map says c02e7708. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol iomem_resource , ksyms_base says c02dccd8, System.map says c02dba98. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol ioport_resource , ksyms_base says c02dccbc, System.map says c02dba7c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol ipv4_specific , ksyms_base says c02f0160, System.map says c02eef20. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol isapnp_cards , ksyms_base says c02ec800, System.map says c02eb5c0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol isapnp_devices , ksyms_base says c02ec808, System.map says c02eb5c8. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol jh_splice_lock , ksyms_base says c02dfb8c, System.map says c02de94c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol journal_oom_retry , ksyms_base says c02dfb90, System.map says c02de950. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol kernel_flag , ksyms_base says c02dbe00, System.map says c02dabc0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol keyboard_tasklet , ksyms_base says c02e722c, System.map says c02e5fec. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol loopback_dev , ksyms_base says c02eb600, System.map says c02ea3c0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol loops_per_jiffy , ksyms_base says c02da6c8, System.map says c02d9488. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol nfsd_linkage , ksyms_base says c02dfb88, System.map says c02de948. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol noop_qdisc , ksyms_base says c02ef980, System.map says c02ee740. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol overflowgid , ksyms_base says c02dda24, System.map says c02dc7e4. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol overflowuid , ksyms_base says c02dda20, System.map says c02dc7e0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol page_cache_size , ksyms_base says c02ddd60, System.map says c02dcb20. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol page_symlink_inode_operations , ksyms_base says c02df9a0, System.map says c02de760. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol partition_name , ksyms_base says c01c3f80, System.map says c0162cf0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol pci_devices , ksyms_base says c02ec4a8, System.map says c02eb268. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol pci_mem_start , ksyms_base says c02db594, System.map says c02da354. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol pci_root_buses , ksyms_base says c02ec4a0, System.map says c02eb260. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol pfifo_qdisc_ops , ksyms_base says c02efae0, System.map says c02ee8a0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol proc_root , ksyms_base says c02dff00, System.map says c02decc0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol qdisc_tree_lock , ksyms_base says c02ef920, System.map says c02ee6e0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol rtc_lock , ksyms_base says c02db9a0, System.map says c02da760. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol rtnl_sem , ksyms_base says c02ef600, System.map says c02ee3c0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol securebits , ksyms_base says c02dc800, System.map says c02db5c0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol smp_num_cpus , ksyms_base says c02dbe10, System.map says c02dabd0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol socket_file_ops , ksyms_base says c02eed80, System.map says c02edb40. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sys_call_table , ksyms_base says c02da8e4, System.map says c02d96a4. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sys_tux_ptr , ksyms_base says c02eee74, System.map says c02edc34. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_ip_default_ttl , ksyms_base says c02f0080, System.map says c02eee40. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_local_port_range , ksyms_base says c02f0124, System.map says c02eeee4. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_max_syn_backlog , ksyms_base says c02f0130, System.map says c02eeef0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_rmem_max , ksyms_base says c02eee84, System.map says c02edc44. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_tcp_ecn , ksyms_base says c02f00e0, System.map says c02eeea0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_tcp_reordering , ksyms_base says c02f00dc, System.map says c02eee9c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_tcp_rmem , ksyms_base says c02f00b0, System.map says c02eee70. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_tcp_tw_recycle , ksyms_base says c02f0620, System.map says c02ef3e0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_tcp_wmem , ksyms_base says c02f00a4, System.map says c02eee64. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol sysctl_wmem_max , ksyms_base says c02eee80, System.map says c02edc40. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol system_utsname , ksyms_base says c02da6e0, System.map says c02d94a0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol tcp_port_rover , ksyms_base says c02f012c, System.map says c02eeeec. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol tcp_prot , ksyms_base says c02f01a0, System.map says c02eef60. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol tq_disk , ksyms_base says c02e8940, System.map says c02e7700. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol tq_immediate , ksyms_base says c02dd9c4, System.map says c02dc784. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol tq_timer , ksyms_base says c02dd9bc, System.map says c02dc77c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol tqueue_lock , ksyms_base says c02dd9f8, System.map says c02dc7b8. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol tux_module , ksyms_base says c02eee78, System.map says c02edc38. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol tux_module_lock , ksyms_base says c02eee7c, System.map says c02edc3c. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol udp_hash_lock , ksyms_base says c02f0b40, System.map says c02ef900. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol udp_prot , ksyms_base says c02f0b60, System.map says c02ef920. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol uts_sem , ksyms_base says c02dda50, System.map says c02dc810. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol vm_max_readahead , ksyms_base says c02ddd64, System.map says c02dcb24. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol vm_min_readahead , ksyms_base says c02ddd68, System.map says c02dcb28. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol _nv000173rm , nvidia says f8c757c0, /lib/modules/2.4.9-31enterprise/kernel/drivers/video/nvidia.o says f8c755a0. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/video/nvidia.o entry
Warning (compare_maps): mismatch on symbol nlmsvc_grace_period , lockd says f8a008f4, /lib/modules/2.4.9-31enterprise/kernel/fs/lockd/lockd.o says f89ffd68. Ignoring /lib/modules/2.4.9-31enterprise/kernel/fs/lockd/lockd.o entry
Warning (compare_maps): mismatch on symbol nlmsvc_ops , lockd says f8a008f0, /lib/modules/2.4.9-31enterprise/kernel/fs/lockd/lockd.o says f89ffd64. Ignoring /lib/modules/2.4.9-31enterprise/kernel/fs/lockd/lockd.o entry
Warning (compare_maps): mismatch on symbol nlmsvc_timeout , lockd says f8a008f8, /lib/modules/2.4.9-31enterprise/kernel/fs/lockd/lockd.o says f89ffd6c. Ignoring /lib/modules/2.4.9-31enterprise/kernel/fs/lockd/lockd.o entry
Warning (compare_maps): mismatch on symbol nfs_debug , sunrpc says f89f0f00, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0be0. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol nfsd_debug , sunrpc says f89f0f04, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0be4. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol nlm_debug , sunrpc says f89f0f08, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0be8. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol rpc_debug , sunrpc says f89f0efc, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0bdc. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol rpc_garbage_args , sunrpc says f89f0edc, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0bbc. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol rpc_success , sunrpc says f89f0ecc, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0bac. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol rpc_system_err , sunrpc says f89f0ee0, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0bc0. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol xdr_one , sunrpc says f89f0ec4, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0ba4. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol xdr_two , sunrpc says f89f0ec8, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0ba8. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol xdr_zero , sunrpc says f89f0ec0, /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o says f89f0ba0. Ignoring /lib/modules/2.4.9-31enterprise/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol usb_devfs_handle , usbcore says f895faa0, /lib/modules/2.4.9-31enterprise/kernel/drivers/usb/usbcore.o says f895f5c0. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/usb/usbcore.o entry
Warning (map_ksym_to_module): cannot match loaded module ext3 to a unique module object. Trace may not be reliable.
Warning (compare_maps): mismatch on symbol DmpService , mptbase says f8826084, /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o says f8826004. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o entry
Warning (compare_maps): mismatch on symbol mpt_ASCQ_TableSz , mptbase says f8826090, /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o says f8826010. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o entry
Warning (compare_maps): mismatch on symbol mpt_ScsiOpcodesPtr , mptbase says f882608c, /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o says f882600c. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o entry
Warning (compare_maps): mismatch on symbol mpt_adapters , mptbase says f8826040, /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o says f8825fc0. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o entry
Warning (compare_maps): mismatch on symbol mpt_proc_root_dir , mptbase says f8826080, /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o says f8826000. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o entry
Warning (compare_maps): mismatch on symbol mpt_v_ASCQ_TablePtr , mptbase says f8826088, /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o says f8826008. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/message/fusion/mptbase.o entry
Warning (compare_maps): mismatch on symbol sd , sd_mod says f881cda0, /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/sd_mod.o says f881cd00. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/sd_mod.o entry
Warning (compare_maps): mismatch on symbol proc_scsi , scsi_mod says f88181dc, /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o says f8816a14. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o entry
Warning (compare_maps): mismatch on symbol scsi_devicelist , scsi_mod says f8818208, /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o says f8816a40. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o entry
Warning (compare_maps): mismatch on symbol scsi_hostlist , scsi_mod says f8818204, /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o says f8816a3c. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o entry
Warning (compare_maps): mismatch on symbol scsi_hosts , scsi_mod says f881820c, /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o says f8816a44. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o entry
Warning (compare_maps): mismatch on symbol scsi_logging_level , scsi_mod says f88181d8, /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o says f8816a10. Ignoring /lib/modules/2.4.9-31enterprise/kernel/drivers/scsi/scsi_mod.o entry
Unable to handle kernel NULL pointer dereference at virtual address 00000000
f89b669f
*pde = 35db8001
Oops: 0002
CPU: 0
EIP: 0010:[<f89b669f>] Tainted: P
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 0000011c ebx: 00000000 ecx: 00000071 edx: 0000011c
esi: 000001fb edi: f77b0140 ebp: c02f5f18 esp: c02f5ef0
ds: 0018 es: 0018 ss: 0018
Process swapper (pid: 0, stackpage=c02f5000)
Stack: f77b2230 f5e749d8 c02f5f18 00000246 f77b1a1c f77b2230 00000001 f77b0140
00000000 000000c3 c02f5f48 f89a3e1f f77b0140 00005804 00000001 c037dea0
c0123bc1 00000046 00000001 f7cd7620 00000000 04000001 00000030 c0108a6e
Call Trace: [<f89a3e1f>] bcm5700_interrupt [bcm5700] 0x12f
[<c0123bc1>] __run_timers [kernel] 0xd1
[<c0108a6e>] handle_IRQ_event [kernel] 0x5e
[<c0108c74>] do_IRQ [kernel] 0xa4
[<c0105410>] default_idle [kernel] 0x0
[<c0105410>] default_idle [kernel] 0x0
[<c022fab0>] call_do_IRQ [kernel] 0x5
[<c0105410>] default_idle [kernel] 0x0
[<c0105410>] default_idle [kernel] 0x0
[<c010543d>] default_idle [kernel] 0x2d
[<c01054c2>] cpu_idle [kernel] 0x52
[<c0105000>] stext [kernel] 0x0
[<c022e7c0>] .rodata.str1.32 [kernel] 0x5c0
Code: c7 03 00 00 00 00 8b 4d ec 89 5c 24 04 89 0c 24 e8 7c 69 00
>>EIP; f89b669f <[bcm5700]LM_ServiceInterrupts+cf/230> <=====
Trace; f89a3e1f <[bcm5700]bcm5700_interrupt+12f/410>
Trace; c0123bc1 <__run_timers+d1/110>
Trace; c0108a6e <handle_IRQ_event+5e/90>
Trace; c0108c74 <do_IRQ+a4/f0>
Trace; c0105410 <default_idle+0/40>
Trace; c0105410 <default_idle+0/40>
Trace; c022fab0 <IRQ0x6f_interrupt+8/c>
Trace; c0105410 <default_idle+0/40>
Trace; c0105410 <default_idle+0/40>
Trace; c010543d <default_idle+2d/40>
Trace; c01054c2 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>
Trace; c022e7c0 <llc_oui+990/1740>
Code; f89b669f <[bcm5700]LM_ServiceInterrupts+cf/230>
00000000 <_EIP>:
Code; f89b669f <[bcm5700]LM_ServiceInterrupts+cf/230> <=====
0: c7 03 00 00 00 00 movl $0x0,(%ebx) <=====
Code; f89b66a5 <[bcm5700]LM_ServiceInterrupts+d5/230>
6: 8b 4d ec mov 0xffffffec(%ebp),%ecx
Code; f89b66a8 <[bcm5700]LM_ServiceInterrupts+d8/230>
9: 89 5c 24 04 mov %ebx,0x4(%esp,1)
Code; f89b66ac <[bcm5700]LM_ServiceInterrupts+dc/230>
d: 89 0c 24 mov %ecx,(%esp,1)
Code; f89b66af <[bcm5700]LM_ServiceInterrupts+df/230>
10: e8 7c 69 00 00 call 6991 <_EIP+0x6991> f89bd030 <[bcm5700]QQ_PushTail+0/40>
<0>Kernel panic: Aiee, killing interrupt handler!
133 warnings and 7 errors issued. Results may not be reliable.