2.6.11-rc4-RT-V0.7.39-02 crash out of nowhere

[Gene Heskett]

Greetings;

I just came back in here after a 2 hour nap, to find the box crashed,
100%.  Uptime was at least 2 days for 2.6.11-rc4-RT-V0.7.39-02, and
no hints of impending doom.  The .config is attached.

The system was quiet except for kmail's timed mail fetching runs, 
x-6.8.1 (built on site) was running, as was setiathome.  Nothing
in the crontabs covers this time frame as rsync was scheduled to
start half an hour later, with amdump following it at 1 am.  So
if anything triggered it, it might have been kmail.  This is kde
3.3.0, konstruct built, and much more stable than the rpms have
been so far.  I haven't ever seen 99% of the crashes and problems
reported on the kde mailing lists.
 
Here is the first of the crash, up and including a fraction of the
next full second of the log. I won't include any more except a
couple of lines at the bottom to show how long it looped before
totally dying as the full log is 107+ megabytes.

Feb 20 23:08:01 coyote kernel: move_wait_queue+0x43/0x60 (12)
Feb 20 23:08:01 coyote kernel:  [<c012ad23>] remove_wait_queue+0x43/0x60 (24)
Feb 20 23:08:01 coyote kernel:  [<c0168f44>] poll_freewait+0x24/0x50 (24)
Feb 20 23:08:01 coyote kernel:  [<c01692e9>] do_select+0x1c9/0x2d0 (16)
Feb 20 23:08:01 coyote kernel:  [<c0168f70>] __pollwait+0x0/0xd0 (84)
Feb 20 23:08:01 coyote kernel:  [<c01696df>] sys_select+0x2bf/0x4d0 (32)
Feb 20 23:08:01 coyote kernel:  [<c012d44e>] up_mutex+0xbe/0x120 (16)
Feb 20 23:08:01 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (76)
Feb 20 23:08:01 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2604
Feb 20 23:08:01 coyote kernel:  [<c013f765>] kfree+0x135/0x140 (8)
Feb 20 23:08:01 coyote kernel:  [<c0169654>] sys_select+0x234/0x4d0 (44)
Feb 20 23:08:01 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (92)
Feb 20 23:08:01 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2608
Feb 20 23:08:01 coyote kernel:  [<c0169654>] sys_select+0x234/0x4d0 (8)
Feb 20 23:08:01 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (92)
Feb 20 23:08:01 coyote kernel: Warning: kfree_skb on hard IRQ c0351eea
Feb 20 23:08:01 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2604
Feb 20 23:08:01 coyote kernel:  [<c013f765>] kfree+0x135/0x140 (8)
Feb 20 23:08:01 coyote kernel:  [<c02f9363>] kfree_skbmem+0x13/0x30 (44)
Feb 20 23:08:01 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:01 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:01 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:01 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:01 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:01 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:01 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:01 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2608
Feb 20 23:08:02 coyote kernel:  [<c02f9363>] kfree_skbmem+0x13/0x30 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kmem_cache_free at mm/slab.c:2562
Feb 20 23:08:02 coyote kernel:  [<c013f5af>] kmem_cache_free+0x11f/0x130 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9374>] kfree_skbmem+0x24/0x30 (44)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kmem_cache_free at mm/slab.c:2564
Feb 20 23:08:02 coyote kernel:  [<c02f9374>] kfree_skbmem+0x24/0x30 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __cache_alloc at mm/slab.c:2168
Feb 20 23:08:02 coyote kernel:  [<c013f474>] __kmalloc+0x134/0x150 (8)
Feb 20 23:08:02 coyote kernel:  [<c016940c>] select_bits_alloc+0x1c/0x20 (44)
Feb 20 23:08:02 coyote kernel:  [<c0169514>] sys_select+0xf4/0x4d0 (12)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c011a280>] sys_gettimeofday+0x40/0xc0 (40)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (36)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __cache_alloc at mm/slab.c:2180
Feb 20 23:08:02 coyote kernel:  [<c013f41b>] __kmalloc+0xdb/0x150 (8)
Feb 20 23:08:02 coyote kernel:  [<c016940c>] select_bits_alloc+0x1c/0x20 (44)
Feb 20 23:08:02 coyote kernel:  [<c0169514>] sys_select+0xf4/0x4d0 (12)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c011a280>] sys_gettimeofday+0x40/0xc0 (40)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (36)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in _down_mutex at kernel/rt.c:1293
Feb 20 23:08:02 coyote kernel:  [<c012d1a8>] _down_mutex+0x98/0xc0 (8)
Feb 20 23:08:02 coyote kernel:  [<c012dcc2>] __spin_lock+0x32/0x50 (36)
Feb 20 23:08:02 coyote kernel:  [<c012ac61>] add_wait_queue+0x21/t+0x2bf/0x4d0 (32)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (76)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __up_mutex at kernel/rt.c:1146
Feb 20 23:08:02 coyote kernel:  [<c012c8bf>] __up_mutex+0x9f/0x4d0 (8)
Feb 20 23:08:02 coyote kernel:  [<c012ad23>] remove_wait_queue+0x43/0x60 (32)
Feb 20 23:08:02 coyote kernel:  [<c012ad23>] remove_wait_queue+0x43/0x60 (16)
Feb 20 23:08:02 coyote kernel:  [<c012d44e>] up_mutex+0xbe/0x120 (12)
Feb 20 23:08:02 coyote kernel:  [<c012ad23>] remove_wait_queue+0x43/0x60 (12)
Feb 20 23:08:02 coyote kernel:  [<c012ad23>] remove_wait_queue+0x43/0x60 (24)
Feb 20 23:08:02 coyote kernel:  [<c0168f44>] poll_freewait+0x24/0x50 (24)
Feb 20 23:08:02 coyote kernel:  [<c01692e9>] do_select+0x1c9/0x2d0 (16)
Feb 20 23:08:02 coyote kernel:  [<c0168f70>] __pollwait+0x0/0xd0 (84)
Feb 20 23:08:02 coyote kernel:  [<c01696df>] sys_select+0x2bf/0x4d0 (32)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (76)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2604
Feb 20 23:08:02 coyote kernel:  [<c013f765>] kfree+0x135/0x140 (8)
Feb 20 23:08:02 coyote kernel:  [<c0169654>] sys_select+0x234/0x4d0 (44)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (92)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2608
Feb 20 23:08:02 coyote kernel:  [<c0169654>] sys_select+0x234/0x4d0 (8)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (92)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kmem_cache_free at mm/slab.c:2562
Feb 20 23:08:02 coyote kernel:  [<c013f5af>] kmem_cache_free+0x11f/0x130 (8)
Feb 20 23:08:02 coyote kernel:  [<c0120057>] __dequeue_signal+0x117/0x1a0 (44)
Feb 20 23:08:02 coyote kernel:  [<c012018b>] dequeue_signal+0xab/0xc0 (36)
Feb 20 23:08:02 coyote kernel:  [<c0121d48>] get_signal_to_deliver+0x78/0x310 (32)
Feb 20 23:08:02 coyote kernel:  [<c0102514>] do_signal+0xa4/0x140 (40)
Feb 20 23:08:02 coyote kernel:  [<c0116274>] vprintk+0x104/0x160 (28)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (24)
Feb 20 23:08:02 coyote kernel:  [<c0102c6c>] dump_stack+0x1c/0x20 (48)
Feb 20 23:08:02 coyote kernel:  [<c0169654>] sys_select+0x234/0x4d0 (16)
Feb 20 23:08:02 coyote kernel:  [<c01025e5>] do_notify_resume+0x35/0x38 (84)
Feb 20 23:08:02 coyote kernel:  [<c0102766>] work_notifysig+0x13/0x15 (8)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kmem_cache_free at mm/slab.c:2564
Feb 20 23:08:02 coyote kernel:  [<c0120057>] __dequeue_signal+0x117/0x1a0 (8)
Feb 20 23:08:02 coyote kernel:  [<c012018b>] dequeue_signal+0xab/0xc0 (36)
Feb 20 23:08:02 coyote kernel:  [<c0121d48>] get_signal_to_deliver+0x78/0x310 (32)
Feb 20 23:08:02 coyote kernel:  [<c0102514>] do_signal+0xa4/0x140 (40)
Feb 20 23:08:02 coyote kernel:  [<c0116274>] vprintk+0x104/0x160 (28)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (24)
Feb 20 23:08:02 coyote kernel:  [<c0102c6c>] dump_stack+0x1c/0x20 (48)
Feb 20 23:08:02 coyote kernel:  [<c0169654>] sys_select+0x234/0x4d0 (16)
Feb 20 23:08:02 coyote kernel:  [<c01025e5>] do_notify_resume+0x35/0x38 (84)
Feb 20 23:08:02 coyote kernel:  [<c0102766>] work_notifysig+0x13/0x15 (8)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __cache_alloc at mm/slab.c:2168
Feb 20 23:08:02 coyote kernel:  [<c013f474>] __kmalloc+0x134/0x150 (8)
Feb 20 23:08:02 coyote kernel:  [<c016940c>] select_bits_alloc+0x1c/0x20 (44)
Feb 20 23:08:02 coyote kernel:  [<c0169514>] sys_select+0xf4/0x4d0 (12)
Feb 20 23:08:02 coyote kernel:  [<c012d44e>] up_mutex+0xbe/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c011a280>] sys_gettimeofday+0x40/0xc0 (40)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (36)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __cache_alloc at mm/slab.c:2180
Feb 20 23:08:02 coyote kernel:  [<c013f41b>] __kmalloc+0xdb/0x150 (8)
Feb 20 23:08:02 coyote kernel:  [<c016940c>] select_bits_alloc+0x1c/0x20 (44)
Feb 20 23:08:02 coyote kernel:  [<c0169514>] sys_select+0xf4/0x4d0 (12)
Feb 20 23:08:02 coyote kernel:  [<c012d44e>] up_mutex+0xbe/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c011a280>] sys_gettimeofday+0x40/0xc0 (40)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (36)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in _down_mutex at kernel/rt.c:1293
Feb 20 23:08:02 coyote kernel:  [<c012d1a8>] _down_mutex+0x98/0xc0 (8)
Feb 20 23:08:02 coyote kernel:  [<c012dcc2>] __spin_lock+0x32/0x50 (36)
Feb 20 23:08:02 coyote kernel:  [<c012ac61>] add_wait_queue+0x21/0x50 (8)
Feb 20 23:08:02 coyote kernel:  [<c012dd57>] _spin_lock_irqsave+0x17/0x20 (12)
Feb 20 23:08:02 coyote kernel:  [<c012ac61>] add_wait_queue+0x21/0x50 (8)
Feb 20 23:08:02 coyote kernel:  [<c012ac61>] add_wait_queue+0x21/0x50 (4)
Feb 20 23:08:02 coyote kernel:  [<c025dd60>] normal_poll+0x30/0x190 (24)
Feb 20 23:08:02 coyote kernel:  [<c0259920>] tty_poll+0xa0/0xb0 (32)
Feb 20 23:08:02 coyote kernel:  [<c016938f>] do_select+0x26f/0x2d0 (36)
Feb 20 23:08:02 coyote kernel:  [<c0168f70>] __pollwait+0x0/0xd0 (84)
Feb 20 23:08:02 coyote kernel:  [<c01696df>] sys_select+0x2bf/0x4d0 (32)
Feb 20 23:08:02 coyote kernel:  [<c012d44e>] up_mutex+0xbe/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (76)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in up_mutex at kernel/rt.c:1340
Feb 20 23:08:02 coyote kernel:  [<c012d42c>] up_mutex+0x9c/0x120 (8)
Feb 20 23:08:02 coyote kernel:  [<c012ac7f>] add_wait_queue+0x3f/0x50 (36)
Feb 20 23:08:02 coyote kernel:  [<c025dd60>] normal_poll+0x30/0x190 (24)
Feb 20 23:08:02 coyote kernel:  [<c0259920>] tty_poll+0xa0/0xb0 (32)
Feb 20 23:08:02 coyote kernel:  [<c016938f>] do_select+0x26f/0x2d0 (36)
Feb 20 23:08:02 coyote kernel:  [<c0168f70>] __pollwait+0x0/0xd0 (84)
Feb 20 23:08:02 coyote kernel:  [<c01696df>] sys_select+0x2bf/0x4d0 (32)
Feb 20 23:08:02 coyote kernel:  [<c012d44e>] up_mutex+0xbe/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (76)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __up_mutex at kernel/rt.c:1146
Feb 20 23:08:02 coyote kernel:  [<c012c8bf>] __up_mutex+0x9f/0x4d0 (8)
Feb 20 23:08:02 coyote kernel:  [<c012ac7f>] add_wait_queue+0x3f/0x50 (32)
Feb 20 23:08:02 coyote kernel:  [<c012ac7f>] add_wait_queue+0x3f/0x50 (16)
Feb 20 23:08:02 coyote kernel:  [<c012d44e>] up_mutex+0xbe/0x120 (12)
Feb 20 23:08:02 coyote kernel: <unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kmem_cache_free at mm/slab.c:2562
Feb 20 23:08:02 coyote kernel:  [<c013f5af>] kmem_cache_free+0x11f/0x130 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9374>] kfree_skbmem+0x24/0x30 (44)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kmem_cache_free at mm/slab.c:2564
Feb 20 23:08:02 coyote kernel:  [<c02f9374>] kfree_skbmem+0x24/0x30 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __cache_alloc at mm/slab.c:2168
Feb 20 23:08:02 coyote kernel:  [<c013f474>] __kmalloc+0x134/0x150 (8)
Feb 20 23:08:02 coyote kernel:  [<c016940c>] select_bits_alloc+0x1c/0x20 (44)
Feb 20 23:08:02 coyote kernel:  [<c0169514>] sys_select+0xf4/0x4d0 (12)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c011a280>] sys_gettimeofday+0x40/0xc0 (40)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (36)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __cache_alloc at mm/slab.c:2180
Feb 20 23:08:02 coyote kernel:  [<c013f41b>] __kmalloc+0xdb/0x150 (8)
Feb 20 23:08:02 coyote kernel:  [<c016940c>] select_bits_alloc+0x1c/0x20 (44)
Feb 20 23:08:02 coyote kernel:  [<c0169514>] sys_select+0xf4/0x4d0 (12)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c011a280>] sys_gettimeofday+0x40/0xc0 (40)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (36)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in _down_mutex at kernel/rt.c:1293
Feb 20 23:08:02 coyote kernel:  [<c012d1a8>] _down_mutex+0x98/0xc0 (8)
Feb 20 23:08:02 coyote kernel:  [<c012dcc2>] __spin_lock+0x32/0x50 (36)
Feb 20 23:08:02 coyote kernel:  [<c012ac61>] add_wait_queue+0x21/0x50 (8)
Feb 20 23:08:02 coyote kernel:  [<c012dd57>] _spin_lock_irqsave+0x17/0x20 (12)
Feb 20 23:08:02 coyote kernel:  [<c012ac61>] add_wait_queue+0x21/0x50 (8)
Feb 20 23:08:02 coyote kernel:  [<c012ac61>] add_wait_queue+0x21/0x50 (4)
Feb 20 23:08:02 coyote kernel:  [<c025dd60>] normal_poll+0x30/0x190 (24)
Feb 20 23:08:02 coyote kernel:  [<c0259920>] tty_poll+0xa0/0xb0 (32)
Feb 20 23:08:02 coyote kernel:  [<c016938f>] do_select+0x26f/0x2d0 (36)
Feb 20 23:08:02 coyote kernel:  [<c0168f70>] __pollwait+0x0/0xd0 (84)
Feb 20 23:08:02 coyote kernel:  [<c01696df>] sys_select+0x2bf/0x4d0 (32)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (76)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in up_mutex at kernel/rt.c:1340
Feb 20 23:08:02 coyote kernel:  [<c012d42c>] up_mutex+0x9c/0x120 (8)
Feb 20 23:08:02 coyote kernel:  [<c012ac7f>] add_wait_queue+0x3f/0x50 (36)
Feb 20 23:08:02 coyote kernel:  [<c025dd60>] normal_poll+0x30/0x190 (24)
Feb 20 23:08:02 coyote kernel:  [<c0259920>] tty_poll+0xa0/0xb0 (32)
Feb 20 23:08:02 coyote kernel:  [<c016938f>] do_select+0x26f/0x2d0 (36)
Feb 20 23:08:02 coyote kernel:  [<c0168f70>] __pollwait+0x0/0xd0 (84)
Feb 20 23:08:02 coyote kernel:  [<c01696df>] sys_select+0x2bf/0x4d0 (32)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (76)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in __up_mutex at kernel/rt.c:1146
Feb 20 23:08:02 coyote kernel:  [<c012c8bf>] __up_mutex+0x9f/0x4d0 (8)
Feb 20 23:08:02 coyote kernel:  [<c012ac7f>] add_wait_queue+0x3f/0x50 (32)
Feb 20 23:08:02 coyote kernel:  [<c012ac7f>] add_wait_queue+0x3f/0x50 (16)
Feb 20 23:08:02 coyote kernel:  [<c012d44e>] up_mutex+0xbe/0x120 (12)
Feb 20 23:08:02 coyote kernel:  [<c012ac7f>] add_wait_queue+0x3f/0x50 (12)
Feb 20 23:08:02 coyote kernel:  [<c012ac7f>] add_wait_queue+0x3f/0x50 (24)
Feb 20 23:08:02 coyote kernel:  [<c025dd60>] normal_poll+0x30/0x190 (24)
Feb 20 23:08:02 coyote kernel:  [<c0259920>] tty_poll+0xa0/0xb0 (32)
Feb 20 23:08:02 coyote kernel:  [<c016938f>] do_select+0x26f/0x2d0 (36)
Feb 20 23:08:02 coyote kernel:  [<c0168f70>] __pollwait+0x0/0xd0 (84)
Feb 20 23:08:02 coyote kernel:  [<c01696df>] sys_select+0x2bf/0x4d0 (32)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (16)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (76)
Feb 20 23:08:02 coyote kernel: BUG: scheduling 5/0x140 (8)
Feb 20 23:08:02 coyote kernel:  [<c0169654>] sys_select+0x234/0x4d0 (44)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (92)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2608
Feb 20 23:08:02 coyote kernel:  [<c0169654>] sys_select+0x234/0x4d0 (8)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (92)
Feb 20 23:08:02 coyote kernel: Warning: kfree_skb on hard IRQ c0351eea
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2604
Feb 20 23:08:02 coyote kernel:  [<c013f765>] kfree+0x135/0x140 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9363>] kfree_skbmem+0x13/0x30 (44)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kfree at mm/slab.c:2608
Feb 20 23:08:02 coyote kernel:  [<c02f9363>] kfree_skbmem+0x13/0x30 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kmem_cache_free at mm/slab.c:2562
Feb 20 23:08:02 coyote kernel:  [<c013f5af>] kmem_cache_free+0x11f/0x130 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9374>] kfree_skbmem+0x24/0x30 (44)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (8)
Feb 20 23:08:02 coyote kernel:  [<c0351eea>] unix_stream_recvmsg+0x21a/0x480 (24)
Feb 20 23:08:02 coyote kernel:  [<c02f53c5>] sock_aio_read+0xf5/0x110 (116)
Feb 20 23:08:02 coyote kernel:  [<c015525e>] do_sync_read+0xbe/0xf0 (116)
Feb 20 23:08:02 coyote kernel:  [<c011e5fe>] __mod_timer+0xfe/0x170 (28)
Feb 20 23:08:02 coyote kernel:  [<c012dcf7>] _spin_lock+0x17/0x20 (36)
Feb 20 23:08:02 coyote kernel:  [<c012aeb0>] autoremove_wake_function+0x0/0x60 (52)
Feb 20 23:08:02 coyote kernel:  [<c01553cc>] vfs_read+0x13c/0x150 (48)
Feb 20 23:08:02 coyote kernel:  [<c0155671>] sys_read+0x51/0x80 (36)
Feb 20 23:08:02 coyote kernel:  [<c01026c5>] sysenter_past_esp+0x52/0x75 (40)
Feb 20 23:08:02 coyote kernel: X/28686: BUG in kmem_cache_free at mm/slab.c:2564
Feb 20 23:08:02 coyote kernel:  [<c02f9374>] kfree_skbmem+0x24/0x30 (8)
Feb 20 23:08:02 coyote kernel:  [<c02f9405>] __kfree_skb+0x85/0x120 (16)

And this cut & paste is getting ridiculous, this is not even a full
seconds worth, so on to the last line before even the logging died:

Feb 20 23:35:05 coyote kernel:  [<c01696df>] sys_select+0x2bf/0x4d0 (32)
Feb 21 01:16:16 coyote syslogd 1.4.1: restart.

Is this enough to point a finger at?

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.33% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Attachment: config-2.6.11-rc4-RT-V0.7.39-02.gz
Description: GNU Zip compressed data