Re: kblockd/1: page allocation failure in 2.6.9
From: Frank Steiner
Date: Thu Dec 23 2004 - 10:57:06 EST
Hi,
Denis Vlasenko wrote
On Thursday 23 December 2004 11:26, Frank Steiner wrote:
No one able to give me least some hints what this message could mean?
If it is sth. like a kernel oops or sth. else? If it could point to
a hardware error, or kernel bug, or kernel configuration problem or sth?
As long as I don't even know that this message means or where it comes
from, I cannot really debug things :-(
Details?
what exactly do you mean by details? The details of the kernel messages?
They were in my first email, but I post them again below in case the first
mail got lost! Or do you need more details about my system?
I quote the complete first mail to the lkml below, because it contains some details
about the host, too.
cu,
Frank
Frank Steiner wrote
Hi,
we got sth. that looks similar to but not exactly like a kernel oops on one
of our hosts saying "kblockd/1: page allocation failure. order:0, mode:0x21"
(the full output is below). The kernel is 2.6.9.
After that happend, "free" told me that 1.7GB of the 2GB was used and nothing
was buffered (i.e., the "-/+ buffers" shopwed the same amount of used memory),
which was strange because there was no process running that did any real
work. A tomcat server and a mysql database are running on this machine, but
the database is still empty and the tomcat does not yet offer any services.
Also, "ps -aux" showed only 2/3 of the processes then hang until I killed it.
I've never seen "ps -aux" hang! I had to reboot to get rid of this.
I guess the problem occured after copying a large amount of data to the disks.
The host is a dual xeon 1.7GHz with an internal 147GB Raid-1 (ICP vortex controlled
with gdth modul) and two external 600GB IDE-Raid5, connected to an Adaptec 29160
with the aic7xxx modul.
Can someone tell me what this kblockd messages mean? What is kblockd? Could
that point to a hardware problem (CPU, memory, mainboard) or a driver problems
with one the scsi controllers or sth.? The log says sth about scsi_get_command.
And are these messages sth. like a kernel oops or not? I've never seen sth.
like this before so I'm a bit lost :-)
Thanks for any hints!
Best regards,
Frank
Dec 20 13:17:21 turing kernel: kblockd/1: page allocation failure. order:0, mode:0x21
Dec 20 13:17:21 turing kernel: [__alloc_pages+824/832] __alloc_pages+0x338/0x340
Dec 20 13:17:21 turing kernel: [<c0142648>] __alloc_pages+0x338/0x340
Dec 20 13:17:21 turing kernel: [__get_free_pages+24/48] __get_free_pages+0x18/0x30
Dec 20 13:17:21 turing kernel: [<c0142668>] __get_free_pages+0x18/0x30
Dec 20 13:17:21 turing kernel: [kmem_getpages+36/224] kmem_getpages+0x24/0xe0
Dec 20 13:17:21 turing kernel: [<c0145884>] kmem_getpages+0x24/0xe0
Dec 20 13:17:21 turing kernel: [cache_grow+168/352] cache_grow+0xa8/0x160
Dec 20 13:17:21 turing kernel: [<c0146578>] cache_grow+0xa8/0x160
Dec 20 13:17:21 turing kernel: [cache_alloc_refill+354/560]
cache_alloc_refill+0x162/0x230
Dec 20 13:17:21 turing kernel: [<c0146792>] cache_alloc_refill+0x162/0x230
Dec 20 13:17:21 turing kernel: [kmem_cache_alloc+52/64] kmem_cache_alloc+0x34/0x40
Dec 20 13:17:22 turing kernel: [<c0146a54>] kmem_cache_alloc+0x34/0x40
Dec 20 13:17:22 turing kernel: [__scsi_get_command+21/96] __scsi_get_command+0x15/0x60
Dec 20 13:17:22 turing kernel: [<c02eba35>] __scsi_get_command+0x15/0x60
Dec 20 13:17:22 turing kernel: [scsi_get_command+15/144] scsi_get_command+0xf/0x90
Dec 20 13:17:22 turing kernel: [<c02eba8f>] scsi_get_command+0xf/0x90
Dec 20 13:17:22 turing kernel: [scsi_prep_fn+227/448] scsi_prep_fn+0xe3/0x1c0
Dec 20 13:17:22 turing kernel: [<c02f0f93>] scsi_prep_fn+0xe3/0x1c0
Dec 20 13:17:22 turing kernel: [elv_next_request+85/224] elv_next_request+0x55/0xe0
Dec 20 13:17:22 turing kernel: [<c029a795>] elv_next_request+0x55/0xe0
Dec 20 13:17:22 turing kernel: [scsi_request_fn+501/960] scsi_request_fn+0x1f5/0x3c0
Dec 20 13:17:22 turing kernel: [<c02f1265>] scsi_request_fn+0x1f5/0x3c0
Dec 20 13:17:22 turing kernel: [__generic_unplug_device+46/48]
__generic_unplug_device+0x2e/0x30
Dec 20 13:17:22 turing kernel: [<c029c29e>] __generic_unplug_device+0x2e/0x30
Dec 20 13:17:22 turing kernel: [generic_unplug_device+21/48]
generic_unplug_device+0x15/0x30
Dec 20 13:17:22 turing kernel: [<c029c2b5>] generic_unplug_device+0x15/0x30
Dec 20 13:17:22 turing kernel: [blk_unplug_work+6/16] blk_unplug_work+0x6/0x10
Dec 20 13:17:22 turing kernel: [<c029c2f6>] blk_unplug_work+0x6/0x10
Dec 20 13:17:22 turing kernel: [worker_thread+424/560] worker_thread+0x1a8/0x230
Dec 20 13:17:22 turing kernel: [<c0130048>] worker_thread+0x1a8/0x230
Dec 20 13:17:22 turing kernel: [blk_unplug_work+0/16] blk_unplug_work+0x0/0x10
Dec 20 13:17:22 turing kernel: [<c029c2f0>] blk_unplug_work+0x0/0x10
Dec 20 13:17:22 turing kernel: [default_wake_function+0/16]
default_wake_function+0x0/0x10
Dec 20 13:17:22 turing kernel: [<c011e470>] default_wake_function+0x0/0x10
Dec 20 13:17:22 turing kernel: [default_wake_function+0/16]
default_wake_function+0x0/0x10
Dec 20 13:17:22 turing kernel: [<c011e470>] default_wake_function+0x0/0x10
Dec 20 13:17:22 turing kernel: [worker_thread+0/560] worker_thread+0x0/0x230
Dec 20 13:17:22 turing kernel: [<c012fea0>] worker_thread+0x0/0x230
Dec 20 13:17:22 turing kernel: [kthread+134/176] kthread+0x86/0xb0
Dec 20 13:17:22 turing kernel: [<c0133b46>] kthread+0x86/0xb0
Dec 20 13:17:22 turing kernel: [kthread+0/176] kthread+0x0/0xb0
Dec 20 13:17:22 turing kernel: [<c0133ac0>] kthread+0x0/0xb0
Dec 20 13:17:22 turing kernel: [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
Dec 20 13:17:22 turing kernel: [<c0105275>] kernel_thread_helper+0x5/0x10
Dec 20 13:17:22 turing kernel: klogd: page allocation failure. order:0, mode:0x21
Dec 20 13:17:22 turing kernel: [__alloc_pages+824/832] __alloc_pages+0x338/0x340
Dec 20 13:17:22 turing kernel: [<c0142648>] __alloc_pages+0x338/0x340
Dec 20 13:17:22 turing kernel: [__get_free_pages+24/48] __get_free_pages+0x18/0x30
Dec 20 13:17:22 turing kernel: [<c0142668>] __get_free_pages+0x18/0x30
Dec 20 13:17:22 turing kernel: [kmem_getpages+36/224] kmem_getpages+0x24/0xe0
Dec 20 13:17:22 turing kernel: [<c0145884>] kmem_getpages+0x24/0xe0
Dec 20 13:17:22 turing kernel: [cache_grow+168/352] cache_grow+0xa8/0x160
Dec 20 13:17:22 turing kernel: [<c0146578>] cache_grow+0xa8/0x160
Dec 20 13:17:22 turing kernel: [cache_alloc_refill+354/560]
cache_alloc_refill+0x162/0x230
Dec 20 13:17:22 turing kernel: [<c0146792>] cache_alloc_refill+0x162/0x230
Dec 20 13:17:22 turing kernel: [kmem_cache_alloc+52/64] kmem_cache_alloc+0x34/0x40
Dec 20 13:17:22 turing kernel: [<c0146a54>] kmem_cache_alloc+0x34/0x40
Dec 20 13:17:22 turing kernel: [__scsi_get_command+21/96] __scsi_get_command+0x15/0x60
Dec 20 13:17:22 turing kernel: [<c02eba35>] __scsi_get_command+0x15/0x60
Dec 20 13:17:22 turing kernel: [scsi_get_command+15/144] scsi_get_command+0xf/0x90
Dec 20 13:17:22 turing kernel: [<c02eba8f>] scsi_get_command+0xf/0x90
Dec 20 13:17:22 turing kernel: [scsi_prep_fn+227/448] scsi_prep_fn+0xe3/0x1c0
Dec 20 13:17:22 turing kernel: [<c02f0f93>] scsi_prep_fn+0xe3/0x1c0
Dec 20 13:17:22 turing kernel: [elv_next_request+85/224] elv_next_request+0x55/0xe0
Dec 20 13:17:22 turing kernel: [<c029a795>] elv_next_request+0x55/0xe0
Dec 20 13:17:22 turing kernel: [__generic_unplug_device+34/48]
__generic_unplug_device+0x22/0x30
Dec 20 13:17:22 turing kernel: [<c029c292>] __generic_unplug_device+0x22/0x30
Dec 20 13:17:22 turing kernel: [generic_unplug_device+21/48]
generic_unplug_device+0x15/0x30
Dec 20 13:17:22 turing kernel: [<c029c2b5>] generic_unplug_device+0x15/0x30
Dec 20 13:17:22 turing kernel: [blk_backing_dev_unplug+18/32]
blk_backing_dev_unplug+0x12/0x20
Dec 20 13:17:22 turing kernel: [<c029c2e2>] blk_backing_dev_unplug+0x12/0x20
Dec 20 13:17:22 turing kernel: [swap_unplug_io_fn+90/144] swap_unplug_io_fn+0x5a/0x90
Dec 20 13:17:22 turing kernel: [<c015347a>] swap_unplug_io_fn+0x5a/0x90
Dec 20 13:17:22 turing kernel: [swap_unplug_io_fn+0/144] swap_unplug_io_fn+0x0/0x90
Dec 20 13:17:22 turing kernel: [<c0153420>] swap_unplug_io_fn+0x0/0x90
Dec 20 13:17:22 turing kernel: [block_sync_page+68/80] block_sync_page+0x44/0x50
Dec 20 13:17:22 turing kernel: [<c015f284>] block_sync_page+0x44/0x50
Dec 20 13:17:22 turing kernel: [__lock_page+235/256] __lock_page+0xeb/0x100
Dec 20 13:17:22 turing kernel: [<c013e42b>] __lock_page+0xeb/0x100
Dec 20 13:17:22 turing kernel: [page_wake_function+0/64] page_wake_function+0x0/0x40
Dec 20 13:17:22 turing kernel: [<c013e130>] page_wake_function+0x0/0x40
Dec 20 13:17:22 turing kernel: [page_wake_function+0/64] page_wake_function+0x0/0x40
Dec 20 13:17:22 turing kernel: [<c013e130>] page_wake_function+0x0/0x40
Dec 20 13:17:22 turing kernel: [grab_swap_token+158/176] grab_swap_token+0x9e/0xb0
Dec 20 13:17:22 turing kernel: [<c015584e>] grab_swap_token+0x9e/0xb0
Dec 20 13:17:22 turing kernel: [do_swap_page+327/656] do_swap_page+0x147/0x290
Dec 20 13:17:22 turing kernel: [<c014d077>] do_swap_page+0x147/0x290
Dec 20 13:17:22 turing kernel: [handle_mm_fault+244/336] handle_mm_fault+0xf4/0x150
Dec 20 13:17:22 turing kernel: [<c014d7b4>] handle_mm_fault+0xf4/0x150
Dec 20 13:17:22 turing kernel: [do_page_fault+443/1400] do_page_fault+0x1bb/0x578
Dec 20 13:17:22 turing kernel: [<c011af4b>] do_page_fault+0x1bb/0x578
Dec 20 13:17:22 turing kernel: [avc_has_perm_noaudit+188/416]
avc_has_perm_noaudit+0xbc/0x1a0
Dec 20 13:17:22 turing kernel: [<c020fb2c>] avc_has_perm_noaudit+0xbc/0x1a0
Dec 20 13:17:22 turing kernel: [recalc_task_prio+330/480] recalc_task_prio+0x14a/0x1e0
Dec 20 13:17:22 turing kernel: [<c011c5ea>] recalc_task_prio+0x14a/0x1e0
Dec 20 13:17:22 turing kernel: [finish_task_switch+50/112] finish_task_switch+0x32/0x70
Dec 20 13:17:22 turing kernel: [<c011cfa2>] finish_task_switch+0x32/0x70
Dec 20 13:17:22 turing kernel: [schedule+848/2832] schedule+0x350/0xb10
Dec 20 13:17:22 turing kernel: [<c03908b0>] schedule+0x350/0xb10
Dec 20 13:17:22 turing kernel: [do_page_fault+0/1400] do_page_fault+0x0/0x578
Dec 20 13:17:22 turing kernel: [<c011ad90>] do_page_fault+0x0/0x578
Dec 20 13:17:22 turing kernel: [error_code+45/64] error_code+0x2d/0x40
Dec 20 13:17:22 turing kernel: [<c0107f5d>] error_code+0x2d/0x40
Dec 20 13:17:22 turing kernel: [do_syslog+779/992] do_syslog+0x30b/0x3e0
Dec 20 13:17:22 turing kernel: [<c012197b>] do_syslog+0x30b/0x3e0
Dec 20 13:17:22 turing kernel: [autoremove_wake_function+0/48]
autoremove_wake_function+0x0/0x30
Dec 20 13:17:22 turing kernel: [<c011f9e0>] autoremove_wake_function+0x0/0x30
Dec 20 13:17:22 turing kernel: [autoremove_wake_function+0/48]
autoremove_wake_function+0x0/0x30
Dec 20 13:17:22 turing kernel: [<c011f9e0>] autoremove_wake_function+0x0/0x30
Dec 20 13:17:22 turing kernel: [kmsg_read+0/64] kmsg_read+0x0/0x40
Dec 20 13:17:22 turing kernel: [<c018a620>] kmsg_read+0x0/0x40
Dec 20 13:17:22 turing kernel: [vfs_read+174/240] vfs_read+0xae/0xf0
Dec 20 13:17:22 turing kernel: [<c015aa8e>] vfs_read+0xae/0xf0
Dec 20 13:17:22 turing kernel: [sys_read+60/112] sys_read+0x3c/0x70
Dec 20 13:17:22 turing kernel: [<c015acec>] sys_read+0x3c/0x70
Dec 20 13:17:22 turing kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Dec 20 13:17:22 turing kernel: [<c0106e27>] syscall_call+0x7/0xb
Dec 20 13:17:22 turing kernel: klogd: page allocation failure. order:0, mode:0x21
Dec 20 13:17:22 turing kernel: [__alloc_pages+824/832] __alloc_pages+0x338/0x340
Dec 20 13:17:22 turing kernel: [<c0142648>] __alloc_pages+0x338/0x340
Dec 20 13:17:22 turing kernel: [__get_free_pages+24/48] __get_free_pages+0x18/0x30
Dec 20 13:17:22 turing kernel: [<c0142668>] __get_free_pages+0x18/0x30
Dec 20 13:17:22 turing kernel: [kmem_getpages+36/224] kmem_getpages+0x24/0xe0
Dec 20 13:17:22 turing kernel: [<c0145884>] kmem_getpages+0x24/0xe0
Dec 20 13:17:22 turing kernel: [cache_grow+168/352] cache_grow+0xa8/0x160
Dec 20 13:17:22 turing kernel: [<c0146578>] cache_grow+0xa8/0x160
Dec 20 13:17:22 turing kernel: [cache_alloc_refill+354/560]
cache_alloc_refill+0x162/0x230
Dec 20 13:17:22 turing kernel: [<c0146792>] cache_alloc_refill+0x162/0x230
Dec 20 13:17:22 turing kernel: [kmem_cache_alloc+52/64] kmem_cache_alloc+0x34/0x40
Dec 20 13:17:22 turing kernel: [<c0146a54>] kmem_cache_alloc+0x34/0x40
Dec 20 13:17:22 turing kernel: [__scsi_get_command+21/96] __scsi_get_command+0x15/0x60
Dec 20 13:17:22 turing kernel: [<c02eba35>] __scsi_get_command+0x15/0x60
Dec 20 13:17:22 turing kernel: [scsi_get_command+15/144] scsi_get_command+0xf/0x90
Dec 20 13:17:22 turing kernel: [<c02eba8f>] scsi_get_command+0xf/0x90
Dec 20 13:17:22 turing kernel: [scsi_prep_fn+227/448] scsi_prep_fn+0xe3/0x1c0
Dec 20 13:17:22 turing kernel: [<c02f0f93>] scsi_prep_fn+0xe3/0x1c0
Dec 20 13:17:22 turing kernel: [elv_next_request+85/224] elv_next_request+0x55/0xe0
Dec 20 13:17:22 turing kernel: [<c029a795>] elv_next_request+0x55/0xe0
Dec 20 13:17:22 turing kernel: [__generic_unplug_device+34/48]
__generic_unplug_device+0x22/0x30
Dec 20 13:17:22 turing kernel: [<c029c292>] __generic_unplug_device+0x22/0x30
Dec 20 13:17:22 turing kernel: [generic_unplug_device+21/48]
generic_unplug_device+0x15/0x30
Dec 20 13:17:22 turing kernel: [<c029c2b5>] generic_unplug_device+0x15/0x30
Dec 20 13:17:22 turing kernel: [blk_backing_dev_unplug+0/32]
blk_backing_dev_unplug+0x0/0x20
Dec 20 13:17:22 turing kernel: [<c029c2d0>] blk_backing_dev_unplug+0x0/0x20
Dec 20 13:17:22 turing kernel: [blk_backing_dev_unplug+18/32]
blk_backing_dev_unplug+0x12/0x20
Dec 20 13:17:22 turing kernel: [<c029c2e2>] blk_backing_dev_unplug+0x12/0x20
Dec 20 13:17:22 turing kernel: [block_sync_page+68/80] block_sync_page+0x44/0x50
Dec 20 13:17:22 turing kernel: [<c015f284>] block_sync_page+0x44/0x50
Dec 20 13:17:22 turing kernel: [__lock_page+235/256] __lock_page+0xeb/0x100
Dec 20 13:17:22 turing kernel: [<c013e42b>] __lock_page+0xeb/0x100
Dec 20 13:17:22 turing kernel: [page_wake_function+0/64] page_wake_function+0x0/0x40
Dec 20 13:17:22 turing kernel: [<c013e130>] page_wake_function+0x0/0x40
Dec 20 13:17:22 turing kernel: [page_wake_function+0/64] page_wake_function+0x0/0x40
Dec 20 13:17:22 turing kernel: [<c013e130>] page_wake_function+0x0/0x40
Dec 20 13:17:22 turing kernel: [find_get_page+57/80] find_get_page+0x39/0x50
Dec 20 13:17:22 turing kernel: [<c013e479>] find_get_page+0x39/0x50
Dec 20 13:17:22 turing kernel: [filemap_nopage+697/864] filemap_nopage+0x2b9/0x360
Dec 20 13:17:22 turing kernel: [<c013f489>] filemap_nopage+0x2b9/0x360
Dec 20 13:17:22 turing kernel: [do_no_page+180/688] do_no_page+0xb4/0x2b0
Dec 20 13:17:22 turing kernel: [<c014d3f4>] do_no_page+0xb4/0x2b0
Dec 20 13:17:22 turing kernel: [handle_mm_fault+272/336] handle_mm_fault+0x110/0x150
Dec 20 13:17:22 turing kernel: [<c014d7d0>] handle_mm_fault+0x110/0x150
Dec 20 13:17:22 turing kernel: [do_page_fault+443/1400] do_page_fault+0x1bb/0x578
Dec 20 13:17:22 turing kernel: [<c011af4b>] do_page_fault+0x1bb/0x578
Dec 20 13:17:22 turing kernel: [do_syslog+722/992] do_syslog+0x2d2/0x3e0
Dec 20 13:17:22 turing kernel: [<c0121942>] do_syslog+0x2d2/0x3e0
Dec 20 13:17:22 turing kernel: [autoremove_wake_function+0/48]
autoremove_wake_function+0x0/0x30
Dec 20 13:17:22 turing kernel: [<c011f9e0>] autoremove_wake_function+0x0/0x30
Dec 20 13:17:22 turing kernel: [autoremove_wake_function+0/48]
autoremove_wake_function+0x0/0x30
Dec 20 13:17:22 turing kernel: [<c011f9e0>] autoremove_wake_function+0x0/0x30
Dec 20 13:17:22 turing kernel: [kmsg_read+0/64] kmsg_read+0x0/0x40
Dec 20 13:17:22 turing kernel: [<c018a620>] kmsg_read+0x0/0x40
Dec 20 13:17:22 turing kernel: [dnotify_parent+31/112] dnotify_parent+0x1f/0x70
Dec 20 13:17:22 turing kernel: [<c017463f>] dnotify_parent+0x1f/0x70
Dec 20 13:17:22 turing kernel: [vfs_read+194/240] vfs_read+0xc2/0xf0
Dec 20 13:17:22 turing kernel: [<c015aaa2>] vfs_read+0xc2/0xf0
Dec 20 13:17:22 turing kernel: [sys_read+60/112] sys_read+0x3c/0x70
Dec 20 13:17:22 turing kernel: [<c015acec>] sys_read+0x3c/0x70
Dec 20 13:17:22 turing kernel: [do_page_fault+0/1400] do_page_fault+0x0/0x578
Dec 20 13:17:22 turing kernel: [<c011ad90>] do_page_fault+0x0/0x578
Dec 20 13:17:22 turing kernel: [error_code+45/64] error_code+0x2d/0x40
Dec 20 13:17:22 turing kernel: [<c0107f5d>] error_code+0x2d/0x40
--
Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17 Phone: +49 89 2180-4049
80333 Muenchen, Germany Fax: +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/