Re: [PATCH] vmscan: raise the bar to PAGEOUT_IO_SYNC stalls

From: Wu Fengguang
Date: Sat Jul 31 2010 - 13:10:02 EST

Next message: Christoph Hellwig: "Re: [PATCH] vmscan: raise the bar to PAGEOUT_IO_SYNC stalls"
Previous message: David Howells: "Re: [PATCH 02/18] xstat: Add a pair of system calls to make extended file stats available [ver #6]"
In reply to: Mel Gorman: "Re: [PATCH] vmscan: raise the bar to PAGEOUT_IO_SYNC stalls"
Next in thread: Christoph Hellwig: "Re: [PATCH] vmscan: raise the bar to PAGEOUT_IO_SYNC stalls"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Wed, Jul 28, 2010 at 03:17:05PM +0800, Wu Fengguang wrote:
> Fix "system goes unresponsive under memory pressure and lots of
> dirty/writeback pages" bug.
>
> http://lkml.org/lkml/2010/4/4/86
>
> In the above thread, Andreas Mohr described that
>
> Invoking any command locked up for minutes (note that I'm
> talking about attempted additional I/O to the _other_,
> _unaffected_ main system HDD - such as loading some shell
> binaries -, NOT the external SSD18M!!).
>
> This happens when the two conditions are both meet:
> - under memory pressure
> - writing heavily to a slow device
>
> OOM also happens in Andreas' system. The OOM trace shows that 3
> processes are stuck in wait_on_page_writeback() in the direct reclaim
> path. One in do_fork() and the other two in unix_stream_sendmsg(). They
> are blocked on this condition:
>
> (sc->order && priority < DEF_PRIORITY - 2)
>
> which was introduced in commit 78dc583d (vmscan: low order lumpy reclaim
> also should use PAGEOUT_IO_SYNC) one year ago. That condition may be too
> permissive. In Andreas' case, 512MB/1024 = 512KB. If the direct reclaim
> for the order-1 fork() allocation runs into a range of 512KB
> hard-to-reclaim LRU pages, it will be stalled.
>
> It's a severe problem in three ways.

FYI I did some memory stress test and find there are much more order-1
(and higher) users than fork(). This means lots of running applications
may stall on direct reclaim.

Basically all of these slab caches will do high order allocations:

$ grep -v '1 :' /proc/slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
nf_conntrack_expect 0 0 352 23 2 : tunables 0 0 0 : slabdata 0 0 0
nf_conntrack_ffffffff826e7340 57 102 472 17 2 : tunables 0 0 0 : slabdata 6 6 0
ip6_dst_cache 52 72 448 18 2 : tunables 0 0 0 : slabdata 4 4 0
ndisc_cache 32 32 512 16 2 : tunables 0 0 0 : slabdata 2 2 0
RAWv6 20 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0
UDPLITEv6 0 0 1600 20 8 : tunables 0 0 0 : slabdata 0 0 0
UDPv6 40 40 1600 20 8 : tunables 0 0 0 : slabdata 2 2 0
TCPv6 24 24 2688 12 8 : tunables 0 0 0 : slabdata 2 2 0
btrfs_transaction_cache 13 13 592 13 2 : tunables 0 0 0 : slabdata 1 1 0
btrfs_inode_cache 29 42 2256 14 8 : tunables 0 0 0 : slabdata 3 3 0
fat_inode_cache 0 0 1376 23 8 : tunables 0 0 0 : slabdata 0 0 0
reiser_inode_cache 276868 278300 1592 20 8 : tunables 0 0 0 : slabdata 13915 13915 0
kvm_vcpu 0 0 10608 3 8 : tunables 0 0 0 : slabdata 0 0 0
mqueue_inode_cache 19 19 1664 19 8 : tunables 0 0 0 : slabdata 1 1 0
fuse_request 0 0 760 21 4 : tunables 0 0 0 : slabdata 0 0 0
fuse_inode 0 0 1472 22 8 : tunables 0 0 0 : slabdata 0 0 0
nfs_write_data 38 38 832 19 4 : tunables 0 0 0 : slabdata 2 2 0
nfs_read_data 42 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0
nfs_inode_cache 0 0 1648 19 8 : tunables 0 0 0 : slabdata 0 0 0
hugetlbfs_inode_cache 12 12 1312 12 4 : tunables 0 0 0 : slabdata 1 1 0
ext4_inode_cache 68415 68748 1912 17 8 : tunables 0 0 0 : slabdata 4044 4044 0
ext3_inode_cache 232870 237481 1664 19 8 : tunables 0 0 0 : slabdata 12499 12499 0
kioctx 0 0 704 23 4 : tunables 0 0 0 : slabdata 0 0 0
rpc_buffers 15 15 2176 15 8 : tunables 0 0 0 : slabdata 1 1 0
rpc_tasks 21 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0
rpc_inode_cache 20 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0
UNIX 205 240 1600 20 8 : tunables 0 0 0 : slabdata 12 12 0
UDP-Lite 0 0 1472 22 8 : tunables 0 0 0 : slabdata 0 0 0
xfrm_dst_cache 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0
ip_dst_cache 75 90 448 18 2 : tunables 0 0 0 : slabdata 5 5 0
arp_cache 32 32 512 16 2 : tunables 0 0 0 : slabdata 2 2 0
RAW 46 46 1408 23 8 : tunables 0 0 0 : slabdata 2 2 0
UDP 49 66 1472 22 8 : tunables 0 0 0 : slabdata 3 3 0
TCP 37 60 2560 12 8 : tunables 0 0 0 : slabdata 5 5 0
sgpool-128 16 21 4224 7 8 : tunables 0 0 0 : slabdata 3 3 0
sgpool-64 30 30 2176 15 8 : tunables 0 0 0 : slabdata 2 2 0
sgpool-32 28 28 1152 14 4 : tunables 0 0 0 : slabdata 2 2 0
sgpool-16 24 24 640 12 2 : tunables 0 0 0 : slabdata 2 2 0
sgpool-8 44 63 384 21 2 : tunables 0 0 0 : slabdata 3 3 0
blkdev_queue 30 30 2984 10 8 : tunables 0 0 0 : slabdata 3 3 0
blkdev_requests 48 60 408 20 2 : tunables 0 0 0 : slabdata 3 3 0
biovec-256 7 7 4224 7 8 : tunables 0 0 0 : slabdata 1 1 0
biovec-128 30 30 2176 15 8 : tunables 0 0 0 : slabdata 2 2 0
biovec-64 28 28 1152 14 4 : tunables 0 0 0 : slabdata 2 2 0
biovec-16 42 42 384 21 2 : tunables 0 0 0 : slabdata 2 2 0
bip-256 7 7 4288 7 8 : tunables 0 0 0 : slabdata 1 1 0
bip-128 0 0 2240 14 8 : tunables 0 0 0 : slabdata 0 0 0
bip-64 0 0 1216 13 4 : tunables 0 0 0 : slabdata 0 0 0
bip-16 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0
sock_inode_cache 283 322 1408 23 8 : tunables 0 0 0 : slabdata 14 14 0
skbuff_fclone_cache 28 28 576 14 2 : tunables 0 0 0 : slabdata 2 2 0
shmem_inode_cache 2632 2646 1560 21 8 : tunables 0 0 0 : slabdata 126 126 0
taskstats 40 40 400 20 2 : tunables 0 0 0 : slabdata 2 2 0
proc_inode_cache 848 888 1288 12 4 : tunables 0 0 0 : slabdata 74 74 0
bdev_cache 54 54 1728 18 8 : tunables 0 0 0 : slabdata 3 3 0
filp 3704 3948 384 21 2 : tunables 0 0 0 : slabdata 188 188 0
inode_cache 4546 4836 1240 13 4 : tunables 0 0 0 : slabdata 372 372 0
names_cache 14 14 4224 7 8 : tunables 0 0 0 : slabdata 2 2 0
radix_tree_node 37173 37817 624 13 2 : tunables 0 0 0 : slabdata 2909 2909 0
mm_struct 125 144 1280 12 4 : tunables 0 0 0 : slabdata 12 12 0
files_cache 129 144 896 18 4 : tunables 0 0 0 : slabdata 8 8 0
signal_cache 227 247 1216 13 4 : tunables 0 0 0 : slabdata 19 19 0
sighand_cache 227 238 2304 14 8 : tunables 0 0 0 : slabdata 17 17 0
task_xstate 210 228 640 12 2 : tunables 0 0 0 : slabdata 19 19 0
task_struct 310 318 9104 3 8 : tunables 0 0 0 : slabdata 106 106 0
idr_layer_cache 336 338 616 13 2 : tunables 0 0 0 : slabdata 26 26 0
kmalloc-8192 41 42 8264 3 8 : tunables 0 0 0 : slabdata 14 14 0
kmalloc-4096 546 560 4168 7 8 : tunables 0 0 0 : slabdata 80 80 0
kmalloc-2048 451 495 2120 15 8 : tunables 0 0 0 : slabdata 33 33 0
kmalloc-1024 928 986 1096 29 8 : tunables 0 0 0 : slabdata 34 34 0
kmalloc-512 5011 5012 584 14 2 : tunables 0 0 0 : slabdata 358 358 0

In particular the radix_tree_node slab cache turns up a lot as in the
below trace. It happens when the process tries to swap in a page (this
happens a lot!) and need to allocate a new radix tree node for the
swapper_space.

Imagine the user want to switch to an application which involves many
swap-in reads, and many of them will stall in wait_on_page_writeback()
for seconds waiting for some unrelated dirty page to be synced!

[ 8603.802898] usemem D 00000001001f68a1 2224 3920 1 0x00000004
[ 8603.803028] ffff8800aac3b5c8 0000000000000006 00000000ffffffff 00000000001d55c0
[ 8603.803246] ffff8800b4d1a350 00000000001d55c0 ffff8800aac3bfd8 ffff8800aac3bfd8
[ 8603.803464] 00000000001d55c0 ffff8800b4d1a6b8 00000000001d55c0 00000000001d55c0
[ 8603.803681] Call Trace:
[ 8603.803747] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.803820] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.803896] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.803966] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.804037] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.804108] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.804179] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.804250] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.804322] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.804393] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.804465] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.804536] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.804608] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.804679] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.804751] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.804824] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.804895] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.804965] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.805036] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.805108] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.805179] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.805250] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.805322] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.805393] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.805465] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.805536] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.805606] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.805677] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.805749] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.805820] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.805889] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.805962] [<ffffffff81b83b45>] page_fault+0x25/0x30

Thanks,
Fengguang

The more sysrq-t output after `shutdown`. The system is so
stressed that it cannot be shutdown at all..

[ 8603.783334] cp D 00000001001fb070 1832 3528 3513 0x00000004
[ 8603.783464] ffff88009f505a08 0000000000000006 0000000000000000 00000000001d55c0
[ 8603.783681] ffff8800b9a3a350 00000000001d55c0 ffff88009f505fd8 ffff88009f505fd8
[ 8603.783898] 00000000001d55c0 ffff8800b9a3a6b8 00000000001d55c0 00000000001d55c0
[ 8603.784117] Call Trace:
[ 8603.784183] [<ffffffff8123f890>] ? inode_wait+0x0/0x20
[ 8603.784253] [<ffffffff8123f8a5>] inode_wait+0x15/0x20
[ 8603.784323] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.784395] [<ffffffff81251279>] inode_wait_for_writeback+0xb9/0xf0
[ 8603.784467] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.784538] [<ffffffff8115992c>] ? check_for_new_grace_period+0x1ac/0x1e0
[ 8603.784612] [<ffffffff8125133f>] writeback_single_inode+0x8f/0x3e0
[ 8603.784684] [<ffffffff812516d1>] sync_inode+0x41/0x70
[ 8603.784753] [<ffffffff81384d2a>] nfs_wb_all+0x4a/0x60
[ 8603.784824] [<ffffffff81370108>] nfs_do_fsync+0x38/0x90
[ 8603.784894] [<ffffffff813703fa>] nfs_file_flush+0x8a/0xd0
[ 8603.784965] [<ffffffff8121d164>] filp_close+0x54/0xd0
[ 8603.785036] [<ffffffff810b9881>] put_files_struct+0x111/0x210
[ 8603.785106] [<ffffffff810b97b0>] ? put_files_struct+0x40/0x210
[ 8603.785178] [<ffffffff810b9a7e>] exit_files+0x6e/0x90
[ 8603.785248] [<ffffffff810ba1b6>] do_exit+0x1e6/0xc40
[ 8603.785319] [<ffffffff810fee3b>] ? trace_hardirqs_off+0x1b/0x30
[ 8603.785390] [<ffffffff81b8342c>] ? _raw_spin_unlock_irq+0x4c/0x70
[ 8603.785461] [<ffffffff810baf7d>] do_group_exit+0x6d/0x130
[ 8603.785533] [<ffffffff810d1e9e>] get_signal_to_deliver+0x3de/0x6a0
[ 8603.785606] [<ffffffff8104c843>] do_signal+0x83/0xae0
[ 8603.785676] [<ffffffff8126d1fe>] ? fsnotify+0x35e/0x3d0
[ 8603.785747] [<ffffffff811cdb46>] ? might_fault+0xd6/0xf0
[ 8603.785818] [<ffffffff8104d35c>] do_notify_resume+0x8c/0xc0
[ 8603.785889] [<ffffffff81b82017>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 8603.785961] [<ffffffff8104dec3>] int_signal+0x12/0x17
[ 8603.786030] flush-0:23 D 00000001001f6fae 2440 3529 2 0x00000000
[ 8603.786160] ffff88003deeba50 0000000000000006 ffff880000000000 00000000001d55c0
[ 8603.786379] ffff8800b9a3c6a0 00000000001d55c0 ffff88003deebfd8 ffff88003deebfd8
[ 8603.786597] 00000000001d55c0 ffff8800b9a3ca08 00000000001d55c0 00000000001d55c0
[ 8603.786814] Call Trace:
[ 8603.786881] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.786951] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.787020] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.787090] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.787160] [<ffffffff811a565d>] ? find_get_pages_tag+0x16d/0x280
[ 8603.787231] [<ffffffff811a54f0>] ? find_get_pages_tag+0x0/0x280
[ 8603.787302] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.787373] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.787444] [<ffffffff811b5b3c>] ? pagevec_lookup_tag+0x2c/0x40
[ 8603.787515] [<ffffffff811a756e>] filemap_fdatawait_range+0x1de/0x250
[ 8603.787587] [<ffffffff811a7611>] filemap_fdatawait+0x31/0x40
[ 8603.787658] [<ffffffff8125158f>] writeback_single_inode+0x2df/0x3e0
[ 8603.787730] [<ffffffff81251cc6>] generic_writeback_sb_inodes+0x106/0x230
[ 8603.787802] [<ffffffff81251e2f>] do_writeback_sb_inodes+0x3f/0x50
[ 8603.787874] [<ffffffff81252ca2>] wb_writeback+0x172/0x510
[ 8603.787944] [<ffffffff810fee3b>] ? trace_hardirqs_off+0x1b/0x30
[ 8603.788015] [<ffffffff812531ae>] wb_do_writeback+0xce/0x290
[ 8603.788086] [<ffffffff8125351a>] bdi_writeback_thread+0x1aa/0x3e0
[ 8603.788157] [<ffffffff81253370>] ? bdi_writeback_thread+0x0/0x3e0
[ 8603.788229] [<ffffffff810e3dad>] kthread+0xcd/0xe0
[ 8603.788299] [<ffffffff8104e9e4>] kernel_thread_helper+0x4/0x10
[ 8603.788370] [<ffffffff81b83910>] ? restore_args+0x0/0x30
[ 8603.788440] [<ffffffff810e3ce0>] ? kthread+0x0/0xe0
[ 8603.788510] [<ffffffff8104e9e0>] ? kernel_thread_helper+0x0/0x10
[ 8603.788580] mem-stress-te S 000000010015b965 4056 3605 3202 0x00000000
[ 8603.788710] ffff880067ccde78 0000000000000002 0000000000000000 00000000001d55c0
[ 8603.788928] ffff8800b9b446a0 00000000001d55c0 ffff880067ccdfd8 ffff880067ccdfd8
[ 8603.789146] 00000000001d55c0 ffff8800b9b44a08 00000000001d55c0 00000000001d55c0
[ 8603.789364] Call Trace:
[ 8603.789429] [<ffffffff810b9475>] do_wait+0x245/0x340
[ 8603.789500] [<ffffffff810bb436>] sys_wait4+0x86/0x150
[ 8603.789570] [<ffffffff810b7860>] ? child_wait_callback+0x0/0xb0
[ 8603.789641] [<ffffffff8104db72>] system_call_fastpath+0x16/0x1b
[ 8603.789712] cp D 00000001001fb07b 1904 3612 3605 0x00000004
[ 8603.789843] ffff880008c41a08 0000000000000006 0000000000000000 00000000001d55c0
[ 8603.790061] ffff8800b4d40000 00000000001d55c0 ffff880008c41fd8 ffff880008c41fd8
[ 8603.790280] 00000000001d55c0 ffff8800b4d40368 00000000001d55c0 00000000001d55c0
[ 8603.790498] Call Trace:
[ 8603.790565] [<ffffffff8123f890>] ? inode_wait+0x0/0x20
[ 8603.790636] [<ffffffff8123f8a5>] inode_wait+0x15/0x20
[ 8603.790706] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.790777] [<ffffffff81251279>] inode_wait_for_writeback+0xb9/0xf0
[ 8603.790849] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.790921] [<ffffffff8115992c>] ? check_for_new_grace_period+0x1ac/0x1e0
[ 8603.790994] [<ffffffff8125133f>] writeback_single_inode+0x8f/0x3e0
[ 8603.791065] [<ffffffff812516d1>] sync_inode+0x41/0x70
[ 8603.791135] [<ffffffff81384d2a>] nfs_wb_all+0x4a/0x60
[ 8603.791206] [<ffffffff81370108>] nfs_do_fsync+0x38/0x90
[ 8603.791276] [<ffffffff813703fa>] nfs_file_flush+0x8a/0xd0
[ 8603.791346] [<ffffffff8121d164>] filp_close+0x54/0xd0
[ 8603.791988] [<ffffffff810b9881>] put_files_struct+0x111/0x210
[ 8603.792059] [<ffffffff810b97b0>] ? put_files_struct+0x40/0x210
[ 8603.792130] [<ffffffff810b9a7e>] exit_files+0x6e/0x90
[ 8603.792200] [<ffffffff810ba1b6>] do_exit+0x1e6/0xc40
[ 8603.792269] [<ffffffff810fee3b>] ? trace_hardirqs_off+0x1b/0x30
[ 8603.792340] [<ffffffff81b8342c>] ? _raw_spin_unlock_irq+0x4c/0x70
[ 8603.792412] [<ffffffff810baf7d>] do_group_exit+0x6d/0x130
[ 8603.792482] [<ffffffff810d1e9e>] get_signal_to_deliver+0x3de/0x6a0
[ 8603.792555] [<ffffffff8104c843>] do_signal+0x83/0xae0
[ 8603.792625] [<ffffffff8126d1fe>] ? fsnotify+0x35e/0x3d0
[ 8603.792695] [<ffffffff8104d35c>] do_notify_resume+0x8c/0xc0
[ 8603.792766] [<ffffffff81b82017>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 8603.792837] [<ffffffff8104dec3>] int_signal+0x12/0x17
[ 8603.792907] su S 000000010012dd7d 4136 3857 3417 0x00000000
[ 8603.793037] ffff880099a71e78 0000000000000002 0000000000000000 00000000001d55c0
[ 8603.793254] ffff880073db2350 00000000001d55c0 ffff880099a71fd8 ffff880099a71fd8
[ 8603.793473] 00000000001d55c0 ffff880073db26b8 00000000001d55c0 00000000001d55c0
[ 8603.793690] Call Trace:
[ 8603.793755] [<ffffffff810b9475>] do_wait+0x245/0x340
[ 8603.793826] [<ffffffff810bb436>] sys_wait4+0x86/0x150
[ 8603.793895] [<ffffffff810b7860>] ? child_wait_callback+0x0/0xb0
[ 8603.793967] [<ffffffff8104db72>] system_call_fastpath+0x16/0x1b
[ 8603.794037] zsh S 0000000100196400 3760 3860 3857 0x00000000
[ 8603.794166] ffff880016f37f38 0000000000000006 0000000000000000 00000000001d55c0
[ 8603.794383] ffff8800a6180000 00000000001d55c0 ffff880016f37fd8 ffff880016f37fd8
[ 8603.794601] 00000000001d55c0 ffff8800a6180368 00000000001d55c0 00000000001d55c0
[ 8603.794820] Call Trace:
[ 8603.794886] [<ffffffff810d3cad>] sys_rt_sigsuspend+0xfd/0x150
[ 8603.794957] [<ffffffff8104db72>] system_call_fastpath+0x16/0x1b
[ 8603.795027] mem-stress-te S 00000001001fa536 4056 3907 3202 0x00000000
[ 8603.795157] ffff8800489b3e78 0000000000000006 0000000000000000 00000000001d55c0
[ 8603.795376] ffff880073db0000 00000000001d55c0 ffff8800489b3fd8 ffff8800489b3fd8
[ 8603.795593] 00000000001d55c0 ffff880073db0368 00000000001d55c0 00000000001d55c0
[ 8603.795811] Call Trace:
[ 8603.795876] [<ffffffff810b9475>] do_wait+0x245/0x340
[ 8603.795946] [<ffffffff810bb436>] sys_wait4+0x86/0x150
[ 8603.796017] [<ffffffff810b7860>] ? child_wait_callback+0x0/0xb0
[ 8603.796089] [<ffffffff8104db72>] system_call_fastpath+0x16/0x1b
[ 8603.796159] cp D 00000001001b3f13 2152 3917 3907 0x00000004
[ 8603.796289] ffff880033ae7338 0000000000000006 ffff880000000000 00000000001d55c0
[ 8603.796507] ffff880073db46a0 00000000001d55c0 ffff880033ae7fd8 ffff880033ae7fd8
[ 8603.796725] 00000000001d55c0 ffff880073db4a08 00000000001d55c0 00000000001d55c0
[ 8603.796943] Call Trace:
[ 8603.797010] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.797079] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.797150] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.797220] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.797291] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.797362] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.797433] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.797505] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.797576] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.797647] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.797719] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.797790] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.797862] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.797933] [<ffffffff81104f08>] ? __lock_acquire+0x618/0x2a20
[ 8603.798004] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.798076] [<ffffffff810ee46d>] ? local_clock+0x9d/0xb0
[ 8603.798147] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.798218] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.798288] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.798360] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.798431] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.798502] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.798573] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.798644] [<ffffffff811a650a>] add_to_page_cache_locked+0x8a/0x210
[ 8603.798716] [<ffffffff811a66d9>] add_to_page_cache_lru+0x49/0xc0
[ 8603.798787] [<ffffffff8126ac75>] mpage_readpages+0xd5/0x190
[ 8603.798859] [<ffffffff812f4960>] ? ext4_get_block+0x0/0x30
[ 8603.798929] [<ffffffff812f4960>] ? ext4_get_block+0x0/0x30
[ 8603.799000] [<ffffffff812eec04>] ext4_readpages+0x24/0x30
[ 8603.799071] [<ffffffff811b4f97>] __do_page_cache_readahead+0x1e7/0x300
[ 8603.799142] [<ffffffff811b4e7b>] ? __do_page_cache_readahead+0xcb/0x300
[ 8603.799215] [<ffffffff810fec5d>] ? trace_hardirqs_off_caller+0x2d/0x1f0
[ 8603.799287] [<ffffffff811b5568>] ra_submit+0x28/0x40
[ 8603.799357] [<ffffffff811b5657>] ondemand_readahead+0xd7/0x3b0
[ 8603.799429] [<ffffffff811b59ef>] page_cache_async_readahead+0xbf/0x140
[ 8603.799501] [<ffffffff811a5cf3>] ? find_get_page+0x113/0x1c0
[ 8603.799572] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.799643] [<ffffffff811a88fb>] generic_file_aio_read+0x67b/0xa50
[ 8603.799715] [<ffffffff810fee3b>] ? trace_hardirqs_off+0x1b/0x30
[ 8603.799787] [<ffffffff810ee46d>] ? local_clock+0x9d/0xb0
[ 8603.799858] [<ffffffff8121f700>] do_sync_read+0xf0/0x140
[ 8603.799929] [<ffffffff8126d1fe>] ? fsnotify+0x35e/0x3d0
[ 8603.799999] [<ffffffff812201d1>] vfs_read+0xc1/0x240
[ 8603.800069] [<ffffffff812203b6>] sys_read+0x66/0xb0
[ 8603.800139] [<ffffffff8104db72>] system_call_fastpath+0x16/0x1b
[ 8603.800209] cp D 00000001001fb07e 2184 3919 3907 0x00000004
[ 8603.800339] ffff88006b77fa08 0000000000000002 0000000000000000 00000000001d55c0
[ 8603.800557] ffff8800b988c6a0 00000000001d55c0 ffff88006b77ffd8 ffff88006b77ffd8
[ 8603.800776] 00000000001d55c0 ffff8800b988ca08 00000000001d55c0 00000000001d55c0
[ 8603.800993] Call Trace:
[ 8603.801059] [<ffffffff8123f890>] ? inode_wait+0x0/0x20
[ 8603.801129] [<ffffffff8123f8a5>] inode_wait+0x15/0x20
[ 8603.801199] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.801270] [<ffffffff81251279>] inode_wait_for_writeback+0xb9/0xf0
[ 8603.801342] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.801413] [<ffffffff8115992c>] ? check_for_new_grace_period+0x1ac/0x1e0
[ 8603.801486] [<ffffffff8125133f>] writeback_single_inode+0x8f/0x3e0
[ 8603.801557] [<ffffffff812516d1>] sync_inode+0x41/0x70
[ 8603.801627] [<ffffffff81384d2a>] nfs_wb_all+0x4a/0x60
[ 8603.801697] [<ffffffff81370108>] nfs_do_fsync+0x38/0x90
[ 8603.801767] [<ffffffff813703fa>] nfs_file_flush+0x8a/0xd0
[ 8603.801837] [<ffffffff8121d164>] filp_close+0x54/0xd0
[ 8603.801907] [<ffffffff810b9881>] put_files_struct+0x111/0x210
[ 8603.801978] [<ffffffff810b97b0>] ? put_files_struct+0x40/0x210
[ 8603.802048] [<ffffffff810b9a7e>] exit_files+0x6e/0x90
[ 8603.802118] [<ffffffff810ba1b6>] do_exit+0x1e6/0xc40
[ 8603.802188] [<ffffffff810fee3b>] ? trace_hardirqs_off+0x1b/0x30
[ 8603.802259] [<ffffffff81b8342c>] ? _raw_spin_unlock_irq+0x4c/0x70
[ 8603.802330] [<ffffffff810baf7d>] do_group_exit+0x6d/0x130
[ 8603.802401] [<ffffffff810d1e9e>] get_signal_to_deliver+0x3de/0x6a0
[ 8603.802473] [<ffffffff8104c843>] do_signal+0x83/0xae0
[ 8603.802543] [<ffffffff8126d1fe>] ? fsnotify+0x35e/0x3d0
[ 8603.802614] [<ffffffff811cdb46>] ? might_fault+0xd6/0xf0
[ 8603.802685] [<ffffffff8104d35c>] do_notify_resume+0x8c/0xc0
[ 8603.802756] [<ffffffff81b82017>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 8603.802828] [<ffffffff8104dec3>] int_signal+0x12/0x17
[ 8603.802898] usemem D 00000001001f68a1 2224 3920 1 0x00000004
[ 8603.803028] ffff8800aac3b5c8 0000000000000006 00000000ffffffff 00000000001d55c0
[ 8603.803246] ffff8800b4d1a350 00000000001d55c0 ffff8800aac3bfd8 ffff8800aac3bfd8
[ 8603.803464] 00000000001d55c0 ffff8800b4d1a6b8 00000000001d55c0 00000000001d55c0
[ 8603.803681] Call Trace:
[ 8603.803747] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.803820] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.803896] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.803966] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.804037] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.804108] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.804179] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.804250] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.804322] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.804393] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.804465] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.804536] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.804608] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.804679] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.804751] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.804824] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.804895] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.804965] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.805036] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.805108] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.805179] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.805250] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.805322] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.805393] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.805465] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.805536] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.805606] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.805677] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.805749] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.805820] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.805889] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.805962] [<ffffffff81b83b45>] page_fault+0x25/0x30
[ 8603.806032] usemem D 00000001001f66b7 2560 3921 1 0x00000004
[ 8603.806163] ffff88006c98b5c8 0000000000000006 0000000000000000 00000000001d55c0
[ 8603.806383] ffff880008d5c6a0 00000000001d55c0 ffff88006c98bfd8 ffff88006c98bfd8
[ 8603.806602] 00000000001d55c0 ffff880008d5ca08 00000000001d55c0 00000000001d55c0
[ 8603.807389] Call Trace:
[ 8603.807455] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.807526] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.807596] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.807665] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.807736] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.807807] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.807878] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.807949] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.808019] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.808092] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.808163] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.808234] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.808305] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.808377] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.808448] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.808520] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.808591] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.808662] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.808733] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.808803] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.808875] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.808946] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.809017] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.809089] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.809161] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.809231] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.809302] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.809372] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.809444] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.809514] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.809584] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.809656] [<ffffffff81b83b45>] page_fault+0x25/0x30
[ 8603.809725] usemem D 00000001001f6ec4 2616 3922 1 0x00000004
[ 8603.809855] ffff8800954ff5c8 0000000000000006 00000000ffffffff 00000000001d55c0
[ 8603.810073] ffff880003540000 00000000001d55c0 ffff8800954fffd8 ffff8800954fffd8
[ 8603.810290] 00000000001d55c0 ffff880003540368 00000000001d55c0 00000000001d55c0
[ 8603.810507] Call Trace:
[ 8603.810573] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.810643] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.810713] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.810783] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.810854] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.810924] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.810996] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.811066] [<ffffffff8115eb48>] ? delayacct_end+0xa8/0xd0
[ 8603.811137] [<ffffffff8115ed04>] ? __delayacct_blkio_end+0x64/0x70
[ 8603.811208] [<ffffffff81b7e660>] ? io_schedule_timeout+0xe0/0x130
[ 8603.811280] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.811352] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.811424] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.811495] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.811566] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.811637] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.811709] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.811781] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.811853] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.811924] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.811994] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.812065] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.812137] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.812208] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.812281] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.812355] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.812429] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.812499] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.812570] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.812641] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.812712] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.812783] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.812853] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.812925] [<ffffffff81b83b45>] page_fault+0x25/0x30
[ 8603.812995] usemem D 00000001001f6604 2616 3923 1 0x00000004
[ 8603.813125] ffff8800229d75c8 0000000000000006 0000000000000000 00000000001d55c0
[ 8603.813343] ffff880003542350 00000000001d55c0 ffff8800229d7fd8 ffff8800229d7fd8
[ 8603.813561] 00000000001d55c0 ffff8800035426b8 00000000001d55c0 00000000001d55c0
[ 8603.813779] Call Trace:
[ 8603.813845] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.813915] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.813985] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.814055] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.814125] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.814197] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.814268] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.814340] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.814410] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.814482] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.814554] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.814625] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.814695] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.814766] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.814838] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.814910] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.814980] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.815050] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.815121] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.815192] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.815263] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.815334] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.815405] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.815477] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.815548] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.815619] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.815689] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.815760] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.815831] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.815902] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.815971] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.816043] [<ffffffff81b83b45>] page_fault+0x25/0x30
[ 8603.816112] usemem D 00000001001f663a 2616 3924 1 0x00000004
[ 8603.816242] ffff88000341d5c8 0000000000000002 00000000ffffffff 00000000001d55c0
[ 8603.816459] ffff8800035446a0 00000000001d55c0 ffff88000341dfd8 ffff88000341dfd8
[ 8603.816677] 00000000001d55c0 ffff880003544a08 00000000001d55c0 00000000001d55c0
[ 8603.816895] Call Trace:
[ 8603.816961] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.817032] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.817101] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.817171] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.817242] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.817313] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.817384] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.817455] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.817526] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.817598] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.817669] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.817740] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.817811] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.817883] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.817954] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.818026] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.818097] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.818167] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.818238] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.818309] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.818381] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.818452] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.818523] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.818595] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.818666] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.818737] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.818807] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.818878] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.818950] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.819020] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.819090] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.819162] [<ffffffff81b83b45>] page_fault+0x25/0x30
[ 8603.819231] usemem D 00000001001eefd8 2144 3925 1 0x00000004
[ 8603.819362] ffff880015edd5c8 0000000000000006 0000000000000000 00000000001d55c0
[ 8603.819581] ffff8800034e0000 00000000001d55c0 ffff880015eddfd8 ffff880015eddfd8
[ 8603.819800] 00000000001d55c0 ffff8800034e0368 00000000001d55c0 00000000001d55c0
[ 8603.820018] Call Trace:
[ 8603.820084] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.820155] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.820225] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.820295] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.820366] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.820437] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.820508] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.820580] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.820650] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.820723] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.820794] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.820866] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.820936] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.821008] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.821080] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.821152] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.821795] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.821865] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.821936] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.822008] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.822079] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.822150] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.822221] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.822292] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.822364] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.822435] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.822506] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.822577] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.822648] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.822719] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.822789] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.822861] [<ffffffff81b83b45>] page_fault+0x25/0x30
[ 8603.822930] usemem D 00000001001f6a75 2616 3926 1 0x00000004
[ 8603.823059] ffff88007c4bf5c8 0000000000000002 00000000ffffffff 00000000001d55c0
[ 8603.823276] ffff8800034e2350 00000000001d55c0 ffff88007c4bffd8 ffff88007c4bffd8
[ 8603.823495] 00000000001d55c0 ffff8800034e26b8 00000000001d55c0 00000000001d55c0
[ 8603.823713] Call Trace:
[ 8603.823779] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.823849] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.823919] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.823989] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.824060] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.824131] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.824202] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.824274] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.824345] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.824416] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.824488] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.824559] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.824630] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.824701] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.824772] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.824845] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.824916] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.824986] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.825056] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.825127] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.825199] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.825270] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.825341] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.825412] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.825484] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.825554] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.825625] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.825696] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.825767] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.825838] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.825908] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.825980] [<ffffffff81b83b45>] page_fault+0x25/0x30
[ 8603.826050] usemem D 00000001001ec073 2552 3927 1 0x00000004
[ 8603.826181] ffff8800325415c8 0000000000000002 0000000000000000 00000000001d55c0
[ 8603.826399] ffff8800034e46a0 00000000001d55c0 ffff880032541fd8 ffff880032541fd8
[ 8603.826618] 00000000001d55c0 ffff8800034e4a08 00000000001d55c0 00000000001d55c0
[ 8603.826837] Call Trace:
[ 8603.826903] [<ffffffff811a5ee0>] ? sync_page+0x0/0xb0
[ 8603.826974] [<ffffffff81b7e046>] io_schedule+0x96/0x110
[ 8603.827044] [<ffffffff811a5f4f>] sync_page+0x6f/0xb0
[ 8603.827113] [<ffffffff81b7ee0d>] __wait_on_bit+0x8d/0xe0
[ 8603.827184] [<ffffffff811a62ff>] wait_on_page_bit+0x8f/0xa0
[ 8603.827256] [<ffffffff810e4530>] ? wake_bit_function+0x0/0x70
[ 8603.827327] [<ffffffff811bc331>] shrink_page_list+0x241/0xcf0
[ 8603.827398] [<ffffffff810e46ef>] ? finish_wait+0x7f/0xb0
[ 8603.827468] [<ffffffff810e44d0>] ? autoremove_wake_function+0x0/0x60
[ 8603.827540] [<ffffffff811bd3bc>] shrink_inactive_list+0x27c/0x4b0
[ 8603.827612] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.827683] [<ffffffff811bdea9>] shrink_zone+0x499/0x610
[ 8603.827754] [<ffffffff811be52b>] do_try_to_free_pages+0x13b/0x540
[ 8603.827825] [<ffffffff811beb59>] try_to_free_pages+0x99/0x180
[ 8603.827897] [<ffffffff811b1e50>] __alloc_pages_nodemask+0x6f0/0xb50
[ 8603.827969] [<ffffffff811f759b>] alloc_pages_current+0xcb/0x160
[ 8603.828041] [<ffffffff81204b74>] new_slab+0x274/0x3b0
[ 8603.828111] [<ffffffff8120581f>] __slab_alloc+0x4bf/0x840
[ 8603.828182] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.828253] [<ffffffff8120665b>] kmem_cache_alloc+0x22b/0x240
[ 8603.828324] [<ffffffff81578bd4>] ? radix_tree_preload+0x44/0xd0
[ 8603.828396] [<ffffffff81578bd4>] radix_tree_preload+0x44/0xd0
[ 8603.828467] [<ffffffff811e94d3>] read_swap_cache_async+0x83/0x1f0
[ 8603.828538] [<ffffffff811ef20f>] ? valid_swaphandles+0x1cf/0x240
[ 8603.828610] [<ffffffff811e96eb>] swapin_readahead+0xab/0xf0
[ 8603.828681] [<ffffffff811a5be0>] ? find_get_page+0x0/0x1c0
[ 8603.828752] [<ffffffff811d2bdd>] handle_mm_fault+0xabd/0xe70
[ 8603.828822] [<ffffffff81b88534>] ? do_page_fault+0x144/0x770
[ 8603.828893] [<ffffffff81b885c9>] do_page_fault+0x1d9/0x770
[ 8603.828964] [<ffffffff81b83d56>] ? error_sti+0x5/0x6
[ 8603.829033] [<ffffffff81b82056>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 8603.829105] [<ffffffff81b83b45>] page_fault+0x25/0x30
[ 8603.829175] flush-8:0 S ffff8800b9320000 3296 3971 2 0x00000000
[ 8603.829304] ffff88008a509d50 0000000000000002 0000000000000002 00000000001d55c0
[ 8603.829522] ffff8800b9320000 00000000001d55c0 ffff88008a509fd8 ffff88008a509fd8
[ 8603.829740] 00000000001d55c0 ffff8800b9320368 00000000001d55c0 00000000001d55c0
[ 8603.829956] Call Trace:
[ 8603.830022] [<ffffffff81b7e930>] schedule_timeout+0x230/0x450
[ 8603.830093] [<ffffffff810cb150>] ? process_timeout+0x0/0x20
[ 8603.830164] [<ffffffff81b7ebd5>] schedule_timeout_interruptible+0x25/0x30
[ 8603.830236] [<ffffffff8125361b>] bdi_writeback_thread+0x2ab/0x3e0
[ 8603.830307] [<ffffffff81253370>] ? bdi_writeback_thread+0x0/0x3e0
[ 8603.830379] [<ffffffff810e3dad>] kthread+0xcd/0xe0
[ 8603.830449] [<ffffffff8104e9e4>] kernel_thread_helper+0x4/0x10
[ 8603.830519] [<ffffffff81b83910>] ? restore_args+0x0/0x30
[ 8603.830590] [<ffffffff810e3ce0>] ? kthread+0x0/0xe0
[ 8603.830659] [<ffffffff8104e9e0>] ? kernel_thread_helper+0x0/0x10
[ 8603.830730] shutdown D 000000010019756c 4120 3972 3860 0x00000000
[ 8603.830859] ffff880042b5bc48 0000000000000002 0000000000000000 00000000001d55c0
[ 8603.831077] ffff880042a5c6a0 00000000001d55c0 ffff880042b5bfd8 ffff880042b5bfd8
[ 8603.831295] 00000000001d55c0 ffff880042a5ca08 00000000001d55c0 00000000001d55c0
[ 8603.831512] Call Trace:
[ 8603.831578] [<ffffffff81b7ea0c>] schedule_timeout+0x30c/0x450
[ 8603.831649] [<ffffffff811023e0>] ? mark_held_locks+0x90/0xd0
[ 8603.831720] [<ffffffff81b8342c>] ? _raw_spin_unlock_irq+0x4c/0x70
[ 8603.831792] [<ffffffff81b7e333>] wait_for_common+0x183/0x220
[ 8603.831863] [<ffffffff810ac590>] ? default_wake_function+0x0/0x30
[ 8603.831935] [<ffffffff81b7e524>] wait_for_completion+0x24/0x30
[ 8603.832007] [<ffffffff81251f6d>] sync_inodes_sb+0x12d/0x2d0
[ 8603.832078] [<ffffffff81b7e216>] ? wait_for_common+0x66/0x220
[ 8603.832149] [<ffffffff812585b0>] ? sync_one_sb+0x0/0x40
[ 8603.832220] [<ffffffff8125857f>] __sync_filesystem+0xcf/0x100
[ 8603.832291] [<ffffffff812585e5>] sync_one_sb+0x35/0x40
[ 8603.832361] [<ffffffff812241fb>] iterate_supers+0x8b/0x130
[ 8603.832432] [<ffffffff81258447>] sync_filesystems+0x27/0x30
[ 8603.832502] [<ffffffff812586c6>] sys_sync+0x36/0x60
[ 8603.832572] [<ffffffff8104db72>] system_call_fastpath+0x16/0x1b
[ 8603.832643] su S 00000001001fbdf1 4456 4049 3202 0x00000000
[ 8603.832774] ffff880017bf1e78 0000000000000002 0000000000000000 00000000001d55c0
[ 8603.832991] ffff8800b918a350 00000000001d55c0 ffff880017bf1fd8 ffff880017bf1fd8
[ 8603.833209] 00000000001d55c0 ffff8800b918a6b8 00000000001d55c0 00000000001d55c0
[ 8603.833427] Call Trace:
[ 8603.833494] [<ffffffff810b9475>] do_wait+0x245/0x340
[ 8603.833564] [<ffffffff810bb436>] sys_wait4+0x86/0x150
[ 8603.833634] [<ffffffff810b7860>] ? child_wait_callback+0x0/0xb0
[ 8603.833705] [<ffffffff8104db72>] system_call_fastpath+0x16/0x1b
[ 8603.833776] zsh R running task 4216 4052 4049 0x00000000
[ 8603.833906] ffff880077ba5858 ffffffff81056c98 0000000000000020 ffff880077ba5898
[ 8603.834124] ffffc90000663758 000000000000002f ffff880077ba5928 ffffffff817c1a0d
[ 8603.834343] 0000000000000020 00000000b996d800 ffff88000000006e 0000002f810fec5d
[ 8603.834562] Call Trace:
[ 8603.834628] [<ffffffff81056c98>] ? nommu_map_page+0x58/0xb0
[ 8603.834699] [<ffffffff817c1a0d>] ? e1000_xmit_frame+0xc6d/0x1030
[ 8603.834772] [<ffffffff81056c40>] ? nommu_map_page+0x0/0xb0
[ 8603.834843] [<ffffffff81a6b6a6>] ? netpoll_send_skb+0x356/0x3b0
[ 8603.834915] [<ffffffff810fec5d>] ? trace_hardirqs_off_caller+0x2d/0x1f0
[ 8603.834988] [<ffffffff810fee3b>] ? trace_hardirqs_off+0x1b/0x30
[ 8603.835059] [<ffffffff81a6b6a6>] ? netpoll_send_skb+0x356/0x3b0
[ 8603.835130] [<ffffffff81b834ed>] ? _raw_spin_unlock_irqrestore+0x9d/0xb0
[ 8603.835203] [<ffffffff810fec5d>] ? trace_hardirqs_off_caller+0x2d/0x1f0
[ 8603.835276] [<ffffffff810fee3b>] ? trace_hardirqs_off+0x1b/0x30
[ 8603.835347] [<ffffffff81b8261b>] ? _raw_spin_lock_irqsave+0x4b/0xf0
[ 8603.835419] [<ffffffff81b834ed>] ? _raw_spin_unlock_irqrestore+0x9d/0xb0
[ 8603.835491] [<ffffffff81b834ed>] ? _raw_spin_unlock_irqrestore+0x9d/0xb0
[ 8603.835564] [<ffffffff810fec5d>] ? trace_hardirqs_off_caller+0x2d/0x1f0
[ 8603.835636] [<ffffffff810fee3b>] ? trace_hardirqs_off+0x1b/0x30
[ 8603.835707] [<ffffffff81b834ed>] ? _raw_spin_unlock_irqrestore+0x9d/0xb0
[ 8603.835779] [<ffffffff810b4e25>] ? release_console_sem+0x2f5/0x390
[ 8603.835851] [<ffffffff81b8261b>] ? _raw_spin_lock_irqsave+0x4b/0xf0
[ 8603.835922] [<ffffffff81b834ed>] ? _raw_spin_unlock_irqrestore+0x9d/0xb0
[ 8603.835995] [<ffffffff81b834ed>] ? _raw_spin_unlock_irqrestore+0x9d/0xb0
[ 8603.836067] [<ffffffff81b834ed>] ? _raw_spin_unlock_irqrestore+0x9d/0xb0
[ 8603.836139] [<ffffffff81b7ce94>] ? printk+0x48/0x51
[ 8603.836209] [<ffffffff81b7ce94>] ? printk+0x48/0x51
[ 8603.836849] [<ffffffff81116a39>] ? __module_text_address+0x19/0xb0
[ 8603.836920] [<ffffffff81052c6f>] ? printk_address+0x2f/0x50
[ 8603.836992] [<ffffffff8111c965>] ? is_module_text_address+0x15/0x30
[ 8603.837063] [<ffffffff810dfdf7>] ? __kernel_text_address+0x47/0xb0
[ 8603.837135] [<ffffffff81052b32>] ? print_context_stack+0xc2/0x1d0
[ 8603.837207] [<ffffffff81051200>] ? dump_trace+0x2d0/0x4d0
[ 8603.837278] [<ffffffff81052d6a>] show_trace_log_lvl+0x6a/0x90
[ 8603.837349] [<ffffffff810514c6>] show_stack_log_lvl+0xc6/0x210
[ 8603.837420] [<ffffffff81052de3>] show_stack+0x23/0x30
[ 8603.837490] [<ffffffff810aef4c>] sched_show_task+0xec/0x180
[ 8603.837561] [<ffffffff810af081>] show_state_filter+0xa1/0x120
[ 8603.837632] [<ffffffff81675dcc>] ? __handle_sysrq+0x3c/0x250
[ 8603.837703] [<ffffffff816757e7>] sysrq_handle_showstate+0x17/0x20
[ 8603.837775] [<ffffffff81675f3c>] __handle_sysrq+0x1ac/0x250
[ 8603.837846] [<ffffffff81675fe0>] ? write_sysrq_trigger+0x0/0x80
[ 8603.837916] [<ffffffff8167604a>] write_sysrq_trigger+0x6a/0x80
[ 8603.837988] [<ffffffff81296ebe>] proc_reg_write+0x9e/0x100
[ 8603.838059] [<ffffffff8121ff94>] vfs_write+0xc4/0x240
[ 8603.838129] [<ffffffff81220466>] sys_write+0x66/0xb0
[ 8603.838199] [<ffffffff8104db72>] system_call_fastpath+0x16/0x1b
[ 8603.838270] Sched Debug Version: v0.09, 2.6.35-rc6-mm1+ #1
[ 8603.838339] now at 8628482.555712 msecs
[ 8603.838406] .jiffies : 4297049416
[ 8603.838477] .sysctl_sched_latency : 24.000000
[ 8603.838546] .sysctl_sched_min_granularity : 8.000000
[ 8603.838617] .sysctl_sched_wakeup_granularity : 4.000000
[ 8603.838686] .sysctl_sched_child_runs_first : 0
[ 8603.838756] .sysctl_sched_features : 15471
[ 8603.838825] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 8603.838896]
[ 8603.838897] cpu#0, 3200.784 MHz
[ 8603.839023] .nr_running : 0
[ 8603.839090] .load : 0
[ 8603.839158] .nr_switches : 1156603
[ 8603.839227] .nr_load_updates : 294710
[ 8603.839295] .nr_uninterruptible : 7
[ 8603.839363] .next_balance : 4297.049409
[ 8603.839432] .curr->pid : 0
[ 8603.839499] .clock : 8603803.856528
[ 8603.839569] .cpu_load[0] : 0
[ 8603.839636] .cpu_load[1] : 0
[ 8603.839704] .cpu_load[2] : 0
[ 8603.839772] .cpu_load[3] : 0
[ 8603.839839] .cpu_load[4] : 0
[ 8603.839906] .yld_count : 0
[ 8603.839974] .sched_switch : 0
[ 8603.840041] .sched_count : 1473751
[ 8603.840110] .sched_goidle : 517579
[ 8603.840178] .avg_idle : 1000000
[ 8603.840247] .ttwu_count : 623917
[ 8603.840315] .ttwu_local : 514058
[ 8603.840383] .bkl_count : 56
[ 8603.840451]
[ 8603.840452] cfs_rq[0]:
[ 8603.840577] .exec_clock : 322781.673156
[ 8603.840648] .MIN_vruntime : 0.000001
[ 8603.840718] .min_vruntime : 305286.495000
[ 8603.840788] .max_vruntime : 0.000001
[ 8603.840857] .spread : 0.000000
[ 8603.840925] .spread0 : 0.000000
[ 8603.840993] .nr_running : 0
[ 8603.841062] .load : 0
[ 8603.841130] .nr_spread_over : 159
[ 8603.841199]
[ 8603.841199] rt_rq[0]:
[ 8603.841325] .rt_nr_running : 0
[ 8603.841393] .rt_throttled : 0
[ 8603.841460] .rt_time : 0.000000
[ 8603.841529] .rt_runtime : 950.000000
[ 8603.841598]
[ 8603.841598] runnable tasks:
[ 8603.841599] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 8603.841600] ----------------------------------------------------------------------------------------------------------
[ 8603.841950]
[ 8603.841950] cpu#1, 3200.784 MHz
[ 8603.842077] .nr_running : 0
[ 8603.842144] .load : 0
[ 8603.842212] .nr_switches : 391795
[ 8603.842280] .nr_load_updates : 142602
[ 8603.842348] .nr_uninterruptible : 1
[ 8603.842415] .next_balance : 4297.048752
[ 8603.842484] .curr->pid : 0
[ 8603.842552] .clock : 8601183.612562
[ 8603.842621] .cpu_load[0] : 0
[ 8603.842689] .cpu_load[1] : 0
[ 8603.842756] .cpu_load[2] : 0
[ 8603.842824] .cpu_load[3] : 0
[ 8603.842891] .cpu_load[4] : 0
[ 8603.842959] .yld_count : 0
[ 8603.843026] .sched_switch : 0
[ 8603.843094] .sched_count : 396119
[ 8603.843162] .sched_goidle : 169313
[ 8603.843230] .avg_idle : 1000000
[ 8603.843298] .ttwu_count : 208609
[ 8603.843367] .ttwu_local : 69252
[ 8603.843435] .bkl_count : 1
[ 8603.843503]
[ 8603.843503] cfs_rq[1]:
[ 8603.843629] .exec_clock : 245715.789291
[ 8603.843699] .MIN_vruntime : 0.000001
[ 8603.843768] .min_vruntime : 271798.833400
[ 8603.843837] .max_vruntime : 0.000001
[ 8603.843905] .spread : 0.000000
[ 8603.843974] .spread0 : -33487.661600
[ 8603.844043] .nr_running : 0
[ 8603.844110] .load : 0
[ 8603.844178] .nr_spread_over : 199
[ 8603.844246]
[ 8603.844246] rt_rq[1]:
[ 8603.844372] .rt_nr_running : 0
[ 8603.844440] .rt_throttled : 0
[ 8603.844507] .rt_time : 0.000000
[ 8603.844575] .rt_runtime : 950.000000
[ 8603.844644]
[ 8603.844644] runnable tasks:
[ 8603.844645] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 8603.844646] ----------------------------------------------------------------------------------------------------------
[ 8603.844992]
[ 8603.844992] cpu#2, 3200.784 MHz
[ 8603.845118] .nr_running : 0
[ 8603.845186] .load : 0
[ 8603.845253] .nr_switches : 560256
[ 8603.845322] .nr_load_updates : 165111
[ 8603.845390] .nr_uninterruptible : 0
[ 8603.845457] .next_balance : 4297.049442
[ 8603.845526] .curr->pid : 0
[ 8603.845593] .clock : 8603812.358700
[ 8603.845663] .cpu_load[0] : 1024
[ 8603.845731] .cpu_load[1] : 512
[ 8603.845799] .cpu_load[2] : 256
[ 8603.845866] .cpu_load[3] : 128
[ 8603.845934] .cpu_load[4] : 64
[ 8603.846002] .yld_count : 0
[ 8603.846070] .sched_switch : 0
[ 8603.846137] .sched_count : 780384
[ 8603.846206] .sched_goidle : 266659
[ 8603.846274] .avg_idle : 1000000
[ 8603.846341] .ttwu_count : 285780
[ 8603.846409] .ttwu_local : 258161
[ 8603.846478] .bkl_count : 20
[ 8603.846546]
[ 8603.846547] cfs_rq[2]:
[ 8603.846673] .exec_clock : 280267.553066
[ 8603.846743] .MIN_vruntime : 0.000001
[ 8603.846811] .min_vruntime : 294921.089281
[ 8603.846880] .max_vruntime : 0.000001
[ 8603.846949] .spread : 0.000000
[ 8603.847018] .spread0 : -10365.405719
[ 8603.847087] .nr_running : 0
[ 8603.847155] .load : 0
[ 8603.847222] .nr_spread_over : 2
[ 8603.847290]
[ 8603.847290] rt_rq[2]:
[ 8603.847416] .rt_nr_running : 0
[ 8603.847483] .rt_throttled : 0
[ 8603.847551] .rt_time : 0.000000
[ 8603.847620] .rt_runtime : 950.000000
[ 8603.847690]
[ 8603.847690] runnable tasks:
[ 8603.847691] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 8603.847692] ----------------------------------------------------------------------------------------------------------
[ 8603.848039]
[ 8603.848040] cpu#3, 3200.784 MHz
[ 8603.848167] .nr_running : 0
[ 8603.848234] .load : 0
[ 8603.848302] .nr_switches : 229642
[ 8603.848370] .nr_load_updates : 115984
[ 8603.848438] .nr_uninterruptible : 0
[ 8603.848507] .next_balance : 4297.049306
[ 8603.848576] .curr->pid : 0
[ 8603.848643] .clock : 8603812.390353
[ 8603.848712] .cpu_load[0] : 1024
[ 8603.848780] .cpu_load[1] : 512
[ 8603.848848] .cpu_load[2] : 256
[ 8603.849487] .cpu_load[3] : 128
[ 8603.849555] .cpu_load[4] : 68
[ 8603.849624] .yld_count : 0
[ 8603.849692] .sched_switch : 0
[ 8603.849759] .sched_count : 234049
[ 8603.849827] .sched_goidle : 94559
[ 8603.849895] .avg_idle : 1000000
[ 8603.849964] .ttwu_count : 120423
[ 8603.850032] .ttwu_local : 49148
[ 8603.850100] .bkl_count : 1
[ 8603.850168]
[ 8603.850168] cfs_rq[3]:
[ 8603.850294] .exec_clock : 169922.393756
[ 8603.850363] .MIN_vruntime : 0.000001
[ 8603.850432] .min_vruntime : 220358.984921
[ 8603.850501] .max_vruntime : 0.000001
[ 8603.850570] .spread : 0.000000
[ 8603.850639] .spread0 : -84927.510079
[ 8603.850708] .nr_running : 0
[ 8603.850775] .load : 0
[ 8603.850870] .nr_spread_over : 55
[ 8603.850939]
[ 8603.850939] rt_rq[3]:
[ 8603.851065] .rt_nr_running : 0
[ 8603.851133] .rt_throttled : 0
[ 8603.851201] .rt_time : 0.000000
[ 8603.851269] .rt_runtime : 950.000000
[ 8603.851338]
[ 8603.851339] runnable tasks:
[ 8603.851339] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 8603.851340] ----------------------------------------------------------------------------------------------------------
[ 8603.851686]
[ 8603.851686] cpu#4, 3200.784 MHz
[ 8603.851813] .nr_running : 1
[ 8603.851880] .load : 1024
[ 8603.851948] .nr_switches : 269553
[ 8603.852016] .nr_load_updates : 155276
[ 8603.852084] .nr_uninterruptible : 4
[ 8603.852152] .next_balance : 4297.049127
[ 8603.852220] .curr->pid : 4052
[ 8603.852289] .clock : 8603554.183086
[ 8603.852358] .cpu_load[0] : 0
[ 8603.852425] .cpu_load[1] : 0
[ 8603.852493] .cpu_load[2] : 0
[ 8603.852560] .cpu_load[3] : 0
[ 8603.852628] .cpu_load[4] : 0
[ 8603.852695] .yld_count : 0
[ 8603.852763] .sched_switch : 0
[ 8603.852830] .sched_count : 274122
[ 8603.852898] .sched_goidle : 107904
[ 8603.852967] .avg_idle : 906404
[ 8603.853036] .ttwu_count : 144344
[ 8603.853104] .ttwu_local : 77278
[ 8603.853173] .bkl_count : 8
[ 8603.853241]
[ 8603.853241] cfs_rq[4]:
[ 8603.853367] .exec_clock : 159892.509962
[ 8603.853437] .MIN_vruntime : 0.000001
[ 8603.853505] .min_vruntime : 181519.598867
[ 8603.853575] .max_vruntime : 0.000001
[ 8603.853644] .spread : 0.000000
[ 8603.853713] .spread0 : -123766.896133
[ 8603.853782] .nr_running : 1
[ 8603.853849] .load : 1024
[ 8603.853917] .nr_spread_over : 14
[ 8603.853986]
[ 8603.853986] rt_rq[4]:
[ 8603.854113] .rt_nr_running : 0
[ 8603.854184] .rt_throttled : 0
[ 8603.854255] .rt_time : 0.000000
[ 8603.854324] .rt_runtime : 950.000000
[ 8603.854393]
[ 8603.854394] runnable tasks:
[ 8603.854394] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 8603.854395] ----------------------------------------------------------------------------------------------------------
[ 8603.854741] R zsh 4052 181519.598867 244 120 181519.598867 136.870513 6324.549747
[ 8603.854938]
[ 8603.854939] cpu#5, 3200.784 MHz
[ 8603.855066] .nr_running : 0
[ 8603.855133] .load : 0
[ 8603.855201] .nr_switches : 185920
[ 8603.855269] .nr_load_updates : 122336
[ 8603.855337] .nr_uninterruptible : 0
[ 8603.855404] .next_balance : 4297.049252
[ 8603.855473] .curr->pid : 0
[ 8603.855541] .clock : 8603177.836247
[ 8603.855610] .cpu_load[0] : 0
[ 8603.855678] .cpu_load[1] : 0
[ 8603.855745] .cpu_load[2] : 0
[ 8603.855812] .cpu_load[3] : 0
[ 8603.855880] .cpu_load[4] : 0
[ 8603.855947] .yld_count : 1
[ 8603.856015] .sched_switch : 0
[ 8603.856082] .sched_count : 190458
[ 8603.856150] .sched_goidle : 78018
[ 8603.856218] .avg_idle : 1000000
[ 8603.856286] .ttwu_count : 94704
[ 8603.856353] .ttwu_local : 55316
[ 8603.856422] .bkl_count : 3
[ 8603.856491]
[ 8603.856491] cfs_rq[5]:
[ 8603.856618] .exec_clock : 108412.314731
[ 8603.856688] .MIN_vruntime : 0.000001
[ 8603.856757] .min_vruntime : 163961.230573
[ 8603.856826] .max_vruntime : 0.000001
[ 8603.856895] .spread : 0.000000
[ 8603.856964] .spread0 : -141325.264427
[ 8603.857033] .nr_running : 0
[ 8603.857101] .load : 0
[ 8603.857169] .nr_spread_over : 6
[ 8603.857236]
[ 8603.857237] rt_rq[5]:
[ 8603.857362] .rt_nr_running : 0
[ 8603.857430] .rt_throttled : 0
[ 8603.857497] .rt_time : 0.000000
[ 8603.857566] .rt_runtime : 950.000000
[ 8603.857634]
[ 8603.857635] runnable tasks:
[ 8603.857635] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 8603.857636] ----------------------------------------------------------------------------------------------------------
[ 8603.857983]
[ 8603.857983] cpu#6, 3200.784 MHz
[ 8603.858110] .nr_running : 0
[ 8603.858177] .load : 0
[ 8603.858244] .nr_switches : 276975
[ 8603.858312] .nr_load_updates : 155233
[ 8603.858380] .nr_uninterruptible : 1
[ 8603.858447] .next_balance : 4297.049377
[ 8603.858516] .curr->pid : 0
[ 8603.858584] .clock : 8603676.300923
[ 8603.858653] .cpu_load[0] : 0
[ 8603.858721] .cpu_load[1] : 0
[ 8603.858788] .cpu_load[2] : 0
[ 8603.858855] .cpu_load[3] : 0
[ 8603.858922] .cpu_load[4] : 0
[ 8603.858989] .yld_count : 0
[ 8603.859056] .sched_switch : 0
[ 8603.859124] .sched_count : 13643133
[ 8603.859192] .sched_goidle : 109193
[ 8603.859260] .avg_idle : 1000000
[ 8603.859328] .ttwu_count : 142634
[ 8603.859396] .ttwu_local : 60067
[ 8603.859464] .bkl_count : 6
[ 8603.859532]
[ 8603.859532] cfs_rq[6]:
[ 8603.859658] .exec_clock : 273546.895137
[ 8603.859728] .MIN_vruntime : 0.000001
[ 8603.859797] .min_vruntime : 314787.690335
[ 8603.859866] .max_vruntime : 0.000001
[ 8603.859934] .spread : 0.000000
[ 8603.860004] .spread0 : 9501.195335
[ 8603.860072] .nr_running : 0
[ 8603.860141] .load : 0
[ 8603.860209] .nr_spread_over : 85
[ 8603.860277]
[ 8603.860277] rt_rq[6]:
[ 8603.860402] .rt_nr_running : 0
[ 8603.860469] .rt_throttled : 0
[ 8603.860537] .rt_time : 0.000000
[ 8603.860606] .rt_runtime : 950.000000
[ 8603.860675]
[ 8603.860676] runnable tasks:
[ 8603.860676] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 8603.860677] ----------------------------------------------------------------------------------------------------------
[ 8603.861024]
[ 8603.861024] cpu#7, 3200.784 MHz
[ 8603.861151] .nr_running : 0
[ 8603.861218] .load : 0
[ 8603.861286] .nr_switches : 134607
[ 8603.861354] .nr_load_updates : 108783
[ 8603.861423] .nr_uninterruptible : 1
[ 8603.862060] .next_balance : 4297.048502
[ 8603.862128] .curr->pid : 0
[ 8603.862196] .clock : 8600186.437864
[ 8603.862265] .cpu_load[0] : 0
[ 8603.862332] .cpu_load[1] : 0
[ 8603.862399] .cpu_load[2] : 0
[ 8603.862467] .cpu_load[3] : 0
[ 8603.862533] .cpu_load[4] : 0
[ 8603.862601] .yld_count : 0
[ 8603.862668] .sched_switch : 0
[ 8603.862735] .sched_count : 138870
[ 8603.862803] .sched_goidle : 55637
[ 8603.862871] .avg_idle : 1000000
[ 8603.862939] .ttwu_count : 69313
[ 8603.863007] .ttwu_local : 36525
[ 8603.863075] .bkl_count : 1
[ 8603.863143]
[ 8603.863144] cfs_rq[7]:
[ 8603.863269] .exec_clock : 88588.952951
[ 8603.863339] .MIN_vruntime : 0.000001
[ 8603.863408] .min_vruntime : 128455.709256
[ 8603.863477] .max_vruntime : 0.000001
[ 8603.863545] .spread : 0.000000
[ 8603.863614] .spread0 : -176830.785744
[ 8603.863683] .nr_running : 0
[ 8603.863751] .load : 0
[ 8603.863819] .nr_spread_over : 3
[ 8603.863887]
[ 8603.863887] rt_rq[7]:
[ 8603.864013] .rt_nr_running : 0
[ 8603.864080] .rt_throttled : 0
[ 8603.864148] .rt_time : 0.000000
[ 8603.864217] .rt_runtime : 950.000000
[ 8603.864286]
[ 8603.864287] runnable tasks:
[ 8603.864287] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 8603.864288] ----------------------------------------------------------------------------------------------------------
[ 8603.864635]
[ 8603.864698]
[ 8603.864699] Showing all locks held in the system:
[ 8603.864831] 1 lock held by getty/3094:
[ 8603.864898] #0: (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff81657b70>] n_tty_read+0x7e0/0xca0
[ 8603.865121] 1 lock held by getty/3095:
[ 8603.865187] #0: (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff81657b70>] n_tty_read+0x7e0/0xca0
[ 8603.865409] 1 lock held by getty/3096:
[ 8603.865475] #0: (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff81657b70>] n_tty_read+0x7e0/0xca0
[ 8603.865696] 1 lock held by getty/3097:
[ 8603.865762] #0: (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff81657b70>] n_tty_read+0x7e0/0xca0
[ 8603.865982] 1 lock held by getty/3098:
[ 8603.866048] #0: (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff81657b70>] n_tty_read+0x7e0/0xca0
[ 8603.866269] 1 lock held by getty/3099:
[ 8603.866335] #0: (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff81657b70>] n_tty_read+0x7e0/0xca0
[ 8603.866556] 1 lock held by usemem/3920:
[ 8603.866623] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81b88534>] do_page_fault+0x144/0x770
[ 8603.866845] 1 lock held by usemem/3921:
[ 8603.866911] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81b88534>] do_page_fault+0x144/0x770
[ 8603.867132] 1 lock held by usemem/3922:
[ 8603.867198] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81b88534>] do_page_fault+0x144/0x770
[ 8603.867419] 1 lock held by usemem/3923:
[ 8603.867485] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81b88534>] do_page_fault+0x144/0x770
[ 8603.867705] 1 lock held by usemem/3924:
[ 8603.867772] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81b88534>] do_page_fault+0x144/0x770
[ 8603.867992] 1 lock held by usemem/3925:
[ 8603.868059] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81b88534>] do_page_fault+0x144/0x770
[ 8603.868278] 1 lock held by usemem/3926:
[ 8603.868345] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81b88534>] do_page_fault+0x144/0x770
[ 8603.868565] 1 lock held by usemem/3927:
[ 8603.868631] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81b88534>] do_page_fault+0x144/0x770
[ 8603.868852] 1 lock held by shutdown/3972:
[ 8603.868918] #0: (&type->s_umount_key#41){.+.+..}, at: [<ffffffff812241dd>] iterate_supers+0x6d/0x130
[ 8603.869168] 2 locks held by zsh/4052:
[ 8603.869234] #0: (sysrq_key_table_lock){......}, at: [<ffffffff81675dcc>] __handle_sysrq+0x3c/0x250
[ 8603.869454] #1: (tasklist_lock){.+.+..}, at: [<ffffffff810ffa3c>] debug_show_all_locks+0x4c/0x280
[ 8603.869676]
[ 8603.869739] =============================================
[ 8603.869740]
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Christoph Hellwig: "Re: [PATCH] vmscan: raise the bar to PAGEOUT_IO_SYNC stalls"
Previous message: David Howells: "Re: [PATCH 02/18] xstat: Add a pair of system calls to make extended file stats available [ver #6]"
In reply to: Mel Gorman: "Re: [PATCH] vmscan: raise the bar to PAGEOUT_IO_SYNC stalls"
Next in thread: Christoph Hellwig: "Re: [PATCH] vmscan: raise the bar to PAGEOUT_IO_SYNC stalls"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]