Re: PROBLEM + POSS FIX: kernel stack overflow, xfs, many disks, heavywrite load, 8k stack, x86-64

From: John Berthels
Date: Tue Apr 13 2010 - 05:53:12 EST

Chris Mason wrote:
shrink_zone on my box isn't 500 bytes, but lets try the easy stuff
first. This is against .34, if you have any trouble applying to .32,
just add the word noinline after the word static on the function
definitions.

Hi Chris,

Thanks for this, we've been soaking it for a while and get the stack trace below (which is still >8k), which still has shrink_zone at 528 bytes.

I find it odd that the shrink_zone stack usage is different on our systems. This is a stock kernel 2.6.33.2 kernel, x86_64 arch (plus your patch + Dave Chinner's patch) built using ubuntu make-kpkg, with gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3 (.vmscan.o.cmd with full build options is below, gzipped .config attached).

Can you see any difference between your system and ours which might explain the discrepancy? I note -g and -pg in there. (Does -pg have any stack overhead? It seems to be enabled in ubuntu release kernels).

regards,

jb



mm/.vmscan.o.cmd:

cmd_mm/vmscan.o := gcc -Wp,-MD,mm/.vmscan.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/4.3.3/include -I/usr/local/src/kern/linux-2.6.33.2/arch/x86/include -Iinclude -include include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -m64 -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -D"KBUILD_STR(s)=\#s" -D"KBUILD_BASENAME=KBUILD_STR(vmscan)" -D"KBUILD_MODNAME=KBUILD_STR(vmscan)" -c -o mm/.tmp_vmscan.o mm/vmscan.c



Apr 12 22:06:35 nas17 kernel: [36346.599076] apache2 used greatest stack
depth: 7904 bytes left
Depth Size Location (56 entries)
----- ---- --------
0) 7904 48 __call_rcu+0x67/0x190
1) 7856 16 call_rcu_sched+0x15/0x20
2) 7840 16 call_rcu+0xe/0x10
3) 7824 272 radix_tree_delete+0x159/0x2e0
4) 7552 32 __remove_from_page_cache+0x21/0x110
5) 7520 64 __remove_mapping+0xe8/0x130
6) 7456 384 shrink_page_list+0x400/0x860
7) 7072 528 shrink_zone+0x636/0xdc0
8) 6544 112 do_try_to_free_pages+0xc2/0x3c0
9) 6432 112 try_to_free_pages+0x64/0x70
10) 6320 256 __alloc_pages_nodemask+0x3d2/0x710
11) 6064 48 alloc_pages_current+0x8c/0xe0
12) 6016 32 __page_cache_alloc+0x67/0x70
13) 5984 80 find_or_create_page+0x50/0xb0
14) 5904 160 _xfs_buf_lookup_pages+0x145/0x350 [xfs]
15) 5744 64 xfs_buf_get+0x74/0x1d0 [xfs]
16) 5680 48 xfs_buf_read+0x2f/0x110 [xfs]
17) 5632 80 xfs_trans_read_buf+0x2bf/0x430 [xfs]
18) 5552 80 xfs_btree_read_buf_block+0x5d/0xb0 [xfs]
19) 5472 176 xfs_btree_rshift+0xd7/0x530 [xfs]
20) 5296 96 xfs_btree_make_block_unfull+0x5b/0x190 [xfs]
21) 5200 224 xfs_btree_insrec+0x39c/0x5b0 [xfs]
22) 4976 128 xfs_btree_insert+0x86/0x180 [xfs]
23) 4848 96 xfs_alloc_fixup_trees+0x1fa/0x350 [xfs]
24) 4752 144 xfs_alloc_ag_vextent_near+0x916/0xb30 [xfs]
25) 4608 32 xfs_alloc_ag_vextent+0xe5/0x140 [xfs]
26) 4576 96 xfs_alloc_vextent+0x49f/0x630 [xfs]
27) 4480 160 xfs_bmbt_alloc_block+0xbe/0x1d0 [xfs]
28) 4320 208 xfs_btree_split+0xb3/0x6a0 [xfs]
29) 4112 96 xfs_btree_make_block_unfull+0x151/0x190 [xfs]
30) 4016 224 xfs_btree_insrec+0x39c/0x5b0 [xfs]
31) 3792 128 xfs_btree_insert+0x86/0x180 [xfs]
32) 3664 352 xfs_bmap_add_extent_delay_real+0x41e/0x1670 [xfs]
33) 3312 208 xfs_bmap_add_extent+0x41c/0x450 [xfs]
34) 3104 448 xfs_bmapi+0x982/0x1200 [xfs]
35) 2656 256 xfs_iomap_write_allocate+0x248/0x3c0 [xfs]
36) 2400 208 xfs_iomap+0x3d8/0x410 [xfs]
37) 2192 32 xfs_map_blocks+0x2c/0x30 [xfs]
38) 2160 256 xfs_page_state_convert+0x443/0x730 [xfs]
39) 1904 64 xfs_vm_writepage+0xab/0x160 [xfs]
40) 1840 32 __writepage+0x1a/0x60
41) 1808 288 write_cache_pages+0x1f7/0x400
42) 1520 16 generic_writepages+0x27/0x30
43) 1504 48 xfs_vm_writepages+0x5a/0x70 [xfs]
44) 1456 16 do_writepages+0x24/0x40
45) 1440 64 writeback_single_inode+0xf1/0x3e0
46) 1376 128 writeback_inodes_wb+0x31e/0x510
47) 1248 16 writeback_inodes_wbc+0x1e/0x20
48) 1232 224 balance_dirty_pages_ratelimited_nr+0x277/0x410
49) 1008 192 generic_file_buffered_write+0x19b/0x240
50) 816 288 xfs_write+0x849/0x930 [xfs]
51) 528 16 xfs_file_aio_write+0x5b/0x70 [xfs]
52) 512 272 do_sync_write+0xd1/0x120
53) 240 48 vfs_write+0xcb/0x1a0
54) 192 64 sys_write+0x55/0x90
55) 128 128 system_call_fastpath+0x16/0x1b

Attachment: config.gz
Description: GNU Zip compressed data