Re: [RFC PATCH 1/2] mm, vmscan: account the number of isolated pages per zone
From: Tetsuo Handa
Date: Fri Feb 03 2017 - 06:00:18 EST
Michal Hocko wrote:
> On Mon 30-01-17 09:55:46, Michal Hocko wrote:
> > On Sun 29-01-17 00:27:27, Tetsuo Handa wrote:
> [...]
> > > Regarding [1], it helped avoiding the too_many_isolated() issue. I can't
> > > tell whether it has any negative effect, but I got on the first trial that
> > > all allocating threads are blocked on wait_for_completion() from flush_work()
> > > in drain_all_pages() introduced by "mm, page_alloc: drain per-cpu pages from
> > > workqueue context". There was no warn_alloc() stall warning message afterwords.
> >
> > That patch is buggy and there is a follow up [1] which is not sitting in the
> > mmotm (and thus linux-next) yet. I didn't get to review it properly and
> > I cannot say I would be too happy about using WQ from the page
> > allocator. I believe even the follow up needs to have WQ_RECLAIM WQ.
> >
> > [1] http://lkml.kernel.org/r/20170125083038.rzb5f43nptmk7aed@xxxxxxxxxxxxxxxxxxx
>
> Did you get chance to test with this follow up patch? It would be
> interesting to see whether OOM situation can still starve the waiter.
> The current linux-next should contain this patch.
So far I can't reproduce problems except two listed below (cond_resched() trap
in printk() and IDLE priority trap are excluded from the list). But I agree that
the follow up patch needs to use a WQ_RECLAIM WQ. It is theoretically possible
that an allocation request which can trigger the OOM killer waits for the
system_wq while there is already a work which is in system_wq which is looping
forever inside the page allocator without triggering the OOM killer.
Maybe the follow up patch can share the vmstat WQ?
(1) I got an assertion failure.
[ 969.626518] Killed process 6262 (oom-write) total-vm:2166856kB, anon-rss:1128732kB, file-rss:4kB, shmem-rss:0kB
[ 969.958307] oom_reaper: reaped process 6262 (oom-write), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 972.114644] XFS: Assertion failed: oldlen > newlen, file: fs/xfs/libxfs/xfs_bmap.c, line: 2867
[ 972.125085] ------------[ cut here ]------------
[ 972.129261] WARNING: CPU: 0 PID: 6280 at fs/xfs/xfs_message.c:105 asswarn+0x33/0x40 [xfs]
[ 972.136146] Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack coretemp crct10dif_pclmul ppdev crc32_pclmul ghash_clmulni_intel ip_set nfnetlink ebtable_nat aesni_intel crypto_simd cryptd ebtable_broute glue_helper vmw_balloon bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 pcspkr nf_nat_ipv6 ip6table_mangle ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sg parport_pc parport shpchp i2c_piix4 vmw_vsock_vmci_transport vsock vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi crc32c_intel serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect
[ 972.163630] sysimgblt fb_sys_fops ttm drm ata_piix ahci libahci mptspi scsi_transport_spi mptscsih e1000 libata i2c_core mptbase
[ 972.172535] CPU: 0 PID: 6280 Comm: write Not tainted 4.10.0-rc6-next-20170202 #498
[ 972.175126] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 972.178381] Call Trace:
[ 972.180003] dump_stack+0x85/0xc9
[ 972.181682] __warn+0xd1/0xf0
[ 972.183374] warn_slowpath_null+0x1d/0x20
[ 972.185223] asswarn+0x33/0x40 [xfs]
[ 972.186950] xfs_bmap_add_extent_hole_delay+0xb7f/0xdf0 [xfs]
[ 972.189055] xfs_bmapi_reserve_delalloc+0x297/0x440 [xfs]
[ 972.191263] ? xfs_ilock+0x1c9/0x360 [xfs]
[ 972.193414] xfs_file_iomap_begin+0x880/0x1140 [xfs]
[ 972.195300] ? iomap_write_end+0x80/0x80
[ 972.196980] iomap_apply+0x6c/0x130
[ 972.198539] iomap_file_buffered_write+0x68/0xa0
[ 972.200316] ? iomap_write_end+0x80/0x80
[ 972.201950] xfs_file_buffered_aio_write+0x132/0x390 [xfs]
[ 972.203868] ? _raw_spin_unlock+0x27/0x40
[ 972.205470] xfs_file_write_iter+0x90/0x130 [xfs]
[ 972.207167] __vfs_write+0xe5/0x140
[ 972.208752] vfs_write+0xc7/0x1f0
[ 972.210233] ? syscall_trace_enter+0x1d0/0x380
[ 972.211809] SyS_write+0x58/0xc0
[ 972.213166] do_int80_syscall_32+0x6c/0x1f0
[ 972.214676] entry_INT80_compat+0x38/0x50
[ 972.216168] RIP: 0023:0x8048076
[ 972.217494] RSP: 002b:00000000ff997020 EFLAGS: 00000202 ORIG_RAX: 0000000000000004
[ 972.219635] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000000008048000
[ 972.221679] RDX: 0000000000001000 RSI: 0000000000000000 RDI: 0000000000000000
[ 972.223774] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 972.225905] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 972.227946] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 972.230064] ---[ end trace d498098daec56c11 ]---
[ 984.210890] vmtoolsd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
[ 984.224191] vmtoolsd cpuset=/ mems_allowed=0
[ 984.231022] CPU: 0 PID: 689 Comm: vmtoolsd Tainted: G W 4.10.0-rc6-next-20170202 #498
(2) I got a lockdep warning. (A new false positive?)
[ 243.036975] =====================================================
[ 243.042976] WARNING: RECLAIM_FS-safe -> RECLAIM_FS-unsafe lock order detected
[ 243.051211] 4.10.0-rc6-next-20170202 #46 Not tainted
[ 243.054619] -----------------------------------------------------
[ 243.057395] awk/8767 [HC0[0]:SC0[0]:HE1:SE1] is trying to acquire:
[ 243.060310] (cpu_hotplug.dep_map){++++++}, at: [<ffffffff8108ddf2>] get_online_cpus+0x32/0x80
[ 243.063462]
[ 243.063462] and this task is already holding:
[ 243.066851] (&xfs_dir_ilock_class){++++-.}, at: [<ffffffffa02a4af4>] xfs_ilock+0x114/0x290 [xfs]
[ 243.069949] which would create a new lock dependency:
[ 243.072143] (&xfs_dir_ilock_class){++++-.} -> (cpu_hotplug.dep_map){++++++}
[ 243.074789]
[ 243.074789] but this new dependency connects a RECLAIM_FS-irq-safe lock:
[ 243.078735] (&xfs_dir_ilock_class){++++-.}
[ 243.078739]
[ 243.078739] ... which became RECLAIM_FS-irq-safe at:
[ 243.084175]
[ 243.084180] [<ffffffff810ef934>] __lock_acquire+0x344/0x1bb0
[ 243.087257]
[ 243.087261] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.090027]
[ 243.090033] [<ffffffff810ea7e9>] down_write_nested+0x59/0xc0
[ 243.092838]
[ 243.092888] [<ffffffffa02a4b2e>] xfs_ilock+0x14e/0x290 [xfs]
[ 243.095453]
[ 243.095485] [<ffffffffa02986a5>] xfs_reclaim_inode+0x135/0x340 [xfs]
[ 243.098083]
[ 243.098109] [<ffffffffa0298b7a>] xfs_reclaim_inodes_ag+0x2ca/0x4f0 [xfs]
[ 243.100668]
[ 243.100692] [<ffffffffa029af9e>] xfs_reclaim_inodes_nr+0x2e/0x40 [xfs]
[ 243.103191]
[ 243.103221] [<ffffffffa02b32c4>] xfs_fs_free_cached_objects+0x14/0x20 [xfs]
[ 243.105710]
[ 243.105714] [<ffffffff81261dbc>] super_cache_scan+0x17c/0x190
[ 243.107947]
[ 243.107950] [<ffffffff811d375a>] shrink_slab+0x29a/0x710
[ 243.110133]
[ 243.110135] [<ffffffff811d876d>] shrink_node+0x23d/0x320
[ 243.112262]
[ 243.112264] [<ffffffff811d9e24>] kswapd+0x354/0xa10
[ 243.114323]
[ 243.114326] [<ffffffff810b5caa>] kthread+0x10a/0x140
[ 243.116448]
[ 243.116452] [<ffffffff81715081>] ret_from_fork+0x31/0x40
[ 243.118692]
[ 243.118692] to a RECLAIM_FS-irq-unsafe lock:
[ 243.120636] (cpu_hotplug.dep_map){++++++}
[ 243.120638]
[ 243.120638] ... which became RECLAIM_FS-irq-unsafe at:
[ 243.124021] ...
[ 243.124022]
[ 243.124820] [<ffffffff810ef051>] mark_held_locks+0x71/0x90
[ 243.127033]
[ 243.127035] [<ffffffff810f3405>] lockdep_trace_alloc+0xc5/0x110
[ 243.129228]
[ 243.129231] [<ffffffff8122f8ca>] kmem_cache_alloc_node_trace+0x4a/0x410
[ 243.131534]
[ 243.131536] [<ffffffff810ba350>] __smpboot_create_thread.part.3+0x30/0xf0
[ 243.133850]
[ 243.133852] [<ffffffff810ba7a1>] smpboot_create_threads+0x61/0x90
[ 243.136113]
[ 243.136119] [<ffffffff8108e2cb>] cpuhp_invoke_callback+0xbb/0xb70
[ 243.138319]
[ 243.138320] [<ffffffff8108fc82>] cpuhp_up_callbacks+0x32/0xb0
[ 243.140479]
[ 243.140480] [<ffffffff810900f4>] _cpu_up+0x84/0xf0
[ 243.142484]
[ 243.142485] [<ffffffff810901e4>] do_cpu_up+0x84/0xd0
[ 243.144716]
[ 243.144719] [<ffffffff8109023e>] cpu_up+0xe/0x10
[ 243.146684]
[ 243.146687] [<ffffffff81f6f446>] smp_init+0xd5/0x141
[ 243.148755]
[ 243.148758] [<ffffffff81f3f35b>] kernel_init_freeable+0x17d/0x2a7
[ 243.150932]
[ 243.150936] [<ffffffff817048e9>] kernel_init+0x9/0x100
[ 243.153088]
[ 243.153092] [<ffffffff81715081>] ret_from_fork+0x31/0x40
[ 243.155135]
[ 243.155135] other info that might help us debug this:
[ 243.155135]
[ 243.157724] Possible interrupt unsafe locking scenario:
[ 243.157724]
[ 243.159877] CPU0 CPU1
[ 243.161047] ---- ----
[ 243.162210] lock(cpu_hotplug.dep_map);
[ 243.163279] local_irq_disable();
[ 243.164669] lock(&xfs_dir_ilock_class);
[ 243.166148] lock(cpu_hotplug.dep_map);
[ 243.167653] <Interrupt>
[ 243.168594] lock(&xfs_dir_ilock_class);
[ 243.169694]
[ 243.169694] *** DEADLOCK ***
[ 243.169694]
[ 243.171864] 3 locks held by awk/8767:
[ 243.172872] #0: (&type->i_mutex_dir_key#3){++++++}, at: [<ffffffff8126e2dc>] path_openat+0x53c/0xa90
[ 243.174791] #1: (&xfs_dir_ilock_class){++++-.}, at: [<ffffffffa02a4af4>] xfs_ilock+0x114/0x290 [xfs]
[ 243.176899] #2: (pcpu_drain_mutex){+.+...}, at: [<ffffffff811bf39a>] drain_all_pages.part.80+0x1a/0x320
[ 243.178875]
[ 243.178875] the dependencies between RECLAIM_FS-irq-safe lock and the holding lock:
[ 243.181262] -> (&xfs_dir_ilock_class){++++-.} ops: 17348 {
[ 243.182610] HARDIRQ-ON-W at:
[ 243.183603]
[ 243.183606] [<ffffffff810efd84>] __lock_acquire+0x794/0x1bb0
[ 243.186056]
[ 243.186059] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.188419]
[ 243.188422] [<ffffffff810ea7e9>] down_write_nested+0x59/0xc0
[ 243.190909]
[ 243.190941] [<ffffffffa02a4b2e>] xfs_ilock+0x14e/0x290 [xfs]
[ 243.193257]
[ 243.193281] [<ffffffffa02a4c9b>] xfs_ilock_data_map_shared+0x2b/0x30 [xfs]
[ 243.195795]
[ 243.195814] [<ffffffffa02559f4>] xfs_dir_lookup+0xd4/0x1c0 [xfs]
[ 243.198204]
[ 243.198227] [<ffffffffa02a62ff>] xfs_lookup+0x7f/0x250 [xfs]
[ 243.200570]
[ 243.200593] [<ffffffffa02a1fcb>] xfs_vn_lookup+0x6b/0xb0 [xfs]
[ 243.203086]
[ 243.203089] [<ffffffff8126ce2c>] lookup_open+0x54c/0x790
[ 243.205417]
[ 243.205420] [<ffffffff8126e2fa>] path_openat+0x55a/0xa90
[ 243.207711]
[ 243.207713] [<ffffffff8126f9ec>] do_filp_open+0x8c/0x100
[ 243.210092]
[ 243.210095] [<ffffffff81263c41>] do_open_execat+0x71/0x180
[ 243.212427]
[ 243.212429] [<ffffffff812641b6>] open_exec+0x26/0x40
[ 243.214664]
[ 243.214668] [<ffffffff812c43ee>] load_elf_binary+0x2be/0x15f0
[ 243.217045]
[ 243.217048] [<ffffffff812644b0>] search_binary_handler+0x80/0x1e0
[ 243.219501]
[ 243.219503] [<ffffffff812663ca>] do_execveat_common.isra.40+0x68a/0xa00
[ 243.222056]
[ 243.222058] [<ffffffff81266767>] do_execve+0x27/0x30
[ 243.224471]
[ 243.224475] [<ffffffff812669c0>] SyS_execve+0x20/0x30
[ 243.226787]
[ 243.226790] [<ffffffff81003c17>] do_syscall_64+0x67/0x1f0
[ 243.229178]
[ 243.229182] [<ffffffff81714ec9>] return_from_SYSCALL_64+0x0/0x7a
[ 243.231695] HARDIRQ-ON-R at:
[ 243.232709]
[ 243.232712] [<ffffffff810ef8c0>] __lock_acquire+0x2d0/0x1bb0
[ 243.235161]
[ 243.235164] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.237547]
[ 243.237551] [<ffffffff810ea672>] down_read_nested+0x52/0xb0
[ 243.239930]
[ 243.239962] [<ffffffffa02a4af4>] xfs_ilock+0x114/0x290 [xfs]
[ 243.242353]
[ 243.242385] [<ffffffffa02a4c9b>] xfs_ilock_data_map_shared+0x2b/0x30 [xfs]
[ 243.244978]
[ 243.244998] [<ffffffffa02559f4>] xfs_dir_lookup+0xd4/0x1c0 [xfs]
[ 243.247493]
[ 243.247515] [<ffffffffa02a62ff>] xfs_lookup+0x7f/0x250 [xfs]
[ 243.249910]
[ 243.249930] [<ffffffffa02a1fcb>] xfs_vn_lookup+0x6b/0xb0 [xfs]
[ 243.252407]
[ 243.252412] [<ffffffff8126902e>] lookup_slow+0x12e/0x220
[ 243.254747]
[ 243.254750] [<ffffffff8126d2c6>] walk_component+0x1a6/0x2b0
[ 243.257126]
[ 243.257128] [<ffffffff8126d55c>] link_path_walk+0x18c/0x580
[ 243.259495]
[ 243.259497] [<ffffffff8126de41>] path_openat+0xa1/0xa90
[ 243.261804]
[ 243.261806] [<ffffffff8126f9ec>] do_filp_open+0x8c/0x100
[ 243.264184]
[ 243.264188] [<ffffffff8125c0ea>] do_sys_open+0x13a/0x200
[ 243.266595]
[ 243.266599] [<ffffffff8125c1c9>] SyS_open+0x19/0x20
[ 243.268984]
[ 243.268989] [<ffffffff81714e01>] entry_SYSCALL_64_fastpath+0x1f/0xc2
[ 243.271702] SOFTIRQ-ON-W at:
[ 243.272726]
[ 243.272729] [<ffffffff810ef8ed>] __lock_acquire+0x2fd/0x1bb0
[ 243.275109]
[ 243.275111] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.277426]
[ 243.277429] [<ffffffff810ea7e9>] down_write_nested+0x59/0xc0
[ 243.279790]
[ 243.279823] [<ffffffffa02a4b2e>] xfs_ilock+0x14e/0x290 [xfs]
[ 243.282192]
[ 243.282216] [<ffffffffa02a4c9b>] xfs_ilock_data_map_shared+0x2b/0x30 [xfs]
[ 243.284794]
[ 243.284816] [<ffffffffa02559f4>] xfs_dir_lookup+0xd4/0x1c0 [xfs]
[ 243.287259]
[ 243.287284] [<ffffffffa02a62ff>] xfs_lookup+0x7f/0x250 [xfs]
[ 243.289735]
[ 243.289763] [<ffffffffa02a1fcb>] xfs_vn_lookup+0x6b/0xb0 [xfs]
[ 243.292205]
[ 243.292208] [<ffffffff8126ce2c>] lookup_open+0x54c/0x790
[ 243.294555]
[ 243.294558] [<ffffffff8126e2fa>] path_openat+0x55a/0xa90
[ 243.296897]
[ 243.296900] [<ffffffff8126f9ec>] do_filp_open+0x8c/0x100
[ 243.299242]
[ 243.299244] [<ffffffff81263c41>] do_open_execat+0x71/0x180
[ 243.301754]
[ 243.301759] [<ffffffff812641b6>] open_exec+0x26/0x40
[ 243.304037]
[ 243.304042] [<ffffffff812c43ee>] load_elf_binary+0x2be/0x15f0
[ 243.306531]
[ 243.306534] [<ffffffff812644b0>] search_binary_handler+0x80/0x1e0
[ 243.308976]
[ 243.308979] [<ffffffff812663ca>] do_execveat_common.isra.40+0x68a/0xa00
[ 243.311506]
[ 243.311508] [<ffffffff81266767>] do_execve+0x27/0x30
[ 243.313777]
[ 243.313779] [<ffffffff812669c0>] SyS_execve+0x20/0x30
[ 243.316067]
[ 243.316070] [<ffffffff81003c17>] do_syscall_64+0x67/0x1f0
[ 243.318429]
[ 243.318434] [<ffffffff81714ec9>] return_from_SYSCALL_64+0x0/0x7a
[ 243.320884] SOFTIRQ-ON-R at:
[ 243.321860]
[ 243.321862] [<ffffffff810ef8ed>] __lock_acquire+0x2fd/0x1bb0
[ 243.324251]
[ 243.324252] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.326601]
[ 243.326604] [<ffffffff810ea672>] down_read_nested+0x52/0xb0
[ 243.328966]
[ 243.328998] [<ffffffffa02a4af4>] xfs_ilock+0x114/0x290 [xfs]
[ 243.331384]
[ 243.331407] [<ffffffffa02a4c9b>] xfs_ilock_data_map_shared+0x2b/0x30 [xfs]
[ 243.333978]
[ 243.334001] [<ffffffffa02559f4>] xfs_dir_lookup+0xd4/0x1c0 [xfs]
[ 243.336492]
[ 243.336516] [<ffffffffa02a62ff>] xfs_lookup+0x7f/0x250 [xfs]
[ 243.338926]
[ 243.338948] [<ffffffffa02a1fcb>] xfs_vn_lookup+0x6b/0xb0 [xfs]
[ 243.341365]
[ 243.341368] [<ffffffff8126902e>] lookup_slow+0x12e/0x220
[ 243.343694]
[ 243.343696] [<ffffffff8126d2c6>] walk_component+0x1a6/0x2b0
[ 243.346074]
[ 243.346076] [<ffffffff8126d55c>] link_path_walk+0x18c/0x580
[ 243.348443]
[ 243.348444] [<ffffffff8126de41>] path_openat+0xa1/0xa90
[ 243.350753]
[ 243.350755] [<ffffffff8126f9ec>] do_filp_open+0x8c/0x100
[ 243.353240]
[ 243.353244] [<ffffffff8125c0ea>] do_sys_open+0x13a/0x200
[ 243.355581]
[ 243.355583] [<ffffffff8125c1c9>] SyS_open+0x19/0x20
[ 243.358015]
[ 243.358019] [<ffffffff81714e01>] entry_SYSCALL_64_fastpath+0x1f/0xc2
[ 243.360586] IN-RECLAIM_FS-W at:
[ 243.361628]
[ 243.361630] [<ffffffff810ef934>] __lock_acquire+0x344/0x1bb0
[ 243.364273]
[ 243.364275] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.366710]
[ 243.366713] [<ffffffff810ea7e9>] down_write_nested+0x59/0xc0
[ 243.369153]
[ 243.369182] [<ffffffffa02a4b2e>] xfs_ilock+0x14e/0x290 [xfs]
[ 243.371597]
[ 243.371619] [<ffffffffa02986a5>] xfs_reclaim_inode+0x135/0x340 [xfs]
[ 243.374339]
[ 243.374366] [<ffffffffa0298b7a>] xfs_reclaim_inodes_ag+0x2ca/0x4f0 [xfs]
[ 243.377009]
[ 243.377032] [<ffffffffa029af9e>] xfs_reclaim_inodes_nr+0x2e/0x40 [xfs]
[ 243.379659]
[ 243.379686] [<ffffffffa02b32c4>] xfs_fs_free_cached_objects+0x14/0x20 [xfs]
[ 243.382349]
[ 243.382352] [<ffffffff81261dbc>] super_cache_scan+0x17c/0x190
[ 243.384907]
[ 243.384911] [<ffffffff811d375a>] shrink_slab+0x29a/0x710
[ 243.387690]
[ 243.387693] [<ffffffff811d876d>] shrink_node+0x23d/0x320
[ 243.390148]
[ 243.390150] [<ffffffff811d9e24>] kswapd+0x354/0xa10
[ 243.392517]
[ 243.392520] [<ffffffff810b5caa>] kthread+0x10a/0x140
[ 243.394851]
[ 243.394853] [<ffffffff81715081>] ret_from_fork+0x31/0x40
[ 243.397246] INITIAL USE at:
[ 243.398227]
[ 243.398229] [<ffffffff810ef960>] __lock_acquire+0x370/0x1bb0
[ 243.400646]
[ 243.400648] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.402997]
[ 243.402999] [<ffffffff810ea672>] down_read_nested+0x52/0xb0
[ 243.405351]
[ 243.405397] [<ffffffffa02a4af4>] xfs_ilock+0x114/0x290 [xfs]
[ 243.407778]
[ 243.407799] [<ffffffffa02a4c9b>] xfs_ilock_data_map_shared+0x2b/0x30 [xfs]
[ 243.410364]
[ 243.410390] [<ffffffffa02559f4>] xfs_dir_lookup+0xd4/0x1c0 [xfs]
[ 243.412989]
[ 243.413011] [<ffffffffa02a62ff>] xfs_lookup+0x7f/0x250 [xfs]
[ 243.415416]
[ 243.415437] [<ffffffffa02a1fcb>] xfs_vn_lookup+0x6b/0xb0 [xfs]
[ 243.417871]
[ 243.417874] [<ffffffff8126902e>] lookup_slow+0x12e/0x220
[ 243.420641]
[ 243.420644] [<ffffffff8126d2c6>] walk_component+0x1a6/0x2b0
[ 243.423039]
[ 243.423041] [<ffffffff8126d55c>] link_path_walk+0x18c/0x580
[ 243.425553]
[ 243.425555] [<ffffffff8126de41>] path_openat+0xa1/0xa90
[ 243.427891]
[ 243.427892] [<ffffffff8126f9ec>] do_filp_open+0x8c/0x100
[ 243.430249]
[ 243.430251] [<ffffffff8125c0ea>] do_sys_open+0x13a/0x200
[ 243.432586]
[ 243.432588] [<ffffffff8125c1c9>] SyS_open+0x19/0x20
[ 243.434839]
[ 243.434843] [<ffffffff81714e01>] entry_SYSCALL_64_fastpath+0x1f/0xc2
[ 243.437343] }
[ 243.438115] ... key at: [<ffffffffa031dfcc>] xfs_dir_ilock_class+0x0/0xfffffffffffc3f6e [xfs]
[ 243.440082] ... acquired at:
[ 243.441047]
[ 243.441049] [<ffffffff810ee7ea>] check_irq_usage+0x4a/0xb0
[ 243.443169]
[ 243.443171] [<ffffffff810f0954>] __lock_acquire+0x1364/0x1bb0
[ 243.445366]
[ 243.445368] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.447471]
[ 243.447474] [<ffffffff8108de18>] get_online_cpus+0x58/0x80
[ 243.449601]
[ 243.449604] [<ffffffff811bf3a7>] drain_all_pages.part.80+0x27/0x320
[ 243.452123]
[ 243.452125] [<ffffffff811c2039>] drain_all_pages+0x19/0x20
[ 243.454264]
[ 243.454266] [<ffffffff811c4854>] __alloc_pages_nodemask+0x784/0x1630
[ 243.456596]
[ 243.456599] [<ffffffff8122e1bf>] cache_grow_begin+0xcf/0x630
[ 243.458774]
[ 243.458776] [<ffffffff8122eb45>] fallback_alloc+0x1e5/0x290
[ 243.460952]
[ 243.460955] [<ffffffff8122e955>] ____cache_alloc_node+0x235/0x240
[ 243.463199]
[ 243.463201] [<ffffffff8122f30c>] kmem_cache_alloc+0x26c/0x3e0
[ 243.465482]
[ 243.465510] [<ffffffffa02b9211>] kmem_zone_alloc+0x91/0x120 [xfs]
[ 243.467754]
[ 243.467774] [<ffffffffa024e2f5>] xfs_da_state_alloc+0x15/0x20 [xfs]
[ 243.470083]
[ 243.470101] [<ffffffffa025f333>] xfs_dir2_node_lookup+0x53/0x2b0 [xfs]
[ 243.472427]
[ 243.472445] [<ffffffffa0255ac5>] xfs_dir_lookup+0x1a5/0x1c0 [xfs]
[ 243.474705]
[ 243.474726] [<ffffffffa02a62ff>] xfs_lookup+0x7f/0x250 [xfs]
[ 243.476933]
[ 243.476954] [<ffffffffa02a1fcb>] xfs_vn_lookup+0x6b/0xb0 [xfs]
[ 243.479178]
[ 243.479180] [<ffffffff8126ce2c>] lookup_open+0x54c/0x790
[ 243.481350]
[ 243.481352] [<ffffffff8126e2fa>] path_openat+0x55a/0xa90
[ 243.483907]
[ 243.483910] [<ffffffff8126f9ec>] do_filp_open+0x8c/0x100
[ 243.486070]
[ 243.486073] [<ffffffff8125c0ea>] do_sys_open+0x13a/0x200
[ 243.488334]
[ 243.488338] [<ffffffff8125c1c9>] SyS_open+0x19/0x20
[ 243.490476]
[ 243.490480] [<ffffffff81003c17>] do_syscall_64+0x67/0x1f0
[ 243.492619]
[ 243.492623] [<ffffffff81714ec9>] return_from_SYSCALL_64+0x0/0x7a
[ 243.494864]
[ 243.495618]
[ 243.495618] the dependencies between the lock to be acquired
[ 243.495619] and RECLAIM_FS-irq-unsafe lock:
[ 243.498973] -> (cpu_hotplug.dep_map){++++++} ops: 838 {
[ 243.500297] HARDIRQ-ON-W at:
[ 243.501292]
[ 243.501295] [<ffffffff810efd84>] __lock_acquire+0x794/0x1bb0
[ 243.503718]
[ 243.503719] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.506059]
[ 243.506061] [<ffffffff8108ff5e>] cpu_hotplug_begin+0x6e/0xe0
[ 243.508471]
[ 243.508473] [<ffffffff8109009d>] _cpu_up+0x2d/0xf0
[ 243.510708]
[ 243.510709] [<ffffffff810901e4>] do_cpu_up+0x84/0xd0
[ 243.512997]
[ 243.512999] [<ffffffff8109023e>] cpu_up+0xe/0x10
[ 243.515556]
[ 243.515561] [<ffffffff81f6f446>] smp_init+0xd5/0x141
[ 243.517807]
[ 243.517810] [<ffffffff81f3f35b>] kernel_init_freeable+0x17d/0x2a7
[ 243.520271]
[ 243.520275] [<ffffffff817048e9>] kernel_init+0x9/0x100
[ 243.522538]
[ 243.522540] [<ffffffff81715081>] ret_from_fork+0x31/0x40
[ 243.524833] HARDIRQ-ON-R at:
[ 243.525801]
[ 243.525803] [<ffffffff810ef8c0>] __lock_acquire+0x2d0/0x1bb0
[ 243.528152]
[ 243.528153] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.530416]
[ 243.530419] [<ffffffff8108de18>] get_online_cpus+0x58/0x80
[ 243.532696]
[ 243.532698] [<ffffffff811ec375>] kmem_cache_create+0x35/0x2d0
[ 243.535039]
[ 243.535041] [<ffffffff81f87d4a>] debug_objects_mem_init+0x48/0x5c5
[ 243.537451]
[ 243.537453] [<ffffffff81f3f108>] start_kernel+0x3ec/0x4c2
[ 243.539744]
[ 243.539746] [<ffffffff81f3e5d6>] x86_64_start_reservations+0x2a/0x2c
[ 243.542186]
[ 243.542188] [<ffffffff81f3e724>] x86_64_start_kernel+0x14c/0x16f
[ 243.544603]
[ 243.544605] [<ffffffff810001c4>] verify_cpu+0x0/0xfc
[ 243.547245] SOFTIRQ-ON-W at:
[ 243.548241]
[ 243.548243] [<ffffffff810ef8ed>] __lock_acquire+0x2fd/0x1bb0
[ 243.550559]
[ 243.550561] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.552841]
[ 243.552842] [<ffffffff8108ff5e>] cpu_hotplug_begin+0x6e/0xe0
[ 243.555186]
[ 243.555187] [<ffffffff8109009d>] _cpu_up+0x2d/0xf0
[ 243.557404]
[ 243.557405] [<ffffffff810901e4>] do_cpu_up+0x84/0xd0
[ 243.559654]
[ 243.559656] [<ffffffff8109023e>] cpu_up+0xe/0x10
[ 243.561824]
[ 243.561827] [<ffffffff81f6f446>] smp_init+0xd5/0x141
[ 243.564048]
[ 243.564050] [<ffffffff81f3f35b>] kernel_init_freeable+0x17d/0x2a7
[ 243.566455]
[ 243.566457] [<ffffffff817048e9>] kernel_init+0x9/0x100
[ 243.568731]
[ 243.568733] [<ffffffff81715081>] ret_from_fork+0x31/0x40
[ 243.571014] SOFTIRQ-ON-R at:
[ 243.571975]
[ 243.571976] [<ffffffff810ef8ed>] __lock_acquire+0x2fd/0x1bb0
[ 243.574328]
[ 243.574330] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.576610]
[ 243.576612] [<ffffffff8108de18>] get_online_cpus+0x58/0x80
[ 243.579161]
[ 243.579165] [<ffffffff811ec375>] kmem_cache_create+0x35/0x2d0
[ 243.581537]
[ 243.581539] [<ffffffff81f87d4a>] debug_objects_mem_init+0x48/0x5c5
[ 243.583982]
[ 243.583984] [<ffffffff81f3f108>] start_kernel+0x3ec/0x4c2
[ 243.586304]
[ 243.586306] [<ffffffff81f3e5d6>] x86_64_start_reservations+0x2a/0x2c
[ 243.588819]
[ 243.588821] [<ffffffff81f3e724>] x86_64_start_kernel+0x14c/0x16f
[ 243.591227]
[ 243.591229] [<ffffffff810001c4>] verify_cpu+0x0/0xfc
[ 243.593507] RECLAIM_FS-ON-W at:
[ 243.594519]
[ 243.594520] [<ffffffff810ef051>] mark_held_locks+0x71/0x90
[ 243.596888]
[ 243.596895] [<ffffffff810f3405>] lockdep_trace_alloc+0xc5/0x110
[ 243.599331]
[ 243.599334] [<ffffffff8122f8ca>] kmem_cache_alloc_node_trace+0x4a/0x410
[ 243.601872]
[ 243.601874] [<ffffffff810ba350>] __smpboot_create_thread.part.3+0x30/0xf0
[ 243.604460]
[ 243.604461] [<ffffffff810ba7a1>] smpboot_create_threads+0x61/0x90
[ 243.606950]
[ 243.606952] [<ffffffff8108e2cb>] cpuhp_invoke_callback+0xbb/0xb70
[ 243.609463]
[ 243.609465] [<ffffffff8108fc82>] cpuhp_up_callbacks+0x32/0xb0
[ 243.612282]
[ 243.612285] [<ffffffff810900f4>] _cpu_up+0x84/0xf0
[ 243.614604]
[ 243.614606] [<ffffffff810901e4>] do_cpu_up+0x84/0xd0
[ 243.616929]
[ 243.616930] [<ffffffff8109023e>] cpu_up+0xe/0x10
[ 243.619208]
[ 243.619211] [<ffffffff81f6f446>] smp_init+0xd5/0x141
[ 243.621518]
[ 243.621520] [<ffffffff81f3f35b>] kernel_init_freeable+0x17d/0x2a7
[ 243.624018]
[ 243.624020] [<ffffffff817048e9>] kernel_init+0x9/0x100
[ 243.626374]
[ 243.626376] [<ffffffff81715081>] ret_from_fork+0x31/0x40
[ 243.628771] RECLAIM_FS-ON-R at:
[ 243.629802]
[ 243.629803] [<ffffffff810ef051>] mark_held_locks+0x71/0x90
[ 243.632201]
[ 243.632203] [<ffffffff810f3405>] lockdep_trace_alloc+0xc5/0x110
[ 243.634692]
[ 243.634695] [<ffffffff8122f8ca>] kmem_cache_alloc_node_trace+0x4a/0x410
[ 243.637277]
[ 243.637279] [<ffffffff8100cbb4>] allocate_shared_regs+0x24/0x70
[ 243.639777]
[ 243.639779] [<ffffffff8100cc32>] intel_pmu_cpu_prepare+0x32/0x140
[ 243.643062]
[ 243.643066] [<ffffffff810053db>] x86_pmu_prepare_cpu+0x3b/0x40
[ 243.645553]
[ 243.645556] [<ffffffff8108e2cb>] cpuhp_invoke_callback+0xbb/0xb70
[ 243.648095]
[ 243.648097] [<ffffffff8108f29c>] cpuhp_issue_call+0xec/0x160
[ 243.650536]
[ 243.650539] [<ffffffff8108f6bb>] __cpuhp_setup_state+0x13b/0x1a0
[ 243.653126]
[ 243.653130] [<ffffffff81f427e9>] init_hw_perf_events+0x402/0x5b6
[ 243.655652]
[ 243.655655] [<ffffffff8100217c>] do_one_initcall+0x4c/0x1b0
[ 243.658127]
[ 243.658130] [<ffffffff81f3f333>] kernel_init_freeable+0x155/0x2a7
[ 243.660653]
[ 243.660656] [<ffffffff817048e9>] kernel_init+0x9/0x100
[ 243.663048]
[ 243.663050] [<ffffffff81715081>] ret_from_fork+0x31/0x40
[ 243.665436] INITIAL USE at:
[ 243.666403]
[ 243.666405] [<ffffffff810ef960>] __lock_acquire+0x370/0x1bb0
[ 243.668790]
[ 243.668791] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.671093]
[ 243.671095] [<ffffffff8108de18>] get_online_cpus+0x58/0x80
[ 243.673455]
[ 243.673458] [<ffffffff8108f5be>] __cpuhp_setup_state+0x3e/0x1a0
[ 243.676126]
[ 243.676130] [<ffffffff81f7660e>] page_alloc_init+0x23/0x3a
[ 243.678510]
[ 243.678512] [<ffffffff81f3eebe>] start_kernel+0x1a2/0x4c2
[ 243.680851]
[ 243.680853] [<ffffffff81f3e5d6>] x86_64_start_reservations+0x2a/0x2c
[ 243.683367]
[ 243.683369] [<ffffffff81f3e724>] x86_64_start_kernel+0x14c/0x16f
[ 243.685812]
[ 243.685815] [<ffffffff810001c4>] verify_cpu+0x0/0xfc
[ 243.688133] }
[ 243.688907] ... key at: [<ffffffff81c56848>] cpu_hotplug+0x108/0x140
[ 243.690542] ... acquired at:
[ 243.691514]
[ 243.691517] [<ffffffff810ee7ea>] check_irq_usage+0x4a/0xb0
[ 243.693655]
[ 243.693656] [<ffffffff810f0954>] __lock_acquire+0x1364/0x1bb0
[ 243.695820]
[ 243.695822] [<ffffffff810f1840>] lock_acquire+0xe0/0x2a0
[ 243.697926]
[ 243.697929] [<ffffffff8108de18>] get_online_cpus+0x58/0x80
[ 243.700042]
[ 243.700044] [<ffffffff811bf3a7>] drain_all_pages.part.80+0x27/0x320
[ 243.702285]
[ 243.702286] [<ffffffff811c2039>] drain_all_pages+0x19/0x20
[ 243.704405]
[ 243.704407] [<ffffffff811c4854>] __alloc_pages_nodemask+0x784/0x1630
[ 243.706721]
[ 243.706724] [<ffffffff8122e1bf>] cache_grow_begin+0xcf/0x630
[ 243.708867]
[ 243.708870] [<ffffffff8122eb45>] fallback_alloc+0x1e5/0x290
[ 243.711000]
[ 243.711002] [<ffffffff8122e955>] ____cache_alloc_node+0x235/0x240
[ 243.713211]
[ 243.713213] [<ffffffff8122f30c>] kmem_cache_alloc+0x26c/0x3e0
[ 243.715366]
[ 243.715410] [<ffffffffa02b9211>] kmem_zone_alloc+0x91/0x120 [xfs]
[ 243.717625]
[ 243.717644] [<ffffffffa024e2f5>] xfs_da_state_alloc+0x15/0x20 [xfs]
[ 243.719889]
[ 243.719918] [<ffffffffa025f333>] xfs_dir2_node_lookup+0x53/0x2b0 [xfs]
[ 243.722224]
[ 243.722242] [<ffffffffa0255ac5>] xfs_dir_lookup+0x1a5/0x1c0 [xfs]
[ 243.724493]
[ 243.724514] [<ffffffffa02a62ff>] xfs_lookup+0x7f/0x250 [xfs]
[ 243.726690]
[ 243.726710] [<ffffffffa02a1fcb>] xfs_vn_lookup+0x6b/0xb0 [xfs]
[ 243.728933]
[ 243.728936] [<ffffffff8126ce2c>] lookup_open+0x54c/0x790
[ 243.731064]
[ 243.731066] [<ffffffff8126e2fa>] path_openat+0x55a/0xa90
[ 243.733192]
[ 243.733194] [<ffffffff8126f9ec>] do_filp_open+0x8c/0x100
[ 243.735312]
[ 243.735315] [<ffffffff8125c0ea>] do_sys_open+0x13a/0x200
[ 243.737523]
[ 243.737527] [<ffffffff8125c1c9>] SyS_open+0x19/0x20
[ 243.739577]
[ 243.739579] [<ffffffff81003c17>] do_syscall_64+0x67/0x1f0
[ 243.741702]
[ 243.741706] [<ffffffff81714ec9>] return_from_SYSCALL_64+0x0/0x7a
[ 243.743932]
[ 243.744661]
[ 243.744661] stack backtrace:
[ 243.746302] CPU: 1 PID: 8767 Comm: awk Not tainted 4.10.0-rc6-next-20170202 #46
[ 243.747963] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
[ 243.750166] Call Trace:
[ 243.751071] dump_stack+0x85/0xc9
[ 243.752110] check_usage+0x4f9/0x680
[ 243.753188] check_irq_usage+0x4a/0xb0
[ 243.754280] __lock_acquire+0x1364/0x1bb0
[ 243.755410] lock_acquire+0xe0/0x2a0
[ 243.756467] ? get_online_cpus+0x32/0x80
[ 243.757580] get_online_cpus+0x58/0x80
[ 243.758664] ? get_online_cpus+0x32/0x80
[ 243.759764] drain_all_pages.part.80+0x27/0x320
[ 243.760972] drain_all_pages+0x19/0x20
[ 243.762039] __alloc_pages_nodemask+0x784/0x1630
[ 243.763249] ? rcu_read_lock_sched_held+0x91/0xa0
[ 243.764466] ? __alloc_pages_nodemask+0x2e6/0x1630
[ 243.765689] ? mark_held_locks+0x71/0x90
[ 243.766780] ? cache_grow_begin+0x4ac/0x630
[ 243.767912] cache_grow_begin+0xcf/0x630
[ 243.768985] ? ____cache_alloc_node+0x1bf/0x240
[ 243.770173] fallback_alloc+0x1e5/0x290
[ 243.771233] ____cache_alloc_node+0x235/0x240
[ 243.772403] ? kmem_zone_alloc+0x91/0x120 [xfs]
[ 243.773576] kmem_cache_alloc+0x26c/0x3e0
[ 243.774671] kmem_zone_alloc+0x91/0x120 [xfs]
[ 243.775816] xfs_da_state_alloc+0x15/0x20 [xfs]
[ 243.776989] xfs_dir2_node_lookup+0x53/0x2b0 [xfs]
[ 243.778188] xfs_dir_lookup+0x1a5/0x1c0 [xfs]
[ 243.779327] xfs_lookup+0x7f/0x250 [xfs]
[ 243.780394] xfs_vn_lookup+0x6b/0xb0 [xfs]
[ 243.781466] lookup_open+0x54c/0x790
[ 243.782440] path_openat+0x55a/0xa90
[ 243.783412] do_filp_open+0x8c/0x100
[ 243.784377] ? _raw_spin_unlock+0x22/0x30
[ 243.785418] ? __alloc_fd+0xf2/0x210
[ 243.786378] do_sys_open+0x13a/0x200
[ 243.787361] SyS_open+0x19/0x20
[ 243.788252] do_syscall_64+0x67/0x1f0
[ 243.789228] entry_SYSCALL64_slow_path+0x25/0x25
[ 243.790347] RIP: 0033:0x7fcf8dda06c7
[ 243.791299] RSP: 002b:00007ffd883327b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[ 243.792895] RAX: ffffffffffffffda RBX: 00007ffd883328a8 RCX: 00007fcf8dda06c7
[ 243.794424] RDX: 00007fcf8dfa9148 RSI: 0000000000080000 RDI: 00007fcf8dfa6b08
[ 243.795949] RBP: 00007ffd88332810 R08: 00007ffd88332890 R09: 0000000000000000
[ 243.797480] R10: 00007fcf8dfa6b08 R11: 0000000000000246 R12: 0000000000000000
[ 243.799002] R13: 0000000000000000 R14: 0000000000000000 R15: 00007ffd88332890
[ 253.543441] awk invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
[ 253.546121] awk cpuset=/ mems_allowed=0
[ 253.547233] CPU: 3 PID: 8767 Comm: awk Not tainted 4.10.0-rc6-next-20170202 #46