Re: printk() + memory offline deadlock (WAS Re: page_alloc.shuffle=1 + CONFIG_PROVE_LOCKING=y = arm64 hang)

From: Qian Cai
Date: Mon Sep 23 2019 - 09:09:27 EST

Next message: Anup Patel: "Re: [PATCH v7 10/21] RISC-V: KVM: Handle MMIO exits for VCPU"
Previous message: Daniel Vetter: "Re: [PATCH 01/36] drm/fourcc: Add 2 plane YCbCr 10bit format support"
In reply to: Sergey Senozhatsky: "Re: printk() + memory offline deadlock (WAS Re: page_alloc.shuffle=1 + CONFIG_PROVE_LOCKING=y = arm64 hang)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Mon, 2019-09-23 at 19:21 +0900, Sergey Senozhatsky wrote:
> On (09/18/19 12:10), Qian Cai wrote:
> [..]
> > > So you have debug objects enabled. Right? This thing does not behave
> > > when it comes to printing. debug_objects are slightly problematic.
> >
> > Yes, but there is an also a similar splat without the debug_objects. It looks
> > like anything try to allocate memory in that path will trigger it anyway.
>
> Appears to be different, yet somehow very familiar.
>
> > [ÂÂ297.425908] WARNING: possible circular locking dependency detected
> > [ÂÂ297.425908] 5.3.0-next-20190917 #8 Not tainted
> > [ÂÂ297.425909] ------------------------------------------------------
> > [ÂÂ297.425910] test.sh/8653 is trying to acquire lock:
> > [ÂÂ297.425911] ffffffff865a4460 (console_owner){-.-.}, at:
> > console_unlock+0x207/0x750
> >
> > [ÂÂ297.425914] but task is already holding lock:
> > [ÂÂ297.425915] ffff88883fff3c58 (&(&zone->lock)->rlock){-.-.}, at:
> > __offline_isolated_pages+0x179/0x3e0
> >
> > [ÂÂ297.425919] which lock already depends on the new lock.
> >
> >
> > [ÂÂ297.425920] the existing dependency chain (in reverse order) is:
> >
> > [ÂÂ297.425922] -> #3 (&(&zone->lock)->rlock){-.-.}:
> > [ÂÂ297.425925]ÂÂÂÂÂÂÂÂ__lock_acquire+0x5b3/0xb40
> > [ÂÂ297.425925]ÂÂÂÂÂÂÂÂlock_acquire+0x126/0x280
> > [ÂÂ297.425926]ÂÂÂÂÂÂÂÂ_raw_spin_lock+0x2f/0x40
> > [ÂÂ297.425927]ÂÂÂÂÂÂÂÂrmqueue_bulk.constprop.21+0xb6/0x1160
> > [ÂÂ297.425928]ÂÂÂÂÂÂÂÂget_page_from_freelist+0x898/0x22c0
> > [ÂÂ297.425928]ÂÂÂÂÂÂÂÂ__alloc_pages_nodemask+0x2f3/0x1cd0
> > [ÂÂ297.425929]ÂÂÂÂÂÂÂÂalloc_pages_current+0x9c/0x110
> > [ÂÂ297.425930]ÂÂÂÂÂÂÂÂallocate_slab+0x4c6/0x19c0
> > [ÂÂ297.425931]ÂÂÂÂÂÂÂÂnew_slab+0x46/0x70
> > [ÂÂ297.425931]ÂÂÂÂÂÂÂÂ___slab_alloc+0x58b/0x960
> > [ÂÂ297.425932]ÂÂÂÂÂÂÂÂ__slab_alloc+0x43/0x70
> > [ÂÂ297.425933]ÂÂÂÂÂÂÂÂ__kmalloc+0x3ad/0x4b0
> > [ÂÂ297.425933]ÂÂÂÂÂÂÂÂ__tty_buffer_request_room+0x100/0x250
> > [ÂÂ297.425934]ÂÂÂÂÂÂÂÂtty_insert_flip_string_fixed_flag+0x67/0x110
> > [ÂÂ297.425935]ÂÂÂÂÂÂÂÂpty_write+0xa2/0xf0
> > [ÂÂ297.425936]ÂÂÂÂÂÂÂÂn_tty_write+0x36b/0x7b0
> > [ÂÂ297.425936]ÂÂÂÂÂÂÂÂtty_write+0x284/0x4c0
> > [ÂÂ297.425937]ÂÂÂÂÂÂÂÂ__vfs_write+0x50/0xa0
> > [ÂÂ297.425938]ÂÂÂÂÂÂÂÂvfs_write+0x105/0x290
> > [ÂÂ297.425939]ÂÂÂÂÂÂÂÂredirected_tty_write+0x6a/0xc0
> > [ÂÂ297.425939]ÂÂÂÂÂÂÂÂdo_iter_write+0x248/0x2a0
> > [ÂÂ297.425940]ÂÂÂÂÂÂÂÂvfs_writev+0x106/0x1e0
> > [ÂÂ297.425941]ÂÂÂÂÂÂÂÂdo_writev+0xd4/0x180
> > [ÂÂ297.425941]ÂÂÂÂÂÂÂÂ__x64_sys_writev+0x45/0x50
> > [ÂÂ297.425942]ÂÂÂÂÂÂÂÂdo_syscall_64+0xcc/0x76c
> > [ÂÂ297.425943]ÂÂÂÂÂÂÂÂentry_SYSCALL_64_after_hwframe+0x49/0xbe
> >
> > [ÂÂ297.425944] -> #2 (&(&port->lock)->rlock){-.-.}:
> > [ÂÂ297.425946]ÂÂÂÂÂÂÂÂ__lock_acquire+0x5b3/0xb40
> > [ÂÂ297.425947]ÂÂÂÂÂÂÂÂlock_acquire+0x126/0x280
> > [ÂÂ297.425948]ÂÂÂÂÂÂÂÂ_raw_spin_lock_irqsave+0x3a/0x50
> > [ÂÂ297.425949]ÂÂÂÂÂÂÂÂtty_port_tty_get+0x20/0x60
> > [ÂÂ297.425949]ÂÂÂÂÂÂÂÂtty_port_default_wakeup+0xf/0x30
> > [ÂÂ297.425950]ÂÂÂÂÂÂÂÂtty_port_tty_wakeup+0x39/0x40
> > [ÂÂ297.425951]ÂÂÂÂÂÂÂÂuart_write_wakeup+0x2a/0x40
> > [ÂÂ297.425952]ÂÂÂÂÂÂÂÂserial8250_tx_chars+0x22e/0x440
> > [ÂÂ297.425952]ÂÂÂÂÂÂÂÂserial8250_handle_irq.part.8+0x14a/0x170
> > [ÂÂ297.425953]ÂÂÂÂÂÂÂÂserial8250_default_handle_irq+0x5c/0x90
> > [ÂÂ297.425954]ÂÂÂÂÂÂÂÂserial8250_interrupt+0xa6/0x130
> > [ÂÂ297.425955]ÂÂÂÂÂÂÂÂ__handle_irq_event_percpu+0x78/0x4f0
> > [ÂÂ297.425955]ÂÂÂÂÂÂÂÂhandle_irq_event_percpu+0x70/0x100
> > [ÂÂ297.425956]ÂÂÂÂÂÂÂÂhandle_irq_event+0x5a/0x8b
> > [ÂÂ297.425957]ÂÂÂÂÂÂÂÂhandle_edge_irq+0x117/0x370
> > [ÂÂ297.425958]ÂÂÂÂÂÂÂÂdo_IRQ+0x9e/0x1e0
> > [ÂÂ297.425958]ÂÂÂÂÂÂÂÂret_from_intr+0x0/0x2a
> > [ÂÂ297.425959]ÂÂÂÂÂÂÂÂcpuidle_enter_state+0x156/0x8e0
> > [ÂÂ297.425960]ÂÂÂÂÂÂÂÂcpuidle_enter+0x41/0x70
> > [ÂÂ297.425960]ÂÂÂÂÂÂÂÂcall_cpuidle+0x5e/0x90
> > [ÂÂ297.425961]ÂÂÂÂÂÂÂÂdo_idle+0x333/0x370
> > [ÂÂ297.425962]ÂÂÂÂÂÂÂÂcpu_startup_entry+0x1d/0x1f
> > [ÂÂ297.425962]ÂÂÂÂÂÂÂÂstart_secondary+0x290/0x330
> > [ÂÂ297.425963]ÂÂÂÂÂÂÂÂsecondary_startup_64+0xb6/0xc0
> >
> > [ÂÂ297.425964] -> #1 (&port_lock_key){-.-.}:
> > [ÂÂ297.425967]ÂÂÂÂÂÂÂÂ__lock_acquire+0x5b3/0xb40
> > [ÂÂ297.425967]ÂÂÂÂÂÂÂÂlock_acquire+0x126/0x280
> > [ÂÂ297.425968]ÂÂÂÂÂÂÂÂ_raw_spin_lock_irqsave+0x3a/0x50
> > [ÂÂ297.425969]ÂÂÂÂÂÂÂÂserial8250_console_write+0x3e4/0x450
> > [ÂÂ297.425970]ÂÂÂÂÂÂÂÂuniv8250_console_write+0x4b/0x60
> > [ÂÂ297.425970]ÂÂÂÂÂÂÂÂconsole_unlock+0x501/0x750
> > [ÂÂ297.425971]ÂÂÂÂÂÂÂÂvprintk_emit+0x10d/0x340
> > [ÂÂ297.425972]ÂÂÂÂÂÂÂÂvprintk_default+0x1f/0x30
> > [ÂÂ297.425972]ÂÂÂÂÂÂÂÂvprintk_func+0x44/0xd4
> > [ÂÂ297.425973]ÂÂÂÂÂÂÂÂprintk+0x9f/0xc5
> > [ÂÂ297.425974]ÂÂÂÂÂÂÂÂregister_console+0x39c/0x520
> > [ÂÂ297.425975]ÂÂÂÂÂÂÂÂuniv8250_console_init+0x23/0x2d
> > [ÂÂ297.425975]ÂÂÂÂÂÂÂÂconsole_init+0x338/0x4cd
> > [ÂÂ297.425976]ÂÂÂÂÂÂÂÂstart_kernel+0x534/0x724
> > [ÂÂ297.425977]ÂÂÂÂÂÂÂÂx86_64_start_reservations+0x24/0x26
> > [ÂÂ297.425977]ÂÂÂÂÂÂÂÂx86_64_start_kernel+0xf4/0xfb
> > [ÂÂ297.425978]ÂÂÂÂÂÂÂÂsecondary_startup_64+0xb6/0xc0
> >
> > [ÂÂ297.425979] -> #0 (console_owner){-.-.}:
> > [ÂÂ297.425982]ÂÂÂÂÂÂÂÂcheck_prev_add+0x107/0xea0
> > [ÂÂ297.425982]ÂÂÂÂÂÂÂÂvalidate_chain+0x8fc/0x1200
> > [ÂÂ297.425983]ÂÂÂÂÂÂÂÂ__lock_acquire+0x5b3/0xb40
> > [ÂÂ297.425984]ÂÂÂÂÂÂÂÂlock_acquire+0x126/0x280
> > [ÂÂ297.425984]ÂÂÂÂÂÂÂÂconsole_unlock+0x269/0x750
> > [ÂÂ297.425985]ÂÂÂÂÂÂÂÂvprintk_emit+0x10d/0x340
> > [ÂÂ297.425986]ÂÂÂÂÂÂÂÂvprintk_default+0x1f/0x30
> > [ÂÂ297.425987]ÂÂÂÂÂÂÂÂvprintk_func+0x44/0xd4
> > [ÂÂ297.425987]ÂÂÂÂÂÂÂÂprintk+0x9f/0xc5
> > [ÂÂ297.425988]ÂÂÂÂÂÂÂÂ__offline_isolated_pages.cold.52+0x2f/0x30a
> > [ÂÂ297.425989]ÂÂÂÂÂÂÂÂoffline_isolated_pages_cb+0x17/0x30
> > [ÂÂ297.425990]ÂÂÂÂÂÂÂÂwalk_system_ram_range+0xda/0x160
> > [ÂÂ297.425990]ÂÂÂÂÂÂÂÂ__offline_pages+0x79c/0xa10
> > [ÂÂ297.425991]ÂÂÂÂÂÂÂÂoffline_pages+0x11/0x20
> > [ÂÂ297.425992]ÂÂÂÂÂÂÂÂmemory_subsys_offline+0x7e/0xc0
> > [ÂÂ297.425992]ÂÂÂÂÂÂÂÂdevice_offline+0xd5/0x110
> > [ÂÂ297.425993]ÂÂÂÂÂÂÂÂstate_store+0xc6/0xe0
> > [ÂÂ297.425994]ÂÂÂÂÂÂÂÂdev_attr_store+0x3f/0x60
> > [ÂÂ297.425995]ÂÂÂÂÂÂÂÂsysfs_kf_write+0x89/0xb0
> > [ÂÂ297.425995]ÂÂÂÂÂÂÂÂkernfs_fop_write+0x188/0x240
> > [ÂÂ297.425996]ÂÂÂÂÂÂÂÂ__vfs_write+0x50/0xa0
> > [ÂÂ297.425997]ÂÂÂÂÂÂÂÂvfs_write+0x105/0x290
> > [ÂÂ297.425997]ÂÂÂÂÂÂÂÂksys_write+0xc6/0x160
> > [ÂÂ297.425998]ÂÂÂÂÂÂÂÂ__x64_sys_write+0x43/0x50
> > [ÂÂ297.425999]ÂÂÂÂÂÂÂÂdo_syscall_64+0xcc/0x76c
> > [ÂÂ297.426000]ÂÂÂÂÂÂÂÂentry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> I suppose you run with CONFIG_DEBUG_VM...
>
> So we have
>
> port->lock -> MM -> zone->lock
> // from pty_write()->__tty_buffer_request_room()->kmalloc()
>
> vs
>
> zone->lock -> printk() -> port->lock
> // from __offline_pages()->__offline_isolated_pages()->printk()
>
>
> A number of debugging options make the kernel less stable.
> Sad but true.

I am afraid it does not matter, as it still trigger a splat without either
CONFIG_DEBUG_VM and debug objects because offline_pages() will still call
printk() while holding zone->lock in a different path.

offline_pages()
start_isolate_page_range()
set_migratetype_isolate()
has_unmovable_pages()
dump_page()
printk()

[ÂÂ377.525562] WARNING: possible circular locking dependency detected
[ÂÂ377.525563] 5.3.0-next-20190920+ #9 Not tainted
[ÂÂ377.525564] ------------------------------------------------------
[ÂÂ377.525565] test.sh/8876 is trying to acquire lock:
[ÂÂ377.525565] ffffffffa2da1b80 (console_owner){-.-.}, at:
console_unlock+0x207/0x750

[ÂÂ377.525569] but task is already holding lock:
[ÂÂ377.525569] ffff88883fff4318 (&(&zone->lock)->rlock){-.-.}, at:
start_isolate_page_range+0x1ea/0x540

[ÂÂ377.525573] which lock already depends on the new lock.

[ÂÂ377.525575] the existing dependency chain (in reverse order) is:

[ÂÂ377.525576] -> #3 (&(&zone->lock)->rlock){-.-.}:
[ÂÂ377.525579]ÂÂÂÂÂÂÂÂ__lock_acquire+0x5b3/0xb40
[ÂÂ377.525579]ÂÂÂÂÂÂÂÂlock_acquire+0x126/0x280
[ÂÂ377.525580]ÂÂÂÂÂÂÂÂ_raw_spin_lock+0x2f/0x40
[ÂÂ377.525581]ÂÂÂÂÂÂÂÂrmqueue_bulk.constprop.21+0xb6/0xf70
[ÂÂ377.525582]ÂÂÂÂÂÂÂÂget_page_from_freelist+0x89a/0x20d0
[ÂÂ377.525582]ÂÂÂÂÂÂÂÂ__alloc_pages_nodemask+0x2b1/0x1c80
[ÂÂ377.525583]ÂÂÂÂÂÂÂÂalloc_pages_current+0x9c/0x110
[ÂÂ377.525584]ÂÂÂÂÂÂÂÂallocate_slab+0xa8f/0x16d0
[ÂÂ377.525585]ÂÂÂÂÂÂÂÂnew_slab+0x46/0x70
[ÂÂ377.525586]ÂÂÂÂÂÂÂÂ___slab_alloc+0x424/0x630
[ÂÂ377.525586]ÂÂÂÂÂÂÂÂ__slab_alloc+0x43/0x70
[ÂÂ377.525587]ÂÂÂÂÂÂÂÂ__kmalloc+0x3e3/0x490
[ÂÂ377.525588]ÂÂÂÂÂÂÂÂ__tty_buffer_request_room+0x100/0x250
[ÂÂ377.525589]ÂÂÂÂÂÂÂÂtty_insert_flip_string_fixed_flag+0x67/0x110
[ÂÂ377.525590]ÂÂÂÂÂÂÂÂpty_write+0xa2/0xf0
[ÂÂ377.525591]ÂÂÂÂÂÂÂÂn_tty_write+0x36b/0x7b0
[ÂÂ377.525592]ÂÂÂÂÂÂÂÂtty_write+0x284/0x4c0
[ÂÂ377.525593]ÂÂÂÂÂÂÂÂ__vfs_write+0x50/0xa0
[ÂÂ377.525593]ÂÂÂÂÂÂÂÂvfs_write+0x105/0x290
[ÂÂ377.525595]ÂÂÂÂÂÂÂÂksys_write+0xc6/0x160
[ÂÂ377.525596]ÂÂÂÂÂÂÂÂ__x64_sys_write+0x43/0x50
[ÂÂ377.525596]ÂÂÂÂÂÂÂÂdo_syscall_64+0xcc/0x76c
[ÂÂ377.525597]ÂÂÂÂÂÂÂÂentry_SYSCALL_64_after_hwframe+0x49/0xbe

[ÂÂ377.525598] -> #2 (&(&port->lock)->rlock){-.-.}:
[ÂÂ377.525601]ÂÂÂÂÂÂÂÂ__lock_acquire+0x5b3/0xb40
[ÂÂ377.525601]ÂÂÂÂÂÂÂÂlock_acquire+0x126/0x280
[ÂÂ377.525602]ÂÂÂÂÂÂÂÂ_raw_spin_lock_irqsave+0x3a/0x50
[ÂÂ377.525603]ÂÂÂÂÂÂÂÂtty_port_tty_get+0x20/0x60
[ÂÂ377.525604]ÂÂÂÂÂÂÂÂtty_port_default_wakeup+0xf/0x30
[ÂÂ377.525605]ÂÂÂÂÂÂÂÂtty_port_tty_wakeup+0x39/0x40
[ÂÂ377.525606]ÂÂÂÂÂÂÂÂuart_write_wakeup+0x2a/0x40
[ÂÂ377.525607]ÂÂÂÂÂÂÂÂserial8250_tx_chars+0x22e/0x440
[ÂÂ377.525607]ÂÂÂÂÂÂÂÂserial8250_handle_irq.part.8+0x14a/0x170
[ÂÂ377.525608]ÂÂÂÂÂÂÂÂserial8250_default_handle_irq+0x5c/0x90
[ÂÂ377.525609]ÂÂÂÂÂÂÂÂserial8250_interrupt+0xa6/0x130
[ÂÂ377.525610]ÂÂÂÂÂÂÂÂ__handle_irq_event_percpu+0x78/0x4f0
[ÂÂ377.525611]ÂÂÂÂÂÂÂÂhandle_irq_event_percpu+0x70/0x100
[ÂÂ377.525612]ÂÂÂÂÂÂÂÂhandle_irq_event+0x5a/0x8b
[ÂÂ377.525613]ÂÂÂÂÂÂÂÂhandle_edge_irq+0x117/0x370
[ÂÂ377.525614]ÂÂÂÂÂÂÂÂdo_IRQ+0x9e/0x1e0
[ÂÂ377.525614]ÂÂÂÂÂÂÂÂret_from_intr+0x0/0x2a
[ÂÂ377.525615]ÂÂÂÂÂÂÂÂcpuidle_enter_state+0x156/0x8e0
[ÂÂ377.525616]ÂÂÂÂÂÂÂÂcpuidle_enter+0x41/0x70
[ÂÂ377.525616]ÂÂÂÂÂÂÂÂcall_cpuidle+0x5e/0x90
[ÂÂ377.525617]ÂÂÂÂÂÂÂÂdo_idle+0x333/0x370
[ÂÂ377.525618]ÂÂÂÂÂÂÂÂcpu_startup_entry+0x1d/0x1f
[ÂÂ377.525619]ÂÂÂÂÂÂÂÂstart_secondary+0x290/0x330
[ÂÂ377.525619]ÂÂÂÂÂÂÂÂsecondary_startup_64+0xb6/0xc0

[ÂÂ377.525620] -> #1 (&port_lock_key){-.-.}:
[ÂÂ377.525623]ÂÂÂÂÂÂÂÂ__lock_acquire+0x5b3/0xb40
[ÂÂ377.525623]ÂÂÂÂÂÂÂÂlock_acquire+0x126/0x280
[ÂÂ377.525624]ÂÂÂÂÂÂÂÂ_raw_spin_lock_irqsave+0x3a/0x50
[ÂÂ377.525625]ÂÂÂÂÂÂÂÂserial8250_console_write+0x3e4/0x450
[ÂÂ377.525626]ÂÂÂÂÂÂÂÂuniv8250_console_write+0x4b/0x60
[ÂÂ377.525627]ÂÂÂÂÂÂÂÂconsole_unlock+0x501/0x750
[ÂÂ377.525627]ÂÂÂÂÂÂÂÂvprintk_emit+0x10d/0x340
[ÂÂ377.525628]ÂÂÂÂÂÂÂÂvprintk_default+0x1f/0x30
[ÂÂ377.525629]ÂÂÂÂÂÂÂÂvprintk_func+0x44/0xd4
[ÂÂ377.525629]ÂÂÂÂÂÂÂÂprintk+0x9f/0xc5
[ÂÂ377.525630]ÂÂÂÂÂÂÂÂregister_console+0x39c/0x520
[ÂÂ377.525631]ÂÂÂÂÂÂÂÂuniv8250_console_init+0x23/0x2d
[ÂÂ377.525632]ÂÂÂÂÂÂÂÂconsole_init+0x338/0x4cd
[ÂÂ377.525632]ÂÂÂÂÂÂÂÂstart_kernel+0x52a/0x71a
[ÂÂ377.525633]ÂÂÂÂÂÂÂÂx86_64_start_reservations+0x24/0x26
[ÂÂ377.525634]ÂÂÂÂÂÂÂÂx86_64_start_kernel+0xf4/0xfb
[ÂÂ377.525635]ÂÂÂÂÂÂÂÂsecondary_startup_64+0xb6/0xc0

[ÂÂ377.525636] -> #0 (console_owner){-.-.}:
[ÂÂ377.525638]ÂÂÂÂÂÂÂÂcheck_prev_add+0x107/0xea0
[ÂÂ377.525639]ÂÂÂÂÂÂÂÂvalidate_chain+0x8fc/0x1200
[ÂÂ377.525640]ÂÂÂÂÂÂÂÂ__lock_acquire+0x5b3/0xb40
[ÂÂ377.525641]ÂÂÂÂÂÂÂÂlock_acquire+0x126/0x280
[ÂÂ377.525641]ÂÂÂÂÂÂÂÂconsole_unlock+0x269/0x750
[ÂÂ377.525642]ÂÂÂÂÂÂÂÂvprintk_emit+0x10d/0x340
[ÂÂ377.525643]ÂÂÂÂÂÂÂÂvprintk_default+0x1f/0x30
[ÂÂ377.525643]ÂÂÂÂÂÂÂÂvprintk_func+0x44/0xd4
[ÂÂ377.525644]ÂÂÂÂÂÂÂÂprintk+0x9f/0xc5
[ÂÂ377.525645]ÂÂÂÂÂÂÂÂ__dump_page.cold.0+0x73/0x20a
[ÂÂ377.525645]ÂÂÂÂÂÂÂÂdump_page+0x12/0x46
[ÂÂ377.525646]ÂÂÂÂÂÂÂÂhas_unmovable_pages+0x2ff/0x360
[ÂÂ377.525647]ÂÂÂÂÂÂÂÂstart_isolate_page_range+0x3bb/0x540
[ÂÂ377.525648]ÂÂÂÂÂÂÂÂ__offline_pages+0x258/0xfc0
[ÂÂ377.525649]ÂÂÂÂÂÂÂÂoffline_pages+0x11/0x20
[ÂÂ377.525649]ÂÂÂÂÂÂÂÂmemory_subsys_offline+0x7e/0xc0
[ÂÂ377.525650]ÂÂÂÂÂÂÂÂdevice_offline+0xd5/0x110
[ÂÂ377.525651]ÂÂÂÂÂÂÂÂstate_store+0xc6/0xe0
[ÂÂ377.525651]ÂÂÂÂÂÂÂÂdev_attr_store+0x3f/0x60
[ÂÂ377.525652]ÂÂÂÂÂÂÂÂsysfs_kf_write+0x89/0xb0
[ÂÂ377.525653]ÂÂÂÂÂÂÂÂkernfs_fop_write+0x188/0x240
[ÂÂ377.525654]ÂÂÂÂÂÂÂÂ__vfs_write+0x50/0xa0
[ÂÂ377.525654]ÂÂÂÂÂÂÂÂvfs_write+0x105/0x290
[ÂÂ377.525655]ÂÂÂÂÂÂÂÂksys_write+0xc6/0x160
[ÂÂ377.525656]ÂÂÂÂÂÂÂÂ__x64_sys_write+0x43/0x50
[ÂÂ377.525656]ÂÂÂÂÂÂÂÂdo_syscall_64+0xcc/0x76c
[ÂÂ377.525657]ÂÂÂÂÂÂÂÂentry_SYSCALL_64_after_hwframe+0x49/0xbe

[ÂÂ377.525658] other info that might help us debug this:

[ÂÂ377.525660] Chain exists of:
[ÂÂ377.525660]ÂÂÂconsole_owner --> &(&port->lock)->rlock --> &(&zone->lock)-
>rlock

[ÂÂ377.525664]ÂÂPossible unsafe locking scenario:

[ÂÂ377.525665]ÂÂÂÂÂÂÂÂCPU0ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂCPU1
[ÂÂ377.525666]ÂÂÂÂÂÂÂÂ----ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ----
[ÂÂ377.525667]ÂÂÂlock(&(&zone->lock)->rlock);
[ÂÂ377.525669]ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂlock(&(&port->lock)->rlock);
[ÂÂ377.525671]ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂlock(&(&zone->lock)->rlock);
[ÂÂ377.525672]ÂÂÂlock(console_owner);

[ÂÂ377.525675]ÂÂ*** DEADLOCK ***

[ÂÂ377.525676] 9 locks held by test.sh/8876:
[ÂÂ377.525677]ÂÂ#0: ffff8883bca59408 (sb_writers#4){.+.+}, at:
vfs_write+0x25f/0x290
[ÂÂ377.525681]ÂÂ#1: ffff888eefb58880 (&of->mutex){+.+.}, at:
kernfs_fop_write+0x128/0x240
[ÂÂ377.525684]ÂÂ#2: ffff889feb6dcc40 (kn->count#115){.+.+}, at:
kernfs_fop_write+0x138/0x240
[ÂÂ377.525687]ÂÂ#3: ffffffffa30f7e60 (device_hotplug_lock){+.+.}, at:
lock_device_hotplug_sysfs+0x16/0x50
[ÂÂ377.525690]ÂÂ#4: ffff88981f55c990 (&dev->mutex){....}, at:
device_offline+0x70/0x110
[ÂÂ377.525693]ÂÂ#5: ffffffffa2d12d70 (cpu_hotplug_lock.rw_sem){++++}, at:
__offline_pages+0xdb/0xfc0
[ÂÂ377.525696]ÂÂ#6: ffffffffa2f007b0 (mem_hotplug_lock.rw_sem){++++}, at:
percpu_down_write+0x87/0x2f0
[ÂÂ377.525699]ÂÂ#7: ffff88883fff4318 (&(&zone->lock)->rlock){-.-.}, at:
start_isolate_page_range+0x1ea/0x540
[ÂÂ377.525703]ÂÂ#8: ffffffffa2da2040 (console_lock){+.+.}, at:
vprintk_emit+0x100/0x340
[ÂÂ377.525706] stack backtrace:
[ÂÂ377.525707] CPU: 10 PID: 8876 Comm: test.sh Not tainted 5.3.0-next-20190920+
#9
[ÂÂ377.525708] Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10,
BIOS U34 05/21/2019
[ÂÂ377.525709] Call Trace:
[ÂÂ377.525710]ÂÂdump_stack+0x86/0xca
[ÂÂ377.525710]ÂÂprint_circular_bug.cold.31+0x243/0x26e
[ÂÂ377.525711]ÂÂcheck_noncircular+0x29e/0x2e0
[ÂÂ377.525712]ÂÂ? stack_trace_save+0x87/0xb0
[ÂÂ377.525712]ÂÂ? print_circular_bug+0x120/0x120
[ÂÂ377.525713]ÂÂcheck_prev_add+0x107/0xea0
[ÂÂ377.525714]ÂÂvalidate_chain+0x8fc/0x1200
[ÂÂ377.525714]ÂÂ? check_prev_add+0xea0/0xea0
[ÂÂ377.525715]ÂÂ__lock_acquire+0x5b3/0xb40
[ÂÂ377.525716]ÂÂlock_acquire+0x126/0x280
[ÂÂ377.525717]ÂÂ? console_unlock+0x207/0x750
[ÂÂ377.525717]ÂÂ? __kasan_check_read+0x11/0x20
[ÂÂ377.525718]ÂÂconsole_unlock+0x269/0x750
[ÂÂ377.525719]ÂÂ? console_unlock+0x207/0x750
[ÂÂ377.525719]ÂÂvprintk_emit+0x10d/0x340
[ÂÂ377.525720]ÂÂvprintk_default+0x1f/0x30
[ÂÂ377.525721]ÂÂvprintk_func+0x44/0xd4
[ÂÂ377.525721]ÂÂprintk+0x9f/0xc5
[ÂÂ377.525722]ÂÂ? kmsg_dump_rewind_nolock+0x64/0x64
[ÂÂ377.525723]ÂÂ? lockdep_hardirqs_off+0x74/0x140
[ÂÂ377.525723]ÂÂ__dump_page.cold.0+0x73/0x20a
[ÂÂ377.525724]ÂÂdump_page+0x12/0x46
[ÂÂ377.525725]ÂÂhas_unmovable_pages+0x2ff/0x360
[ÂÂ377.525725]ÂÂstart_isolate_page_range+0x3bb/0x540
[ÂÂ377.525726]ÂÂ? unset_migratetype_isolate+0x260/0x260
[ÂÂ377.525727]ÂÂ? rcu_read_lock_bh_held+0xc0/0xc0
[ÂÂ377.525728]ÂÂ__offline_pages+0x258/0xfc0
[ÂÂ377.525728]ÂÂ? __lock_acquire+0x670/0xb40
[ÂÂ377.525729]ÂÂ? __add_memory+0xc0/0xc0
[ÂÂ377.525730]ÂÂ? lock_acquire+0x126/0x280
[ÂÂ377.525730]ÂÂ? device_offline+0x70/0x110
[ÂÂ377.525731]ÂÂ? __kasan_check_write+0x14/0x20
[ÂÂ377.525732]ÂÂ? __mutex_lock+0x344/0xcd0
[ÂÂ377.525732]ÂÂ? _raw_spin_unlock_irqrestore+0x49/0x50
[ÂÂ377.525733]ÂÂ? device_offline+0x70/0x110
[ÂÂ377.525734]ÂÂ? klist_next+0x1c1/0x1e0
[ÂÂ377.525735]ÂÂ? __mutex_add_waiter+0xc0/0xc0
[ÂÂ377.525735]ÂÂ? __device_link_free_srcu+0x80/0x80
[ÂÂ377.525736]ÂÂ? klist_next+0x10b/0x1e0
[ÂÂ377.525737]ÂÂ? klist_iter_exit+0x16/0x40
[ÂÂ377.525737]ÂÂ? device_for_each_child+0xd0/0x110
[ÂÂ377.525738]ÂÂoffline_pages+0x11/0x20
[ÂÂ377.525739]ÂÂmemory_subsys_offline+0x7e/0xc0
[ÂÂ377.525739]ÂÂdevice_offline+0xd5/0x110
[ÂÂ377.525740]ÂÂ? auto_online_blocks_show+0x70/0x70
[ÂÂ377.525741]ÂÂstate_store+0xc6/0xe0
[ÂÂ377.525741]ÂÂdev_attr_store+0x3f/0x60
[ÂÂ377.525742]ÂÂ? device_match_name+0x40/0x40
[ÂÂ377.525743]ÂÂsysfs_kf_write+0x89/0xb0
[ÂÂ377.525744]ÂÂ? sysfs_file_ops+0xa0/0xa0
[ÂÂ377.525745]ÂÂkernfs_fop_write+0x188/0x240
[ÂÂ377.525745]ÂÂ__vfs_write+0x50/0xa0
[ÂÂ377.525746]ÂÂvfs_write+0x105/0x290
[ÂÂ377.525747]ÂÂksys_write+0xc6/0x160
[ÂÂ377.525748]ÂÂ? __x64_sys_read+0x50/0x50
[ÂÂ377.525748]ÂÂ? do_syscall_64+0x79/0x76c
[ÂÂ377.525749]ÂÂ? do_syscall_64+0x79/0x76c
[ÂÂ377.525750]ÂÂ__x64_sys_write+0x43/0x50
[ÂÂ377.525751]ÂÂdo_syscall_64+0xcc/0x76c
[ÂÂ377.525752]ÂÂ? trace_hardirqs_on_thunk+0x1a/0x20
[ÂÂ377.525753]ÂÂ? syscall_return_slowpath+0x210/0x210
[ÂÂ377.525754]ÂÂ? entry_SYSCALL_64_after_hwframe+0x3e/0xbe
[ÂÂ377.525754]ÂÂ? trace_hardirqs_off_caller+0x3a/0x150
[ÂÂ377.525755]ÂÂ? trace_hardirqs_off_thunk+0x1a/0x20
[ÂÂ377.525756]ÂÂentry_SYSCALL_64_after_hwframe+0x49/0xbe

Next message: Anup Patel: "Re: [PATCH v7 10/21] RISC-V: KVM: Handle MMIO exits for VCPU"
Previous message: Daniel Vetter: "Re: [PATCH 01/36] drm/fourcc: Add 2 plane YCbCr 10bit format support"
In reply to: Sergey Senozhatsky: "Re: printk() + memory offline deadlock (WAS Re: page_alloc.shuffle=1 + CONFIG_PROVE_LOCKING=y = arm64 hang)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]