Re: [REGRESSION] lxc-stop hang on 5.17.x kernels

[Daniel Harding]

[Resend with a smaller trace]

On 5/3/22 02:14, Pavel Begunkov wrote:
> On 5/2/22 19:49, Daniel Harding wrote:
> > On 5/2/22 20:40, Pavel Begunkov wrote:
> > > On 5/2/22 18:00, Jens Axboe wrote:
> > > > On 5/2/22 7:59 AM, Jens Axboe wrote:
> > > > > On 5/2/22 7:36 AM, Daniel Harding wrote:
> > > > > > On 5/2/22 16:26, Jens Axboe wrote:
> > > > > > > On 5/2/22 7:17 AM, Daniel Harding wrote:
> > > > > > > > I use lxc-4.0.12 on Gentoo, built with io-uring support
> > > > > > > > (--enable-liburing), targeting liburing-2.1.  My kernel config is a
> > > > > > > > very lightly modified version of Fedora's generic kernel 
> > > > > > > > config. After
> > > > > > > > moving from the 5.16.x series to the 5.17.x kernel series, I 
> > > > > > > > started
> > > > > > > > noticed frequent hangs in lxc-stop.  It doesn't happen 100% of the
> > > > > > > > time, but definitely more than 50% of the time. Bisecting narrowed
> > > > > > > > down the issue to commit aa43477b040251f451db0d844073ac00a8ab66ee:
> > > > > > > > io_uring: poll rework. Testing indicates the problem is still 
> > > > > > > > present
> > > > > > > > in 5.18-rc5. Unfortunately I do not have the expertise with the
> > > > > > > > codebases of either lxc or io-uring to try to debug the problem
> > > > > > > > further on my own, but I can easily apply patches to any of the
> > > > > > > > involved components (lxc, liburing, kernel) and rebuild for 
> > > > > > > > testing or
> > > > > > > > validation.  I am also happy to provide any further information 
> > > > > > > > that
> > > > > > > > would be helpful with reproducing or debugging the problem.
> > > > > > > Do you have a recipe to reproduce the hang? That would make it
> > > > > > > significantly easier to figure out.
> > > > > >
> > > > > > I can reproduce it with just the following:
> > > > > >
> > > > > >      sudo lxc-create --n lxc-test --template download --bdev dir 
> > > > > > --dir /var/lib/lxc/lxc-test/rootfs -- -d ubuntu -r bionic -a amd64
> > > > > >      sudo lxc-start -n lxc-test
> > > > > >      sudo lxc-stop -n lxc-test
> > > > > >
> > > > > > The lxc-stop command never exits and the container continues running.
> > > > > > If that isn't sufficient to reproduce, please let me know.
> > > > >
> > > > > Thanks, that's useful! I'm at a conference this week and hence have
> > > > > limited amount of time to debug, hopefully Pavel has time to take 
> > > > > a look
> > > > > at this.
> > > >
> > > > Didn't manage to reproduce. Can you try, on both the good and bad
> > > > kernel, to do:
> > >
> > > Same here, it doesn't reproduce for me
> > OK, sorry it wasn't something simple.
> > > # echo 1 > /sys/kernel/debug/tracing/events/io_uring/enable
> > > > run lxc-stop
> > > >
> > > > # cp /sys/kernel/debug/tracing/trace ~/iou-trace
> > > >
> > > > so we can see what's going on? Looking at the source, lxc is just using
> > > > plain POLL_ADD, so I'm guessing it's not getting a notification when it
> > > > expects to, or it's POLL_REMOVE not doing its job. If we have a trace
> > > > from both a working and broken kernel, that might shed some light 
> > > > on it.
> > It's late in my timezone, but I'll try to work on getting those 
> > traces tomorrow.
>
> I think I got it, I've attached a trace.
>
> What's interesting is that it issues a multi shot poll but I don't
> see any kind of cancellation, neither cancel requests nor task/ring
> exit. Perhaps have to go look at lxc to see how it's supposed
> to work

Yes, that looks exactly like my bad trace.  I've attached good trace 
(captured with linux-5.16.19) and a bad trace (captured with 
linux-5.17.5).  These are the differences I noticed with just a visual scan:

* Both traces have three io_uring_submit_sqe calls at the very 
beginning, but in the good trace, there are further io_uring_submit_sqe 
calls throughout the trace, while in the bad trace, there are none.
* The good trace uses a mask of c3 for io_uring_task_add much more often 
than the bad trace:  the bad trace uses a mask of c3 only for the very 
last call to io_uring_task_add, but a mask of 41 for the other calls.
* In the good trace, many of the io_uring_complete calls have a result 
of 195, while in the bad trace, they all have a result of 1.

I don't know whether any of those things are significant or not, but 
that's what jumped out at me.

I have also attached a copy of the script I used to generate the 
traces.  If there is anything further I can to do help debug, please let 
me know.

--
Regards,

Daniel Harding
# tracer: nop
#
# entries-in-buffer/entries-written: 145/145   #P:16
#
#                                _-----=> irqs-off
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
       lxc-start-5701    [008] .....   537.440889: io_uring_create: ring 00000000b31688a5, fd 3 sq size 512, cq size 1024, flags 0
       lxc-start-5701    [008] .....   537.440918: io_uring_create: ring 0000000076cc4f29, fd 4 sq size 512, cq size 1024, flags 0
       lxc-start-5701    [008] .....   537.440928: io_uring_file_get: ring 00000000b31688a5, fd 7
       lxc-start-5701    [008] .....   537.440928: io_uring_submit_sqe: ring 00000000b31688a5, req 000000002a7a691b, op 6, data 0x55dbcf28cd80, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [008] .....   537.440931: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [008] .....   537.440931: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000105632b5, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [008] .....   537.440933: io_uring_file_get: ring 00000000b31688a5, fd 5
       lxc-start-5701    [008] .....   537.440933: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000dcef163d, op 6, data 0x55dbcf28d470, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [008] .....   537.440935: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [008] d..1.   537.442660: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.442693: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.442708: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.442709: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000435a6b59, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.442713: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [008] d..1.   537.442882: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.442907: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.442914: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.442915: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000df1dac20, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.442917: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
        lxc-stop-5751    [006] d..1.   537.448788: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
       lxc-start-5701    [010] ...1.   537.448814: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
       lxc-start-5701    [010] .....   537.448836: io_uring_file_get: ring 00000000b31688a5, fd 24
       lxc-start-5701    [010] .....   537.448837: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000c3421fbe, op 6, data 0x55dbcf28d4c0, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] ...1.   537.448838: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d4c0, result 1, cflags 0
       lxc-start-5701    [010] .....   537.448857: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
        lxc-stop-5751    [012] d..1.   537.448888: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
       lxc-start-5701    [010] ...1.   537.448908: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
       lxc-start-5701    [010] .....   537.448916: io_uring_file_get: ring 00000000b31688a5, fd 24
       lxc-start-5701    [010] .....   537.448916: io_uring_submit_sqe: ring 00000000b31688a5, req 0000000091e8a675, op 6, data 0x55dbcf28a380, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] ...1.   537.448916: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a380, result 1, cflags 0
       lxc-start-5701    [010] .....   537.448924: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
        lxc-stop-5751    [012] d..1.   537.448965: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
       lxc-start-5701    [010] ...1.   537.448982: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
       lxc-start-5701    [010] .....   537.448991: io_uring_file_get: ring 00000000b31688a5, fd 24
       lxc-start-5701    [010] .....   537.448991: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000404191e3, op 6, data 0x55dbcf28a3d0, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] ...1.   537.448992: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a3d0, result 1, cflags 0
       lxc-start-5701    [010] .....   537.449002: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
        lxc-stop-5751    [012] d..1.   537.449018: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
       lxc-start-5701    [010] ...1.   537.449038: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
       lxc-start-5701    [010] .....   537.449046: io_uring_file_get: ring 00000000b31688a5, fd 24
       lxc-start-5701    [010] .....   537.449046: io_uring_submit_sqe: ring 00000000b31688a5, req 000000004c87bdb8, op 6, data 0x55dbcf28a420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] ...1.   537.449047: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a420, result 1, cflags 0
       lxc-start-5701    [010] .....   537.449057: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
        lxc-stop-5751    [012] d..1.   537.449095: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
       lxc-start-5701    [010] ...1.   537.449113: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
       lxc-start-5701    [010] .....   537.449120: io_uring_file_get: ring 00000000b31688a5, fd 24
       lxc-start-5701    [010] .....   537.449120: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000b8df304d, op 6, data 0x55dbcf28a470, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] ...1.   537.449121: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a470, result 1, cflags 0
       lxc-start-5701    [010] .....   537.449131: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
        lxc-stop-5751    [012] d..1.   537.449207: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d470, mask c3
       lxc-start-5701    [010] ...1.   537.449223: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result 195, cflags 2
       lxc-start-5701    [010] .....   537.449231: io_uring_file_get: ring 00000000b31688a5, fd 24
       lxc-start-5701    [010] .....   537.449231: io_uring_submit_sqe: ring 00000000b31688a5, req 0000000028c2db2d, op 6, data 0x55dbcf28a4c0, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] ...1.   537.449232: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a4c0, result 1, cflags 0
       lxc-start-5701    [010] .....   537.449241: io_uring_file_get: ring 00000000b31688a5, fd 24
       lxc-start-5701    [010] .....   537.449241: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000a0ddd401, op 6, data 0x55dbcf28a4c0, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.449242: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.535100: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.535155: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.535170: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.535171: io_uring_submit_sqe: ring 00000000b31688a5, req 000000004b218709, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.535175: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.535942: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.535969: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.535976: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.535976: io_uring_submit_sqe: ring 00000000b31688a5, req 0000000006afc786, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
   kworker/u32:3-138     [012] d..1.   537.536000: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.536030: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.536036: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.536037: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000deda76cd, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.536038: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.536129: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.536155: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.536161: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.536162: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000d59ff773, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.536164: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.536199: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.536225: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.536234: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.536235: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000ecd22ed2, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
   kworker/u32:3-138     [012] d..1.   537.536246: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.536262: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.536264: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.536264: io_uring_submit_sqe: ring 00000000b31688a5, req 000000007e28f103, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.536265: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.536273: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.536286: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.536288: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.536288: io_uring_submit_sqe: ring 00000000b31688a5, req 000000002a8f1fc3, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.536289: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.536666: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.536694: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.536701: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.536701: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000228168af, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.536703: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.537049: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.537077: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.537084: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.537084: io_uring_submit_sqe: ring 00000000b31688a5, req 000000003f0080b8, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.537087: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.537813: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [010] ...1.   537.537842: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [010] .....   537.537851: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [010] .....   537.537851: io_uring_submit_sqe: ring 00000000b31688a5, req 000000001baf187d, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [010] .....   537.537854: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.538189: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [013] ...1.   537.538203: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [013] .....   537.538208: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [013] .....   537.538208: io_uring_submit_sqe: ring 00000000b31688a5, req 000000000fb0d9a9, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [013] .....   537.538210: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.538614: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [013] ...1.   537.538625: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [013] .....   537.538634: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [013] .....   537.538635: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000c9d4b035, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [013] .....   537.538637: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.538718: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [013] ...1.   537.538729: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [013] .....   537.538736: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [013] .....   537.538736: io_uring_submit_sqe: ring 00000000b31688a5, req 000000009f6c3d45, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [013] .....   537.538737: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [010] d..1.   537.539270: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [013] ...1.   537.539272: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [013] .....   537.539278: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [013] .....   537.539278: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000658d5a67, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [013] .....   537.539279: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [012] d..1.   537.540199: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [013] ...1.   537.540213: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [013] .....   537.540221: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [013] .....   537.540222: io_uring_submit_sqe: ring 00000000b31688a5, req 000000003ad51d84, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [013] .....   537.540223: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
   kworker/u32:3-138     [010] d..1.   537.540415: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28d420, mask 41
       lxc-start-5701    [013] ...1.   537.540424: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result 65, cflags 0
       lxc-start-5701    [013] .....   537.540428: io_uring_file_get: ring 00000000b31688a5, fd 53
       lxc-start-5701    [013] .....   537.540428: io_uring_submit_sqe: ring 00000000b31688a5, req 00000000297b5537, op 6, data 0x55dbcf28d420, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [013] .....   537.540430: io_uring_cqring_wait: ring 00000000b31688a5, min_events 1
 systemd-shutdow-5702    [008] dN.3.   537.572892: io_uring_task_add: ring 00000000b31688a5, op 6, data 0x55dbcf28cd80, mask 0
       lxc-start-5701    [013] ...1.   537.572909: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28cd80, result 1, cflags 2
       lxc-start-5701    [013] .....   537.572929: io_uring_file_get: ring 0000000076cc4f29, fd 53
       lxc-start-5701    [013] .....   537.572930: io_uring_submit_sqe: ring 0000000076cc4f29, req 00000000e482f889, op 6, data 0x55dbcf28a510, flags 524288, non block 1, sq_thread 0
       lxc-start-5701    [013] .....   537.572938: io_uring_cqring_wait: ring 0000000076cc4f29, min_events 1
       lxc-start-5701    [013] ...1.   537.572967: io_uring_complete: ring 0000000076cc4f29, user_data 0x55dbcf28a510, result -125, cflags 0
       lxc-start-5701    [013] ...1.   537.572984: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28a4c0, result -125, cflags 0
       lxc-start-5701    [013] ...1.   537.572984: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d470, result -125, cflags 0
       lxc-start-5701    [013] ...1.   537.572984: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28d420, result -125, cflags 0
       lxc-start-5701    [013] ...1.   537.572984: io_uring_complete: ring 00000000b31688a5, user_data 0x55dbcf28cd80, result -125, cflags 0
# tracer: nop
#
# entries-in-buffer/entries-written: 183/183   #P:16
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
       lxc-start-2249    [007] .....    47.766086: io_uring_create: ring 00000000ad366d59, fd 3 sq size 512, cq size 1024, flags 0
       lxc-start-2249    [007] .....    47.766128: io_uring_create: ring 00000000a3f46d45, fd 4 sq size 512, cq size 1024, flags 0
       lxc-start-2249    [007] .....    47.766143: io_uring_submit_sqe: ring 00000000ad366d59, req 00000000c89e5524, op 6, data 0x56049a674b80, flags 524288, non block 1, sq_thread 0
       lxc-start-2249    [007] .....    47.766144: io_uring_file_get: ring 00000000ad366d59, fd 7
       lxc-start-2249    [007] .....    47.766146: io_uring_submit_sqe: ring 00000000ad366d59, req 000000004114f7f5, op 6, data 0x56049a674bd0, flags 524288, non block 1, sq_thread 0
       lxc-start-2249    [007] .....    47.766147: io_uring_file_get: ring 00000000ad366d59, fd 53
       lxc-start-2249    [007] .....    47.766152: io_uring_submit_sqe: ring 00000000ad366d59, req 00000000476a2670, op 6, data 0x56049a671380, flags 524288, non block 1, sq_thread 0
       lxc-start-2249    [007] .....    47.766152: io_uring_file_get: ring 00000000ad366d59, fd 5
       lxc-start-2249    [007] .....    47.766155: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [011] d..1.    47.792070: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [007] ...1.    47.792101: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [007] .....    47.792114: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [007] d..1.    47.792800: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.792834: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.792840: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [007] d..1.    47.792933: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.792959: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.792965: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.902763: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.903010: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.903023: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.903040: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.903049: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.903051: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.903718: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.903742: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.903749: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.904085: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.904107: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.904113: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.904175: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.904198: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.904204: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.904525: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.904548: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.904554: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.904586: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.904609: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.904614: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.904700: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.904725: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.904731: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.905406: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.905414: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.905421: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:8-156     [005] d..1.    47.906170: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.906202: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.906212: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:8-156     [005] d..1.    47.908746: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.908768: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.908778: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:8-156     [005] d..1.    47.909269: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.909291: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.909299: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:8-156     [005] d..1.    47.911856: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.911893: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.911909: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:8-156     [005] d..1.    47.912580: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.912592: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.912601: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.918821: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.918835: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.918844: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.919048: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.919055: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.919056: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.919090: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.919097: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.919098: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.919752: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.919760: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.919761: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
  kworker/u32:11-2181    [000] d..1.    47.921724: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.921733: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.921734: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.922163: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.922171: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.922172: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.922643: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [014] ...1.    47.922654: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [014] .....    47.922662: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.928228: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [014] ...1.    47.928235: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [014] .....    47.928240: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.928339: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [014] ...1.    47.928344: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [014] .....    47.928345: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:2-107     [012] d..1.    47.928370: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [014] ...1.    47.928375: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [014] .....    47.928376: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.930346: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [014] ...1.    47.930358: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [014] .....    47.930363: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [010] d..1.    47.930820: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [014] ...1.    47.930831: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [014] .....    47.930832: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [005] d..1.    47.937915: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.937946: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.937958: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [005] d..1.    47.938370: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.938399: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.938406: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [005] d..1.    47.942927: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.942955: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.942966: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [010] d..1.    47.943029: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.943057: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.943063: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [010] d..1.    47.943247: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.943310: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.943317: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [010] d..1.    47.943323: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.943335: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.943337: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [010] d..1.    47.943389: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.943418: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.943424: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [012] d..1.    47.943452: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.943479: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.943485: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [012] d..1.    47.943656: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.943684: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.943691: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [012] d..1.    47.943709: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.943724: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.943727: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [005] d..1.    47.943742: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [006] ...1.    47.943768: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [006] .....    47.943774: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [005] d..1.    47.944355: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [000] ...1.    47.944382: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [000] .....    47.944389: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [005] d..1.    47.944882: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.944908: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.944914: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:6-148     [005] d..1.    47.945357: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.945382: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.945388: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [003] d..1.    47.954651: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.954665: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.954676: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [006] d..1.    47.955367: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.955381: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.955392: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [000] d..1.    47.955791: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.955798: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.955800: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [006] d..1.    47.956563: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.956574: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.956585: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [006] d..1.    47.956808: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.956816: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.956819: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [006] d..1.    47.957256: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.957265: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.957275: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [006] d..1.    47.957353: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [002] ...1.    47.957361: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [002] .....    47.957362: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:7-151     [004] d..1.    47.960507: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [005] ...1.    47.960518: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [005] .....    47.960529: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:9-1078    [000] d..1.    47.961340: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [001] ...1.    47.961355: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [001] .....    47.961360: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
  kworker/u32:11-2181    [014] d..1.    48.029152: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [005] ...1.    48.029182: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [005] .....    48.029196: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
  kworker/u32:11-2181    [000] d..1.    48.029815: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [005] ...1.    48.029828: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [005] .....    48.029832: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
  kworker/u32:11-2181    [000] d..1.    48.033023: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [005] ...1.    48.033040: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [005] .....    48.033048: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:8-156     [006] d..1.    49.035170: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [005] ...1.    49.035232: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
       lxc-start-2249    [005] .....    49.035261: io_uring_cqring_wait: ring 00000000ad366d59, min_events 1
   kworker/u32:8-156     [006] d..1.    49.042316: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [005] ...1.    49.042361: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
   kworker/u32:7-151     [004] d..1.    49.042367: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a674bd0, mask 41
       lxc-start-2249    [005] ...1.    49.042368: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a674bd0, result 1, cflags 2
        lxc-stop-2534    [003] d..1.    52.782291: io_uring_task_add: ring 00000000ad366d59, op 6, data 0x56049a671380, mask c3
       lxc-start-2249    [005] ...1.    52.782314: io_uring_complete: ring 00000000ad366d59, user_data 0x56049a671380, result 1, cflags 2
#!/bin/bash

sudo -v

echo 0 | sudo dd status=none of=/sys/kernel/debug/tracing/trace

echo 1 | sudo dd status=none of=/sys/kernel/debug/tracing/events/io_uring/enable

sudo lxc-start -n lxc-test

sudo lxc-stop -n lxc-test &

sleep 1

sudo cat /sys/kernel/debug/tracing/trace > ~/lxc-trace

echo 0 | sudo dd status=none of=/sys/kernel/debug/tracing/events/io_uring/enable