Re: d28296d248: stress-ng.sigsegv.ops_per_sec -82.7% regression
From: Eric W. Biederman
Date: Wed Feb 24 2021 - 11:57:43 EST
kernel test robot <oliver.sang@xxxxxxxxx> writes:
> Greeting,
>
> FYI, we noticed a -82.7% regression of stress-ng.sigsegv.ops_per_sec due to commit:
>
>
> commit: d28296d2484fa11e94dff65e93eb25802a443d47 ("[PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts")
> url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
> base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
>
> in testcase: stress-ng
> on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
> with following parameters:
>
> nr_threads: 100%
> disk: 1HDD
> testtime: 60s
> class: interrupt
> test: sigsegv
> cpufreq_governor: performance
> ucode: 0x42e
>
>
> In addition to that, the commit also has significant impact on the
> following tests:
Thank you. Now we have a sense of where we need to test the performance
of these changes carefully.
Eric
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.sigq.ops_per_sec -56.1% regression |
> | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory |
> | test parameters | class=interrupt |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | nr_threads=100% |
> | | test=sigq |
> | | testtime=60s |
> | | ucode=0x42e |
> +------------------+-----------------------------------------------------------------------+
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <oliver.sang@xxxxxxxxx>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp split-job --compatible job.yaml
> bin/lkp run compatible-job.yaml
>
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
> interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/sigsegv/stress-ng/60s/0x42e
>
> commit:
> 4660d663b4 ("Reimplement RLIMIT_MSGQUEUE on top of ucounts")
> d28296d248 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")
>
> 4660d663b4207ce6 d28296d2484fa11e94dff65e93e
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> 14:6 -217% 1:6 perf-profile.children.cycles-pp.error_entry
> 12:6 -179% 1:6 perf-profile.self.cycles-pp.error_entry
> %stddev %change %stddev
> \ | \
> 4.766e+08 -82.7% 82358308 stress-ng.sigsegv.ops
> 7942807 -82.7% 1372639 stress-ng.sigsegv.ops_per_sec
> 29408 -77.5% 6621 ± 61% stress-ng.time.file_system_inputs
> 1566 +69.4% 2653 stress-ng.time.system_time
> 1274 -84.8% 193.22 ± 8% stress-ng.time.user_time
> 12458 ± 5% +37.2% 17097 ± 5% numa-meminfo.node1.Active(anon)
> 51.41 +66.5% 85.59 iostat.cpu.system
> 41.17 -84.2% 6.50 ± 7% iostat.cpu.user
> 3040 ± 4% +37.9% 4193 ± 4% numa-vmstat.node1.nr_active_anon
> 3040 ± 4% +37.9% 4193 ± 4% numa-vmstat.node1.nr_zone_active_anon
> 50.83 +67.2% 85.00 vmstat.cpu.sy
> 40.50 -85.6% 5.83 ± 11% vmstat.cpu.us
> 225.33 -77.7% 50.33 ± 62% vmstat.io.bi
> 7.00 -100.0% 0.00 vmstat.memory.buff
> 20735 ± 2% -14.1% 17812 ± 5% meminfo.Active
> 13506 ± 3% +31.9% 17812 ± 5% meminfo.Active(anon)
> 7228 -100.0% 0.00 meminfo.Active(file)
> 29308 +18.4% 34687 ± 2% meminfo.Shmem
> 202067 -9.5% 182899 meminfo.VmallocUsed
> 0.01 ± 17% -0.0 0.00 ± 10% mpstat.cpu.all.iowait%
> 1.04 -0.1 0.92 mpstat.cpu.all.irq%
> 0.03 ± 8% -0.0 0.02 ± 4% mpstat.cpu.all.soft%
> 51.54 +35.6 87.17 mpstat.cpu.all.sys%
> 42.22 -35.6 6.66 ± 8% mpstat.cpu.all.usr%
> 0.00 ± 70% +191.7% 0.01 ± 26% perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
> 0.00 ± 47% +158.8% 0.01 ± 43% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
> 0.27 ± 25% -55.4% 0.12 ± 37% perf-sched.total_wait_and_delay.average.ms
> 202.17 ± 23% -29.5% 142.50 ± 13% perf-sched.total_wait_and_delay.count.ms
> 0.21 ± 18% -69.3% 0.06 ± 58% perf-sched.total_wait_time.average.ms
> 0.00 ± 70% +191.7% 0.01 ± 26% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
> 8.17 ± 29% -85.7% 1.17 ±125% perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
> 0.00 ± 47% +158.8% 0.01 ± 43% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
> 3.11 ± 11% -76.1% 0.74 ±142% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
> 1790 ± 15% -30.1% 1250 ± 13% slabinfo.dmaengine-unmap-16.active_objs
> 1790 ± 15% -30.1% 1250 ± 13% slabinfo.dmaengine-unmap-16.num_objs
> 123.00 -100.0% 0.00 slabinfo.ext4_pending_reservation.active_objs
> 123.00 -100.0% 0.00 slabinfo.ext4_pending_reservation.num_objs
> 3619 -100.0% 0.00 slabinfo.f2fs_free_nid.active_objs
> 3619 -100.0% 0.00 slabinfo.f2fs_free_nid.num_objs
> 62865 -44.1% 35155 slabinfo.kmalloc-64.active_objs
> 984.33 -43.9% 552.50 slabinfo.kmalloc-64.active_slabs
> 63031 -43.9% 35389 slabinfo.kmalloc-64.num_objs
> 984.33 -43.9% 552.50 slabinfo.kmalloc-64.num_slabs
> 161.00 ± 9% +67.1% 269.00 ± 7% slabinfo.xfs_buf.active_objs
> 161.00 ± 9% +67.1% 269.00 ± 7% slabinfo.xfs_buf.num_objs
> 3399 ± 3% +32.9% 4519 ± 4% proc-vmstat.nr_active_anon
> 1806 -100.0% 0.00 proc-vmstat.nr_active_file
> 9333 +3.0% 9610 proc-vmstat.nr_mapped
> 7344 +18.4% 8698 ± 2% proc-vmstat.nr_shmem
> 16319 -0.9% 16176 proc-vmstat.nr_slab_reclaimable
> 24399 -6.2% 22882 proc-vmstat.nr_slab_unreclaimable
> 3399 ± 3% +32.9% 4519 ± 4% proc-vmstat.nr_zone_active_anon
> 1806 -100.0% 0.00 proc-vmstat.nr_zone_active_file
> 3693 ± 80% -80.1% 736.17 ± 61% proc-vmstat.numa_hint_faults
> 293002 -2.7% 284991 proc-vmstat.numa_hit
> 249530 -3.3% 241180 proc-vmstat.numa_local
> 5007 ±111% -90.5% 478.00 ± 81% proc-vmstat.numa_pages_migrated
> 11443 ± 2% -7.0% 10636 ± 4% proc-vmstat.pgactivate
> 332528 -3.9% 319693 proc-vmstat.pgalloc_normal
> 249148 ± 2% -4.1% 239053 ± 2% proc-vmstat.pgfree
> 5007 ±111% -90.5% 478.00 ± 81% proc-vmstat.pgmigrate_success
> 14704 -77.5% 3310 ± 61% proc-vmstat.pgpgin
> 0.00 +2.1e+105% 2095 proc-vmstat.pgpgout
> 13870 ± 10% -55.1% 6227 ± 28% softirqs.CPU0.RCU
> 9989 ± 3% -62.2% 3775 ± 23% softirqs.CPU1.RCU
> 8625 ± 13% -76.1% 2061 ± 10% softirqs.CPU10.RCU
> 7954 ± 15% -65.9% 2709 ± 18% softirqs.CPU14.RCU
> 9075 ± 14% -78.7% 1929 ± 11% softirqs.CPU17.RCU
> 8522 ± 13% -76.7% 1985 ± 22% softirqs.CPU18.RCU
> 9595 ± 7% -63.3% 3522 ± 22% softirqs.CPU2.RCU
> 8455 ± 11% -74.5% 2152 ± 45% softirqs.CPU20.RCU
> 8320 ± 12% -76.7% 1939 ± 14% softirqs.CPU21.RCU
> 8338 ± 13% -71.7% 2359 ± 32% softirqs.CPU23.RCU
> 8541 ± 12% -75.5% 2089 ± 32% softirqs.CPU26.RCU
> 9639 ± 20% -79.5% 1976 ± 17% softirqs.CPU28.RCU
> 9232 ± 13% -78.0% 2026 ± 6% softirqs.CPU30.RCU
> 7857 ± 17% -68.9% 2446 ± 27% softirqs.CPU34.RCU
> 8619 ± 11% -75.8% 2081 ± 30% softirqs.CPU36.RCU
> 9614 ± 3% -74.3% 2469 ± 15% softirqs.CPU4.RCU
> 8962 ± 10% -77.9% 1981 ± 12% softirqs.CPU41.RCU
> 9027 ± 12% -78.6% 1932 ± 8% softirqs.CPU42.RCU
> 9364 ± 12% -76.5% 2197 ± 8% softirqs.CPU44.RCU
> 8774 ± 13% -75.5% 2147 ± 21% softirqs.CPU47.RCU
> 8783 ± 12% -76.0% 2105 ± 12% softirqs.CPU5.RCU
> 9007 ± 9% -75.8% 2177 ± 8% softirqs.CPU6.RCU
> 417664 ± 7% -72.8% 113621 ± 8% softirqs.RCU
> 12708 ± 4% +13.0% 14362 ± 2% softirqs.TIMER
> 60500 -27.7% 43751 interrupts.CAL:Function_call_interrupts
> 1121 -26.9% 819.17 ± 3% interrupts.CPU10.CAL:Function_call_interrupts
> 1561 ± 43% -48.8% 800.00 ± 5% interrupts.CPU11.CAL:Function_call_interrupts
> 1425 ± 6% -25.3% 1065 ± 6% interrupts.CPU12.CAL:Function_call_interrupts
> 166.17 ± 13% -26.4% 122.33 ± 21% interrupts.CPU13.RES:Rescheduling_interrupts
> 1402 ± 18% -25.6% 1043 ± 22% interrupts.CPU15.CAL:Function_call_interrupts
> 129.17 ± 50% -42.5% 74.33 ± 4% interrupts.CPU17.RES:Rescheduling_interrupts
> 1182 ± 9% -31.0% 815.00 ± 2% interrupts.CPU20.CAL:Function_call_interrupts
> 1120 -29.7% 787.17 ± 4% interrupts.CPU21.CAL:Function_call_interrupts
> 1115 ± 3% -28.2% 801.17 interrupts.CPU23.CAL:Function_call_interrupts
> 1169 ± 7% -27.2% 851.33 ± 5% interrupts.CPU24.CAL:Function_call_interrupts
> 177.33 ± 98% -55.9% 78.17 ± 6% interrupts.CPU25.RES:Rescheduling_interrupts
> 1142 ± 16% -28.8% 813.00 ± 3% interrupts.CPU27.CAL:Function_call_interrupts
> 1229 ± 18% -33.3% 820.33 ± 4% interrupts.CPU28.CAL:Function_call_interrupts
> 1124 ± 4% -28.3% 806.17 interrupts.CPU29.CAL:Function_call_interrupts
> 1123 ± 3% -28.5% 803.00 interrupts.CPU30.CAL:Function_call_interrupts
> 1127 ± 2% -32.0% 766.67 ± 19% interrupts.CPU31.CAL:Function_call_interrupts
> 1066 ± 8% -22.3% 829.33 ± 7% interrupts.CPU32.CAL:Function_call_interrupts
> 1109 -26.0% 820.50 ± 4% interrupts.CPU34.CAL:Function_call_interrupts
> 1315 ± 22% -37.7% 818.83 ± 2% interrupts.CPU38.CAL:Function_call_interrupts
> 1164 ± 4% -29.0% 827.00 ± 3% interrupts.CPU39.CAL:Function_call_interrupts
> 5513 ± 35% +13.1% 6237 ± 33% interrupts.CPU39.NMI:Non-maskable_interrupts
> 5513 ± 35% +13.1% 6237 ± 33% interrupts.CPU39.PMI:Performance_monitoring_interrupts
> 1277 ± 23% -36.1% 815.83 interrupts.CPU40.CAL:Function_call_interrupts
> 97.33 ± 28% -25.7% 72.33 ± 6% interrupts.CPU40.RES:Rescheduling_interrupts
> 1116 -24.8% 839.00 ± 11% interrupts.CPU42.CAL:Function_call_interrupts
> 1130 ± 3% -28.5% 808.67 ± 3% interrupts.CPU43.CAL:Function_call_interrupts
> 1121 -29.8% 787.50 ± 4% interrupts.CPU45.CAL:Function_call_interrupts
> 1119 -27.3% 813.83 interrupts.CPU46.CAL:Function_call_interrupts
> 1167 ± 6% -28.0% 840.67 interrupts.CPU47.CAL:Function_call_interrupts
> 1667 ± 41% -44.3% 928.67 ± 24% interrupts.CPU5.CAL:Function_call_interrupts
> 1369 ± 24% -39.5% 827.67 ± 3% interrupts.CPU6.CAL:Function_call_interrupts
> 96.83 ± 25% -23.1% 74.50 ± 2% interrupts.CPU7.RES:Rescheduling_interrupts
> 1123 -28.1% 807.00 interrupts.CPU9.CAL:Function_call_interrupts
> 0.72 ± 5% +107.4% 1.50 ± 2% perf-stat.i.MPKI
> 8.023e+09 -13.5% 6.943e+09 perf-stat.i.branch-instructions
> 1.08 ± 2% -0.5 0.55 ± 3% perf-stat.i.branch-miss-rate%
> 71073978 ± 2% -67.4% 23149119 ± 3% perf-stat.i.branch-misses
> 29.33 ± 2% +5.9 35.23 perf-stat.i.cache-miss-rate%
> 6189596 +120.2% 13628525 perf-stat.i.cache-misses
> 21228048 +82.4% 38714786 perf-stat.i.cache-references
> 3.36 +34.8% 4.53 perf-stat.i.cpi
> 109.52 ± 2% -22.5% 84.92 perf-stat.i.cpu-migrations
> 22695 -56.5% 9882 perf-stat.i.cycles-between-cache-misses
> 1.15 ± 7% -0.9 0.30 ± 4% perf-stat.i.dTLB-load-miss-rate%
> 1.398e+08 ± 7% -83.4% 23247225 ± 4% perf-stat.i.dTLB-load-misses
> 1.154e+10 -34.4% 7.564e+09 perf-stat.i.dTLB-loads
> 1.17 -0.0 1.13 perf-stat.i.dTLB-store-miss-rate%
> 1.321e+08 -82.1% 23679743 perf-stat.i.dTLB-store-misses
> 1.071e+10 -81.3% 2.005e+09 perf-stat.i.dTLB-stores
> 41869658 -74.5% 10693569 ± 56% perf-stat.i.iTLB-load-misses
> 19932113 ± 38% -88.3% 2325708 ± 64% perf-stat.i.iTLB-loads
> 3.945e+10 -25.9% 2.924e+10 perf-stat.i.instructions
> 1199 ± 4% +182.7% 3389 ± 26% perf-stat.i.instructions-per-iTLB-miss
> 0.31 -23.7% 0.24 perf-stat.i.ipc
> 634.71 -45.5% 345.81 perf-stat.i.metric.M/sec
> 166710 ± 10% +3369.3% 5783625 perf-stat.i.node-load-misses
> 227268 ± 6% +3063.1% 7188743 perf-stat.i.node-loads
> 48.41 -4.5 43.91 perf-stat.i.node-store-miss-rate%
> 5425859 -11.8% 4783945 perf-stat.i.node-store-misses
> 5599687 +6.1% 5943407 perf-stat.i.node-stores
> 7532204 -82.7% 1305118 perf-stat.i.page-faults
> 0.54 +146.0% 1.32 perf-stat.overall.MPKI
> 0.89 ± 2% -0.6 0.33 ± 3% perf-stat.overall.branch-miss-rate%
> 29.17 ± 2% +6.0 35.20 perf-stat.overall.cache-miss-rate%
> 3.45 +35.0% 4.65 perf-stat.overall.cpi
> 21953 -54.5% 9979 perf-stat.overall.cycles-between-cache-misses
> 1.20 ± 7% -0.9 0.31 ± 4% perf-stat.overall.dTLB-load-miss-rate%
> 1.22 -0.1 1.17 perf-stat.overall.dTLB-store-miss-rate%
> 942.39 +245.5% 3255 ± 28% perf-stat.overall.instructions-per-iTLB-miss
> 0.29 -25.9% 0.21 perf-stat.overall.ipc
> 42.24 ± 2% +2.3 44.58 perf-stat.overall.node-load-miss-rate%
> 49.21 -4.6 44.59 perf-stat.overall.node-store-miss-rate%
> 7.894e+09 -13.5% 6.83e+09 perf-stat.ps.branch-instructions
> 69952381 ± 2% -67.4% 22781972 ± 3% perf-stat.ps.branch-misses
> 6093197 +120.1% 13409962 perf-stat.ps.cache-misses
> 20897937 +82.3% 38097787 perf-stat.ps.cache-references
> 107.78 ± 2% -22.4% 83.62 perf-stat.ps.cpu-migrations
> 1.375e+08 ± 7% -83.4% 22871450 ± 4% perf-stat.ps.dTLB-load-misses
> 1.135e+10 -34.4% 7.442e+09 perf-stat.ps.dTLB-loads
> 1.3e+08 -82.1% 23295591 perf-stat.ps.dTLB-store-misses
> 1.054e+10 -81.3% 1.973e+09 perf-stat.ps.dTLB-stores
> 41193894 -74.5% 10519305 ± 56% perf-stat.ps.iTLB-load-misses
> 19610606 ± 38% -88.3% 2288293 ± 64% perf-stat.ps.iTLB-loads
> 3.882e+10 -25.9% 2.876e+10 perf-stat.ps.instructions
> 164152 ± 10% +3366.2% 5689843 perf-stat.ps.node-load-misses
> 223940 ± 6% +3058.2% 7072454 perf-stat.ps.node-loads
> 5338769 -11.8% 4706549 perf-stat.ps.node-store-misses
> 5510338 +6.1% 5847491 perf-stat.ps.node-stores
> 7410609 -82.7% 1283937 perf-stat.ps.page-faults
> 2.454e+12 -25.9% 1.817e+12 perf-stat.total.instructions
> 33.68 -29.6 4.04 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 18.94 -16.5 2.46 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 13.67 -12.3 1.42 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
> 12.27 -10.9 1.36 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 8.04 -7.0 1.05 ± 2% perf-profile.calltrace.cycles-pp.__entry_text_start
> 6.06 -6.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack
> 6.02 ± 2% -5.4 0.66 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigaction.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 5.51 -4.8 0.70 perf-profile.calltrace.cycles-pp.__setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
> 0.00 +0.6 0.65 perf-profile.calltrace.cycles-pp.inc_rlimit_ucounts_and_test.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault
> 18.91 +27.0 45.89 perf-profile.calltrace.cycles-pp.irqentry_exit_to_user_mode.asm_exc_page_fault
> 15.00 +30.4 45.42 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
> 14.66 +30.7 45.38 perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
> 11.73 +34.2 45.97 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
> 8.85 +36.7 45.54 perf-profile.calltrace.cycles-pp.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
> 8.16 +37.2 45.40 perf-profile.calltrace.cycles-pp.force_sig_fault.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
> 8.04 +37.3 45.39 perf-profile.calltrace.cycles-pp.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
> 6.53 +37.8 44.36 perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
> 7.06 +38.0 45.05 perf-profile.calltrace.cycles-pp.__send_signal.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore.exc_page_fault
> 4.28 ± 2% +39.2 43.46 perf-profile.calltrace.cycles-pp.__sigqueue_free.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
> 4.87 ± 2% +39.9 44.81 perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore
> 0.00 +42.7 42.72 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.get_signal
> 0.00 +43.0 42.99 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.get_signal.arch_do_signal_or_restart
> 0.00 +43.1 43.08 perf-profile.calltrace.cycles-pp.put_ucounts.__sigqueue_free.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare
> 0.00 +43.5 43.52 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal
> 0.00 +44.0 43.97 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal.force_sig_info_to_task
> 0.00 +44.1 44.05 perf-profile.calltrace.cycles-pp.get_ucounts.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault
> 31.89 +60.0 91.94 perf-profile.calltrace.cycles-pp.asm_exc_page_fault
> 33.84 -29.7 4.10 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 19.01 -16.5 2.48 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
> 14.93 -13.3 1.59 perf-profile.children.cycles-pp.syscall_return_via_sysret
> 13.14 -11.7 1.43 ± 2% perf-profile.children.cycles-pp.do_syscall_64
> 10.60 -9.3 1.27 perf-profile.children.cycles-pp.__entry_text_start
> 6.17 ± 2% -5.5 0.68 ± 2% perf-profile.children.cycles-pp.__x64_sys_rt_sigaction
> 5.55 -4.8 0.71 ± 2% perf-profile.children.cycles-pp.__setup_rt_frame
> 3.63 ± 2% -3.2 0.40 ± 2% perf-profile.children.cycles-pp.__x64_sys_rt_sigprocmask
> 3.54 -3.2 0.32 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
> 3.11 ± 3% -2.8 0.35 ± 3% perf-profile.children.cycles-pp._copy_from_user
> 2.91 -2.5 0.37 ± 2% perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
> 2.38 -2.1 0.28 perf-profile.children.cycles-pp.__irqentry_text_end
> 2.40 ± 4% -2.1 0.30 ± 2% perf-profile.children.cycles-pp.__might_fault
> 2.23 -1.9 0.32 perf-profile.children.cycles-pp.native_irq_return_iret
> 2.26 ± 2% -1.9 0.35 ± 5% perf-profile.children.cycles-pp.do_user_addr_fault
> 1.86 -1.6 0.27 ± 3% perf-profile.children.cycles-pp.do_sigaction
> 1.75 -1.5 0.21 ± 3% perf-profile.children.cycles-pp.copy_user_generic_unrolled
> 1.67 ± 3% -1.5 0.19 ± 3% perf-profile.children.cycles-pp.__set_current_blocked
> 1.51 ± 2% -1.3 0.18 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 1.33 ± 3% -1.1 0.19 perf-profile.children.cycles-pp.copy_siginfo_to_user
> 1.24 ± 3% -1.1 0.13 ± 3% perf-profile.children.cycles-pp.recalc_sigpending
> 1.30 ± 4% -1.1 0.18 ± 3% perf-profile.children.cycles-pp._copy_to_user
> 1.23 ± 3% -1.1 0.15 ± 3% perf-profile.children.cycles-pp.___might_sleep
> 1.07 ± 4% -0.9 0.12 ± 3% perf-profile.children.cycles-pp.signal_setup_done
> 0.98 ± 4% -0.9 0.11 ± 4% perf-profile.children.cycles-pp.fpu__clear
> 0.93 ± 4% -0.8 0.12 ± 8% perf-profile.children.cycles-pp.__might_sleep
> 0.89 -0.8 0.14 ± 3% perf-profile.children.cycles-pp.__clear_user
> 0.85 ± 4% -0.7 0.10 ± 3% perf-profile.children.cycles-pp.__set_task_blocked
> 0.81 ± 2% -0.7 0.10 ± 8% perf-profile.children.cycles-pp.sigprocmask
> 0.76 ± 11% -0.7 0.07 ± 10% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
> 0.70 ± 2% -0.6 0.06 ± 9% perf-profile.children.cycles-pp.signal_wake_up_state
> 0.73 -0.6 0.13 ± 7% perf-profile.children.cycles-pp.__perf_sw_event
> 0.50 ± 2% -0.5 0.04 ± 44% perf-profile.children.cycles-pp.complete_signal
> 0.50 ± 4% -0.4 0.08 ± 11% perf-profile.children.cycles-pp.___perf_sw_event
> 0.45 -0.4 0.06 ± 6% perf-profile.children.cycles-pp.sync_regs
> 0.44 -0.4 0.07 ± 10% perf-profile.children.cycles-pp.fixup_vdso_exception
> 0.35 ± 2% -0.3 0.05 perf-profile.children.cycles-pp.prepare_signal
> 0.35 ± 3% -0.3 0.08 ± 5% perf-profile.children.cycles-pp.is_prefetch
> 0.33 ± 11% -0.3 0.07 ± 6% perf-profile.children.cycles-pp.kmem_cache_alloc
> 0.19 ± 4% -0.1 0.05 ± 45% perf-profile.children.cycles-pp.copy_from_kernel_nofault
> 0.33 +0.0 0.37 ± 3% perf-profile.children.cycles-pp.kmem_cache_free
> 0.00 +0.4 0.36 ± 4% perf-profile.children.cycles-pp.dec_rlimit_ucounts
> 0.00 +0.6 0.65 perf-profile.children.cycles-pp.inc_rlimit_ucounts_and_test
> 19.11 +26.8 45.91 perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
> 15.69 +29.8 45.51 perf-profile.children.cycles-pp.exit_to_user_mode_prepare
> 14.70 +30.7 45.38 perf-profile.children.cycles-pp.arch_do_signal_or_restart
> 11.77 +34.2 45.99 perf-profile.children.cycles-pp.exc_page_fault
> 8.90 +36.7 45.55 perf-profile.children.cycles-pp.__bad_area_nosemaphore
> 8.17 +37.2 45.40 perf-profile.children.cycles-pp.force_sig_fault
> 8.10 +37.3 45.39 perf-profile.children.cycles-pp.force_sig_info_to_task
> 6.57 +37.8 44.38 perf-profile.children.cycles-pp.get_signal
> 7.11 +38.0 45.06 perf-profile.children.cycles-pp.__send_signal
> 4.28 ± 2% +39.2 43.46 perf-profile.children.cycles-pp.__sigqueue_free
> 4.89 ± 2% +39.9 44.82 perf-profile.children.cycles-pp.__sigqueue_alloc
> 0.00 +43.1 43.09 perf-profile.children.cycles-pp.put_ucounts
> 0.00 +44.1 44.05 perf-profile.children.cycles-pp.get_ucounts
> 31.95 +60.0 91.95 perf-profile.children.cycles-pp.asm_exc_page_fault
> 0.00 +86.2 86.24 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> 0.32 ± 4% +86.7 87.01 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 18.17 -15.8 2.36 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
> 14.90 -13.3 1.58 perf-profile.self.cycles-pp.syscall_return_via_sysret
> 10.60 -9.3 1.27 perf-profile.self.cycles-pp.__entry_text_start
> 4.55 ± 2% -4.5 0.04 ± 72% perf-profile.self.cycles-pp.__sigqueue_alloc
> 4.09 ± 2% -3.6 0.49 ± 2% perf-profile.self.cycles-pp.irqentry_exit_to_user_mode
> 2.30 ± 5% -2.1 0.19 ± 4% perf-profile.self.cycles-pp.do_syscall_64
> 2.38 -2.1 0.28 perf-profile.self.cycles-pp.__irqentry_text_end
> 2.22 -1.9 0.32 perf-profile.self.cycles-pp.native_irq_return_iret
> 1.92 ± 8% -1.8 0.16 ± 2% perf-profile.self.cycles-pp.__x64_sys_rt_sigaction
> 1.72 ± 3% -1.5 0.20 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
> 1.71 -1.5 0.21 ± 4% perf-profile.self.cycles-pp.copy_fpstate_to_sigframe
> 1.68 -1.5 0.20 ± 3% perf-profile.self.cycles-pp.copy_user_generic_unrolled
> 1.46 ± 4% -1.3 0.12 ± 4% perf-profile.self.cycles-pp.__x64_sys_rt_sigprocmask
> 1.46 ± 2% -1.3 0.17 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irq
> 1.21 ± 6% -1.1 0.14 ± 5% perf-profile.self.cycles-pp.__setup_rt_frame
> 1.21 ± 2% -1.1 0.14 ± 3% perf-profile.self.cycles-pp.___might_sleep
> 1.06 ± 4% -1.0 0.09 ± 5% perf-profile.self.cycles-pp.recalc_sigpending
> 1.01 ± 6% -1.0 0.05 perf-profile.self.cycles-pp.asm_exc_page_fault
> 0.97 ± 4% -0.9 0.10 ± 7% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
> 0.95 ± 2% -0.8 0.12 ± 4% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
> 0.98 -0.8 0.15 ± 4% perf-profile.self.cycles-pp.do_sigaction
> 0.84 ± 4% -0.7 0.10 ± 6% perf-profile.self.cycles-pp.__might_sleep
> 0.72 ± 8% -0.6 0.09 ± 4% perf-profile.self.cycles-pp._copy_from_user
> 0.71 ± 2% -0.6 0.10 ± 7% perf-profile.self.cycles-pp.do_user_addr_fault
> 0.65 ± 14% -0.6 0.06 ± 11% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
> 0.68 ± 6% -0.6 0.10 ± 3% perf-profile.self.cycles-pp.__might_fault
> 0.80 -0.6 0.25 ± 3% perf-profile.self.cycles-pp.get_signal
> 0.64 ± 4% -0.6 0.08 ± 5% perf-profile.self.cycles-pp.fpu__clear
> 0.57 ± 3% -0.5 0.07 ± 7% perf-profile.self.cycles-pp.__send_signal
> 0.53 ± 2% -0.5 0.08 ± 6% perf-profile.self.cycles-pp.__clear_user
> 0.49 ± 5% -0.4 0.04 ± 45% perf-profile.self.cycles-pp.arch_do_signal_or_restart
> 0.43 -0.4 0.06 ± 7% perf-profile.self.cycles-pp.fixup_vdso_exception
> 0.42 ± 4% -0.4 0.06 ± 7% perf-profile.self.cycles-pp.___perf_sw_event
> 0.41 -0.4 0.05 ± 9% perf-profile.self.cycles-pp.sync_regs
> 0.33 ± 3% -0.3 0.04 ± 44% perf-profile.self.cycles-pp.prepare_signal
> 0.31 ± 12% -0.2 0.07 ± 7% perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.22 ± 3% -0.1 0.10 ± 7% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.33 +0.0 0.36 ± 3% perf-profile.self.cycles-pp.kmem_cache_free
> 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.get_ucounts
> 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.put_ucounts
> 0.00 +0.4 0.36 ± 4% perf-profile.self.cycles-pp.dec_rlimit_ucounts
> 0.27 ± 4% +0.5 0.77 ± 2% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.00 +0.6 0.65 perf-profile.self.cycles-pp.inc_rlimit_ucounts_and_test
> 0.00 +86.2 86.24 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>
> stress-ng.time.user_time
>
> 1400 +--------------------------------------------------------------------+
> |.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.++.+.+.++.+.++.+.+. +.+.+.++.|
> 1200 |-+ + |
> | |
> 1000 |-+ |
> | |
> 800 |-+ |
> | |
> 600 |-+ |
> | |
> 400 |-+ O |
> | |
> 200 |-OO O O O O O OO O OO O O OO O O OO O OO O O OO O OO |
> | |
> 0 +--------------------------------------------------------------------+
>
>
> stress-ng.time.system_time
>
> 2800 +--------------------------------------------------------------------+
> | O O O O OO O O O O O |
> 2600 |-OO O O O O OO O OO OO O O O O O O |
> | |
> 2400 |-+ O |
> | |
> 2200 |-+ |
> | |
> 2000 |-+ |
> | |
> 1800 |-+ |
> | |
> 1600 |.++.+.+.++.+. .+.++.+.+.++.+.+.++.+.++.+.+.++.+.+ .+.+.++.+.+.++.|
> | +.++ + |
> 1400 +--------------------------------------------------------------------+
>
>
> stress-ng.sigsegv.ops
>
> 5e+08 +-----------------------------------------------------------------+
> |.++.+.++.+.++.+.++.++.+.++.+.++.+.++.+.++.+.++.+.++.++.+.++.+.++.|
> 4.5e+08 |-+ |
> 4e+08 |-+ |
> | |
> 3.5e+08 |-+ |
> 3e+08 |-+ |
> | |
> 2.5e+08 |-+ |
> 2e+08 |-+ |
> | |
> 1.5e+08 |-+ |
> 1e+08 |-+ |
> | OO O OO O OO O OO OO O OO O OO O OO O OO O OO O OO |
> 5e+07 +-----------------------------------------------------------------+
>
>
> stress-ng.sigsegv.ops_per_sec
>
> 8e+06 +-------------------------------------------------------------------+
> | |
> 7e+06 |-+ |
> | |
> 6e+06 |-+ |
> | |
> 5e+06 |-+ |
> | |
> 4e+06 |-+ |
> | |
> 3e+06 |-+ |
> | |
> 2e+06 |-+ |
> | OO O O OO O OO O OO O O OO O OO O OO O O OO O OO O O |
> 1e+06 +-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
> ***************************************************************************************************
> lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
> interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/sigq/stress-ng/60s/0x42e
>
> commit:
> 4660d663b4 ("Reimplement RLIMIT_MSGQUEUE on top of ucounts")
> d28296d248 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")
>
> 4660d663b4207ce6 d28296d2484fa11e94dff65e93e
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 4.176e+08 -56.1% 1.831e+08 ± 21% stress-ng.sigq.ops
> 6959423 -56.1% 3051857 ± 21% stress-ng.sigq.ops_per_sec
> 2.157e+08 ± 8% -66.1% 73100172 ± 9% stress-ng.time.involuntary_context_switches
> 14467 +1.4% 14674 stress-ng.time.minor_page_faults
> 4513 +1.2% 4569 stress-ng.time.percent_of_cpu_this_job_got
> 2150 +17.9% 2535 ± 3% stress-ng.time.system_time
> 660.96 -53.1% 309.96 ± 30% stress-ng.time.user_time
> 2.478e+08 ± 5% -70.5% 73168836 ± 9% stress-ng.time.voluntary_context_switches
> 7.70 ± 8% -6.8% 7.18 iostat.cpu.idle
> 70.60 +16.6% 82.32 ± 3% iostat.cpu.system
> 21.70 -51.6% 10.49 ± 29% iostat.cpu.user
> 15660 ± 3% +13.7% 17806 ± 3% meminfo.Active
> 15660 ± 3% +13.7% 17806 ± 3% meminfo.Active(anon)
> 31878 ± 2% +8.8% 34688 ± 2% meminfo.Shmem
> 123900 ± 66% -66.9% 41021 ± 30% cpuidle.C1.usage
> 1.507e+08 ± 6% +9.5% 1.65e+08 ± 4% cpuidle.C6.time
> 681599 ± 38% -72.8% 185137 ±113% cpuidle.POLL.time
> 645871 ± 39% -98.5% 9580 ± 75% cpuidle.POLL.usage
> 1.78 ± 3% -0.8 0.95 mpstat.cpu.all.irq%
> 0.03 ± 4% -0.0 0.02 ± 27% mpstat.cpu.all.soft%
> 70.90 +12.5 83.42 ± 3% mpstat.cpu.all.sys%
> 22.35 -11.6 10.75 ± 29% mpstat.cpu.all.usr%
> 825.33 ± 2% -11.6% 729.67 ± 5% slabinfo.file_lock_cache.active_objs
> 825.33 ± 2% -11.6% 729.67 ± 5% slabinfo.file_lock_cache.num_objs
> 1455 ± 11% -32.4% 983.00 ± 13% slabinfo.khugepaged_mm_slot.active_objs
> 1455 ± 11% -32.4% 983.00 ± 13% slabinfo.khugepaged_mm_slot.num_objs
> 69.67 +17.2% 81.67 ± 3% vmstat.cpu.sy
> 21.00 -52.4% 10.00 ± 29% vmstat.cpu.us
> 7139282 ± 7% -68.5% 2251603 ± 9% vmstat.system.cs
> 518916 ± 6% -80.5% 101139 ± 4% vmstat.system.in
> 8082 ± 51% -68.3% 2565 ± 17% softirqs.CPU10.SCHED
> 6261 ± 72% -56.4% 2729 ± 14% softirqs.CPU17.SCHED
> 6147 ± 66% -64.3% 2195 ± 3% softirqs.CPU25.SCHED
> 16334 ± 41% -82.6% 2846 ± 24% softirqs.CPU27.SCHED
> 6280 ± 56% -61.9% 2394 ± 9% softirqs.CPU39.SCHED
> 8248 ± 50% -73.2% 2209 ± 8% softirqs.CPU40.SCHED
> 228327 ± 9% -46.3% 122665 ± 2% softirqs.SCHED
> 3851 ± 4% +13.8% 4381 ± 4% proc-vmstat.nr_active_anon
> 9587 +2.8% 9855 proc-vmstat.nr_mapped
> 7943 ± 2% +8.7% 8630 ± 2% proc-vmstat.nr_shmem
> 3851 ± 4% +13.8% 4381 ± 4% proc-vmstat.nr_zone_active_anon
> 438.33 ±122% +1463.6% 6853 ± 69% proc-vmstat.numa_pages_migrated
> 8816 ± 4% +11.5% 9827 ± 6% proc-vmstat.pgactivate
> 291848 +1.7% 296689 proc-vmstat.pgalloc_normal
> 438.33 ±122% +1463.6% 6853 ± 69% proc-vmstat.pgmigrate_success
> 252.00 ± 19% +30.8% 329.67 ± 7% numa-vmstat.node0.nr_active_anon
> 44373 ± 23% -64.0% 15967 ± 49% numa-vmstat.node0.nr_anon_pages
> 46179 ± 21% -61.8% 17619 ± 38% numa-vmstat.node0.nr_inactive_anon
> 252.00 ± 19% +30.8% 329.67 ± 7% numa-vmstat.node0.nr_zone_active_anon
> 46179 ± 21% -61.8% 17619 ± 38% numa-vmstat.node0.nr_zone_inactive_anon
> 3678 ± 4% +11.5% 4100 ± 4% numa-vmstat.node1.nr_active_anon
> 13880 ± 74% +206.7% 42565 ± 17% numa-vmstat.node1.nr_anon_pages
> 16004 ± 59% +181.1% 44988 ± 14% numa-vmstat.node1.nr_inactive_anon
> 3678 ± 4% +11.5% 4100 ± 4% numa-vmstat.node1.nr_zone_active_anon
> 16004 ± 59% +181.1% 44988 ± 14% numa-vmstat.node1.nr_zone_inactive_anon
> 1007 ± 19% +31.3% 1322 ± 7% numa-meminfo.node0.Active
> 1007 ± 19% +31.3% 1322 ± 7% numa-meminfo.node0.Active(anon)
> 39280 ± 35% -66.1% 13314 ± 74% numa-meminfo.node0.AnonHugePages
> 177224 ± 24% -63.9% 63933 ± 49% numa-meminfo.node0.AnonPages
> 182948 ± 22% -52.3% 87286 ± 59% numa-meminfo.node0.AnonPages.max
> 184457 ± 21% -61.8% 70532 ± 38% numa-meminfo.node0.Inactive
> 184457 ± 21% -61.8% 70532 ± 38% numa-meminfo.node0.Inactive(anon)
> 1080630 ± 5% -12.3% 947332 ± 3% numa-meminfo.node0.MemUsed
> 14267 ± 4% +17.0% 16695 ± 5% numa-meminfo.node1.Active
> 14267 ± 4% +17.0% 16695 ± 5% numa-meminfo.node1.Active(anon)
> 56124 ± 74% +202.8% 169963 ± 17% numa-meminfo.node1.AnonPages
> 75025 ± 56% +139.7% 179816 ± 14% numa-meminfo.node1.AnonPages.max
> 64951 ± 59% +176.3% 179452 ± 14% numa-meminfo.node1.Inactive
> 64951 ± 59% +176.3% 179452 ± 14% numa-meminfo.node1.Inactive(anon)
> 941178 ± 5% +14.6% 1078500 ± 3% numa-meminfo.node1.MemUsed
> 51.14 ± 15% +47.4% 75.37 ± 13% sched_debug.cfs_rq:/.load_avg.avg
> 114.91 ± 14% +38.2% 158.79 ± 13% sched_debug.cfs_rq:/.load_avg.stddev
> 6.99 ± 70% +352.8% 31.65 ± 27% sched_debug.cfs_rq:/.removed.load_avg.avg
> 47.92 ± 70% +153.1% 121.29 ± 12% sched_debug.cfs_rq:/.removed.load_avg.stddev
> 2.38 ± 97% +182.8% 6.74 ± 36% sched_debug.cfs_rq:/.removed.runnable_avg.avg
> 2.38 ± 97% +182.7% 6.73 ± 36% sched_debug.cfs_rq:/.removed.util_avg.avg
> 356.50 ± 3% -24.1% 270.50 ± 31% sched_debug.cfs_rq:/.util_avg.min
> 290.54 ± 2% +10.6% 321.30 ± 5% sched_debug.cfs_rq:/.util_est_enqueued.avg
> 446696 ± 4% +50.1% 670473 ± 7% sched_debug.cpu.avg_idle.avg
> 1120651 ± 11% +20.5% 1350114 ± 10% sched_debug.cpu.avg_idle.max
> 2124 ± 10% +43.0% 3038 ± 15% sched_debug.cpu.avg_idle.min
> 708.23 ± 3% -14.4% 606.42 ± 4% sched_debug.cpu.clock_task.stddev
> 989.50 ± 8% +12.0% 1108 sched_debug.cpu.curr->pid.min
> 21503 ± 71% +114.0% 46014 ± 39% sched_debug.cpu.max_idle_balance_cost.stddev
> 4583665 ± 7% -68.5% 1443748 ± 9% sched_debug.cpu.nr_switches.avg
> 6508139 ± 3% -71.5% 1854831 sched_debug.cpu.nr_switches.max
> 2183394 ± 28% -83.2% 367194 ± 87% sched_debug.cpu.nr_switches.min
> 1135976 ± 11% -69.9% 342489 ± 34% sched_debug.cpu.nr_switches.stddev
> 0.03 ± 59% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.01 ± 12% -36.4% 0.00 ± 26% perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 0.05 ± 57% -100.0% 0.00 perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.19 ± 77% -61.0% 0.07 ± 5% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 7.83 ±140% -99.8% 0.01 ± 38% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
> 0.36 ± 18% -64.4% 0.13 ± 75% perf-sched.total_wait_and_delay.average.ms
> 0.24 ± 5% -72.2% 0.07 ± 66% perf-sched.total_wait_time.average.ms
> 1.38 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.03 ± 41% -61.6% 0.01 ± 80% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
> 0.01 ± 12% -36.4% 0.00 ± 26% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 0.08 ± 17% -30.3% 0.05 ± 3% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 2.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.do_syslog.part.0
> 9.00 -81.5% 1.67 ± 28% perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
> 2.72 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.80 ± 45% -89.3% 0.09 ± 10% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 7.83 ±140% -99.8% 0.01 ± 38% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
> 1.35 ± 8% -100.0% 0.00 perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.03 ± 42% -66.3% 0.01 ± 82% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
> 0.07 ± 24% -27.6% 0.05 ± 3% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 3.21 ± 2% -58.6% 1.33 ± 70% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
> 2.69 ± 8% -100.0% 0.00 perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.77 ± 50% -89.7% 0.08 ± 10% perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 6.12 ± 3% -27.9% 4.42 ± 3% perf-stat.i.MPKI
> 1.008e+10 ± 4% -20.8% 7.989e+09 ± 2% perf-stat.i.branch-instructions
> 1.55 ± 2% -0.7 0.89 perf-stat.i.branch-miss-rate%
> 1.372e+08 ± 2% -60.4% 54290819 perf-stat.i.branch-misses
> 5.24 ± 7% +6.4 11.66 ± 5% perf-stat.i.cache-miss-rate%
> 12610881 ± 8% +30.8% 16498692 ± 4% perf-stat.i.cache-misses
> 2.994e+08 -50.1% 1.495e+08 ± 2% perf-stat.i.cache-references
> 7370989 ± 7% -68.6% 2316288 ± 9% perf-stat.i.context-switches
> 2.75 ± 4% +37.7% 3.78 ± 2% perf-stat.i.cpi
> 37935 ± 20% -99.1% 344.33 ± 39% perf-stat.i.cpu-migrations
> 10857 ± 9% -24.6% 8184 ± 4% perf-stat.i.cycles-between-cache-misses
> 1.33 ± 4% -0.5 0.81 ± 15% perf-stat.i.dTLB-load-miss-rate%
> 1.921e+08 ± 3% -59.4% 78058438 ± 14% perf-stat.i.dTLB-load-misses
> 1.375e+10 ± 4% -32.6% 9.26e+09 perf-stat.i.dTLB-loads
> 0.51 ± 5% +0.1 0.61 ± 9% perf-stat.i.dTLB-store-miss-rate%
> 52874010 -55.6% 23456497 ± 13% perf-stat.i.dTLB-store-misses
> 9.955e+09 ± 3% -63.0% 3.687e+09 ± 4% perf-stat.i.dTLB-stores
> 62946854 ± 4% -59.7% 25392611 ± 3% perf-stat.i.iTLB-load-misses
> 4.88e+10 ± 4% -28.0% 3.514e+10 perf-stat.i.instructions
> 1071 +54.5% 1655 ± 10% perf-stat.i.instructions-per-iTLB-miss
> 0.38 ± 3% -26.8% 0.28 ± 3% perf-stat.i.ipc
> 0.52 ± 7% -70.6% 0.15 ± 11% perf-stat.i.metric.K/sec
> 711.88 ± 3% -38.1% 440.48 perf-stat.i.metric.M/sec
> 47.42 -1.7 45.75 perf-stat.i.node-load-miss-rate%
> 3468825 ± 20% +118.3% 7571866 ± 5% perf-stat.i.node-load-misses
> 3747717 ± 20% +133.9% 8764955 ± 4% perf-stat.i.node-loads
> 47.67 -2.6 45.11 perf-stat.i.node-store-miss-rate%
> 8290895 ± 4% -28.2% 5954593 ± 6% perf-stat.i.node-store-misses
> 8800840 -19.9% 7053271 ± 5% perf-stat.i.node-stores
> 6.14 ± 3% -30.8% 4.25 perf-stat.overall.MPKI
> 1.36 ± 2% -0.7 0.68 perf-stat.overall.branch-miss-rate%
> 4.22 ± 10% +6.8 11.05 ± 6% perf-stat.overall.cache-miss-rate%
> 2.79 ± 4% +38.7% 3.87 perf-stat.overall.cpi
> 10868 ± 8% -23.9% 8267 ± 4% perf-stat.overall.cycles-between-cache-misses
> 1.38 ± 4% -0.5 0.84 ± 15% perf-stat.overall.dTLB-load-miss-rate%
> 0.53 ± 5% +0.1 0.63 ± 8% perf-stat.overall.dTLB-store-miss-rate%
> 775.35 +78.9% 1386 ± 5% perf-stat.overall.instructions-per-iTLB-miss
> 0.36 ± 4% -28.0% 0.26 perf-stat.overall.ipc
> 48.07 -1.7 46.34 perf-stat.overall.node-load-miss-rate%
> 48.49 -2.7 45.76 perf-stat.overall.node-store-miss-rate%
> 9.922e+09 ± 4% -20.8% 7.859e+09 ± 2% perf-stat.ps.branch-instructions
> 1.351e+08 ± 2% -60.4% 53438494 perf-stat.ps.branch-misses
> 12410866 ± 8% +30.8% 16234283 ± 4% perf-stat.ps.cache-misses
> 2.946e+08 -50.1% 1.47e+08 ± 2% perf-stat.ps.cache-references
> 7252050 ± 7% -68.6% 2278311 ± 9% perf-stat.ps.context-switches
> 37323 ± 20% -99.1% 338.80 ± 39% perf-stat.ps.cpu-migrations
> 1.89e+08 ± 3% -59.4% 76787292 ± 14% perf-stat.ps.dTLB-load-misses
> 1.352e+10 ± 4% -32.6% 9.11e+09 perf-stat.ps.dTLB-loads
> 52020930 -55.6% 23075080 ± 13% perf-stat.ps.dTLB-store-misses
> 9.794e+09 ± 3% -63.0% 3.627e+09 ± 4% perf-stat.ps.dTLB-stores
> 61931461 ± 4% -59.7% 24978175 ± 3% perf-stat.ps.iTLB-load-misses
> 4.802e+10 ± 4% -28.0% 3.457e+10 perf-stat.ps.instructions
> 3412948 ± 20% +118.2% 7448345 ± 5% perf-stat.ps.node-load-misses
> 3687448 ± 20% +133.8% 8622075 ± 4% perf-stat.ps.node-loads
> 8157655 ± 4% -28.2% 5857930 ± 6% perf-stat.ps.node-store-misses
> 8659916 -19.9% 6939197 ± 5% perf-stat.ps.node-stores
> 3.035e+12 ± 4% -27.9% 2.189e+12 perf-stat.total.instructions
> 82.33 ± 64% +1322.3% 1171 ±127% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> 17959753 ± 6% -99.7% 45889 ± 2% interrupts.CAL:Function_call_interrupts
> 103593 ±125% -99.5% 569.33 ± 9% interrupts.CPU0.CAL:Function_call_interrupts
> 45339 ±104% -94.7% 2414 ± 30% interrupts.CPU0.RES:Rescheduling_interrupts
> 1078794 ±112% -99.9% 1002 ± 12% interrupts.CPU1.CAL:Function_call_interrupts
> 6507 ± 26% +21.7% 7922 ± 6% interrupts.CPU1.NMI:Non-maskable_interrupts
> 6507 ± 26% +21.7% 7922 ± 6% interrupts.CPU1.PMI:Performance_monitoring_interrupts
> 505526 ±101% -99.8% 1223 ± 46% interrupts.CPU1.RES:Rescheduling_interrupts
> 791738 ± 80% -99.9% 898.67 ± 13% interrupts.CPU10.CAL:Function_call_interrupts
> 406363 ± 71% -99.4% 2378 ± 80% interrupts.CPU10.RES:Rescheduling_interrupts
> 46401 ±115% -98.3% 789.00 ± 21% interrupts.CPU11.CAL:Function_call_interrupts
> 5509 ± 35% +25.8% 6931 ± 28% interrupts.CPU11.NMI:Non-maskable_interrupts
> 5509 ± 35% +25.8% 6931 ± 28% interrupts.CPU11.PMI:Performance_monitoring_interrupts
> 27829 ±104% -87.3% 3526 ± 77% interrupts.CPU11.RES:Rescheduling_interrupts
> 290182 ±130% -99.6% 1299 ± 24% interrupts.CPU12.CAL:Function_call_interrupts
> 170757 ±136% -98.3% 2886 ± 40% interrupts.CPU12.RES:Rescheduling_interrupts
> 131940 ± 43% -99.1% 1206 ± 37% interrupts.CPU13.CAL:Function_call_interrupts
> 68303 ± 32% -97.9% 1445 ±126% interrupts.CPU13.RES:Rescheduling_interrupts
> 241043 ±116% -99.5% 1319 ± 32% interrupts.CPU14.CAL:Function_call_interrupts
> 98573 ± 93% -97.0% 3005 ± 47% interrupts.CPU14.RES:Rescheduling_interrupts
> 874074 ± 72% -99.9% 875.33 ± 6% interrupts.CPU15.CAL:Function_call_interrupts
> 350120 ± 49% -99.9% 412.67 ± 52% interrupts.CPU15.RES:Rescheduling_interrupts
> 577696 ± 23% -99.9% 855.33 ± 3% interrupts.CPU16.CAL:Function_call_interrupts
> 8263 -24.9% 6202 ± 23% interrupts.CPU16.NMI:Non-maskable_interrupts
> 8263 -24.9% 6202 ± 23% interrupts.CPU16.PMI:Performance_monitoring_interrupts
> 237815 ± 15% -99.5% 1220 ± 53% interrupts.CPU16.RES:Rescheduling_interrupts
> 686797 ±112% -99.9% 933.67 ± 13% interrupts.CPU17.CAL:Function_call_interrupts
> 317108 ±105% -99.8% 536.00 ± 49% interrupts.CPU17.RES:Rescheduling_interrupts
> 748163 ±122% -99.9% 863.00 ± 5% interrupts.CPU18.CAL:Function_call_interrupts
> 320920 ±112% -99.5% 1469 ± 95% interrupts.CPU18.RES:Rescheduling_interrupts
> 286835 ±121% -99.7% 943.33 ± 15% interrupts.CPU19.CAL:Function_call_interrupts
> 174455 ±113% -98.7% 2326 ± 69% interrupts.CPU19.RES:Rescheduling_interrupts
> 443377 ± 83% -99.8% 944.33 ± 12% interrupts.CPU2.CAL:Function_call_interrupts
> 57297 ± 99% -98.5% 855.00 ± 6% interrupts.CPU20.CAL:Function_call_interrupts
> 54090 ± 67% -99.0% 535.33 ± 74% interrupts.CPU20.RES:Rescheduling_interrupts
> 27584 ± 80% -96.8% 890.33 ± 6% interrupts.CPU21.CAL:Function_call_interrupts
> 27052 ± 82% -94.3% 1539 ± 58% interrupts.CPU21.RES:Rescheduling_interrupts
> 62804 ±104% -97.8% 1362 ± 47% interrupts.CPU22.CAL:Function_call_interrupts
> 27230 ± 84% -92.6% 2002 ± 86% interrupts.CPU22.RES:Rescheduling_interrupts
> 351930 ± 72% -99.7% 966.00 ± 16% interrupts.CPU23.CAL:Function_call_interrupts
> 136149 ± 62% -98.1% 2565 ± 93% interrupts.CPU23.RES:Rescheduling_interrupts
> 366644 ±138% -99.8% 877.67 ± 5% interrupts.CPU24.CAL:Function_call_interrupts
> 499203 ± 82% -99.8% 862.33 ± 3% interrupts.CPU25.CAL:Function_call_interrupts
> 298172 ± 70% -99.2% 2317 ± 19% interrupts.CPU25.RES:Rescheduling_interrupts
> 11844 ± 61% -92.9% 838.33 ± 2% interrupts.CPU26.CAL:Function_call_interrupts
> 82.33 ± 64% +1322.3% 1171 ±127% interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> 1841110 ± 49% -100.0% 746.33 ± 26% interrupts.CPU27.CAL:Function_call_interrupts
> 5501 ± 35% +25.6% 6909 ± 28% interrupts.CPU27.NMI:Non-maskable_interrupts
> 5501 ± 35% +25.6% 6909 ± 28% interrupts.CPU27.PMI:Performance_monitoring_interrupts
> 1089819 ± 55% -99.8% 1727 ± 80% interrupts.CPU27.RES:Rescheduling_interrupts
> 74718 ±118% -98.7% 942.00 interrupts.CPU28.CAL:Function_call_interrupts
> 5504 ± 35% +25.6% 6912 ± 28% interrupts.CPU28.NMI:Non-maskable_interrupts
> 5504 ± 35% +25.6% 6912 ± 28% interrupts.CPU28.PMI:Performance_monitoring_interrupts
> 67380 ±125% -98.5% 1034 ± 37% interrupts.CPU28.RES:Rescheduling_interrupts
> 367428 ± 77% -99.7% 1054 ± 22% interrupts.CPU29.CAL:Function_call_interrupts
> 239038 ± 82% -99.3% 1767 ± 84% interrupts.CPU29.RES:Rescheduling_interrupts
> 220744 ±117% -99.5% 1046 ± 21% interrupts.CPU3.CAL:Function_call_interrupts
> 191269 ±119% -98.4% 3122 ± 15% interrupts.CPU3.RES:Rescheduling_interrupts
> 247241 ± 81% -99.6% 874.33 ± 5% interrupts.CPU30.CAL:Function_call_interrupts
> 161878 ± 84% -98.1% 3063 ± 37% interrupts.CPU30.RES:Rescheduling_interrupts
> 144713 ±110% -99.3% 1077 ± 23% interrupts.CPU31.CAL:Function_call_interrupts
> 5502 ± 35% +25.6% 6911 ± 28% interrupts.CPU31.NMI:Non-maskable_interrupts
> 5502 ± 35% +25.6% 6911 ± 28% interrupts.CPU31.PMI:Performance_monitoring_interrupts
> 84227 ±100% -96.4% 3013 ± 72% interrupts.CPU31.RES:Rescheduling_interrupts
> 104510 ±109% -99.2% 848.00 ± 4% interrupts.CPU32.CAL:Function_call_interrupts
> 5515 ± 35% +31.4% 7249 ± 20% interrupts.CPU32.NMI:Non-maskable_interrupts
> 5515 ± 35% +31.4% 7249 ± 20% interrupts.CPU32.PMI:Performance_monitoring_interrupts
> 70068 ±101% -96.7% 2320 ±126% interrupts.CPU32.RES:Rescheduling_interrupts
> 87671 ±131% -99.0% 857.33 ± 2% interrupts.CPU33.CAL:Function_call_interrupts
> 35054 ±127% -96.5% 1242 ± 80% interrupts.CPU33.RES:Rescheduling_interrupts
> 77336 ±124% -98.7% 988.33 ± 21% interrupts.CPU34.CAL:Function_call_interrupts
> 66106 ±123% -96.0% 2630 ± 52% interrupts.CPU34.RES:Rescheduling_interrupts
> 136901 ±122% -99.1% 1278 ± 30% interrupts.CPU35.CAL:Function_call_interrupts
> 731054 ± 69% -99.9% 1046 ± 15% interrupts.CPU36.CAL:Function_call_interrupts
> 376136 ± 70% -99.6% 1429 ± 37% interrupts.CPU36.RES:Rescheduling_interrupts
> 941442 ±105% -99.9% 851.33 ± 7% interrupts.CPU37.CAL:Function_call_interrupts
> 300885 ± 80% -99.2% 2379 ± 57% interrupts.CPU37.RES:Rescheduling_interrupts
> 242878 ±117% -99.6% 995.00 ± 16% interrupts.CPU38.CAL:Function_call_interrupts
> 702836 ± 69% -99.9% 855.00 ± 3% interrupts.CPU39.CAL:Function_call_interrupts
> 286450 ± 74% -99.5% 1448 ± 70% interrupts.CPU39.RES:Rescheduling_interrupts
> 37564 ± 55% -96.0% 1505 ± 62% interrupts.CPU4.CAL:Function_call_interrupts
> 6346 ± 26% +24.0% 7870 ± 6% interrupts.CPU4.NMI:Non-maskable_interrupts
> 6346 ± 26% +24.0% 7870 ± 6% interrupts.CPU4.PMI:Performance_monitoring_interrupts
> 24154 ± 65% -94.0% 1444 ± 83% interrupts.CPU4.RES:Rescheduling_interrupts
> 977103 ± 53% -99.9% 837.33 ± 3% interrupts.CPU40.CAL:Function_call_interrupts
> 512103 ± 71% -99.7% 1579 ± 66% interrupts.CPU40.RES:Rescheduling_interrupts
> 190023 ±117% -99.6% 818.67 ± 2% interrupts.CPU41.CAL:Function_call_interrupts
> 346179 ± 54% -99.8% 829.00 ± 2% interrupts.CPU42.CAL:Function_call_interrupts
> 183103 ± 48% -99.5% 850.00 ± 54% interrupts.CPU42.RES:Rescheduling_interrupts
> 371814 ±131% -99.8% 863.33 ± 5% interrupts.CPU43.CAL:Function_call_interrupts
> 158123 ±124% -99.2% 1224 ± 71% interrupts.CPU43.RES:Rescheduling_interrupts
> 578077 ± 61% -99.9% 850.67 ± 3% interrupts.CPU44.CAL:Function_call_interrupts
> 331471 ± 53% -99.7% 1120 ± 91% interrupts.CPU44.RES:Rescheduling_interrupts
> 172794 ±114% -99.5% 838.33 ± 4% interrupts.CPU45.CAL:Function_call_interrupts
> 89408 ±111% -99.6% 326.00 ± 23% interrupts.CPU45.RES:Rescheduling_interrupts
> 34858 ± 59% -97.3% 947.00 ± 13% interrupts.CPU46.CAL:Function_call_interrupts
> 22612 ± 53% -87.6% 2794 ± 27% interrupts.CPU46.RES:Rescheduling_interrupts
> 41349 ± 57% -97.6% 996.67 ± 10% interrupts.CPU47.CAL:Function_call_interrupts
> 27498 ± 54% -93.7% 1725 ± 89% interrupts.CPU47.RES:Rescheduling_interrupts
> 45388 ± 61% -97.5% 1141 ± 57% interrupts.CPU5.CAL:Function_call_interrupts
> 32500 ± 63% -96.7% 1063 ± 95% interrupts.CPU5.RES:Rescheduling_interrupts
> 534772 ±105% -99.8% 843.33 ± 3% interrupts.CPU6.CAL:Function_call_interrupts
> 259978 ± 94% -98.5% 4008 ± 4% interrupts.CPU6.RES:Rescheduling_interrupts
> 589336 ± 71% -99.8% 1042 ± 25% interrupts.CPU7.CAL:Function_call_interrupts
> 311886 ± 69% -99.4% 1966 ± 66% interrupts.CPU7.RES:Rescheduling_interrupts
> 363153 ± 95% -99.8% 858.00 ± 5% interrupts.CPU8.CAL:Function_call_interrupts
> 190915 ± 83% -98.1% 3681 ± 94% interrupts.CPU8.RES:Rescheduling_interrupts
> 78806 ± 46% -98.9% 905.33 ± 7% interrupts.CPU9.CAL:Function_call_interrupts
> 47384 ± 29% -95.9% 1963 ± 58% interrupts.CPU9.RES:Rescheduling_interrupts
> 9209745 ± 15% -99.0% 95188 ± 9% interrupts.RES:Rescheduling_interrupts
> 20.15 ± 4% -14.2 5.98 ± 15% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 15.92 ± 4% -11.9 3.98 ± 7% perf-profile.calltrace.cycles-pp.signal_wake_up_state.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
> 15.71 ± 4% -11.8 3.95 ± 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info.kill_pid_info
> 13.68 ± 6% -10.8 2.93 ± 10% perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 12.23 ± 26% -10.4 1.80 ± 38% perf-profile.calltrace.cycles-pp.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64
> 11.55 ± 27% -10.0 1.56 ± 39% perf-profile.calltrace.cycles-pp.security_task_kill.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
> 11.33 ± 27% -9.8 1.49 ± 40% perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
> 11.78 ± 5% -9.3 2.52 ± 9% perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
> 11.53 ± 5% -9.1 2.47 ± 9% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait
> 8.27 ± 13% -6.3 1.99 ± 7% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 7.32 ± 2% -4.9 2.43 ± 25% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
> 6.20 ± 13% -4.7 1.53 ± 10% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 5.94 ± 13% -4.5 1.46 ± 10% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 6.32 ± 8% -4.3 2.07 ± 6% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info
> 4.57 ± 30% -4.1 0.44 ± 74% perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.group_send_sig_info.kill_pid_info
> 5.55 ± 8% -3.7 1.90 ± 6% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.signal_wake_up_state.__send_signal
> 4.43 ± 7% -3.4 0.99 ± 6% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info
> 4.29 ± 7% -3.3 0.95 ± 6% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.signal_wake_up_state.__send_signal
> 4.19 ± 6% -3.3 0.92 ± 8% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait
> 4.60 -2.9 1.69 ± 28% perf-profile.calltrace.cycles-pp.__entry_text_start
> 1.93 ± 16% -1.6 0.37 ± 70% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
> 2.40 ± 6% -1.5 0.86 ± 6% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.signal_wake_up_state
> 0.00 +0.7 0.67 perf-profile.calltrace.cycles-pp.inc_rlimit_ucounts_and_test.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info
> 0.67 ± 16% +2.4 3.11 ± 53% perf-profile.calltrace.cycles-pp.__lock_task_sighand.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
> 0.42 ± 71% +2.7 3.08 ± 53% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
> 0.00 +3.0 2.98 ± 54% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.kill_pid_info
> 0.00 +3.2 3.23 ± 51% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
> 0.00 +3.3 3.29 ± 51% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 35.47 ± 7% +9.5 45.02 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 34.17 ± 7% +10.4 44.54 perf-profile.calltrace.cycles-pp.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 33.33 ± 7% +10.9 44.22 perf-profile.calltrace.cycles-pp.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 81.67 +12.6 94.25 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 19.16 ± 4% +19.5 38.64 ± 8% perf-profile.calltrace.cycles-pp.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
> 21.31 ± 6% +20.5 41.77 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 20.83 ± 3% +21.5 42.33 ± 3% perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64
> 19.46 ± 5% +21.9 41.31 ± 4% perf-profile.calltrace.cycles-pp.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 59.72 ± 2% +28.1 87.78 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 3.78 ± 8% +30.8 34.58 ± 9% perf-profile.calltrace.cycles-pp.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 3.07 ± 8% +31.3 34.40 ± 9% perf-profile.calltrace.cycles-pp.__dequeue_signal.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
> 1.74 ± 9% +32.1 33.86 ± 9% perf-profile.calltrace.cycles-pp.__sigqueue_free.__dequeue_signal.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait
> 1.94 ± 8% +32.5 34.40 ± 8% perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
> 0.00 +32.8 32.81 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.__dequeue_signal
> 0.00 +33.0 33.02 ± 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal
> 0.00 +33.2 33.22 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.__dequeue_signal.dequeue_signal
> 0.00 +33.3 33.32 ± 9% perf-profile.calltrace.cycles-pp.put_ucounts.__sigqueue_free.__dequeue_signal.dequeue_signal.do_sigtimedwait
> 0.00 +33.5 33.48 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal.do_send_sig_info
> 0.00 +33.6 33.57 ± 8% perf-profile.calltrace.cycles-pp.get_ucounts.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info
> 20.22 ± 4% -14.2 6.00 ± 15% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
> 18.14 ± 8% -14.1 4.06 ± 10% perf-profile.children.cycles-pp.schedule
> 17.92 ± 7% -14.0 3.96 ± 9% perf-profile.children.cycles-pp.__sched_text_start
> 15.92 ± 4% -11.9 3.98 ± 7% perf-profile.children.cycles-pp.signal_wake_up_state
> 15.73 ± 4% -11.8 3.96 ± 7% perf-profile.children.cycles-pp.try_to_wake_up
> 13.71 ± 6% -10.8 2.93 ± 10% perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
> 12.24 ± 26% -10.4 1.80 ± 38% perf-profile.children.cycles-pp.group_send_sig_info
> 11.56 ± 27% -10.0 1.56 ± 39% perf-profile.children.cycles-pp.security_task_kill
> 11.35 ± 27% -9.9 1.49 ± 40% perf-profile.children.cycles-pp.apparmor_task_kill
> 8.45 ± 12% -6.5 2.00 ± 7% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
> 8.04 -5.3 2.71 ± 25% perf-profile.children.cycles-pp.syscall_return_via_sysret
> 6.34 ± 8% -4.3 2.07 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair
> 4.58 ± 30% -4.0 0.55 ± 36% perf-profile.children.cycles-pp.aa_get_task_label
> 6.03 -4.0 2.05 ± 28% perf-profile.children.cycles-pp.__entry_text_start
> 4.70 ± 6% -3.7 0.99 ± 6% perf-profile.children.cycles-pp.ttwu_do_activate
> 5.63 ± 8% -3.7 1.93 ± 6% perf-profile.children.cycles-pp.select_idle_sibling
> 4.57 ± 6% -3.6 0.96 ± 6% perf-profile.children.cycles-pp.enqueue_task_fair
> 4.23 ± 7% -3.3 0.93 ± 8% perf-profile.children.cycles-pp.dequeue_task_fair
> 3.78 ± 12% -2.9 0.91 ± 11% perf-profile.children.cycles-pp.pick_next_task_fair
> 3.44 ± 9% -2.7 0.78 ± 9% perf-profile.children.cycles-pp.switch_mm_irqs_off
> 3.07 ± 10% -2.4 0.68 ± 9% perf-profile.children.cycles-pp.update_load_avg
> 2.96 ± 3% -2.3 0.69 ± 9% perf-profile.children.cycles-pp.update_curr
> 2.35 ± 3% -1.9 0.48 ± 6% perf-profile.children.cycles-pp.enqueue_entity
> 2.26 ± 13% -1.7 0.56 ± 10% perf-profile.children.cycles-pp.load_new_mm_cr3
> 2.44 ± 6% -1.6 0.86 ± 5% perf-profile.children.cycles-pp.available_idle_cpu
> 1.82 ± 4% -1.4 0.41 ± 7% perf-profile.children.cycles-pp.dequeue_entity
> 1.93 ± 3% -1.4 0.52 ± 32% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
> 1.63 ± 10% -1.3 0.28 ± 8% perf-profile.children.cycles-pp._raw_spin_lock
> 1.59 ± 13% -1.3 0.30 ± 10% perf-profile.children.cycles-pp.switch_fpu_return
> 1.63 ± 11% -1.2 0.39 ± 10% perf-profile.children.cycles-pp.reweight_entity
> 1.47 ± 7% -1.2 0.31 ± 11% perf-profile.children.cycles-pp.__switch_to
> 1.25 ± 12% -1.0 0.27 ± 13% perf-profile.children.cycles-pp.__switch_to_asm
> 1.21 ± 10% -0.9 0.27 ± 13% perf-profile.children.cycles-pp.set_next_entity
> 1.07 ± 7% -0.9 0.13 ± 25% perf-profile.children.cycles-pp.finish_task_switch
> 1.30 ± 4% -0.9 0.41 ± 10% perf-profile.children.cycles-pp._copy_from_user
> 1.11 ± 7% -0.8 0.27 ± 12% perf-profile.children.cycles-pp.update_rq_clock
> 1.04 ± 4% -0.8 0.24 ± 15% perf-profile.children.cycles-pp.ttwu_do_wakeup
> 0.99 ± 13% -0.8 0.21 ± 14% perf-profile.children.cycles-pp.hrtimer_start_range_ns
> 0.98 ± 4% -0.8 0.22 ± 13% perf-profile.children.cycles-pp.check_preempt_curr
> 1.03 -0.8 0.28 ± 4% perf-profile.children.cycles-pp.recalc_sigpending
> 0.97 ± 12% -0.7 0.22 ± 10% perf-profile.children.cycles-pp.__update_load_avg_se
> 0.90 ± 8% -0.7 0.18 ± 12% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
> 0.95 ± 10% -0.7 0.23 ± 10% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
> 0.94 ± 33% -0.7 0.23 ± 40% perf-profile.children.cycles-pp.aa_may_signal
> 0.94 ± 6% -0.7 0.23 ± 2% perf-profile.children.cycles-pp.copy_siginfo_to_user
> 0.95 ± 6% -0.7 0.27 ± 9% perf-profile.children.cycles-pp.__might_fault
> 0.85 ± 5% -0.7 0.19 ± 11% perf-profile.children.cycles-pp.check_preempt_wakeup
> 0.83 ± 9% -0.6 0.19 ± 7% perf-profile.children.cycles-pp.___perf_sw_event
> 1.00 ± 2% -0.6 0.36 ± 20% perf-profile.children.cycles-pp.__copy_siginfo_from_user
> 0.79 ± 17% -0.6 0.19 ± 11% perf-profile.children.cycles-pp.put_prev_entity
> 0.80 ± 12% -0.6 0.20 ± 8% perf-profile.children.cycles-pp.pick_next_entity
> 0.82 ± 2% -0.5 0.29 ± 24% perf-profile.children.cycles-pp.__x64_sys_getpid
> 0.68 ± 2% -0.5 0.16 ± 3% perf-profile.children.cycles-pp.update_cfs_group
> 0.87 ± 17% -0.5 0.38 perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
> 0.53 ± 6% -0.5 0.06 ± 16% perf-profile.children.cycles-pp.complete_signal
> 0.58 ± 6% -0.5 0.12 ± 10% perf-profile.children.cycles-pp.sched_clock_cpu
> 0.55 ± 5% -0.4 0.10 ± 12% perf-profile.children.cycles-pp.hrtimer_try_to_cancel
> 0.79 ± 12% -0.4 0.35 ± 7% perf-profile.children.cycles-pp.cpumask_next_wrap
> 0.62 ± 18% -0.4 0.19 ± 10% perf-profile.children.cycles-pp._find_next_bit
> 0.71 ± 2% -0.4 0.28 ± 34% perf-profile.children.cycles-pp.__task_pid_nr_ns
> 0.62 ± 3% -0.4 0.21 ± 30% perf-profile.children.cycles-pp.__x64_sys_getuid
> 0.53 ± 14% -0.4 0.14 ± 17% perf-profile.children.cycles-pp.__calc_delta
> 0.59 ± 8% -0.4 0.20 ± 32% perf-profile.children.cycles-pp.check_kill_permission
> 0.50 ± 6% -0.4 0.11 ± 11% perf-profile.children.cycles-pp.sched_clock
> 0.50 ± 13% -0.4 0.12 ± 10% perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
> 0.47 ± 6% -0.4 0.09 ± 18% perf-profile.children.cycles-pp.perf_tp_event
> 0.48 ± 6% -0.4 0.10 ± 9% perf-profile.children.cycles-pp.native_sched_clock
> 0.48 ± 3% -0.4 0.11 ± 4% perf-profile.children.cycles-pp.___might_sleep
> 0.74 ± 12% -0.4 0.39 perf-profile.children.cycles-pp.kmem_cache_free
> 0.45 ± 8% -0.3 0.13 ± 13% perf-profile.children.cycles-pp._copy_to_user
> 0.36 ± 5% -0.3 0.06 ± 8% perf-profile.children.cycles-pp.__set_task_blocked
> 0.35 ± 22% -0.3 0.07 ± 11% perf-profile.children.cycles-pp.cpumask_next
> 0.38 ± 7% -0.3 0.10 ± 14% perf-profile.children.cycles-pp.ktime_get
> 0.46 ± 10% -0.3 0.19 ± 36% perf-profile.children.cycles-pp.__radix_tree_lookup
> 0.64 ± 4% -0.3 0.37 ± 6% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.33 ± 16% -0.3 0.06 ± 14% perf-profile.children.cycles-pp.cpuacct_charge
> 0.53 -0.3 0.26 ± 33% perf-profile.children.cycles-pp.send_signal
> 0.33 ± 16% -0.3 0.07 ± 12% perf-profile.children.cycles-pp.__wrgsbase_inactive
> 0.34 ± 8% -0.2 0.09 ± 5% perf-profile.children.cycles-pp.update_min_vruntime
> 0.39 ± 3% -0.2 0.15 ± 12% perf-profile.children.cycles-pp.copy_user_generic_unrolled
> 0.32 ± 6% -0.2 0.07 ± 6% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
> 0.28 ± 2% -0.2 0.04 ± 71% perf-profile.children.cycles-pp.lock_hrtimer_base
> 0.30 ± 9% -0.2 0.06 ± 7% perf-profile.children.cycles-pp.set_next_buddy
> 0.35 ± 5% -0.2 0.11 ± 36% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
> 0.30 ± 5% -0.2 0.07 ± 17% perf-profile.children.cycles-pp.rb_erase
> 0.32 ± 2% -0.2 0.10 ± 9% perf-profile.children.cycles-pp.__x86_retpoline_rax
> 0.30 ± 5% -0.2 0.08 ± 14% perf-profile.children.cycles-pp.__might_sleep
> 0.29 ± 7% -0.2 0.07 ± 12% perf-profile.children.cycles-pp.__clear_user
> 0.31 ± 14% -0.2 0.09 ± 9% perf-profile.children.cycles-pp.clear_buddies
> 0.25 ± 3% -0.2 0.05 perf-profile.children.cycles-pp.recalc_sigpending_tsk
> 0.29 ± 12% -0.2 0.09 ± 22% perf-profile.children.cycles-pp.prepare_signal
> 0.30 ± 4% -0.2 0.11 ± 34% perf-profile.children.cycles-pp.from_kuid_munged
> 0.27 ± 4% -0.2 0.09 ± 36% perf-profile.children.cycles-pp.map_id_up
> 0.21 ± 21% -0.2 0.03 ± 70% perf-profile.children.cycles-pp.enqueue_hrtimer
> 0.22 ± 13% -0.2 0.06 ± 8% perf-profile.children.cycles-pp.get_timespec64
> 0.22 ± 17% -0.2 0.06 ± 86% perf-profile.children.cycles-pp.__cgroup_account_cputime
> 0.20 ± 4% -0.2 0.04 ± 76% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
> 0.20 ± 6% -0.2 0.05 perf-profile.children.cycles-pp.__list_del_entry_valid
> 0.19 ± 7% -0.2 0.04 ± 71% perf-profile.children.cycles-pp.perf_trace_sched_switch
> 0.23 ± 8% -0.1 0.08 ± 31% perf-profile.children.cycles-pp.audit_signal_info
> 0.22 ± 5% -0.1 0.08 ± 16% perf-profile.children.cycles-pp.rcu_read_unlock_strict
> 0.17 ± 9% -0.1 0.03 ± 70% perf-profile.children.cycles-pp.read_tsc
> 0.18 ± 7% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.audit_signal_info_syscall
> 0.17 ± 5% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc
> 0.00 +0.5 0.51 ± 3% perf-profile.children.cycles-pp.dec_rlimit_ucounts
> 0.00 +0.7 0.67 ± 2% perf-profile.children.cycles-pp.inc_rlimit_ucounts_and_test
> 0.68 ± 16% +2.4 3.11 ± 53% perf-profile.children.cycles-pp.__lock_task_sighand
> 0.42 ± 6% +2.9 3.30 ± 51% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 35.50 ± 7% +9.5 45.03 perf-profile.children.cycles-pp.__x64_sys_rt_sigqueueinfo
> 34.19 ± 7% +10.4 44.55 perf-profile.children.cycles-pp.do_rt_sigqueueinfo
> 33.35 ± 7% +10.9 44.22 perf-profile.children.cycles-pp.kill_pid_info
> 81.79 +12.5 94.33 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 19.21 ± 4% +19.4 38.65 ± 8% perf-profile.children.cycles-pp.__send_signal
> 21.36 ± 6% +20.4 41.78 ± 4% perf-profile.children.cycles-pp.__x64_sys_rt_sigtimedwait
> 20.86 ± 3% +21.5 42.34 ± 3% perf-profile.children.cycles-pp.do_send_sig_info
> 19.50 ± 5% +21.8 41.33 ± 4% perf-profile.children.cycles-pp.do_sigtimedwait
> 59.88 ± 2% +28.0 87.85 ± 2% perf-profile.children.cycles-pp.do_syscall_64
> 3.81 ± 8% +30.9 34.66 ± 9% perf-profile.children.cycles-pp.dequeue_signal
> 3.09 ± 8% +31.3 34.41 ± 9% perf-profile.children.cycles-pp.__dequeue_signal
> 1.74 ± 9% +32.1 33.87 ± 9% perf-profile.children.cycles-pp.__sigqueue_free
> 1.95 ± 8% +32.5 34.40 ± 8% perf-profile.children.cycles-pp.__sigqueue_alloc
> 0.00 +33.3 33.32 ± 9% perf-profile.children.cycles-pp.put_ucounts
> 0.00 +33.6 33.58 ± 8% perf-profile.children.cycles-pp.get_ucounts
> 1.07 ± 11% +68.8 69.87 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.74 ± 39% +71.3 72.04 ± 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> 11.75 ± 2% -7.8 3.95 ± 26% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
> 8.02 -5.3 2.70 ± 25% perf-profile.self.cycles-pp.syscall_return_via_sysret
> 5.78 ± 25% -5.1 0.71 ± 43% perf-profile.self.cycles-pp.apparmor_task_kill
> 4.55 ± 30% -4.0 0.53 ± 34% perf-profile.self.cycles-pp.aa_get_task_label
> 6.02 -4.0 2.05 ± 28% perf-profile.self.cycles-pp.__entry_text_start
> 1.79 ± 9% -1.7 0.06 ± 13% perf-profile.self.cycles-pp.__sigqueue_alloc
> 2.22 ± 9% -1.7 0.50 ± 9% perf-profile.self.cycles-pp.__sched_text_start
> 2.25 ± 13% -1.7 0.55 ± 11% perf-profile.self.cycles-pp.load_new_mm_cr3
> 2.40 ± 6% -1.5 0.85 ± 6% perf-profile.self.cycles-pp.available_idle_cpu
> 1.57 ± 12% -1.3 0.30 ± 10% perf-profile.self.cycles-pp.switch_fpu_return
> 1.74 ± 6% -1.2 0.50 ± 11% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
> 1.79 ± 8% -1.2 0.60 ± 6% perf-profile.self.cycles-pp.select_idle_sibling
> 1.41 ± 7% -1.1 0.30 ± 11% perf-profile.self.cycles-pp.__switch_to
> 1.25 ± 3% -1.0 0.27 ± 8% perf-profile.self.cycles-pp.update_curr
> 1.23 ± 12% -1.0 0.27 ± 13% perf-profile.self.cycles-pp.__switch_to_asm
> 1.17 ± 7% -0.9 0.23 ± 9% perf-profile.self.cycles-pp.update_load_avg
> 1.15 ± 2% -0.9 0.21 ± 5% perf-profile.self.cycles-pp.switch_mm_irqs_off
> 1.13 ± 3% -0.8 0.28 ± 8% perf-profile.self.cycles-pp._raw_spin_lock
> 0.94 ± 13% -0.8 0.14 ± 9% perf-profile.self.cycles-pp.try_to_wake_up
> 1.06 -0.7 0.32 ± 28% perf-profile.self.cycles-pp.do_syscall_64
> 0.93 ± 12% -0.7 0.22 ± 12% perf-profile.self.cycles-pp.__update_load_avg_se
> 0.93 ± 33% -0.7 0.23 ± 40% perf-profile.self.cycles-pp.aa_may_signal
> 0.90 ± 10% -0.7 0.22 ± 9% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
> 0.77 -0.6 0.15 ± 5% perf-profile.self.cycles-pp.recalc_sigpending
> 0.77 ± 13% -0.6 0.20 ± 7% perf-profile.self.cycles-pp.reweight_entity
> 0.74 ± 7% -0.6 0.19 ± 11% perf-profile.self.cycles-pp.update_rq_clock
> 0.67 ± 7% -0.5 0.14 ± 6% perf-profile.self.cycles-pp.___perf_sw_event
> 0.63 ± 7% -0.5 0.11 ± 11% perf-profile.self.cycles-pp.enqueue_entity
> 0.67 ± 3% -0.5 0.15 ± 3% perf-profile.self.cycles-pp.update_cfs_group
> 0.66 ± 9% -0.5 0.16 ± 12% perf-profile.self.cycles-pp.pick_next_task_fair
> 0.61 ± 13% -0.5 0.12 ± 4% perf-profile.self.cycles-pp.enqueue_task_fair
> 0.60 ± 5% -0.5 0.11 ± 14% perf-profile.self.cycles-pp.dequeue_task_fair
> 0.62 ± 8% -0.5 0.14 ± 5% perf-profile.self.cycles-pp.select_task_rq_fair
> 0.63 ± 6% -0.5 0.16 ± 10% perf-profile.self.cycles-pp.do_sigtimedwait
> 0.61 ± 18% -0.4 0.19 ± 10% perf-profile.self.cycles-pp._find_next_bit
> 0.68 ± 3% -0.4 0.27 ± 33% perf-profile.self.cycles-pp.__task_pid_nr_ns
> 0.54 ± 3% -0.4 0.14 ± 6% perf-profile.self.cycles-pp.__dequeue_signal
> 0.51 -0.4 0.12 ± 20% perf-profile.self.cycles-pp.__send_signal
> 0.53 ± 14% -0.4 0.14 ± 15% perf-profile.self.cycles-pp.__calc_delta
> 0.49 ± 13% -0.4 0.12 ± 10% perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
> 0.46 ± 7% -0.4 0.09 ± 10% perf-profile.self.cycles-pp.native_sched_clock
> 0.46 ± 3% -0.4 0.10 ± 4% perf-profile.self.cycles-pp.___might_sleep
> 0.50 ± 7% -0.3 0.16 ± 23% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
> 0.50 ± 3% -0.3 0.17 ± 29% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
> 0.44 ± 14% -0.3 0.10 ± 16% perf-profile.self.cycles-pp.schedule
> 0.42 ± 5% -0.3 0.09 ± 9% perf-profile.self.cycles-pp.finish_task_switch
> 0.43 ± 11% -0.3 0.10 ± 9% perf-profile.self.cycles-pp.pick_next_entity
> 0.41 ± 10% -0.3 0.08 ± 5% perf-profile.self.cycles-pp.__x64_sys_rt_sigtimedwait
> 0.44 ± 6% -0.3 0.13 ± 6% perf-profile.self.cycles-pp._copy_from_user
> 0.50 ± 3% -0.3 0.20 ± 11% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.63 ± 20% -0.3 0.35 ± 6% perf-profile.self.cycles-pp.kmem_cache_free
> 0.37 ± 3% -0.3 0.10 ± 12% perf-profile.self.cycles-pp.check_preempt_wakeup
> 0.37 ± 7% -0.3 0.10 ± 4% perf-profile.self.cycles-pp.dequeue_entity
> 0.33 ± 15% -0.3 0.06 ± 13% perf-profile.self.cycles-pp.cpuacct_charge
> 0.46 ± 11% -0.3 0.19 ± 36% perf-profile.self.cycles-pp.__radix_tree_lookup
> 0.34 ± 3% -0.3 0.07 ± 11% perf-profile.self.cycles-pp._raw_spin_lock_irq
> 0.29 ± 7% -0.3 0.04 ± 71% perf-profile.self.cycles-pp.perf_tp_event
> 0.36 ± 11% -0.3 0.10 ± 4% perf-profile.self.cycles-pp.__might_fault
> 0.32 ± 7% -0.2 0.08 ± 5% perf-profile.self.cycles-pp.update_min_vruntime
> 0.31 ± 17% -0.2 0.07 ± 12% perf-profile.self.cycles-pp.__wrgsbase_inactive
> 0.37 ± 2% -0.2 0.14 ± 13% perf-profile.self.cycles-pp.copy_user_generic_unrolled
> 0.30 ± 4% -0.2 0.07 ± 7% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
> 0.26 ± 8% -0.2 0.04 ± 71% perf-profile.self.cycles-pp.set_next_buddy
> 0.29 ± 5% -0.2 0.07 ± 17% perf-profile.self.cycles-pp.rb_erase
> 0.28 ± 11% -0.2 0.06 ± 14% perf-profile.self.cycles-pp.set_next_entity
> 0.30 ± 4% -0.2 0.09 ± 30% perf-profile.self.cycles-pp.__x64_sys_getuid
> 0.39 ± 13% -0.2 0.18 ± 9% perf-profile.self.cycles-pp.cpumask_next_wrap
> 0.27 ± 5% -0.2 0.07 ± 17% perf-profile.self.cycles-pp.__might_sleep
> 0.31 ± 3% -0.2 0.12 ± 28% perf-profile.self.cycles-pp.__x64_sys_rt_sigqueueinfo
> 0.24 -0.2 0.05 perf-profile.self.cycles-pp.recalc_sigpending_tsk
> 0.28 ± 5% -0.2 0.09 ± 36% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
> 0.27 ± 12% -0.2 0.08 ± 24% perf-profile.self.cycles-pp.prepare_signal
> 0.27 ± 13% -0.2 0.08 ± 10% perf-profile.self.cycles-pp.clear_buddies
> 0.26 ± 3% -0.2 0.08 ± 12% perf-profile.self.cycles-pp.__x86_retpoline_rax
> 0.21 ± 17% -0.2 0.04 ± 71% perf-profile.self.cycles-pp.put_prev_entity
> 0.26 ± 4% -0.2 0.09 ± 36% perf-profile.self.cycles-pp.map_id_up
> 0.23 ± 13% -0.2 0.07 ± 11% perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock
> 0.19 ± 8% -0.2 0.04 ± 70% perf-profile.self.cycles-pp.ktime_get
> 0.19 ± 9% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.perf_trace_sched_switch
> 0.18 ± 4% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.__list_del_entry_valid
> 0.19 ± 8% -0.1 0.05 ± 78% perf-profile.self.cycles-pp.check_kill_permission
> 0.26 -0.1 0.13 ± 26% perf-profile.self.cycles-pp.send_signal
> 0.21 ± 4% -0.1 0.08 ± 26% perf-profile.self.cycles-pp.__x64_sys_getpid
> 0.20 ± 2% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.security_task_kill
> 0.17 ± 4% -0.1 0.05 ± 78% perf-profile.self.cycles-pp.kill_pid_info
> 0.16 ± 8% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.rcu_read_unlock_strict
> 0.16 ± 10% -0.1 0.05 ± 72% perf-profile.self.cycles-pp.audit_signal_info_syscall
> 0.24 ± 8% -0.1 0.14 ± 3% perf-profile.self.cycles-pp.dequeue_signal
> 0.12 ± 4% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.do_rt_sigqueueinfo
> 0.12 ± 11% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.do_send_sig_info
> 0.15 ± 6% -0.0 0.10 perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.00 +0.1 0.07 ± 11% perf-profile.self.cycles-pp.get_ucounts
> 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.put_ucounts
> 0.88 ± 7% +0.2 1.04 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.00 +0.5 0.51 ± 3% perf-profile.self.cycles-pp.dec_rlimit_ucounts
> 0.00 +0.7 0.67 perf-profile.self.cycles-pp.inc_rlimit_ucounts_and_test
> 0.74 ± 39% +71.3 72.04 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Oliver Sang