[linus:master] [sched] af7f588d8f: will-it-scale.per_thread_ops -13.9% regression

From: kernel test robot
Date: Mon May 15 2023 - 03:04:41 EST

Next message: Marc Kleine-Budde: "Re: [PATCH V7] can: usb: f81604: add Fintek F81604 support"
Previous message: Abel Wu: "Re: [PATCH] sock: Fix misuse of sk_under_memory_pressure()"
Next in thread: Peter Zijlstra: "Re: [linus:master] [sched] af7f588d8f: will-it-scale.per_thread_ops -13.9% regression"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hello,

kernel test robot noticed a -13.9% regression of will-it-scale.per_thread_ops on:

commit: af7f588d8f7355bc4298dd1962d7826358fc95f0 ("sched: Introduce per-memory-map concurrency ID")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

testcase: will-it-scale
test machine: 224 threads 2 sockets (Sapphire Rapids) with 256G memory
parameters:

test: context_switch1
cpufreq_governor: performance

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -73.8% regression |
| test machine | 224 threads 2 sockets (Sapphire Rapids) with 256G memory |
| test parameters | cpufreq_governor=performance |
| | mode=thread |
| | nr_task=16 |
| | test=context_switch1 |
+------------------+----------------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -57.9% regression |
| test machine | 104 threads 2 sockets (Skylake) with 192G memory |
| test parameters | cpufreq_governor=performance |
| | mode=thread |
| | nr_task=16 |
| | test=context_switch1 |
+------------------+----------------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -85.0% regression |
| test machine | 104 threads 2 sockets (Skylake) with 192G memory |
| test parameters | cpufreq_governor=performance |
| | mode=thread |
| | nr_task=50% |
| | test=context_switch1 |
+------------------+----------------------------------------------------------------------------------------------------+
| testcase: change | vm-scalability: vm-scalability.throughput -9.0% regression |
| test machine | 96 threads 2 sockets Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz (Cascade Lake) with 128G memory |
| test parameters | cpufreq_governor=performance |
| | runtime=300s |
| | size=2T |
| | test=shm-xread-seq-mt |
+------------------+----------------------------------------------------------------------------------------------------+

FYI, we noticed that commit 223baf9d17f2 (sched: Fix performance
regression introduced by mm_cid) fixed a sysbench regression, but
will-it-scale context_switch1 benchmark still saw a regression on this
fix commit.

Furthermore, we applied the code diff in below link [1] on mainline, and
the will-it-scale score was restored to the original level before this
patch.

[1] https://lore.kernel.org/lkml/d96164a6-c522-1bfc-8b37-333726cdc573@xxxxxxxxxxxx/

If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <yujie.liu@xxxxxxxxx>
| Link: https://lore.kernel.org/oe-lkp/202305151017.27581d75-yujie.liu@xxxxxxxxx

Details are as below:

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase:
gcc-11/performance/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-spr-r02/context_switch1/will-it-scale

commit:
99babd04b2 ("selftests/rseq: Implement rseq numa node id field selftest")
af7f588d8f ("sched: Introduce per-memory-map concurrency ID")

99babd04b2505471 af7f588d8f7355bc4298dd1962d
---------------- ---------------------------
%stddev %change %stddev
\ | \
629260 -1.2% 621753 will-it-scale.1.linear
629260 -1.2% 621753 will-it-scale.1.processes
613314 -1.5% 604163 will-it-scale.1.threads
70477120 -1.2% 69636432 will-it-scale.112.linear
5615798 ± 4% -72.0% 1571768 ± 3% will-it-scale.112.threads
24.94 ± 2% -90.0% 2.50 ± 4% will-it-scale.112.threads_idle
1.057e+08 -1.2% 1.045e+08 will-it-scale.168.linear
56978825 -20.6% 45250154 ± 2% will-it-scale.168.threads
7.49 -73.8% 1.96 ± 28% will-it-scale.168.threads_idle
1.41e+08 -1.2% 1.393e+08 will-it-scale.224.linear
95368601 +1.9% 97207305 will-it-scale.224.processes
1.082e+08 +1.4% 1.097e+08 will-it-scale.224.threads
35238560 -1.2% 34818216 will-it-scale.56.linear
8809640 ± 2% -76.2% 2092567 will-it-scale.56.threads
68.53 -3.4% 66.22 will-it-scale.56.threads_idle
328629 -13.9% 282968 will-it-scale.per_thread_ops
0.68 +3.2% 0.70 will-it-scale.scalability
4.074e+08 ± 12% -37.1% 2.564e+08 ± 8% will-it-scale.time.involuntary_context_switches
17476 +1.4% 17713 will-it-scale.time.system_time
1623 -15.7% 1369 will-it-scale.time.user_time
6.135e+09 -8.6% 5.607e+09 will-it-scale.time.voluntary_context_switches
4.584e+08 -4.2% 4.39e+08 will-it-scale.workload
4.804e+09 -13.3% 4.166e+09 cpuidle..usage
0.12 ± 4% -0.0 0.09 ± 3% mpstat.cpu.all.soft%
22357676 -62.5% 8384327 ± 4% vmstat.memory.cache
59747111 -6.3% 55960083 vmstat.system.cs
433036 ± 12% +511.7% 2648925 ± 42% numa-numastat.node0.local_node
572995 ± 11% +391.1% 2814064 ± 39% numa-numastat.node0.numa_hit
8059420 -69.0% 2496648 ± 43% numa-numastat.node1.local_node
8122292 -68.6% 2550541 ± 42% numa-numastat.node1.numa_hit
2.202e+08 ± 5% -82.0% 39691191 ± 11% turbostat.C1
0.93 ± 5% -0.8 0.09 ± 14% turbostat.C1%
1.101e+08 ± 2% -56.5% 47858373 turbostat.C1E
3.22 -0.5 2.73 turbostat.POLL%
231050 +1893.9% 4607008 ± 4% meminfo.Active
230970 +1894.6% 4606928 ± 4% meminfo.Active(anon)
219559 ± 5% -22.6% 170030 ± 15% meminfo.AnonHugePages
572890 -24.2% 434172 meminfo.AnonPages
21995823 -62.4% 8263088 ± 4% meminfo.Cached
25221015 -55.0% 11347952 ± 3% meminfo.Committed_AS
19440153 -93.9% 1192582 ± 23% meminfo.Inactive
19440011 -93.9% 1192439 ± 23% meminfo.Inactive(anon)
214830 -14.7% 183232 meminfo.KReclaimable
25424026 -56.0% 11175036 ± 3% meminfo.Memused
214830 -14.7% 183232 meminfo.SReclaimable
19098913 -71.9% 5366191 ± 6% meminfo.Shmem
25832239 -34.2% 16986009 meminfo.max_used_kB
110043 ± 15% -88.5% 12679 ± 98% sched_debug.cfs_rq:/.MIN_vruntime.avg
2804771 ± 2% -69.8% 846737 ± 85% sched_debug.cfs_rq:/.MIN_vruntime.max
504968 ± 9% -82.6% 88033 ± 76% sched_debug.cfs_rq:/.MIN_vruntime.stddev
110043 ± 15% -88.5% 12679 ± 98% sched_debug.cfs_rq:/.max_vruntime.avg
2804771 ± 2% -69.8% 846738 ± 85% sched_debug.cfs_rq:/.max_vruntime.max
504968 ± 9% -82.6% 88033 ± 76% sched_debug.cfs_rq:/.max_vruntime.stddev
854772 ± 2% +12.8% 964472 ± 2% sched_debug.cfs_rq:/.min_vruntime.stddev
73087 ± 59% +734.2% 609692 ± 44% sched_debug.cfs_rq:/.spread0.max
861849 ± 2% +12.1% 966253 ± 2% sched_debug.cfs_rq:/.spread0.stddev
322.78 ± 2% -14.0% 277.54 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.avg
470921 ± 4% +11.5% 525288 ± 4% sched_debug.cpu.avg_idle.avg
10146 ± 89% +197.5% 30184 ± 27% sched_debug.cpu.avg_idle.min
166.82 ± 15% -82.9% 28.54 ± 6% sched_debug.cpu.clock.stddev
0.00 ± 15% -78.3% 0.00 ± 16% sched_debug.cpu.next_balance.stddev
29334462 -9.5% 26550823 sched_debug.cpu.nr_switches.avg
9400046 -12.1% 8262067 sched_debug.cpu.nr_switches.stddev
1494 ± 50% +1.6e+05% 2458393 ± 56% numa-meminfo.node0.Active
1459 ± 53% +1.7e+05% 2458381 ± 56% numa-meminfo.node0.Active(anon)
193090 ± 23% +315.9% 803087 ± 36% numa-meminfo.node0.Inactive
193026 ± 23% +316.0% 803069 ± 36% numa-meminfo.node0.Inactive(anon)
9933 ± 9% +50.4% 14936 ± 4% numa-meminfo.node0.PageTables
6124 ± 15% +49221.4% 3020604 ± 55% numa-meminfo.node0.Shmem
229095 +841.3% 2156586 ± 64% numa-meminfo.node1.Active
229051 +841.5% 2156518 ± 64% numa-meminfo.node1.Active(anon)
145346 ± 16% -45.5% 79245 ± 34% numa-meminfo.node1.AnonHugePages
383868 ± 12% -49.7% 192976 ± 22% numa-meminfo.node1.AnonPages
19095555 -87.0% 2485880 ± 58% numa-meminfo.node1.FilePages
19201636 -98.0% 390419 ± 19% numa-meminfo.node1.Inactive
19201557 -98.0% 390295 ± 19% numa-meminfo.node1.Inactive(anon)
102686 ± 9% -34.7% 67069 ± 15% numa-meminfo.node1.KReclaimable
2789340 ± 3% -49.4% 1410452 ± 52% numa-meminfo.node1.Mapped
1.113e+08 +15.3% 1.283e+08 numa-meminfo.node1.MemFree
20783078 -81.8% 3782627 ± 38% numa-meminfo.node1.MemUsed
16570 ± 5% -38.1% 10254 ± 7% numa-meminfo.node1.PageTables
102686 ± 9% -34.7% 67069 ± 15% numa-meminfo.node1.SReclaimable
19047079 -87.6% 2354397 ± 60% numa-meminfo.node1.Shmem
282546 ± 6% -14.1% 242658 ± 7% numa-meminfo.node1.Slab
364.56 ± 53% +1.7e+05% 615291 ± 56% numa-vmstat.node0.nr_active_anon
48203 ± 23% +317.2% 201111 ± 35% numa-vmstat.node0.nr_inactive_anon
2477 ± 10% +50.9% 3738 ± 4% numa-vmstat.node0.nr_page_table_pages
1530 ± 15% +49298.5% 756126 ± 55% numa-vmstat.node0.nr_shmem
364.56 ± 53% +1.7e+05% 615292 ± 56% numa-vmstat.node0.nr_zone_active_anon
48203 ± 23% +317.2% 201111 ± 35% numa-vmstat.node0.nr_zone_inactive_anon
573038 ± 11% +390.9% 2813153 ± 39% numa-vmstat.node0.numa_hit
433108 ± 12% +511.4% 2648015 ± 42% numa-vmstat.node0.numa_local
57253 +843.2% 540005 ± 64% numa-vmstat.node1.nr_active_anon
95911 ± 12% -49.6% 48299 ± 22% numa-vmstat.node1.nr_anon_pages
4773715 -87.0% 622469 ± 58% numa-vmstat.node1.nr_file_pages
27815434 +15.3% 32063992 numa-vmstat.node1.nr_free_pages
4800160 -98.0% 97769 ± 19% numa-vmstat.node1.nr_inactive_anon
697703 ± 3% -49.3% 353752 ± 52% numa-vmstat.node1.nr_mapped
4135 ± 5% -37.9% 2567 ± 7% numa-vmstat.node1.nr_page_table_pages
4761596 -87.6% 589599 ± 60% numa-vmstat.node1.nr_shmem
25670 ± 9% -34.7% 16769 ± 15% numa-vmstat.node1.nr_slab_reclaimable
57253 +843.2% 540006 ± 64% numa-vmstat.node1.nr_zone_active_anon
4800159 -98.0% 97769 ± 19% numa-vmstat.node1.nr_zone_inactive_anon
8122528 -68.6% 2548812 ± 42% numa-vmstat.node1.numa_hit
8059656 -69.0% 2494919 ± 43% numa-vmstat.node1.numa_local
57680 +1900.0% 1153633 ± 4% proc-vmstat.nr_active_anon
143171 -24.2% 108550 proc-vmstat.nr_anon_pages
5919284 +6.0% 6274221 proc-vmstat.nr_dirty_background_threshold
11853042 +6.0% 12563784 proc-vmstat.nr_dirty_threshold
5493638 -62.4% 2067724 ± 4% proc-vmstat.nr_file_pages
59564263 +6.0% 63118840 proc-vmstat.nr_free_pages
4854706 -93.9% 298169 ± 23% proc-vmstat.nr_inactive_anon
6640 -4.7% 6328 proc-vmstat.nr_page_table_pages
4769410 -71.8% 1343500 ± 6% proc-vmstat.nr_shmem
53693 -14.7% 45811 proc-vmstat.nr_slab_reclaimable
57680 +1900.0% 1153633 ± 4% proc-vmstat.nr_zone_active_anon
4854706 -93.9% 298169 ± 23% proc-vmstat.nr_zone_inactive_anon
683375 ± 10% -50.0% 341609 ± 4% proc-vmstat.numa_hint_faults
519396 ± 17% -60.2% 206845 ± 27% proc-vmstat.numa_hint_faults_local
8698455 -38.3% 5371241 proc-vmstat.numa_hit
938.89 ± 18% -55.8% 414.71 ± 8% proc-vmstat.numa_huge_pte_updates
8495696 -39.4% 5152209 proc-vmstat.numa_local
1313649 ± 5% -46.7% 700042 ± 3% proc-vmstat.numa_pte_updates
61945 +1240.4% 830307 ± 21% proc-vmstat.pgactivate
8928138 -37.8% 5550603 proc-vmstat.pgalloc_normal
2256846 ± 2% -11.1% 2006618 proc-vmstat.pgfault
1520379 ± 2% +268.9% 5607989 proc-vmstat.pgfree
0.38 ± 4% +115.9% 0.82 ± 11% perf-stat.i.MPKI
7.631e+10 -4.8% 7.269e+10 perf-stat.i.branch-instructions
0.67 -0.1 0.61 perf-stat.i.branch-miss-rate%
4.989e+08 -5.5% 4.715e+08 perf-stat.i.branch-misses
6418393 -8.1% 5897662 ± 5% perf-stat.i.cache-misses
50457351 ± 5% +51.6% 76473775 ± 10% perf-stat.i.cache-references
60968060 -6.7% 56856550 perf-stat.i.context-switches
2.14 ± 2% +13.1% 2.42 perf-stat.i.cpi
4.35e+11 +1.7% 4.424e+11 perf-stat.i.cpu-cycles
1058 ± 14% -21.6% 830.16 ± 4% perf-stat.i.cpu-migrations
385252 ± 4% +63.0% 627819 ± 4% perf-stat.i.cycles-between-cache-misses
0.01 ± 22% -0.0 0.00 ± 12% perf-stat.i.dTLB-load-miss-rate%
2628303 ± 7% -65.0% 920193 ± 5% perf-stat.i.dTLB-load-misses
1.104e+11 -6.0% 1.038e+11 perf-stat.i.dTLB-loads
0.00 ± 13% +0.0 0.00 ± 12% perf-stat.i.dTLB-store-miss-rate%
242451 ± 5% -19.0% 196450 ± 11% perf-stat.i.dTLB-store-misses
6.626e+10 -8.4% 6.071e+10 perf-stat.i.dTLB-stores
3.816e+11 -5.8% 3.595e+11 perf-stat.i.instructions
0.80 -6.6% 0.75 perf-stat.i.ipc
1.94 +1.7% 1.97 perf-stat.i.metric.GHz
504.83 ± 2% -4.4% 482.60 perf-stat.i.metric.K/sec
1129 -6.2% 1058 perf-stat.i.metric.M/sec
6259 ± 2% -11.2% 5560 perf-stat.i.minor-faults
58.32 ± 3% +29.9 88.22 ± 6% perf-stat.i.node-load-miss-rate%
1117525 +34.5% 1502746 ± 10% perf-stat.i.node-load-misses
458656 ± 5% -70.6% 134710 ± 30% perf-stat.i.node-loads
6260 ± 2% -11.2% 5562 perf-stat.i.page-faults
0.13 ± 5% +61.1% 0.22 ± 10% perf-stat.overall.MPKI
12.66 ± 5% -4.9 7.76 ± 14% perf-stat.overall.cache-miss-rate%
1.15 +7.5% 1.24 perf-stat.overall.cpi
67823 +10.3% 74790 ± 4% perf-stat.overall.cycles-between-cache-misses
0.00 ± 7% -0.0 0.00 ± 5% perf-stat.overall.dTLB-load-miss-rate%
0.87 -7.0% 0.81 perf-stat.overall.ipc
69.79 ± 2% +21.7 91.45 ± 3% perf-stat.overall.node-load-miss-rate%
260155 -1.1% 257287 perf-stat.overall.path-length
7.514e+10 -4.2% 7.195e+10 perf-stat.ps.branch-instructions
4.914e+08 -5.1% 4.664e+08 perf-stat.ps.branch-misses
50614911 ± 5% +52.5% 77201196 ± 10% perf-stat.ps.cache-references
60049636 -6.4% 56232111 perf-stat.ps.context-switches
4.331e+11 +1.8% 4.408e+11 perf-stat.ps.cpu-cycles
1048 ± 13% -21.2% 826.67 ± 4% perf-stat.ps.cpu-migrations
2706759 ± 7% -65.6% 930018 ± 5% perf-stat.ps.dTLB-load-misses
1.087e+11 -5.5% 1.027e+11 perf-stat.ps.dTLB-loads
242516 ± 5% -19.3% 195733 ± 11% perf-stat.ps.dTLB-store-misses
6.523e+10 -7.9% 6.005e+10 perf-stat.ps.dTLB-stores
3.757e+11 -5.3% 3.558e+11 perf-stat.ps.instructions
6291 ± 2% -11.8% 5547 perf-stat.ps.minor-faults
1103763 +36.5% 1507093 ± 10% perf-stat.ps.node-load-misses
478062 ± 5% -71.3% 137270 ± 29% perf-stat.ps.node-loads
6292 ± 2% -11.8% 5549 perf-stat.ps.page-faults
1.192e+14 -5.3% 1.129e+14 perf-stat.total.instructions
20.06 ± 2% -20.1 0.00 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.pipe_read.vfs_read
22.16 ± 2% -17.8 4.38 ± 3% perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event
22.02 ± 2% -17.7 4.35 ± 3% perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
20.05 ± 2% -16.1 3.97 ± 3% perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
58.25 -9.4 48.81 perf-profile.calltrace.cycles-pp.__libc_read
11.88 ± 2% -9.3 2.53 ± 3% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
11.73 -9.2 2.48 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
10.32 ± 2% -9.1 1.18 ± 3% perf-profile.calltrace.cycles-pp.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
11.28 -8.8 2.45 ± 6% perf-profile.calltrace.cycles-pp.__libc_write
10.60 ± 2% -8.5 2.08 ± 3% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch
10.69 -8.4 2.33 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
10.59 ± 2% -8.3 2.26 ± 3% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.vfs_read
10.62 -8.3 2.31 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
10.49 -8.3 2.24 ± 5% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
10.36 ± 2% -8.2 2.19 ± 3% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
10.61 ± 3% -8.1 2.48 ± 13% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
8.71 ± 2% -8.1 0.66 ± 54% perf-profile.calltrace.cycles-pp.unwind_next_frame.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain
56.47 -8.0 48.46 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
56.28 -7.8 48.43 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
9.88 -7.8 2.08 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
9.76 -7.7 2.02 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
9.76 -7.7 2.10 ± 5% perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
9.48 ± 2% -7.5 1.95 ± 3% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
55.87 -7.5 48.35 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
55.45 -7.2 48.28 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
8.86 ± 2% -7.0 1.84 ± 3% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
8.63 ± 2% -6.7 1.90 ± 5% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
8.41 ± 2% -6.6 1.80 ± 3% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
8.42 ± 2% -6.6 1.86 ± 5% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write.ksys_write
54.70 -6.5 48.19 perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.30 ± 2% -6.5 1.85 ± 5% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write
8.18 ± 2% -6.5 1.73 ± 3% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
8.10 ± 2% -6.4 1.72 ± 3% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
8.20 ± 2% -6.4 1.83 ± 5% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
6.10 ± 2% -6.1 0.00 perf-profile.calltrace.cycles-pp.unwind_next_frame.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
7.43 ± 2% -6.0 1.47 ± 3% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime
6.92 ± 2% -5.4 1.50 ± 3% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
6.85 ± 2% -5.4 1.46 ± 3% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
6.68 ± 2% -5.3 1.43 ± 3% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common
6.70 ± 2% -5.2 1.46 ± 4% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.do_idle
6.50 ± 2% -5.1 1.39 ± 3% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle
6.47 ± 2% -5.1 1.37 ± 3% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function
6.40 ± 2% -5.0 1.36 ± 3% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up
5.69 ± 2% -4.9 0.84 ± 5% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
5.85 ± 2% -4.7 1.14 ± 3% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template
5.83 ± 2% -4.6 1.25 ± 4% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.pipe_read.vfs_read
5.66 ± 2% -4.4 1.21 ± 4% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.pipe_read
52.13 -4.4 47.75 perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
5.50 ± 2% -4.3 1.16 ± 3% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
51.86 -4.1 47.71 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
3.00 ± 6% -2.4 0.58 ± 2% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
3.00 ± 6% -2.4 0.58 ± 2% perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
28.01 +19.1 47.12 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
27.98 +19.1 47.11 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
27.90 +19.2 47.09 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
28.25 +19.3 47.57 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
13.04 +30.9 43.94 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
12.84 +31.1 43.91 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.00 +41.3 41.26 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle
0.00 +41.5 41.49 perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle.do_idle
0.00 +41.6 41.57 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
0.00 +41.8 41.80 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule
0.00 +42.0 42.01 perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule.pipe_read
0.00 +42.1 42.08 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.pipe_read.vfs_read
27.66 ± 2% -21.6 6.09 ± 3% perf-profile.children.cycles-pp.perf_tp_event
26.84 ± 2% -21.0 5.84 ± 3% perf-profile.children.cycles-pp.__perf_event_overflow
26.57 ± 2% -20.8 5.79 ± 3% perf-profile.children.cycles-pp.perf_event_output_forward
21.19 ± 2% -20.4 0.79 ± 8% perf-profile.children.cycles-pp.finish_task_switch
24.08 ± 2% -19.2 4.85 ± 3% perf-profile.children.cycles-pp.perf_prepare_sample
22.32 ± 2% -17.8 4.53 ± 3% perf-profile.children.cycles-pp.perf_callchain
22.19 ± 2% -17.7 4.51 ± 3% perf-profile.children.cycles-pp.get_perf_callchain
68.01 -16.5 51.52 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
67.76 -16.3 51.48 perf-profile.children.cycles-pp.do_syscall_64
20.43 ± 2% -16.3 4.15 ± 3% perf-profile.children.cycles-pp.perf_callchain_kernel
16.08 ± 2% -12.8 3.30 ± 3% perf-profile.children.cycles-pp.unwind_next_frame
12.85 ± 2% -10.0 2.80 ± 4% perf-profile.children.cycles-pp.perf_trace_sched_switch
58.48 -9.6 48.86 perf-profile.children.cycles-pp.__libc_read
11.84 -9.3 2.50 ± 3% perf-profile.children.cycles-pp.cpuidle_idle_call
12.30 ± 2% -9.1 3.21 ± 4% perf-profile.children.cycles-pp.__libc_write
10.55 ± 2% -8.4 2.14 ± 3% perf-profile.children.cycles-pp.__unwind_start
11.30 ± 2% -8.4 2.94 ± 3% perf-profile.children.cycles-pp.ksys_write
10.60 ± 2% -8.3 2.26 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair
10.42 ± 2% -8.2 2.20 ± 3% perf-profile.children.cycles-pp.dequeue_entity
10.93 ± 2% -8.1 2.84 ± 3% perf-profile.children.cycles-pp.vfs_write
9.96 -7.9 2.09 ± 3% perf-profile.children.cycles-pp.cpuidle_enter
9.92 -7.8 2.08 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state
9.80 -7.7 2.11 ± 5% perf-profile.children.cycles-pp.pipe_write
55.91 -7.5 48.37 perf-profile.children.cycles-pp.ksys_read
9.65 ± 2% -7.5 2.12 ± 3% perf-profile.children.cycles-pp.update_curr
55.47 -7.2 48.30 perf-profile.children.cycles-pp.vfs_read
8.95 ± 2% -7.0 1.98 ± 3% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
8.65 ± 2% -6.7 1.91 ± 5% perf-profile.children.cycles-pp.__wake_up_common_lock
54.85 -6.6 48.21 perf-profile.children.cycles-pp.pipe_read
8.43 ± 2% -6.6 1.86 ± 5% perf-profile.children.cycles-pp.__wake_up_common
8.35 ± 2% -6.5 1.85 ± 5% perf-profile.children.cycles-pp.autoremove_wake_function
8.23 ± 2% -6.4 1.84 ± 5% perf-profile.children.cycles-pp.try_to_wake_up
6.86 ± 2% -5.4 1.46 ± 3% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
5.79 ± 2% -4.9 0.87 ± 5% perf-profile.children.cycles-pp.poll_idle
52.14 -4.3 47.81 perf-profile.children.cycles-pp.schedule
3.81 ± 5% -3.7 0.15 ± 6% perf-profile.children.cycles-pp.__switch_to
4.46 ± 2% -3.5 0.91 ± 3% perf-profile.children.cycles-pp.__orc_find
3.03 ± 6% -2.4 0.58 ± 2% perf-profile.children.cycles-pp.mwait_idle_with_hints
3.03 ± 6% -2.4 0.58 ± 2% perf-profile.children.cycles-pp.intel_idle
2.85 ± 2% -2.3 0.58 ± 3% perf-profile.children.cycles-pp.stack_access_ok
2.44 ± 2% -2.0 0.48 ± 3% perf-profile.children.cycles-pp.unwind_get_return_address
2.08 ± 2% -1.7 0.41 ± 3% perf-profile.children.cycles-pp.__kernel_text_address
2.09 ± 2% -1.6 0.44 ± 3% perf-profile.children.cycles-pp.orc_find
2.00 -1.5 0.49 ± 10% perf-profile.children.cycles-pp.sched_ttwu_pending
1.87 -1.5 0.39 ± 6% perf-profile.children.cycles-pp.flush_smp_call_function_queue
1.53 ± 2% -1.2 0.30 ± 3% perf-profile.children.cycles-pp.kernel_text_address
1.37 ± 2% -1.0 0.33 ± 14% perf-profile.children.cycles-pp.menu_select
1.49 -1.0 0.46 ± 17% perf-profile.children.cycles-pp.ttwu_do_activate
1.43 -1.0 0.45 ± 17% perf-profile.children.cycles-pp.enqueue_task_fair
1.26 -0.9 0.34 ± 9% perf-profile.children.cycles-pp.update_load_avg
1.13 ± 2% -0.9 0.23 ± 4% perf-profile.children.cycles-pp.perf_callchain_user
1.24 -0.9 0.35 ± 15% perf-profile.children.cycles-pp.enqueue_entity
1.53 ± 2% -0.8 0.75 ± 4% perf-profile.children.cycles-pp.perf_output_sample
0.96 ± 3% -0.8 0.18 ± 3% perf-profile.children.cycles-pp.__perf_event_header__init_id
0.94 -0.7 0.24 ± 4% perf-profile.children.cycles-pp.sched_clock_cpu
0.79 ± 3% -0.6 0.16 ± 2% perf-profile.children.cycles-pp.__get_user_nocheck_8
0.76 ± 3% -0.6 0.13 ± 8% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.75 ± 2% -0.6 0.13 ± 8% perf-profile.children.cycles-pp.perf_output_begin_forward
0.91 ± 4% -0.6 0.30 ± 15% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.72 -0.6 0.11 ± 4% perf-profile.children.cycles-pp.prepare_to_wait_event
1.16 ± 2% -0.6 0.58 ± 4% perf-profile.children.cycles-pp.__output_copy
0.70 -0.6 0.14 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.74 ± 2% -0.6 0.19 ± 5% perf-profile.children.cycles-pp.native_sched_clock
0.67 ± 2% -0.5 0.14 ± 3% perf-profile.children.cycles-pp.ftrace_graph_ret_addr
0.65 -0.5 0.12 ± 6% perf-profile.children.cycles-pp.__switch_to_asm
1.01 -0.5 0.49 ± 4% perf-profile.children.cycles-pp.perf_output_copy
0.69 ± 2% -0.5 0.19 ± 22% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.59 -0.5 0.11 ± 3% perf-profile.children.cycles-pp.set_next_entity
0.55 ± 2% -0.5 0.10 ± 6% perf-profile.children.cycles-pp.mutex_lock
0.67 ± 6% -0.4 0.23 ± 12% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.52 -0.4 0.11 ± 4% perf-profile.children.cycles-pp.__entry_text_start
0.50 ± 2% -0.4 0.10 ± 4% perf-profile.children.cycles-pp.core_kernel_text
0.58 ± 4% -0.4 0.20 ± 17% perf-profile.children.cycles-pp.sysvec_call_function_single
0.49 ± 2% -0.4 0.13 ± 7% perf-profile.children.cycles-pp.update_rq_clock
0.42 -0.3 0.08 ± 4% perf-profile.children.cycles-pp.copy_page_from_iter
0.45 ± 5% -0.3 0.11 ± 3% perf-profile.children.cycles-pp.__fdget_pos
0.43 ± 2% -0.3 0.11 ± 4% perf-profile.children.cycles-pp.__update_load_avg_se
0.52 ± 8% -0.3 0.20 ± 13% perf-profile.children.cycles-pp.copyin
0.42 ± 5% -0.3 0.10 ± 5% perf-profile.children.cycles-pp.__fget_light
0.43 ± 5% -0.3 0.12 ± 11% perf-profile.children.cycles-pp.ktime_get
0.48 ± 4% -0.3 0.17 ± 18% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.59 ± 3% -0.3 0.29 ± 6% perf-profile.children.cycles-pp.memcpy_erms
0.38 ± 2% -0.3 0.07 ± 7% perf-profile.children.cycles-pp.copy_page_to_iter
0.38 ± 5% -0.3 0.08 ± 8% perf-profile.children.cycles-pp.__fget_files
0.36 ± 2% -0.3 0.07 ± 4% perf-profile.children.cycles-pp._copy_from_iter
0.35 ± 4% -0.3 0.07 ± 7% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.42 ± 3% -0.3 0.16 ± 21% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.34 ± 7% -0.3 0.07 ± 6% perf-profile.children.cycles-pp.__pthread_disable_asynccancel
0.32 -0.3 0.06 perf-profile.children.cycles-pp.kvm_guest_state
0.39 ± 2% -0.3 0.13 ± 30% perf-profile.children.cycles-pp.tick_nohz_next_event
0.31 ± 2% -0.3 0.06 ± 5% perf-profile.children.cycles-pp._copy_to_iter
0.29 ± 5% -0.2 0.05 perf-profile.children.cycles-pp.perf_trace_buf_alloc
0.31 ± 2% -0.2 0.06 ± 5% perf-profile.children.cycles-pp.os_xsave
0.29 ± 2% -0.2 0.05 ± 9% perf-profile.children.cycles-pp.get_stack_info
0.30 ± 2% -0.2 0.07 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.29 ± 2% -0.2 0.06 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.29 ± 2% -0.2 0.06 ± 7% perf-profile.children.cycles-pp.__might_resched
0.28 ± 4% -0.2 0.06 ± 10% perf-profile.children.cycles-pp.perf_trace_buf_update
0.27 ± 3% -0.2 0.05 ± 6% perf-profile.children.cycles-pp.security_file_permission
0.26 ± 2% -0.2 0.05 ± 6% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.26 ± 3% -0.2 0.05 perf-profile.children.cycles-pp.get_next_timer_interrupt
0.24 ± 4% -0.2 0.04 ± 40% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.26 ± 2% -0.2 0.07 ± 10% perf-profile.children.cycles-pp.___perf_sw_event
0.35 ± 11% -0.2 0.17 ± 16% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
0.79 ± 11% -0.2 0.63 ± 7% perf-profile.children.cycles-pp.generic_perform_write
0.22 -0.2 0.06 perf-profile.children.cycles-pp.read_tsc
0.26 ± 3% -0.2 0.10 ± 8% perf-profile.children.cycles-pp.select_task_rq
0.26 ± 3% -0.2 0.11 ± 29% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.79 ± 11% -0.2 0.64 ± 7% perf-profile.children.cycles-pp.__generic_file_write_iter
0.79 ± 11% -0.1 0.65 ± 7% perf-profile.children.cycles-pp.generic_file_write_iter
0.22 ± 3% -0.1 0.09 ± 9% perf-profile.children.cycles-pp.select_task_rq_fair
0.21 ± 3% -0.1 0.10 ± 9% perf-profile.children.cycles-pp.native_irq_return_iret
0.22 ± 6% -0.1 0.11 ± 10% perf-profile.children.cycles-pp.__irq_exit_rcu
0.14 ± 6% -0.1 0.03 ± 86% perf-profile.children.cycles-pp.pick_next_task_idle
0.16 ± 9% -0.1 0.08 ± 11% perf-profile.children.cycles-pp.__do_softirq
0.14 ± 11% -0.1 0.05 ± 43% perf-profile.children.cycles-pp.rebalance_domains
0.16 ± 2% -0.1 0.08 ± 47% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
0.13 ± 5% -0.1 0.05 ± 6% perf-profile.children.cycles-pp.native_apic_msr_eoi_write
0.13 ± 3% -0.1 0.07 ± 44% perf-profile.children.cycles-pp.nohz_run_idle_balance
0.10 ± 4% -0.0 0.05 ± 9% perf-profile.children.cycles-pp.memcpy
0.11 ± 12% -0.0 0.06 ± 18% perf-profile.children.cycles-pp.clockevents_program_event
0.12 ± 6% +0.0 0.16 ± 8% perf-profile.children.cycles-pp.scheduler_tick
0.15 ± 11% +0.1 0.21 ± 6% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.15 ± 6% +0.1 0.23 ± 18% perf-profile.children.cycles-pp.tick_sched_handle
0.15 ± 7% +0.1 0.23 ± 18% perf-profile.children.cycles-pp.update_process_times
0.00 +0.1 0.13 ± 24% perf-profile.children.cycles-pp.update_blocked_averages
0.24 ± 16% +0.2 0.45 ± 12% perf-profile.children.cycles-pp.start_kernel
0.24 ± 16% +0.2 0.45 ± 12% perf-profile.children.cycles-pp.arch_call_rest_init
0.24 ± 16% +0.2 0.45 ± 12% perf-profile.children.cycles-pp.rest_init
0.00 +0.9 0.85 ± 46% perf-profile.children.cycles-pp.update_sg_lb_stats
0.00 +0.9 0.91 ± 45% perf-profile.children.cycles-pp.update_sd_lb_stats
0.00 +0.9 0.93 ± 45% perf-profile.children.cycles-pp.find_busiest_group
0.00 +1.0 1.01 ± 44% perf-profile.children.cycles-pp.load_balance
0.16 ± 4% +1.0 1.20 ± 41% perf-profile.children.cycles-pp.newidle_balance
28.01 +19.1 47.12 perf-profile.children.cycles-pp.start_secondary
28.25 +19.3 47.57 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
28.25 +19.3 47.57 perf-profile.children.cycles-pp.cpu_startup_entry
28.19 +19.4 47.55 perf-profile.children.cycles-pp.do_idle
65.01 +27.1 92.13 perf-profile.children.cycles-pp.__schedule
13.19 +31.2 44.38 perf-profile.children.cycles-pp.schedule_idle
0.00 +83.5 83.49 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.42 ± 2% +83.7 84.07 perf-profile.children.cycles-pp.prepare_task_switch
0.36 ± 2% +83.8 84.12 perf-profile.children.cycles-pp._raw_spin_lock
20.28 ± 2% -19.9 0.37 ± 7% perf-profile.self.cycles-pp.finish_task_switch
14.57 -14.2 0.39 ± 6% perf-profile.self.cycles-pp.__schedule
6.94 ± 2% -5.5 1.41 ± 3% perf-profile.self.cycles-pp.unwind_next_frame
5.60 ± 2% -4.8 0.79 ± 6% perf-profile.self.cycles-pp.poll_idle
3.77 ± 5% -3.6 0.14 ± 4% perf-profile.self.cycles-pp.__switch_to
4.25 ± 2% -3.4 0.87 ± 3% perf-profile.self.cycles-pp.__orc_find
3.03 ± 6% -2.4 0.58 ± 2% perf-profile.self.cycles-pp.mwait_idle_with_hints
2.31 ± 2% -1.8 0.47 ± 3% perf-profile.self.cycles-pp.stack_access_ok
1.87 ± 2% -1.5 0.40 ± 3% perf-profile.self.cycles-pp.orc_find
0.97 ± 2% -0.8 0.19 ± 3% perf-profile.self.cycles-pp.__unwind_start
0.94 ± 2% -0.7 0.19 ± 4% perf-profile.self.cycles-pp.perf_callchain_kernel
0.75 ± 2% -0.6 0.15 ± 3% perf-profile.self.cycles-pp.__get_user_nocheck_8
0.69 ± 2% -0.6 0.13 ± 8% perf-profile.self.cycles-pp.perf_output_begin_forward
0.67 -0.5 0.14 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.64 -0.5 0.11 ± 5% perf-profile.self.cycles-pp.__switch_to_asm
0.70 -0.5 0.18 ± 5% perf-profile.self.cycles-pp.native_sched_clock
0.51 -0.4 0.07 ± 7% perf-profile.self.cycles-pp.pipe_read
0.55 ± 2% -0.4 0.11 ± 4% perf-profile.self.cycles-pp.kernel_text_address
0.65 ± 6% -0.4 0.22 ± 10% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.52 ± 3% -0.4 0.09 ± 3% perf-profile.self.cycles-pp.perf_prepare_sample
0.47 ± 2% -0.4 0.09 ± 5% perf-profile.self.cycles-pp.ftrace_graph_ret_addr
0.48 ± 2% -0.4 0.11 ± 6% perf-profile.self.cycles-pp.menu_select
0.47 ± 2% -0.4 0.09 ± 5% perf-profile.self.cycles-pp.unwind_get_return_address
0.45 ± 2% -0.4 0.09 ± 3% perf-profile.self.cycles-pp.__kernel_text_address
0.45 ± 3% -0.4 0.09 ± 3% perf-profile.self.cycles-pp.get_perf_callchain
0.43 ± 2% -0.3 0.09 ± 3% perf-profile.self.cycles-pp.update_load_avg
0.40 ± 3% -0.3 0.06 ± 5% perf-profile.self.cycles-pp.cpuidle_idle_call
0.43 -0.3 0.11 ± 13% perf-profile.self.cycles-pp.enqueue_entity
0.36 ± 2% -0.3 0.07 ± 4% perf-profile.self.cycles-pp.core_kernel_text
0.36 ± 2% -0.3 0.07 ± 5% perf-profile.self.cycles-pp.do_idle
0.37 ± 5% -0.3 0.08 ± 7% perf-profile.self.cycles-pp.__fget_files
0.38 ± 2% -0.3 0.10 ± 5% perf-profile.self.cycles-pp.__update_load_avg_se
0.55 ± 4% -0.3 0.26 ± 6% perf-profile.self.cycles-pp.memcpy_erms
0.33 ± 2% -0.3 0.06 ± 8% perf-profile.self.cycles-pp.__libc_read
0.33 -0.3 0.06 ± 8% perf-profile.self.cycles-pp.prepare_to_wait_event
0.33 ± 3% -0.3 0.06 ± 7% perf-profile.self.cycles-pp.__perf_event_header__init_id
0.33 ± 6% -0.3 0.07 ± 6% perf-profile.self.cycles-pp.update_curr
0.31 ± 3% -0.3 0.06 perf-profile.self.cycles-pp.__task_pid_nr_ns
0.40 ± 4% -0.2 0.15 ± 21% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.41 ± 3% -0.2 0.17 ± 25% perf-profile.self.cycles-pp.perf_tp_event
0.30 ± 2% -0.2 0.06 ± 8% perf-profile.self.cycles-pp.os_xsave
0.31 ± 7% -0.2 0.07 ± 9% perf-profile.self.cycles-pp.__pthread_disable_asynccancel
0.48 -0.2 0.24 ± 3% perf-profile.self.cycles-pp.__output_copy
0.29 ± 2% -0.2 0.06 perf-profile.self.cycles-pp.perf_callchain_user
0.30 ± 2% -0.2 0.07 ± 7% perf-profile.self.cycles-pp.__libc_write
0.28 -0.2 0.06 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.26 ± 2% -0.2 0.04 ± 40% perf-profile.self.cycles-pp.kvm_guest_state
0.26 -0.2 0.05 perf-profile.self.cycles-pp.__wrgsbase_inactive
0.26 ± 3% -0.2 0.06 perf-profile.self.cycles-pp.__might_resched
0.22 ± 2% -0.2 0.03 ± 86% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.25 ± 4% -0.2 0.06 ± 7% perf-profile.self.cycles-pp.dequeue_entity
0.33 ± 3% -0.2 0.16 ± 2% perf-profile.self.cycles-pp.perf_output_sample
0.22 ± 4% -0.2 0.06 ± 8% perf-profile.self.cycles-pp.update_rq_clock
0.22 ± 10% -0.2 0.07 ± 17% perf-profile.self.cycles-pp.ktime_get
0.21 ± 2% -0.2 0.06 ± 5% perf-profile.self.cycles-pp.read_tsc
0.18 ± 2% -0.1 0.04 ± 65% perf-profile.self.cycles-pp.___perf_sw_event
0.19 ± 5% -0.1 0.06 ± 41% perf-profile.self.cycles-pp.dequeue_task_fair
0.20 ± 5% -0.1 0.08 ± 42% perf-profile.self.cycles-pp.enqueue_task_fair
0.20 -0.1 0.08 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state
0.21 ± 3% -0.1 0.10 ± 9% perf-profile.self.cycles-pp.native_irq_return_iret
0.14 ± 4% -0.1 0.06 ± 43% perf-profile.self.cycles-pp.newidle_balance
0.15 ± 4% -0.1 0.08 ± 13% perf-profile.self.cycles-pp.prepare_task_switch
0.13 ± 5% -0.1 0.05 ± 6% perf-profile.self.cycles-pp.native_apic_msr_eoi_write
0.12 ± 2% -0.1 0.06 ± 5% perf-profile.self.cycles-pp.perf_output_copy
0.12 ± 5% -0.1 0.06 ± 43% perf-profile.self.cycles-pp.nohz_run_idle_balance
0.14 ± 11% +0.1 0.21 ± 7% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.33 +0.3 0.62 ± 7% perf-profile.self.cycles-pp._raw_spin_lock
0.00 +0.7 0.66 ± 45% perf-profile.self.cycles-pp.update_sg_lb_stats
0.00 +83.5 83.49 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath

***************************************************************************************************
lkp-spr-r02: 224 threads 2 sockets (Sapphire Rapids) with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-11/performance/x86_64-rhel-8.3/thread/16/debian-11.1-x86_64-20220510.cgz/lkp-spr-r02/context_switch1/will-it-scale

commit:
99babd04b2 ("selftests/rseq: Implement rseq numa node id field selftest")
af7f588d8f ("sched: Introduce per-memory-map concurrency ID")

99babd04b2505471 af7f588d8f7355bc4298dd1962d
---------------- ---------------------------
%stddev %change %stddev
\ | \
9168456 -73.8% 2400232 ± 10% will-it-scale.16.threads
573027 -73.8% 150014 ± 10% will-it-scale.per_thread_ops
9168456 -73.8% 2400232 ± 10% will-it-scale.workload
2.864e+09 -73.1% 7.715e+08 ± 11% cpuidle..usage
6.10 +1.0 7.15 mpstat.cpu.all.sys%
1.13 ± 3% -0.9 0.27 ± 10% mpstat.cpu.all.usr%
22292871 ± 7% -78.9% 4696291 ± 10% vmstat.memory.cache
19.00 ± 3% -14.7% 16.20 ± 2% vmstat.procs.r
18144385 -73.8% 4753330 ± 9% vmstat.system.cs
483366 +35.8% 656610 ± 5% vmstat.system.in
1555224 ± 5% +60.7% 2499655 ± 18% turbostat.C1
87537260 ± 26% +24.1% 1.086e+08 ± 14% turbostat.C1E
47.94 ± 62% +27.4 75.30 ± 24% turbostat.C1E%
0.59 -66.0% 0.20 ± 8% turbostat.IPC
1.468e+08 +35.9% 1.995e+08 ± 5% turbostat.IRQ
2.746e+09 -76.2% 6.523e+08 ± 14% turbostat.POLL
1.67 ± 2% -1.0 0.68 ± 4% turbostat.POLL%
34.20 -9.9% 30.80 turbostat.PkgTmp
433.23 ± 2% -10.4% 388.15 turbostat.PkgWatt
33394 ± 5% +814.1% 305243 ± 32% numa-vmstat.node1.nr_active_anon
55976 ± 9% -22.7% 43246 ± 10% numa-vmstat.node1.nr_anon_pages
4483729 ± 33% -88.5% 513735 ± 65% numa-vmstat.node1.nr_file_pages
28119032 ± 5% +14.5% 32194305 numa-vmstat.node1.nr_free_pages
4336878 ± 31% -97.7% 97706 ± 85% numa-vmstat.node1.nr_inactive_anon
1758120 ± 22% -91.3% 153214 ± 43% numa-vmstat.node1.nr_mapped
4166 ± 30% -75.4% 1025 ± 21% numa-vmstat.node1.nr_page_table_pages
4314309 ± 32% -91.7% 359713 ± 46% numa-vmstat.node1.nr_shmem
33394 ± 5% +814.1% 305243 ± 32% numa-vmstat.node1.nr_zone_active_anon
4336876 ± 31% -97.7% 97706 ± 85% numa-vmstat.node1.nr_zone_inactive_anon
134596 ± 6% +807.2% 1221042 ± 32% numa-meminfo.node1.Active
134504 ± 6% +807.7% 1220946 ± 32% numa-meminfo.node1.Active(anon)
223873 ± 9% -22.7% 172976 ± 10% numa-meminfo.node1.AnonPages
17932980 ± 33% -88.5% 2055168 ± 65% numa-meminfo.node1.FilePages
17344798 ± 31% -97.7% 391247 ± 85% numa-meminfo.node1.Inactive
17344618 ± 31% -97.7% 391073 ± 85% numa-meminfo.node1.Inactive(anon)
7027715 ± 22% -91.3% 614131 ± 43% numa-meminfo.node1.Mapped
1.125e+08 ± 5% +14.5% 1.288e+08 numa-meminfo.node1.MemFree
19565762 ± 30% -83.3% 3266862 ± 41% numa-meminfo.node1.MemUsed
16664 ± 30% -75.4% 4105 ± 21% numa-meminfo.node1.PageTables
17255300 ± 32% -91.7% 1439083 ± 46% numa-meminfo.node1.Shmem
421445 ± 2% -16.6% 351355 meminfo.AnonPages
22061123 ± 7% -79.5% 4530612 ± 11% meminfo.Cached
20088958 ± 8% -87.8% 2443971 ± 21% meminfo.Committed_AS
19170771 ± 11% -97.0% 574987 ± 60% meminfo.Inactive
19170591 ± 11% -97.0% 574813 ± 60% meminfo.Inactive(anon)
213051 -18.3% 173997 meminfo.KReclaimable
7430615 ± 13% -90.5% 705190 ± 33% meminfo.Mapped
25328866 ± 7% -71.9% 7127544 ± 7% meminfo.Memused
21777 ± 8% -63.7% 7906 ± 7% meminfo.PageTables
213051 -18.3% 173997 meminfo.SReclaimable
19164273 ± 8% -91.5% 1633768 ± 30% meminfo.Shmem
26692881 -39.4% 16170877 ± 7% meminfo.max_used_kB
65901 ± 13% -98.2% 1210 ±143% sched_debug.cfs_rq:/.MIN_vruntime.avg
1327262 ± 10% -93.4% 87886 ±144% sched_debug.cfs_rq:/.MIN_vruntime.max
265536 ± 3% -96.2% 10166 ±144% sched_debug.cfs_rq:/.MIN_vruntime.stddev
65901 ± 13% -98.2% 1210 ±143% sched_debug.cfs_rq:/.max_vruntime.avg
1327262 ± 10% -93.4% 87886 ±144% sched_debug.cfs_rq:/.max_vruntime.max
265536 ± 3% -96.2% 10166 ±144% sched_debug.cfs_rq:/.max_vruntime.stddev
33780 ± 13% +39.7% 47201 ± 13% sched_debug.cfs_rq:/.min_vruntime.min
979.30 ± 2% -15.8% 824.68 ± 4% sched_debug.cfs_rq:/.runnable_avg.max
978.91 ± 2% -17.1% 811.25 ± 3% sched_debug.cfs_rq:/.util_avg.max
860.31 ± 2% -30.0% 601.91 ± 4% sched_debug.cfs_rq:/.util_est_enqueued.max
1736 ± 3% +506.1% 10522 ± 91% sched_debug.cpu.avg_idle.min
0.09 ± 4% -13.4% 0.08 ± 5% sched_debug.cpu.nr_running.avg
10630744 ± 12% -71.4% 3041094 ± 6% sched_debug.cpu.nr_switches.avg
74675547 ± 12% -71.5% 21318321 ± 6% sched_debug.cpu.nr_switches.max
26031282 ± 12% -71.4% 7440127 ± 6% sched_debug.cpu.nr_switches.stddev
105357 ± 2% -16.6% 87842 proc-vmstat.nr_anon_pages
5921091 +7.7% 6375494 proc-vmstat.nr_dirty_background_threshold
11856660 +7.7% 12766578 proc-vmstat.nr_dirty_threshold
5515701 ± 7% -79.5% 1132643 ± 11% proc-vmstat.nr_file_pages
59582345 +7.6% 64133047 proc-vmstat.nr_free_pages
4793162 ± 11% -97.0% 143617 ± 60% proc-vmstat.nr_inactive_anon
1858672 ± 13% -90.6% 175442 ± 33% proc-vmstat.nr_mapped
5443 ± 8% -63.7% 1976 ± 7% proc-vmstat.nr_page_table_pages
4791488 ± 8% -91.5% 408431 ± 30% proc-vmstat.nr_shmem
53263 -18.3% 43499 proc-vmstat.nr_slab_reclaimable
103557 -1.4% 102106 proc-vmstat.nr_slab_unreclaimable
4793162 ± 11% -97.0% 143617 ± 60% proc-vmstat.nr_zone_inactive_anon
260965 ± 3% -73.8% 68247 ± 12% proc-vmstat.numa_hint_faults
210545 ± 4% -91.7% 17501 ± 29% proc-vmstat.numa_hint_faults_local
8964679 -46.3% 4809967 ± 10% proc-vmstat.numa_hit
1384 ± 19% -89.0% 151.80 ± 23% proc-vmstat.numa_huge_pte_updates
8648086 -46.7% 4607108 ± 10% proc-vmstat.numa_local
1033314 ± 14% -80.9% 197728 ± 15% proc-vmstat.numa_pte_updates
8998915 -45.0% 4950106 ± 9% proc-vmstat.pgalloc_normal
1704072 ± 2% -23.0% 1311857 ± 3% proc-vmstat.pgfault
0.45 ± 8% +168.1% 1.19 ± 3% perf-stat.i.MPKI
2.07e+10 -65.5% 7.138e+09 ± 7% perf-stat.i.branch-instructions
1.334e+08 -66.0% 45319480 ± 8% perf-stat.i.branch-misses
8.65 ± 7% -4.0 4.68 ± 6% perf-stat.i.cache-miss-rate%
7550889 ± 5% -48.6% 3883482 ± 15% perf-stat.i.cache-misses
18312903 -73.8% 4797849 ± 10% perf-stat.i.context-switches
0.99 +203.5% 3.00 ± 7% perf-stat.i.cpi
9.998e+10 -1.4% 9.855e+10 perf-stat.i.cpu-cycles
122169 ± 13% +148.0% 302970 ± 5% perf-stat.i.cycles-between-cache-misses
0.02 ± 3% -0.0 0.00 ± 5% perf-stat.i.dTLB-load-miss-rate%
5554209 ± 3% -97.0% 168530 ± 11% perf-stat.i.dTLB-load-misses
2.912e+10 -67.7% 9.412e+09 ± 7% perf-stat.i.dTLB-loads
0.00 ± 13% -0.0 0.00 ± 8% perf-stat.i.dTLB-store-miss-rate%
316990 ± 13% -85.2% 46788 ± 18% perf-stat.i.dTLB-store-misses
1.752e+10 -71.8% 4.938e+09 ± 9% perf-stat.i.dTLB-stores
1.016e+11 -66.7% 3.386e+10 ± 7% perf-stat.i.instructions
1.02 -66.2% 0.34 ± 7% perf-stat.i.ipc
0.45 -1.4% 0.44 perf-stat.i.metric.GHz
299.03 ± 6% -30.0% 209.39 ± 10% perf-stat.i.metric.K/sec
300.64 -68.1% 95.93 ± 8% perf-stat.i.metric.M/sec
4904 -27.1% 3576 ± 4% perf-stat.i.minor-faults
52.06 ± 27% +42.6 94.61 perf-stat.i.node-load-miss-rate%
1131145 ± 23% -47.4% 595531 ± 25% perf-stat.i.node-load-misses
1002828 ± 32% -96.5% 35044 ± 23% perf-stat.i.node-loads
4904 -27.1% 3576 ± 4% perf-stat.i.page-faults
0.46 ± 8% +168.7% 1.23 ± 2% perf-stat.overall.MPKI
16.33 ± 4% -7.0 9.34 ± 6% perf-stat.overall.cache-miss-rate%
0.98 +197.4% 2.93 ± 7% perf-stat.overall.cpi
13283 ± 6% +94.7% 25864 ± 15% perf-stat.overall.cycles-between-cache-misses
0.02 ± 3% -0.0 0.00 ± 6% perf-stat.overall.dTLB-load-miss-rate%
0.00 ± 13% -0.0 0.00 ± 10% perf-stat.overall.dTLB-store-miss-rate%
1.02 -66.2% 0.34 ± 7% perf-stat.overall.ipc
53.52 ± 19% +40.4 93.95 perf-stat.overall.node-load-miss-rate%
3330788 +27.6% 4250478 ± 2% perf-stat.overall.path-length
2.063e+10 -65.5% 7.117e+09 ± 7% perf-stat.ps.branch-instructions
1.33e+08 -66.0% 45185662 ± 8% perf-stat.ps.branch-misses
7528642 ± 5% -48.3% 3888889 ± 15% perf-stat.ps.cache-misses
18252001 -73.8% 4781904 ± 10% perf-stat.ps.context-switches
9.965e+10 -1.4% 9.823e+10 perf-stat.ps.cpu-cycles
5544297 ± 3% -97.0% 168873 ± 11% perf-stat.ps.dTLB-load-misses
2.903e+10 -67.7% 9.384e+09 ± 7% perf-stat.ps.dTLB-loads
316419 ± 13% -85.2% 46843 ± 19% perf-stat.ps.dTLB-store-misses
1.746e+10 -71.8% 4.923e+09 ± 9% perf-stat.ps.dTLB-stores
1.013e+11 -66.7% 3.376e+10 ± 7% perf-stat.ps.instructions
4885 -26.8% 3577 ± 4% perf-stat.ps.minor-faults
1127461 ± 23% -47.3% 593739 ± 25% perf-stat.ps.node-load-misses
1001303 ± 32% -96.4% 36005 ± 22% perf-stat.ps.node-loads
4885 -26.8% 3577 ± 4% perf-stat.ps.page-faults
3.054e+13 -66.7% 1.018e+13 ± 7% perf-stat.total.instructions
34.69 -20.3 14.42 ± 13% perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event
34.45 -20.1 14.34 ± 13% perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
31.23 ± 2% -18.2 13.07 ± 13% perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
20.47 ± 2% -12.3 8.15 ± 13% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
18.88 -11.5 7.40 ± 13% perf-profile.calltrace.cycles-pp.__libc_write
17.86 -10.9 7.01 ± 13% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
17.75 -10.8 6.97 ± 13% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
17.53 -10.6 6.90 ± 13% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
19.27 ± 2% -10.6 8.64 ± 10% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
16.91 ± 2% -10.3 6.65 ± 13% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch
16.38 -9.9 6.47 ± 13% perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
16.36 ± 2% -9.5 6.84 ± 13% perf-profile.calltrace.cycles-pp.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
14.54 -8.7 5.86 ± 13% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
15.76 -8.6 7.19 ± 13% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.vfs_read
14.21 -8.5 5.73 ± 13% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write.ksys_write
14.11 -8.4 5.69 ± 13% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write
15.42 -8.4 7.05 ± 13% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
13.97 -8.3 5.64 ± 13% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
13.81 ± 2% -8.0 5.79 ± 13% perf-profile.calltrace.cycles-pp.unwind_next_frame.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain
14.00 -7.6 6.43 ± 13% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
11.27 ± 2% -7.5 3.75 ± 12% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.pipe_read.vfs_read
10.99 ± 2% -7.3 3.64 ± 12% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.pipe_read
12.19 -7.3 4.89 ± 13% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
10.72 ± 2% -7.2 3.55 ± 12% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
11.91 -7.1 4.78 ± 13% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common
13.13 -7.0 6.08 ± 13% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
12.84 -6.9 5.95 ± 13% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
11.43 ± 2% -6.8 4.62 ± 13% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function
12.56 -6.7 5.82 ± 13% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
11.32 ± 2% -6.7 4.58 ± 13% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up
12.46 -6.7 5.77 ± 13% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
9.69 ± 2% -5.8 3.85 ± 14% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template
10.64 -5.7 4.94 ± 13% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime
10.56 ± 2% -5.6 4.95 ± 13% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
10.24 ± 2% -5.4 4.81 ± 13% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.do_idle
9.23 ± 2% -5.3 3.91 ± 13% perf-profile.calltrace.cycles-pp.unwind_next_frame.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
9.97 -5.3 4.68 ± 13% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle
14.74 -4.4 10.31 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
11.02 -2.5 8.53 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
10.67 -2.4 8.25 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
2.87 -2.1 0.76 ± 42% perf-profile.calltrace.cycles-pp.perf_output_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch
5.01 -1.9 3.08 ± 11% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
3.13 ± 2% -1.9 1.23 ± 15% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
2.76 -1.7 1.07 ± 15% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
2.92 ± 4% -1.5 1.44 ± 18% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
2.07 ± 2% -1.3 0.81 ± 15% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry
1.96 ± 2% -1.2 0.76 ± 15% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle
1.60 ± 2% -1.0 0.63 ± 15% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue
1.30 -1.0 0.34 ± 82% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
1.68 ± 8% -0.8 0.86 ± 29% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
1.46 -0.8 0.64 ± 13% perf-profile.calltrace.cycles-pp.perf_output_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime
1.37 ± 16% -0.5 0.90 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.pipe_read.vfs_read
0.44 ± 50% +0.6 1.05 ± 21% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
0.43 ± 50% +0.6 1.05 ± 21% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.rest_init
0.00 +0.7 0.67 ± 7% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule_idle.do_idle.cpu_startup_entry
41.34 +1.9 43.24 perf-profile.calltrace.cycles-pp.__libc_read
38.47 +3.6 42.12 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
38.19 +3.8 42.01 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
37.53 +4.3 41.80 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
36.83 +4.7 41.56 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
35.95 +5.3 41.23 perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
32.14 +7.7 39.84 perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
31.80 +7.9 39.70 ± 2% perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
34.12 +10.2 44.37 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
34.07 +10.3 44.35 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
33.94 +10.3 44.27 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
35.09 +10.5 45.63 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
14.13 +17.8 31.89 ± 3% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
13.80 +18.0 31.76 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.00 +24.3 24.34 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle
0.00 +25.1 25.13 ± 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule
0.00 +25.3 25.28 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle.do_idle
0.00 +25.5 25.54 ± 8% perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
0.00 +25.9 25.88 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule.pipe_read
0.00 +26.1 26.10 ± 8% perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.pipe_read.vfs_read
46.54 -27.0 19.49 ± 13% perf-profile.children.cycles-pp.perf_tp_event
45.22 -26.2 18.97 ± 13% perf-profile.children.cycles-pp.__perf_event_overflow
44.82 -26.0 18.82 ± 13% perf-profile.children.cycles-pp.perf_event_output_forward
37.76 -22.0 15.72 ± 13% perf-profile.children.cycles-pp.perf_prepare_sample
35.11 -20.4 14.66 ± 13% perf-profile.children.cycles-pp.perf_callchain
34.91 -20.3 14.59 ± 13% perf-profile.children.cycles-pp.get_perf_callchain
32.01 -18.6 13.46 ± 13% perf-profile.children.cycles-pp.perf_callchain_kernel
25.16 ± 2% -14.5 10.65 ± 13% perf-profile.children.cycles-pp.unwind_next_frame
22.28 ± 2% -13.4 8.91 ± 13% perf-profile.children.cycles-pp.perf_trace_sched_switch
21.63 ± 2% -11.9 9.70 ± 10% perf-profile.children.cycles-pp.__libc_write
19.92 ± 2% -10.9 8.98 ± 10% perf-profile.children.cycles-pp.ksys_write
19.32 ± 2% -10.6 8.69 ± 10% perf-profile.children.cycles-pp.vfs_write
16.44 -9.9 6.50 ± 13% perf-profile.children.cycles-pp.pipe_write
16.82 -9.8 7.05 ± 13% perf-profile.children.cycles-pp.__unwind_start
14.58 -8.7 5.87 ± 13% perf-profile.children.cycles-pp.__wake_up_common_lock
15.79 -8.6 7.20 ± 13% perf-profile.children.cycles-pp.dequeue_task_fair
14.23 -8.5 5.73 ± 13% perf-profile.children.cycles-pp.__wake_up_common
15.53 -8.4 7.10 ± 13% perf-profile.children.cycles-pp.dequeue_entity
14.13 -8.4 5.70 ± 13% perf-profile.children.cycles-pp.autoremove_wake_function
14.03 -8.4 5.67 ± 13% perf-profile.children.cycles-pp.try_to_wake_up
14.25 -7.7 6.58 ± 13% perf-profile.children.cycles-pp.update_curr
58.80 -7.5 51.32 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
12.22 -7.3 4.90 ± 13% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
58.45 -7.3 51.19 perf-profile.children.cycles-pp.do_syscall_64
13.24 -7.1 6.18 ± 12% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
15.07 -4.6 10.47 ± 5% perf-profile.children.cycles-pp.cpuidle_idle_call
6.83 -3.9 2.91 ± 13% perf-profile.children.cycles-pp.__orc_find
5.92 -3.5 2.41 ± 13% perf-profile.children.cycles-pp.perf_output_sample
4.58 -2.7 1.86 ± 14% perf-profile.children.cycles-pp.__output_copy
4.53 -2.6 1.91 ± 13% perf-profile.children.cycles-pp.stack_access_ok
11.21 -2.6 8.65 ± 3% perf-profile.children.cycles-pp.cpuidle_enter
11.14 -2.5 8.62 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state
3.96 -2.4 1.60 ± 13% perf-profile.children.cycles-pp.perf_output_copy
5.46 -2.2 3.24 ± 10% perf-profile.children.cycles-pp.poll_idle
3.78 -2.2 1.58 ± 12% perf-profile.children.cycles-pp.unwind_get_return_address
3.24 ± 2% -2.0 1.27 ± 14% perf-profile.children.cycles-pp.flush_smp_call_function_queue
3.27 -2.0 1.31 ± 13% perf-profile.children.cycles-pp.sched_ttwu_pending
3.31 ± 2% -1.9 1.42 ± 13% perf-profile.children.cycles-pp.orc_find
3.20 -1.9 1.34 ± 13% perf-profile.children.cycles-pp.__kernel_text_address
3.04 ± 4% -1.6 1.49 ± 18% perf-profile.children.cycles-pp.menu_select
2.43 -1.5 0.98 ± 13% perf-profile.children.cycles-pp.ttwu_do_activate
2.32 -1.4 0.93 ± 13% perf-profile.children.cycles-pp.enqueue_task_fair
2.29 -1.4 0.90 ± 14% perf-profile.children.cycles-pp.memcpy_erms
2.32 -1.3 0.97 ± 13% perf-profile.children.cycles-pp.kernel_text_address
2.02 -1.2 0.81 ± 13% perf-profile.children.cycles-pp.enqueue_entity
2.04 ± 2% -1.2 0.89 ± 14% perf-profile.children.cycles-pp.update_load_avg
1.83 -1.1 0.72 ± 13% perf-profile.children.cycles-pp.perf_callchain_user
1.77 -0.9 0.83 ± 8% perf-profile.children.cycles-pp.pick_next_task_fair
1.72 -0.9 0.79 ± 8% perf-profile.children.cycles-pp.sched_clock_cpu
1.75 ± 8% -0.9 0.89 ± 28% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
1.45 -0.9 0.60 ± 13% perf-profile.children.cycles-pp.__perf_event_header__init_id
1.26 ± 2% -0.8 0.49 ± 13% perf-profile.children.cycles-pp.__get_user_nocheck_8
1.20 ± 2% -0.7 0.46 ± 13% perf-profile.children.cycles-pp.__switch_to
1.35 -0.7 0.61 ± 8% perf-profile.children.cycles-pp.native_sched_clock
1.14 -0.7 0.48 ± 10% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.03 -0.6 0.38 ± 12% perf-profile.children.cycles-pp.__switch_to_asm
1.46 ± 10% -0.6 0.83 ± 16% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
1.06 ± 2% -0.6 0.45 ± 12% perf-profile.children.cycles-pp.ftrace_graph_ret_addr
1.00 ± 2% -0.6 0.40 ± 13% perf-profile.children.cycles-pp.set_next_entity
0.85 ± 2% -0.6 0.25 ± 16% perf-profile.children.cycles-pp.copy_page_from_iter
0.99 ± 2% -0.6 0.39 ± 13% perf-profile.children.cycles-pp.prepare_to_wait_event
1.06 ± 7% -0.6 0.51 ± 7% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.88 ± 2% -0.5 0.37 ± 12% perf-profile.children.cycles-pp.__entry_text_start
1.14 ± 13% -0.5 0.64 ± 36% perf-profile.children.cycles-pp.tick_nohz_next_event
0.71 ± 2% -0.5 0.22 ± 16% perf-profile.children.cycles-pp._copy_from_iter
1.22 ± 13% -0.5 0.73 ± 16% perf-profile.children.cycles-pp.copyin
0.81 ± 2% -0.5 0.33 ± 13% perf-profile.children.cycles-pp.core_kernel_text
0.78 ± 4% -0.5 0.32 ± 13% perf-profile.children.cycles-pp.mutex_lock
0.65 ± 2% -0.4 0.22 ± 12% perf-profile.children.cycles-pp.copy_page_to_iter
0.76 ± 2% -0.4 0.35 ± 10% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.73 ± 8% -0.4 0.33 ± 10% perf-profile.children.cycles-pp.__fdget_pos
0.72 -0.4 0.33 ± 8% perf-profile.children.cycles-pp.update_rq_clock
0.68 ± 8% -0.4 0.29 ± 11% perf-profile.children.cycles-pp.__fget_light
0.66 ± 5% -0.4 0.29 ± 14% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.72 -0.4 0.35 ± 13% perf-profile.children.cycles-pp.__update_load_avg_se
0.55 ± 2% -0.4 0.18 ± 14% perf-profile.children.cycles-pp._copy_to_iter
0.70 ± 6% -0.4 0.33 ± 7% perf-profile.children.cycles-pp.sysvec_call_function_single
1.78 ± 12% -0.4 1.41 ± 2% perf-profile.children.cycles-pp.finish_task_switch
0.55 -0.3 0.20 ± 14% perf-profile.children.cycles-pp.kvm_guest_state
0.62 -0.3 0.28 ± 13% perf-profile.children.cycles-pp.select_task_rq
0.60 ± 8% -0.3 0.26 ± 10% perf-profile.children.cycles-pp.__fget_files
0.56 ± 5% -0.3 0.22 ± 13% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.51 ± 3% -0.3 0.20 ± 12% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.51 ± 3% -0.3 0.20 ± 13% perf-profile.children.cycles-pp.os_xsave
0.51 -0.3 0.21 ± 14% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.53 -0.3 0.23 ± 14% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.93 ± 14% -0.3 0.63 ± 14% perf-profile.children.cycles-pp.ktime_get
0.46 -0.3 0.17 ± 11% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.48 ± 2% -0.3 0.19 ± 13% perf-profile.children.cycles-pp.get_stack_info
0.53 ± 2% -0.3 0.24 ± 12% perf-profile.children.cycles-pp.select_task_rq_fair
0.92 ± 18% -0.3 0.64 ± 16% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
0.47 ± 2% -0.3 0.19 ± 15% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.54 ± 7% -0.3 0.27 ± 6% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.45 -0.3 0.17 ± 11% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.48 ± 2% -0.3 0.20 ± 8% perf-profile.children.cycles-pp.__might_resched
0.46 -0.3 0.19 ± 11% perf-profile.children.cycles-pp.perf_trace_buf_update
0.44 ± 4% -0.3 0.17 ± 15% perf-profile.children.cycles-pp.security_file_permission
0.35 ± 3% -0.3 0.09 ± 14% perf-profile.children.cycles-pp.__might_fault
0.42 -0.2 0.17 ± 11% perf-profile.children.cycles-pp.memcpy
0.51 ± 2% -0.2 0.26 ± 11% perf-profile.children.cycles-pp.___perf_sw_event
0.39 ± 2% -0.2 0.16 ± 11% perf-profile.children.cycles-pp.perf_trace_buf_alloc
0.35 ± 3% -0.2 0.12 ± 16% perf-profile.children.cycles-pp.touch_atime
0.36 ± 6% -0.2 0.13 ± 15% perf-profile.children.cycles-pp.get_callchain_entry
0.46 ± 2% -0.2 0.23 ± 5% perf-profile.children.cycles-pp.read_tsc
0.37 ± 2% -0.2 0.15 ± 12% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.35 ± 5% -0.2 0.13 ± 12% perf-profile.children.cycles-pp.update_cfs_group
0.34 ± 3% -0.2 0.12 ± 13% perf-profile.children.cycles-pp.copyout
0.35 ± 2% -0.2 0.14 ± 14% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.34 ± 5% -0.2 0.13 ± 17% perf-profile.children.cycles-pp.apparmor_file_permission
0.32 -0.2 0.12 ± 13% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.30 ± 5% -0.2 0.10 ± 14% perf-profile.children.cycles-pp.atime_needs_update
0.31 ± 2% -0.2 0.12 ± 8% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.36 ± 2% -0.2 0.17 ± 7% perf-profile.children.cycles-pp.ct_idle_exit
0.28 ± 2% -0.2 0.10 ± 14% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
0.28 -0.2 0.10 ± 10% perf-profile.children.cycles-pp.fput
0.29 ± 5% -0.2 0.11 ± 12% perf-profile.children.cycles-pp.reweight_entity
0.32 ± 3% -0.2 0.14 ± 11% perf-profile.children.cycles-pp.mutex_unlock
0.28 ± 3% -0.2 0.10 ± 15% perf-profile.children.cycles-pp.__might_sleep
0.28 ± 4% -0.2 0.11 ± 15% perf-profile.children.cycles-pp.perf_misc_flags
0.26 ± 2% -0.2 0.09 ± 12% perf-profile.children.cycles-pp.perf_instruction_pointer
0.26 ± 2% -0.2 0.09 ± 9% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.28 ± 2% -0.2 0.11 ± 10% perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.27 -0.2 0.10 ± 14% perf-profile.children.cycles-pp.finish_wait
0.27 ± 4% -0.2 0.10 ± 14% perf-profile.children.cycles-pp.current_time
0.37 ± 2% -0.2 0.21 ± 8% perf-profile.children.cycles-pp.__pthread_disable_asynccancel
0.25 ± 5% -0.2 0.10 ± 11% perf-profile.children.cycles-pp.perf_swevent_event
0.65 ± 2% -0.2 0.50 ± 8% perf-profile.children.cycles-pp.perf_output_begin_forward
0.24 ± 10% -0.1 0.09 ± 9% perf-profile.children.cycles-pp.perf_output_end
0.24 ± 3% -0.1 0.10 ± 13% perf-profile.children.cycles-pp.get_stack_info_noinstr
0.24 ± 2% -0.1 0.10 ± 14% perf-profile.children.cycles-pp.hrtimer_get_next_event
0.25 -0.1 0.11 ± 15% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.24 ± 4% -0.1 0.10 ± 11% perf-profile.children.cycles-pp.perf_trace_run_bpf_submit
0.23 ± 6% -0.1 0.09 ± 10% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.22 ± 6% -0.1 0.09 ± 14% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.24 ± 3% -0.1 0.11 ± 8% perf-profile.children.cycles-pp.__cond_resched
0.24 ± 2% -0.1 0.11 ± 13% perf-profile.children.cycles-pp.__is_insn_slot_addr
0.23 ± 3% -0.1 0.11 ± 9% perf-profile.children.cycles-pp.ct_kernel_exit_state
0.22 ± 4% -0.1 0.10 ± 14% perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.18 ± 2% -0.1 0.06 ± 10% perf-profile.children.cycles-pp.place_entity
0.23 ± 3% -0.1 0.12 ± 8% perf-profile.children.cycles-pp.ct_kernel_enter
0.59 ± 15% -0.1 0.48 ± 7% perf-profile.children.cycles-pp.shmem_get_folio_gfp
0.21 ± 6% -0.1 0.10 ± 12% perf-profile.children.cycles-pp.file_update_time
0.19 ± 7% -0.1 0.08 ± 21% perf-profile.children.cycles-pp.update_min_vruntime
0.17 ± 2% -0.1 0.06 ± 14% perf-profile.children.cycles-pp.put_prev_task_fair
0.17 ± 3% -0.1 0.06 ± 12% perf-profile.children.cycles-pp.switch_fpu_return
0.17 ± 4% -0.1 0.07 ± 15% perf-profile.children.cycles-pp.__calc_delta
0.18 ± 4% -0.1 0.07 ± 10% perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
0.16 ± 2% -0.1 0.06 ± 12% perf-profile.children.cycles-pp.testcase
0.18 ± 2% -0.1 0.07 ± 18% perf-profile.children.cycles-pp.pick_next_entity
0.21 ± 3% -0.1 0.11 ± 15% perf-profile.children.cycles-pp.select_idle_sibling
0.18 ± 4% -0.1 0.08 ± 11% perf-profile.children.cycles-pp.__smp_call_single_queue
0.15 ± 2% -0.1 0.06 ± 12% perf-profile.children.cycles-pp.__list_del_entry_valid
0.15 ± 6% -0.1 0.06 ± 14% perf-profile.children.cycles-pp.aa_file_perm
0.15 ± 2% -0.1 0.06 ± 14% perf-profile.children.cycles-pp.__perf_event_account_interrupt
0.16 ± 4% -0.1 0.07 ± 13% perf-profile.children.cycles-pp.is_bpf_text_address
0.15 ± 2% -0.1 0.06 ± 18% perf-profile.children.cycles-pp.pick_next_task_idle
0.13 ± 5% -0.1 0.04 ± 50% perf-profile.children.cycles-pp.in_task_stack
0.15 ± 4% -0.1 0.06 ± 12% perf-profile.children.cycles-pp.__list_add_valid
0.13 ± 4% -0.1 0.04 ± 50% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.16 ± 4% -0.1 0.08 ± 13% perf-profile.children.cycles-pp.__pthread_enable_asynccancel
0.16 ± 4% -0.1 0.08 ± 13% perf-profile.children.cycles-pp.nohz_run_idle_balance
0.13 ± 8% -0.1 0.05 ± 52% perf-profile.children.cycles-pp.check_preempt_curr
0.12 ± 8% -0.1 0.03 ± 82% perf-profile.children.cycles-pp.rb_next
0.12 ± 14% -0.1 0.04 ± 50% perf-profile.children.cycles-pp.cpuacct_charge
0.34 ± 4% -0.1 0.25 ± 4% perf-profile.children.cycles-pp.native_irq_return_iret
0.12 ± 6% -0.1 0.03 ± 84% perf-profile.children.cycles-pp.__update_idle_core
0.14 ± 7% -0.1 0.06 ± 12% perf-profile.children.cycles-pp.rcu_note_context_switch
0.19 ± 14% -0.1 0.11 ± 10% perf-profile.children.cycles-pp.shmem_add_to_page_cache
0.11 ± 5% -0.1 0.03 ± 82% perf-profile.children.cycles-pp.is_module_text_address
0.20 ± 15% -0.1 0.12 ± 15% perf-profile.children.cycles-pp.shmem_alloc_and_acct_folio
0.12 ± 3% -0.1 0.04 ± 50% perf-profile.children.cycles-pp.__get_task_ioprio
0.14 ± 2% -0.1 0.07 ± 7% perf-profile.children.cycles-pp.llist_add_batch
0.17 ± 12% -0.1 0.10 ± 32% perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.13 ± 5% -0.1 0.06 ± 12% perf-profile.children.cycles-pp.sched_clock
0.13 ± 5% -0.1 0.06 ± 12% perf-profile.children.cycles-pp.tick_nohz_tick_stopped
0.13 ± 3% -0.1 0.05 ± 9% perf-profile.children.cycles-pp.send_call_function_single_ipi
0.12 ± 4% -0.1 0.04 ± 50% perf-profile.children.cycles-pp.is_ftrace_trampoline
0.11 ± 4% -0.1 0.04 ± 50% perf-profile.children.cycles-pp.schedule_debug
0.20 ± 16% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.ordered_events__queue
0.16 ± 12% -0.1 0.09 ± 33% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
0.13 ± 4% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.rcu_all_qs
0.16 ± 16% -0.1 0.09 ± 14% perf-profile.children.cycles-pp.shmem_alloc_folio
0.20 ± 4% -0.1 0.14 ± 5% perf-profile.children.cycles-pp.native_apic_msr_eoi_write
0.14 ± 16% -0.1 0.09 ± 13% perf-profile.children.cycles-pp.vma_alloc_folio
0.09 ± 15% -0.1 0.04 ± 51% perf-profile.children.cycles-pp.get_page_from_freelist
0.10 ± 6% -0.0 0.05 ± 7% perf-profile.children.cycles-pp.__x2apic_send_IPI_dest
0.11 ± 19% -0.0 0.06 ± 12% perf-profile.children.cycles-pp.folio_add_lru
0.12 ± 16% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.__folio_alloc
0.12 ± 16% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.__alloc_pages
0.10 ± 16% -0.0 0.06 ± 12% perf-profile.children.cycles-pp.folio_batch_move_lru
0.13 ± 3% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.irqtime_account_irq
0.05 ± 50% +0.0 0.09 ± 17% perf-profile.children.cycles-pp.__filemap_get_folio
0.10 ± 14% +0.0 0.14 ± 17% perf-profile.children.cycles-pp.perf_session__deliver_event
0.10 ± 16% +0.0 0.14 ± 18% perf-profile.children.cycles-pp.perf_session__process_user_event
0.10 ± 16% +0.0 0.14 ± 18% perf-profile.children.cycles-pp.__ordered_events__flush
0.04 ± 50% +0.0 0.09 ± 14% perf-profile.children.cycles-pp.update_sg_lb_stats
0.06 ± 8% +0.1 0.11 ± 15% perf-profile.children.cycles-pp.update_sd_lb_stats
0.06 ± 6% +0.1 0.12 ± 13% perf-profile.children.cycles-pp.find_busiest_group
0.00 +0.1 0.07 ± 12% perf-profile.children.cycles-pp.folio_mark_accessed
0.08 ± 10% +0.1 0.15 ± 11% perf-profile.children.cycles-pp.load_balance
0.00 +0.1 0.08 ± 27% perf-profile.children.cycles-pp.__perf_mmap__read_init
0.00 +0.1 0.08 ± 20% perf-profile.children.cycles-pp.update_blocked_averages
0.00 +0.1 0.13 ± 27% perf-profile.children.cycles-pp.perf_mmap__read_init
0.00 +0.2 0.23 ± 18% perf-profile.children.cycles-pp.ring_buffer_read_head
0.00 +0.2 0.24 ± 17% perf-profile.children.cycles-pp.perf_mmap__read_head
0.22 ± 2% +0.5 0.69 ± 7% perf-profile.children.cycles-pp.switch_mm_irqs_off
41.73 +1.7 43.40 perf-profile.children.cycles-pp.__libc_read
37.62 +4.2 41.84 perf-profile.children.cycles-pp.ksys_read
36.89 +4.7 41.60 perf-profile.children.cycles-pp.vfs_read
36.13 +5.2 41.30 perf-profile.children.cycles-pp.pipe_read
32.16 +7.7 39.86 perf-profile.children.cycles-pp.schedule
34.12 +10.2 44.37 perf-profile.children.cycles-pp.start_secondary
35.09 +10.5 45.63 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
35.09 +10.5 45.63 perf-profile.children.cycles-pp.cpu_startup_entry
34.97 +10.6 45.57 perf-profile.children.cycles-pp.do_idle
14.68 +18.3 32.96 ± 4% perf-profile.children.cycles-pp.schedule_idle
46.43 +26.2 72.65 ± 2% perf-profile.children.cycles-pp.__schedule
0.00 +49.6 49.57 ± 8% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.81 ± 2% +50.9 51.66 ± 8% perf-profile.children.cycles-pp.prepare_task_switch
0.58 ± 2% +50.9 51.44 ± 8% perf-profile.children.cycles-pp._raw_spin_lock
10.88 ± 2% -6.3 4.59 ± 13% perf-profile.self.cycles-pp.unwind_next_frame
6.51 -3.7 2.77 ± 13% perf-profile.self.cycles-pp.__orc_find
3.68 -2.1 1.54 ± 13% perf-profile.self.cycles-pp.stack_access_ok
2.97 ± 2% -1.7 1.27 ± 13% perf-profile.self.cycles-pp.orc_find
4.42 ± 2% -1.4 3.00 ± 10% perf-profile.self.cycles-pp.poll_idle
2.09 -1.3 0.82 ± 15% perf-profile.self.cycles-pp.memcpy_erms
1.91 -1.1 0.80 ± 14% perf-profile.self.cycles-pp.__output_copy
2.28 ± 10% -1.0 1.32 perf-profile.self.cycles-pp.__schedule
1.58 -0.9 0.64 ± 12% perf-profile.self.cycles-pp.__unwind_start
1.50 ± 2% -0.9 0.61 ± 14% perf-profile.self.cycles-pp.perf_callchain_kernel
1.19 ± 2% -0.7 0.47 ± 13% perf-profile.self.cycles-pp.__get_user_nocheck_8
1.21 -0.7 0.50 ± 13% perf-profile.self.cycles-pp.perf_output_sample
1.29 ± 2% -0.7 0.58 ± 8% perf-profile.self.cycles-pp.native_sched_clock
1.14 ± 3% -0.7 0.44 ± 13% perf-profile.self.cycles-pp.__switch_to
1.01 ± 2% -0.6 0.37 ± 12% perf-profile.self.cycles-pp.__switch_to_asm
1.08 -0.6 0.46 ± 10% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.43 ± 11% -0.6 0.81 ± 15% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.91 ± 2% -0.5 0.37 ± 14% perf-profile.self.cycles-pp.kernel_text_address
1.47 ± 15% -0.5 0.93 perf-profile.self.cycles-pp.finish_task_switch
0.96 -0.5 0.46 ± 8% perf-profile.self.cycles-pp.menu_select
0.74 ± 2% -0.5 0.28 ± 12% perf-profile.self.cycles-pp.perf_tp_event
0.73 ± 3% -0.4 0.29 ± 14% perf-profile.self.cycles-pp.get_perf_callchain
0.75 ± 3% -0.4 0.31 ± 13% perf-profile.self.cycles-pp.unwind_get_return_address
0.65 ± 2% -0.4 0.21 ± 15% perf-profile.self.cycles-pp.pipe_read
0.72 ± 2% -0.4 0.29 ± 11% perf-profile.self.cycles-pp.perf_prepare_sample
0.70 -0.4 0.27 ± 13% perf-profile.self.cycles-pp.enqueue_entity
0.74 ± 2% -0.4 0.31 ± 12% perf-profile.self.cycles-pp.ftrace_graph_ret_addr
0.71 ± 2% -0.4 0.28 ± 14% perf-profile.self.cycles-pp.update_load_avg
0.72 ± 2% -0.4 0.30 ± 12% perf-profile.self.cycles-pp.__kernel_text_address
0.61 ± 2% -0.4 0.26 ± 14% perf-profile.self.cycles-pp.do_idle
0.60 -0.3 0.26 ± 9% perf-profile.self.cycles-pp.cpuidle_idle_call
0.62 ± 5% -0.3 0.27 ± 14% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.58 ± 2% -0.3 0.24 ± 13% perf-profile.self.cycles-pp.core_kernel_text
0.58 ± 9% -0.3 0.25 ± 12% perf-profile.self.cycles-pp.__fget_files
0.63 -0.3 0.31 ± 13% perf-profile.self.cycles-pp.__update_load_avg_se
0.51 ± 3% -0.3 0.20 ± 14% perf-profile.self.cycles-pp.prepare_to_wait_event
0.50 ± 3% -0.3 0.19 ± 12% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.51 ± 2% -0.3 0.21 ± 7% perf-profile.self.cycles-pp.__libc_write
0.50 ± 3% -0.3 0.20 ± 13% perf-profile.self.cycles-pp.os_xsave
0.49 ± 2% -0.3 0.19 ± 14% perf-profile.self.cycles-pp.perf_callchain_user
0.50 -0.3 0.20 ± 15% perf-profile.self.cycles-pp.perf_output_copy
0.49 ± 5% -0.3 0.19 ± 14% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.49 -0.3 0.20 ± 13% perf-profile.self.cycles-pp.__libc_read
0.49 ± 3% -0.3 0.20 ± 20% perf-profile.self.cycles-pp.update_curr
0.44 ± 2% -0.3 0.16 ± 14% perf-profile.self.cycles-pp.kvm_guest_state
0.44 -0.3 0.17 ± 10% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.47 ± 2% -0.3 0.20 ± 13% perf-profile.self.cycles-pp.__perf_event_header__init_id
0.37 ± 3% -0.2 0.13 ± 13% perf-profile.self.cycles-pp.pipe_write
0.43 ± 2% -0.2 0.18 ± 8% perf-profile.self.cycles-pp.__might_resched
0.39 ± 2% -0.2 0.14 ± 14% perf-profile.self.cycles-pp.vfs_read
0.37 ± 4% -0.2 0.14 ± 13% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.44 ± 2% -0.2 0.22 ± 5% perf-profile.self.cycles-pp.read_tsc
0.41 ± 6% -0.2 0.19 ± 15% perf-profile.self.cycles-pp.dequeue_entity
0.34 ± 7% -0.2 0.12 ± 16% perf-profile.self.cycles-pp.get_callchain_entry
0.35 ± 2% -0.2 0.14 ± 14% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.31 ± 5% -0.2 0.12 ± 15% perf-profile.self.cycles-pp.update_cfs_group
0.32 -0.2 0.12 ± 13% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.29 ± 6% -0.2 0.09 ± 10% perf-profile.self.cycles-pp.try_to_wake_up
0.32 ± 5% -0.2 0.13 ± 12% perf-profile.self.cycles-pp.enqueue_task_fair
0.27 ± 3% -0.2 0.08 ± 14% perf-profile.self.cycles-pp.ksys_read
0.30 ± 2% -0.2 0.11 ± 11% perf-profile.self.cycles-pp.pick_next_task_fair
0.29 ± 2% -0.2 0.11 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.31 ± 3% -0.2 0.13 ± 15% perf-profile.self.cycles-pp.perf_event_output_forward
0.31 ± 4% -0.2 0.14 ± 15% perf-profile.self.cycles-pp.mutex_lock
0.27 ± 3% -0.2 0.09 ± 15% perf-profile.self.cycles-pp.do_syscall_64
0.27 ± 4% -0.2 0.10 ± 14% perf-profile.self.cycles-pp.__perf_event_overflow
0.27 ± 5% -0.2 0.10 ± 13% perf-profile.self.cycles-pp.reweight_entity
0.29 ± 3% -0.2 0.12 ± 8% perf-profile.self.cycles-pp.update_rq_clock
0.29 ± 4% -0.2 0.13 ± 10% perf-profile.self.cycles-pp.mutex_unlock
0.25 ± 3% -0.2 0.08 ± 14% perf-profile.self.cycles-pp.__rdgsbase_inactive
0.26 ± 3% -0.2 0.10 ± 14% perf-profile.self.cycles-pp.dequeue_task_fair
0.26 ± 2% -0.2 0.10 ± 11% perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.36 ± 3% -0.2 0.21 ± 13% perf-profile.self.cycles-pp.___perf_sw_event
0.24 -0.2 0.09 ± 12% perf-profile.self.cycles-pp.fput
0.34 ± 2% -0.2 0.19 ± 9% perf-profile.self.cycles-pp.__pthread_disable_asynccancel
0.23 ± 3% -0.1 0.08 ± 13% perf-profile.self.cycles-pp.__might_sleep
0.25 ± 2% -0.1 0.10 ± 13% perf-profile.self.cycles-pp.set_next_entity
0.29 -0.1 0.14 ± 12% perf-profile.self.cycles-pp.sched_clock_cpu
0.23 ± 4% -0.1 0.09 ± 15% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.63 ± 2% -0.1 0.49 ± 7% perf-profile.self.cycles-pp.perf_output_begin_forward
0.22 ± 2% -0.1 0.08 ± 11% perf-profile.self.cycles-pp.newidle_balance
0.18 ± 2% -0.1 0.05 ± 52% perf-profile.self.cycles-pp._copy_from_iter
0.22 ± 11% -0.1 0.08 ± 9% perf-profile.self.cycles-pp.perf_output_end
0.28 ± 22% -0.1 0.15 ± 38% perf-profile.self.cycles-pp.tick_nohz_next_event
0.22 -0.1 0.09 ± 11% perf-profile.self.cycles-pp.schedule_idle
0.23 ± 2% -0.1 0.10 ± 7% perf-profile.self.cycles-pp.vfs_write
0.54 ± 3% -0.1 0.41 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state
0.20 ± 2% -0.1 0.08 ± 13% perf-profile.self.cycles-pp.perf_callchain
0.21 ± 6% -0.1 0.08 ± 15% perf-profile.self.cycles-pp.perf_swevent_event
0.22 ± 2% -0.1 0.09 ± 14% perf-profile.self.cycles-pp.get_stack_info
0.20 ± 3% -0.1 0.08 ± 15% perf-profile.self.cycles-pp.schedule
0.20 ± 3% -0.1 0.08 ± 9% perf-profile.self.cycles-pp.memcpy
0.21 ± 3% -0.1 0.08 ± 12% perf-profile.self.cycles-pp.perf_trace_buf_alloc
0.22 ± 2% -0.1 0.10 ± 6% perf-profile.self.cycles-pp.ct_kernel_exit_state
0.20 ± 2% -0.1 0.09 ± 8% perf-profile.self.cycles-pp.perf_trace_run_bpf_submit
0.20 ± 2% -0.1 0.08 ± 14% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.19 ± 5% -0.1 0.07 ± 18% perf-profile.self.cycles-pp.apparmor_file_permission
0.18 ± 2% -0.1 0.07 ± 15% perf-profile.self.cycles-pp.sched_ttwu_pending
0.19 ± 8% -0.1 0.08 ± 14% perf-profile.self.cycles-pp.__entry_text_start
0.18 ± 2% -0.1 0.07 ± 16% perf-profile.self.cycles-pp.perf_trace_buf_update
0.17 ± 2% -0.1 0.07 ± 16% perf-profile.self.cycles-pp.cpu_startup_entry
0.16 ± 3% -0.1 0.06 ± 14% perf-profile.self.cycles-pp.switch_fpu_return
0.23 ± 3% -0.1 0.13 ± 15% perf-profile.self.cycles-pp.select_task_rq_fair
0.30 ± 2% -0.1 0.20 ± 17% perf-profile.self.cycles-pp.prepare_task_switch
0.18 ± 2% -0.1 0.09 ± 13% perf-profile.self.cycles-pp.__is_insn_slot_addr
0.13 ± 3% -0.1 0.03 ± 82% perf-profile.self.cycles-pp.tick_nohz_idle_exit
0.15 ± 3% -0.1 0.06 ± 12% perf-profile.self.cycles-pp.current_time
0.15 ± 3% -0.1 0.06 ± 12% perf-profile.self.cycles-pp.testcase
0.16 ± 5% -0.1 0.07 ± 15% perf-profile.self.cycles-pp.__calc_delta
0.15 ± 3% -0.1 0.06 ± 15% perf-profile.self.cycles-pp.perf_misc_flags
0.17 ± 2% -0.1 0.08 ± 9% perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.17 ± 5% -0.1 0.08 ± 18% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.15 ± 3% -0.1 0.06 ± 12% perf-profile.self.cycles-pp.pick_next_entity
0.15 ± 7% -0.1 0.06 ± 14% perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.13 ± 7% -0.1 0.05 ± 52% perf-profile.self.cycles-pp.aa_file_perm
0.13 ± 3% -0.1 0.04 ± 51% perf-profile.self.cycles-pp.__perf_event_account_interrupt
0.15 ± 6% -0.1 0.06 ± 21% perf-profile.self.cycles-pp.update_min_vruntime
0.15 ± 5% -0.1 0.06 ± 12% perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
0.16 -0.1 0.07 ± 13% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.15 ± 5% -0.1 0.07 ± 15% perf-profile.self.cycles-pp.nohz_run_idle_balance
0.34 ± 4% -0.1 0.25 ± 4% perf-profile.self.cycles-pp.native_irq_return_iret
0.14 ± 5% -0.1 0.06 ± 12% perf-profile.self.cycles-pp.get_next_timer_interrupt
0.12 ± 3% -0.1 0.04 ± 51% perf-profile.self.cycles-pp.send_call_function_single_ipi
0.11 ± 3% -0.1 0.03 ± 82% perf-profile.self.cycles-pp.tick_nohz_tick_stopped
0.14 ± 5% -0.1 0.06 ± 14% perf-profile.self.cycles-pp.llist_add_batch
0.12 ± 4% -0.1 0.04 ± 50% perf-profile.self.cycles-pp.__list_add_valid
0.12 ± 15% -0.1 0.04 ± 50% perf-profile.self.cycles-pp.cpuacct_charge
0.13 ± 3% -0.1 0.06 ± 10% perf-profile.self.cycles-pp.ct_idle_exit
0.13 ± 3% -0.1 0.05 ± 9% perf-profile.self.cycles-pp.get_stack_info_noinstr
0.14 ± 2% -0.1 0.07 ± 9% perf-profile.self.cycles-pp.__pthread_enable_asynccancel
0.13 ± 7% -0.1 0.06 ± 14% perf-profile.self.cycles-pp.rcu_note_context_switch
0.12 ± 4% -0.1 0.04 ± 51% perf-profile.self.cycles-pp.ct_kernel_enter
0.13 ± 5% -0.1 0.06 ± 6% perf-profile.self.cycles-pp.__cond_resched
0.11 ± 3% -0.1 0.04 ± 50% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.19 ± 17% -0.1 0.12 ± 27% perf-profile.self.cycles-pp.queue_event
0.13 ± 15% -0.1 0.07 ± 63% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
0.20 ± 5% -0.1 0.13 ± 3% perf-profile.self.cycles-pp.native_apic_msr_eoi_write
0.10 ± 4% -0.0 0.05 ± 7% perf-profile.self.cycles-pp.__x2apic_send_IPI_dest
0.01 ±200% +0.1 0.07 ± 13% perf-profile.self.cycles-pp.update_sg_lb_stats
0.00 +0.2 0.21 ± 18% perf-profile.self.cycles-pp.ring_buffer_read_head
0.21 ± 2% +0.5 0.69 ± 6% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.54 +1.3 1.84 perf-profile.self.cycles-pp._raw_spin_lock
0.00 +49.6 49.57 ± 8% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath

***************************************************************************************************
lkp-skl-fpga01: 104 threads 2 sockets (Skylake) with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-11/performance/x86_64-rhel-8.3/thread/16/debian-11.1-x86_64-20220510.cgz/lkp-skl-fpga01/context_switch1/will-it-scale

commit:
99babd04b2 ("selftests/rseq: Implement rseq numa node id field selftest")
af7f588d8f ("sched: Introduce per-memory-map concurrency ID")

99babd04b2505471 af7f588d8f7355bc4298dd1962d
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:6 -17% :6 kmsg.EDAC_skx:Can't_find_handle_for_NVDIMM_ADR=
1:6 -17% :6 kmsg.microcode:Attempting_late_microcode_loading-it_is_dangerous_and_taints_the_kernel
1:6 -17% :6 kmsg.microcode:You_should_switch_to_early_loading,if_possible
%stddev %change %stddev
\ | \
4324658 ± 3% -57.9% 1818800 will-it-scale.16.threads
270290 ± 3% -57.9% 113674 will-it-scale.per_thread_ops
4324658 ± 3% -57.9% 1818800 will-it-scale.workload
1.329e+09 ± 3% -58.0% 5.581e+08 cpuidle..usage
12.36 ± 2% +2.9 15.22 mpstat.cpu.all.sys%
5.59 -3.0 2.57 ± 4% mpstat.cpu.all.usr%
19.17 -11.3% 17.00 vmstat.procs.r
8547612 ± 2% -58.0% 3593440 vmstat.system.cs
233179 +16.0% 270468 ± 2% vmstat.system.in
158734 ± 25% +182.8% 448837 ± 18% turbostat.C1
0.14 ± 5% -48.8% 0.07 turbostat.IPC
70953881 +16.1% 82393754 ± 2% turbostat.IRQ
1.286e+09 ± 3% -60.0% 5.137e+08 turbostat.POLL
1.67 ± 6% -0.7 1.00 ± 2% turbostat.POLL%
252.48 -8.1% 232.09 turbostat.PkgWatt
114167 ± 20% -93.5% 7417 ± 59% sched_debug.cfs_rq:/.MIN_vruntime.avg
1525472 -76.4% 359932 ± 59% sched_debug.cfs_rq:/.MIN_vruntime.max
387653 ± 11% -87.1% 49952 ± 56% sched_debug.cfs_rq:/.MIN_vruntime.stddev
0.39 ± 6% -24.3% 0.29 ± 2% sched_debug.cfs_rq:/.h_nr_running.avg
22932 ± 5% +13.0% 25920 ± 3% sched_debug.cfs_rq:/.load.avg
59500 ± 4% +55.5% 92526 ± 68% sched_debug.cfs_rq:/.load.max
114167 ± 20% -93.5% 7417 ± 59% sched_debug.cfs_rq:/.max_vruntime.avg
1525472 -76.4% 359932 ± 59% sched_debug.cfs_rq:/.max_vruntime.max
387653 ± 11% -87.1% 49952 ± 56% sched_debug.cfs_rq:/.max_vruntime.stddev
671631 ± 3% -16.0% 564476 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg
1542612 -17.8% 1267561 sched_debug.cfs_rq:/.min_vruntime.max
708750 -18.9% 575136 sched_debug.cfs_rq:/.min_vruntime.stddev
0.37 ± 6% -24.7% 0.28 ± 2% sched_debug.cfs_rq:/.nr_running.avg
-848558 -19.6% -682164 sched_debug.cfs_rq:/.spread0.avg
-1486548 -18.5% -1211535 sched_debug.cfs_rq:/.spread0.min
708763 -18.9% 575149 sched_debug.cfs_rq:/.spread0.stddev
214.42 -10.2% 192.45 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.stddev
3058 ± 4% +113.4% 6526 ± 6% sched_debug.cpu.avg_idle.min
1063 ± 2% -14.8% 906.25 ± 5% sched_debug.cpu.curr->pid.avg
0.20 ± 4% -13.3% 0.17 ± 2% sched_debug.cpu.nr_running.avg
12311145 ± 3% -57.7% 5209198 sched_debug.cpu.nr_switches.avg
40179926 ± 3% -57.8% 16957686 sched_debug.cpu.nr_switches.max
18457770 ± 3% -57.7% 7805110 sched_debug.cpu.nr_switches.stddev
1.48 ± 33% +393.8% 7.31 ± 14% perf-stat.i.MPKI
9.306e+09 -51.2% 4.543e+09 ± 5% perf-stat.i.branch-instructions
1.571e+08 ± 2% -49.6% 79141069 ± 4% perf-stat.i.branch-misses
67978805 ± 32% +129.9% 1.563e+08 ± 13% perf-stat.i.cache-references
8635700 ± 3% -58.0% 3628862 perf-stat.i.context-switches
2.00 +112.1% 4.24 ± 2% perf-stat.i.cpi
152.95 ± 4% +10.5% 168.99 ± 2% perf-stat.i.cpu-migrations
9807863 ± 8% -54.4% 4473079 ± 12% perf-stat.i.dTLB-load-misses
1.313e+10 -52.5% 6.241e+09 ± 4% perf-stat.i.dTLB-loads
7.867e+09 -54.9% 3.549e+09 ± 4% perf-stat.i.dTLB-stores
9070094 ± 3% -58.2% 3793161 ± 49% perf-stat.i.iTLB-load-misses
11272936 ± 7% -50.6% 5566942 ± 8% perf-stat.i.iTLB-loads
4.592e+10 -51.8% 2.214e+10 ± 5% perf-stat.i.instructions
0.50 -51.9% 0.24 ± 4% perf-stat.i.ipc
660.44 ± 31% -85.8% 93.67 ± 4% perf-stat.i.metric.K/sec
291.58 -52.2% 139.32 ± 4% perf-stat.i.metric.M/sec
1.48 ± 32% +375.8% 7.06 ± 12% perf-stat.overall.MPKI
1.99 +108.2% 4.14 ± 4% perf-stat.overall.cpi
0.50 -51.9% 0.24 ± 4% perf-stat.overall.ipc
3197936 ± 3% +14.7% 3668175 ± 4% perf-stat.overall.path-length
9.276e+09 -51.2% 4.528e+09 ± 5% perf-stat.ps.branch-instructions
1.566e+08 ± 2% -49.6% 78882571 ± 4% perf-stat.ps.branch-misses
67767931 ± 32% +129.8% 1.557e+08 ± 13% perf-stat.ps.cache-references
8606808 ± 3% -58.0% 3616809 perf-stat.ps.context-switches
152.61 ± 4% +10.4% 168.54 ± 2% perf-stat.ps.cpu-migrations
9780257 ± 8% -54.4% 4458181 ± 12% perf-stat.ps.dTLB-load-misses
1.309e+10 -52.5% 6.221e+09 ± 4% perf-stat.ps.dTLB-loads
7.842e+09 -54.9% 3.538e+09 ± 4% perf-stat.ps.dTLB-stores
9039813 ± 3% -58.2% 3780389 ± 49% perf-stat.ps.iTLB-load-misses
11235261 ± 7% -50.6% 5548664 ± 8% perf-stat.ps.iTLB-loads
4.577e+10 -51.8% 2.207e+10 ± 5% perf-stat.ps.instructions
1.381e+13 -51.7% 6.671e+12 ± 4% perf-stat.total.instructions
20.96 ± 7% -9.2 11.78 ± 32% perf-profile.calltrace.cycles-pp.__libc_write
15.02 ± 20% -4.7 10.34 ± 8% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
5.46 ± 28% -2.9 2.58 ± 7% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
4.64 ± 32% -2.4 2.25 ± 11% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
4.55 ± 28% -2.4 2.18 ± 8% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
3.70 ± 37% -1.9 1.84 ± 12% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry
3.55 ± 39% -1.8 1.78 ± 12% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle
2.78 ± 33% -1.5 1.31 ± 6% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_read
2.63 ± 39% -1.4 1.28 ± 7% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_write
2.60 ± 32% -1.3 1.33 ± 6% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
2.27 ± 20% -1.2 1.05 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
2.29 ± 35% -1.1 1.15 ± 4% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
2.08 ± 32% -1.1 0.94 ± 12% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
1.74 ± 34% -0.9 0.83 ± 3% perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.25 ± 45% -0.9 0.36 ± 71% perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary
1.18 ± 29% -0.9 0.30 ±100% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
1.07 ± 34% -0.8 0.26 ±100% perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_read
1.06 ± 38% -0.8 0.27 ±100% perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_write
1.03 ± 21% -0.7 0.28 ±100% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.vfs_write.ksys_write.do_syscall_64
1.38 ± 23% -0.7 0.67 ± 4% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle
0.00 +0.8 0.83 ± 11% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule_idle.do_idle.cpu_startup_entry
40.47 ± 4% +2.9 43.39 ± 2% perf-profile.calltrace.cycles-pp.__libc_read
34.36 ± 8% +6.8 41.18 ± 10% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
34.34 ± 8% +6.8 41.17 ± 10% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
34.08 ± 8% +7.0 41.04 ± 10% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
35.35 ± 8% +7.0 42.35 ± 10% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
32.51 ± 14% +7.7 40.21 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
30.86 ± 16% +8.5 39.36 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
27.80 ± 22% +10.0 37.80 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
27.05 ± 23% +10.4 37.42 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
25.85 ± 26% +11.0 36.84 perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
21.15 ± 38% +13.5 34.64 perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
20.97 ± 39% +13.6 34.56 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
9.89 ± 37% +16.4 26.25 ± 18% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
9.57 ± 39% +16.5 26.10 ± 18% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.00 +18.8 18.81 ± 49% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle
0.00 +19.3 19.34 ± 47% perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle.do_idle
0.29 ±100% +19.4 19.72 ± 46% perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
0.00 +19.8 19.76 ± 45% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule
0.00 +20.3 20.31 ± 44% perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule.pipe_read
0.29 ±100% +20.4 20.67 ± 43% perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.pipe_read.vfs_read
22.18 ± 7% -9.1 13.08 ± 38% perf-profile.children.cycles-pp.__libc_write
15.50 ± 20% -4.9 10.61 ± 7% perf-profile.children.cycles-pp.cpuidle_idle_call
5.65 ± 29% -3.0 2.66 ± 7% perf-profile.children.cycles-pp.flush_smp_call_function_queue
5.48 ± 35% -2.8 2.67 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret
5.08 ± 26% -2.5 2.61 ± 4% perf-profile.children.cycles-pp.sched_ttwu_pending
4.74 ± 28% -2.5 2.27 ± 8% perf-profile.children.cycles-pp.menu_select
4.94 ± 33% -2.4 2.55 ± 4% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
4.06 ± 30% -1.9 2.13 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate
3.89 ± 32% -1.8 2.06 ± 5% perf-profile.children.cycles-pp.enqueue_task_fair
3.16 ± 21% -1.6 1.59 ± 8% perf-profile.children.cycles-pp.pick_next_task_fair
3.19 ± 34% -1.4 1.75 ± 6% perf-profile.children.cycles-pp.enqueue_entity
2.62 ± 18% -1.4 1.25 ± 3% perf-profile.children.cycles-pp.update_load_avg
2.41 ± 35% -1.3 1.15 ± 4% perf-profile.children.cycles-pp.__entry_text_start
2.17 ± 32% -1.2 0.98 ± 12% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
1.76 ± 42% -1.1 0.66 ± 13% perf-profile.children.cycles-pp.__switch_to_asm
1.56 ± 48% -1.1 0.51 ± 9% perf-profile.children.cycles-pp.__switch_to
1.97 ± 34% -1.0 0.95 ± 5% perf-profile.children.cycles-pp.ktime_get
1.84 ± 34% -1.0 0.88 ± 3% perf-profile.children.cycles-pp.tick_nohz_idle_exit
1.54 ± 23% -0.8 0.74 ± 2% perf-profile.children.cycles-pp.set_next_entity
0.98 ± 51% -0.7 0.26 ± 20% perf-profile.children.cycles-pp.os_xsave
1.26 ± 30% -0.7 0.55 ± 16% perf-profile.children.cycles-pp.tick_nohz_next_event
1.15 ± 33% -0.6 0.52 ± 7% perf-profile.children.cycles-pp.mutex_lock
1.40 ± 10% -0.6 0.79 ± 12% perf-profile.children.cycles-pp.sched_clock_cpu
1.17 ± 34% -0.6 0.58 ± 8% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.06 ± 32% -0.5 0.52 ± 2% perf-profile.children.cycles-pp.read_tsc
0.93 ± 26% -0.5 0.41 ± 5% perf-profile.children.cycles-pp.select_task_rq
1.04 ± 21% -0.5 0.52 ± 10% perf-profile.children.cycles-pp.copy_page_from_iter
1.18 ± 12% -0.5 0.66 ± 9% perf-profile.children.cycles-pp.native_sched_clock
0.92 ± 34% -0.5 0.47 ± 3% perf-profile.children.cycles-pp.__fdget_pos
0.88 ± 36% -0.4 0.45 ± 3% perf-profile.children.cycles-pp.__fget_light
0.85 ± 36% -0.4 0.42 ± 6% perf-profile.children.cycles-pp.security_file_permission
0.88 ± 34% -0.4 0.45 ± 5% perf-profile.children.cycles-pp.prepare_to_wait_event
0.78 ± 23% -0.4 0.35 ± 5% perf-profile.children.cycles-pp.__update_load_avg_se
0.75 ± 38% -0.4 0.32 ± 10% perf-profile.children.cycles-pp.copy_page_to_iter
0.76 ± 31% -0.4 0.34 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair
0.65 ± 39% -0.4 0.26 ± 21% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.71 ± 42% -0.4 0.32 ± 11% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.75 ± 35% -0.4 0.37 ± 4% perf-profile.children.cycles-pp.apparmor_file_permission
0.80 ± 22% -0.4 0.43 ± 12% perf-profile.children.cycles-pp._copy_from_iter
0.64 ± 34% -0.4 0.26 ± 6% perf-profile.children.cycles-pp._copy_to_iter
0.71 ± 18% -0.4 0.34 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.67 ± 38% -0.3 0.34 ± 3% perf-profile.children.cycles-pp.__fget_files
0.75 ± 29% -0.3 0.42 ± 9% perf-profile.children.cycles-pp.update_cfs_group
0.64 ± 40% -0.3 0.32 ± 16% perf-profile.children.cycles-pp.reweight_entity
0.66 ± 22% -0.3 0.34 ± 7% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.65 ± 20% -0.3 0.36 ± 5% perf-profile.children.cycles-pp.update_rq_clock
0.44 ± 42% -0.3 0.16 ± 6% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.52 ± 31% -0.3 0.25 ± 5% perf-profile.children.cycles-pp.__might_resched
0.47 ± 18% -0.3 0.20 ± 36% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.55 ± 24% -0.3 0.29 ± 8% perf-profile.children.cycles-pp.___perf_sw_event
0.47 ± 29% -0.3 0.22 ± 5% perf-profile.children.cycles-pp.put_prev_task_fair
0.54 ± 5% -0.3 0.29 ± 20% perf-profile.children.cycles-pp.touch_atime
0.47 ± 10% -0.3 0.22 ± 12% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.45 ± 35% -0.2 0.22 ± 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.43 ± 23% -0.2 0.21 ± 9% perf-profile.children.cycles-pp.__might_fault
0.36 ± 37% -0.2 0.15 ± 6% perf-profile.children.cycles-pp.__might_sleep
0.37 ± 42% -0.2 0.15 ± 4% perf-profile.children.cycles-pp.mutex_unlock
0.35 ± 26% -0.2 0.15 ± 23% perf-profile.children.cycles-pp.__calc_delta
0.43 ± 6% -0.2 0.22 ± 23% perf-profile.children.cycles-pp.atime_needs_update
0.35 ± 45% -0.2 0.14 ± 6% perf-profile.children.cycles-pp.file_update_time
0.36 ± 27% -0.2 0.16 ± 8% perf-profile.children.cycles-pp.pick_next_entity
0.53 ± 16% -0.2 0.34 ± 6% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.39 ± 35% -0.2 0.19 ± 7% perf-profile.children.cycles-pp.__cond_resched
0.38 ± 37% -0.2 0.19 ± 6% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.28 ± 52% -0.2 0.09 ± 9% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.32 ± 30% -0.2 0.13 ± 12% perf-profile.children.cycles-pp.copyout
0.34 ± 15% -0.2 0.16 ± 24% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.34 ± 8% -0.2 0.18 ± 23% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.33 ± 33% -0.2 0.16 ± 15% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
0.31 ± 38% -0.2 0.15 ± 10% perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.28 ± 30% -0.2 0.12 ± 8% perf-profile.children.cycles-pp.select_idle_sibling
0.29 ± 31% -0.2 0.14 ± 5% perf-profile.children.cycles-pp.copy_user_short_string
0.30 ± 26% -0.2 0.16 ± 9% perf-profile.children.cycles-pp.pick_next_task_idle
0.27 ± 34% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.fput
0.25 ± 39% -0.1 0.11 ± 5% perf-profile.children.cycles-pp.cpuacct_charge
0.28 ± 30% -0.1 0.14 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.28 ± 9% -0.1 0.14 ± 17% perf-profile.children.cycles-pp.current_time
0.24 ± 32% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.22 ± 29% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.finish_wait
0.15 ± 17% -0.1 0.03 ±100% perf-profile.children.cycles-pp.llist_reverse_order
0.22 ± 27% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.23 ± 28% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.put_prev_entity
0.21 ± 26% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
0.20 ± 38% -0.1 0.09 ± 19% perf-profile.children.cycles-pp.rcu_nocb_flush_deferred_wakeup
0.25 ± 28% -0.1 0.14 ± 13% perf-profile.children.cycles-pp.__pthread_disable_asynccancel
0.18 ± 18% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.update_min_vruntime
0.22 ± 13% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.tick_nohz_tick_stopped
0.19 ± 36% -0.1 0.08 ± 11% perf-profile.children.cycles-pp.switch_fpu_return
0.19 ± 29% -0.1 0.09 ± 13% perf-profile.children.cycles-pp.ct_kernel_exit_state
0.16 ± 21% -0.1 0.06 ± 9% perf-profile.children.cycles-pp.__get_task_ioprio
0.20 ± 14% -0.1 0.10 ± 19% perf-profile.children.cycles-pp.check_preempt_curr
0.19 ± 31% -0.1 0.10 ± 16% perf-profile.children.cycles-pp.rcu_note_context_switch
0.19 ± 21% -0.1 0.09 ± 12% perf-profile.children.cycles-pp.anon_pipe_buf_release
0.20 ± 8% -0.1 0.10 ± 23% perf-profile.children.cycles-pp.get_cpu_device
0.16 ± 9% -0.1 0.07 ± 20% perf-profile.children.cycles-pp.rb_erase
0.14 ± 9% -0.1 0.05 ± 45% perf-profile.children.cycles-pp.clear_buddies
0.20 ± 25% -0.1 0.12 ± 10% perf-profile.children.cycles-pp.__update_idle_core
0.16 ± 25% -0.1 0.08 ± 16% perf-profile.children.cycles-pp.llist_add_batch
0.17 ± 22% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.testcase
0.18 ± 34% -0.1 0.09 ± 13% perf-profile.children.cycles-pp.cpuidle_not_available
0.18 ± 9% -0.1 0.10 ± 22% perf-profile.children.cycles-pp.__list_del_entry_valid
0.19 ± 20% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.__smp_call_single_queue
0.17 ± 39% -0.1 0.09 ± 17% perf-profile.children.cycles-pp.tick_check_broadcast_expired
0.13 ± 11% -0.1 0.05 ± 7% perf-profile.children.cycles-pp.put_prev_task_idle
0.18 ± 15% -0.1 0.10 ± 33% perf-profile.children.cycles-pp.nr_iowait_cpu
0.12 ± 44% -0.1 0.04 ± 73% perf-profile.children.cycles-pp.timekeeping_max_deferment
0.11 ± 9% -0.1 0.03 ±100% perf-profile.children.cycles-pp.schedule_debug
0.16 ± 29% -0.1 0.08 ± 7% perf-profile.children.cycles-pp.__pthread_enable_asynccancel
0.14 ± 14% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick
0.18 ± 26% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.nohz_run_idle_balance
0.10 ± 23% -0.1 0.03 ±100% perf-profile.children.cycles-pp.send_call_function_single_ipi
0.14 ± 24% -0.1 0.07 perf-profile.children.cycles-pp.ct_kernel_exit
0.13 ± 23% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.__list_add_valid
0.13 ± 38% -0.1 0.06 ± 14% perf-profile.children.cycles-pp.__x64_sys_write
0.08 ± 13% -0.1 0.03 ±100% perf-profile.children.cycles-pp.init_wait_entry
0.12 ± 30% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.__x64_sys_read
0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.child
0.10 ± 19% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.00 +0.1 0.07 ± 25% perf-profile.children.cycles-pp._find_first_zero_bit
0.28 ± 23% +0.6 0.85 ± 11% perf-profile.children.cycles-pp.switch_mm_irqs_off
40.66 ± 4% +2.8 43.50 ± 2% perf-profile.children.cycles-pp.__libc_read
34.36 ± 8% +6.8 41.18 ± 10% perf-profile.children.cycles-pp.start_secondary
35.35 ± 8% +7.0 42.35 ± 10% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
35.35 ± 8% +7.0 42.35 ± 10% perf-profile.children.cycles-pp.cpu_startup_entry
35.24 ± 8% +7.0 42.28 ± 10% perf-profile.children.cycles-pp.do_idle
27.86 ± 22% +10.0 37.84 perf-profile.children.cycles-pp.ksys_read
27.09 ± 23% +10.4 37.45 perf-profile.children.cycles-pp.vfs_read
25.98 ± 26% +10.9 36.90 perf-profile.children.cycles-pp.pipe_read
21.16 ± 38% +13.5 34.65 perf-profile.children.cycles-pp.schedule
10.22 ± 36% +16.9 27.08 ± 18% perf-profile.children.cycles-pp.schedule_idle
31.01 ± 38% +30.5 61.54 ± 8% perf-profile.children.cycles-pp.__schedule
0.00 +38.8 38.77 ± 47% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.66 ± 29% +39.5 40.18 ± 45% perf-profile.children.cycles-pp._raw_spin_lock
1.00 ± 18% +39.6 40.61 ± 44% perf-profile.children.cycles-pp.prepare_task_switch
5.48 ± 35% -2.8 2.67 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret
4.56 ± 33% -2.2 2.36 ± 5% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
3.36 ± 34% -1.6 1.72 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
2.10 ± 36% -1.1 0.99 ± 4% perf-profile.self.cycles-pp.__entry_text_start
1.75 ± 42% -1.1 0.65 ± 12% perf-profile.self.cycles-pp.__switch_to_asm
1.94 ± 28% -1.0 0.98 ± 5% perf-profile.self.cycles-pp.menu_select
1.42 ± 47% -0.9 0.48 ± 9% perf-profile.self.cycles-pp.__switch_to
2.45 ± 29% -0.8 1.63 perf-profile.self.cycles-pp.__schedule
0.97 ± 51% -0.7 0.26 ± 20% perf-profile.self.cycles-pp.os_xsave
1.21 ± 33% -0.6 0.62 ± 4% perf-profile.self.cycles-pp.enqueue_entity
1.14 ± 33% -0.6 0.57 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.11 ± 15% -0.6 0.55 ± 4% perf-profile.self.cycles-pp.update_load_avg
1.01 ± 31% -0.5 0.50 perf-profile.self.cycles-pp.read_tsc
1.15 ± 12% -0.5 0.65 ± 9% perf-profile.self.cycles-pp.native_sched_clock
0.95 ± 36% -0.5 0.45 ± 12% perf-profile.self.cycles-pp.ktime_get
0.81 ± 34% -0.4 0.36 ± 6% perf-profile.self.cycles-pp.pipe_read
0.73 ± 17% -0.4 0.29 ± 5% perf-profile.self.cycles-pp.dequeue_entity
0.84 ± 24% -0.4 0.40 ± 7% perf-profile.self.cycles-pp.pipe_write
0.80 ± 41% -0.4 0.37 ± 17% perf-profile.self.cycles-pp.do_idle
0.75 ± 22% -0.4 0.34 ± 5% perf-profile.self.cycles-pp.__update_load_avg_se
0.70 ± 27% -0.4 0.31 ± 6% perf-profile.self.cycles-pp.enqueue_task_fair
0.64 ± 13% -0.4 0.28 ± 5% perf-profile.self.cycles-pp.pick_next_task_fair
0.67 ± 34% -0.3 0.33 ± 4% perf-profile.self.cycles-pp.apparmor_file_permission
0.65 ± 18% -0.3 0.32 ± 5% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.66 ± 38% -0.3 0.33 ± 4% perf-profile.self.cycles-pp.__fget_files
0.63 ± 40% -0.3 0.31 ± 16% perf-profile.self.cycles-pp.reweight_entity
0.71 ± 27% -0.3 0.40 ± 8% perf-profile.self.cycles-pp.update_cfs_group
0.55 ± 20% -0.3 0.25 ± 6% perf-profile.self.cycles-pp.dequeue_task_fair
0.53 ± 25% -0.3 0.26 ± 2% perf-profile.self.cycles-pp.set_next_entity
0.44 ± 42% -0.3 0.16 ± 6% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.52 ± 31% -0.3 0.25 ± 6% perf-profile.self.cycles-pp.__might_resched
0.45 ± 32% -0.3 0.20 ± 11% perf-profile.self.cycles-pp.vfs_write
0.48 ± 37% -0.2 0.23 ± 5% perf-profile.self.cycles-pp.prepare_to_wait_event
0.47 ± 29% -0.2 0.23 ± 6% perf-profile.self.cycles-pp.mutex_lock
0.44 ± 35% -0.2 0.21 ± 3% perf-profile.self.cycles-pp.vfs_read
0.42 ± 18% -0.2 0.18 ± 39% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.47 ± 30% -0.2 0.23 ± 6% perf-profile.self.cycles-pp.newidle_balance
0.45 ± 29% -0.2 0.22 ± 4% perf-profile.self.cycles-pp.__libc_write
0.51 ± 9% -0.2 0.28 ± 25% perf-profile.self.cycles-pp.update_curr
0.48 ± 23% -0.2 0.25 ± 11% perf-profile.self.cycles-pp.___perf_sw_event
0.42 ± 34% -0.2 0.19 ± 5% perf-profile.self.cycles-pp.select_task_rq_fair
0.43 ± 33% -0.2 0.22 ± 9% perf-profile.self.cycles-pp.__libc_read
0.36 ± 43% -0.2 0.14 ± 6% perf-profile.self.cycles-pp.mutex_unlock
0.34 ± 24% -0.2 0.14 ± 23% perf-profile.self.cycles-pp.__calc_delta
0.28 ± 52% -0.2 0.09 ± 9% perf-profile.self.cycles-pp.__rdgsbase_inactive
0.30 ± 36% -0.2 0.12 ± 6% perf-profile.self.cycles-pp.__might_sleep
0.30 ± 30% -0.2 0.12 ± 7% perf-profile.self.cycles-pp.pick_next_entity
0.34 ± 37% -0.2 0.16 ± 8% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.28 ± 34% -0.2 0.11 ± 13% perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.30 ± 7% -0.2 0.14 ± 38% perf-profile.self.cycles-pp.try_to_wake_up
0.29 ± 46% -0.2 0.14 ± 4% perf-profile.self.cycles-pp.schedule_idle
0.28 ± 37% -0.2 0.13 ± 4% perf-profile.self.cycles-pp.__cond_resched
0.28 ± 30% -0.1 0.14 ± 8% perf-profile.self.cycles-pp.copy_user_short_string
0.24 ± 32% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.put_prev_task_fair
0.27 ± 7% -0.1 0.13 ± 14% perf-profile.self.cycles-pp.sched_ttwu_pending
0.29 ± 22% -0.1 0.15 ± 10% perf-profile.self.cycles-pp._copy_from_iter
0.30 ± 16% -0.1 0.16 ± 8% perf-profile.self.cycles-pp.do_syscall_64
0.22 ± 20% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.copy_page_from_iter
0.28 ± 29% -0.1 0.14 ± 8% perf-profile.self.cycles-pp.tick_nohz_next_event
0.24 ± 35% -0.1 0.11 ± 8% perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.24 ± 39% -0.1 0.11 ± 8% perf-profile.self.cycles-pp.cpuacct_charge
0.26 ± 30% -0.1 0.14 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.27 ± 23% -0.1 0.15 ± 11% perf-profile.self.cycles-pp.update_rq_clock
0.23 ± 17% -0.1 0.11 ± 8% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.40 ± 12% -0.1 0.28 ± 17% perf-profile.self.cycles-pp.tick_nohz_idle_exit
0.21 ± 36% -0.1 0.10 ± 9% perf-profile.self.cycles-pp.fput
0.15 ± 18% -0.1 0.04 ± 44% perf-profile.self.cycles-pp.__get_task_ioprio
0.17 ± 35% -0.1 0.06 ± 11% perf-profile.self.cycles-pp._copy_to_iter
0.20 ± 32% -0.1 0.10 ± 3% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.18 ± 7% -0.1 0.07 ± 5% perf-profile.self.cycles-pp.select_task_rq
0.20 ± 15% -0.1 0.09 ± 21% perf-profile.self.cycles-pp.current_time
0.18 ± 37% -0.1 0.08 ± 7% perf-profile.self.cycles-pp.switch_fpu_return
0.19 ± 35% -0.1 0.09 ± 13% perf-profile.self.cycles-pp.autoremove_wake_function
0.16 ± 23% -0.1 0.06 ± 15% perf-profile.self.cycles-pp.ttwu_do_activate
0.18 ± 39% -0.1 0.08 ± 20% perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
0.22 ± 24% -0.1 0.12 ± 9% perf-profile.self.cycles-pp.ksys_read
0.16 ± 15% -0.1 0.06 ± 9% perf-profile.self.cycles-pp.update_min_vruntime
0.16 ± 20% -0.1 0.07 ± 23% perf-profile.self.cycles-pp.check_preempt_curr
0.19 ± 29% -0.1 0.09 ± 7% perf-profile.self.cycles-pp.put_prev_entity
0.14 ± 9% -0.1 0.05 ± 46% perf-profile.self.cycles-pp.cpuidle_governor_latency_req
0.18 ± 31% -0.1 0.09 ± 13% perf-profile.self.cycles-pp.ct_kernel_exit_state
0.18 ± 21% -0.1 0.09 ± 7% perf-profile.self.cycles-pp.tick_nohz_tick_stopped
0.18 ± 23% -0.1 0.08 ± 8% perf-profile.self.cycles-pp.anon_pipe_buf_release
0.22 ± 26% -0.1 0.13 ± 12% perf-profile.self.cycles-pp.__pthread_disable_asynccancel
0.12 ± 8% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.put_prev_task_idle
0.18 ± 32% -0.1 0.09 ± 14% perf-profile.self.cycles-pp.rcu_note_context_switch
0.18 ± 9% -0.1 0.09 ± 26% perf-profile.self.cycles-pp.ksys_write
0.19 ± 8% -0.1 0.10 ± 23% perf-profile.self.cycles-pp.get_cpu_device
0.19 ± 32% -0.1 0.10 ± 13% perf-profile.self.cycles-pp.__fget_light
0.17 ± 33% -0.1 0.08 ± 10% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.15 ± 31% -0.1 0.07 ± 10% perf-profile.self.cycles-pp.select_idle_sibling
0.20 ± 2% -0.1 0.12 ± 27% perf-profile.self.cycles-pp.sched_clock_cpu
0.16 ± 27% -0.1 0.08 ± 15% perf-profile.self.cycles-pp.llist_add_batch
0.17 ± 19% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
0.15 ± 30% -0.1 0.06 ± 21% perf-profile.self.cycles-pp.get_next_timer_interrupt
0.16 ± 37% -0.1 0.08 ± 13% perf-profile.self.cycles-pp.tick_check_broadcast_expired
0.14 ± 11% -0.1 0.06 ± 47% perf-profile.self.cycles-pp.rb_erase
0.17 ± 26% -0.1 0.09 ± 13% perf-profile.self.cycles-pp.nohz_run_idle_balance
0.17 ± 20% -0.1 0.09 ± 13% perf-profile.self.cycles-pp.__update_idle_core
0.16 ± 22% -0.1 0.08 ± 8% perf-profile.self.cycles-pp.testcase
0.17 ± 16% -0.1 0.09 ± 31% perf-profile.self.cycles-pp.nr_iowait_cpu
0.12 ± 31% -0.1 0.04 ± 45% perf-profile.self.cycles-pp.__wake_up_common
0.18 ± 10% -0.1 0.10 ± 36% perf-profile.self.cycles-pp.atime_needs_update
0.14 ± 43% -0.1 0.06 ± 13% perf-profile.self.cycles-pp.file_update_time
0.12 ± 15% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.tick_nohz_idle_retain_tick
0.11 ± 9% -0.1 0.03 ±100% perf-profile.self.cycles-pp.schedule_debug
0.16 ± 16% -0.1 0.08 ± 12% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
0.16 ± 35% -0.1 0.09 ± 12% perf-profile.self.cycles-pp.cpuidle_not_available
0.14 ± 19% -0.1 0.07 ± 27% perf-profile.self.cycles-pp.ttwu_do_wakeup
0.15 ± 30% -0.1 0.08 ± 7% perf-profile.self.cycles-pp.__pthread_enable_asynccancel
0.10 ± 23% -0.1 0.03 ±100% perf-profile.self.cycles-pp.send_call_function_single_ipi
0.15 ± 8% -0.1 0.08 ± 22% perf-profile.self.cycles-pp.__list_del_entry_valid
0.14 ± 34% -0.1 0.08 ± 16% perf-profile.self.cycles-pp.cpu_startup_entry
0.12 ± 21% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.__list_add_valid
0.12 ± 38% -0.1 0.06 ± 14% perf-profile.self.cycles-pp.__x64_sys_write
0.08 ± 26% -0.1 0.03 ±100% perf-profile.self.cycles-pp.rb_insert_color
0.14 ± 8% -0.1 0.08 ± 8% perf-profile.self.cycles-pp.__might_fault
0.12 ± 28% -0.1 0.06 ± 6% perf-profile.self.cycles-pp.__x64_sys_read
0.10 ± 29% -0.1 0.05 ± 7% perf-profile.self.cycles-pp.ct_kernel_exit
0.10 ± 21% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.touch_atime
0.08 ± 10% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.child
0.00 +0.1 0.07 ± 25% perf-profile.self.cycles-pp._find_first_zero_bit
0.28 ± 23% +0.6 0.85 ± 11% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.65 ± 30% +0.8 1.41 ± 2% perf-profile.self.cycles-pp._raw_spin_lock
0.00 +38.8 38.76 ± 47% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath

***************************************************************************************************
lkp-skl-fpga01: 104 threads 2 sockets (Skylake) with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-11/performance/x86_64-rhel-8.3/thread/50%/debian-11.1-x86_64-20220510.cgz/lkp-skl-fpga01/context_switch1/will-it-scale

commit:
99babd04b2 ("selftests/rseq: Implement rseq numa node id field selftest")
af7f588d8f ("sched: Introduce per-memory-map concurrency ID")

99babd04b2505471 af7f588d8f7355bc4298dd1962d
---------------- ---------------------------
%stddev %change %stddev
\ | \
8413415 ± 2% -85.0% 1259200 will-it-scale.52.threads
22.36 -83.3% 3.74 will-it-scale.52.threads_idle
161796 ± 2% -85.0% 24214 will-it-scale.per_thread_ops
8413415 ± 2% -85.0% 1259200 will-it-scale.workload
10216 ± 2% -40.9% 6040 ± 2% uptime.idle
2.233e+09 -73.2% 5.989e+08 cpuidle..time
2.46e+09 ± 2% -89.5% 2.577e+08 ± 3% cpuidle..usage
329714 ± 11% +232.0% 1094566 ± 72% numa-numastat.node0.local_node
374570 ± 9% +205.7% 1145060 ± 69% numa-numastat.node0.numa_hit
4491204 ± 2% -53.1% 2106039 ± 36% numa-numastat.node1.local_node
4540680 ± 2% -52.7% 2149877 ± 35% numa-numastat.node1.numa_hit
22.83 -18.4 4.43 ± 2% mpstat.cpu.all.idle%
1.44 +0.2 1.68 mpstat.cpu.all.irq%
0.12 ± 2% -0.0 0.10 mpstat.cpu.all.soft%
59.94 +30.2 90.11 mpstat.cpu.all.sys%
15.67 -12.0 3.68 mpstat.cpu.all.usr%
22.83 -78.1% 5.00 vmstat.cpu.id
60.33 +50.6% 90.83 vmstat.cpu.sy
15.00 -80.0% 3.00 vmstat.cpu.us
14294773 ± 2% -53.0% 6714843 ± 9% vmstat.memory.cache
59.50 -12.3% 52.17 vmstat.procs.r
16450432 ± 2% -85.4% 2405190 vmstat.system.cs
276754 +52.0% 420615 ± 2% vmstat.system.in
1431257 ± 8% +1030.1% 16174000 ± 6% turbostat.C1
0.02 ± 17% +0.3 0.29 ± 7% turbostat.C1%
0.77 ± 6% +31.1% 1.01 ± 4% turbostat.CPU%c1
53.67 ± 2% -14.6% 45.83 ± 2% turbostat.CoreTmp
0.09 ± 4% -78.2% 0.02 turbostat.IPC
84431937 +52.1% 1.284e+08 ± 2% turbostat.IRQ
6.83 ± 3% -6.8 0.00 turbostat.PKG_%
2.458e+09 ± 2% -90.2% 2.41e+08 ± 3% turbostat.POLL
6.16 -5.5 0.69 ± 5% turbostat.POLL%
53.50 ± 2% -13.1% 46.50 ± 3% turbostat.PkgTmp
365.46 -17.4% 301.75 turbostat.PkgWatt
92948 +1761.8% 1730551 ± 4% meminfo.Active
92857 +1763.6% 1730458 ± 4% meminfo.Active(anon)
475097 -26.0% 351530 meminfo.AnonPages
14131358 ± 2% -53.3% 6596409 ± 9% meminfo.Cached
12167680 ± 2% -62.9% 4516022 ± 5% meminfo.Committed_AS
11089942 ± 2% -83.8% 1794273 ± 16% meminfo.Inactive
11089749 ± 2% -83.8% 1794082 ± 16% meminfo.Inactive(anon)
146766 -11.5% 129954 meminfo.KReclaimable
5019103 ± 3% -69.4% 1533664 ± 7% meminfo.Mapped
16059430 ± 2% -48.9% 8206619 ± 7% meminfo.Memused
14911 ± 2% -47.7% 7796 ± 2% meminfo.PageTables
146766 -11.5% 129954 meminfo.SReclaimable
10707913 ± 2% -70.4% 3172409 ± 6% meminfo.Shmem
16146332 ± 2% -25.6% 12012181 ± 4% meminfo.max_used_kB
382.33 ± 13% +30777.3% 118054 ± 98% numa-vmstat.node0.nr_active_anon
44471 ± 32% +315.1% 184580 ± 82% numa-vmstat.node0.nr_inactive_anon
44053 +211.7% 137333 ± 75% numa-vmstat.node0.nr_mapped
1598 ± 2% +15438.7% 248438 ±106% numa-vmstat.node0.nr_shmem
382.33 ± 13% +30777.3% 118054 ± 98% numa-vmstat.node0.nr_zone_active_anon
44471 ± 32% +315.1% 184580 ± 82% numa-vmstat.node0.nr_zone_inactive_anon
374056 ± 9% +206.0% 1144611 ± 69% numa-vmstat.node0.numa_hit
329200 ± 11% +232.4% 1094117 ± 72% numa-vmstat.node0.numa_local
22812 +1275.5% 313777 ± 42% numa-vmstat.node1.nr_active_anon
75507 ± 19% -55.4% 33711 ± 22% numa-vmstat.node1.nr_anon_pages
2702384 ± 2% -78.4% 583457 ± 37% numa-vmstat.node1.nr_file_pages
2727949 ± 2% -90.3% 264439 ± 35% numa-vmstat.node1.nr_inactive_anon
1210795 ± 3% -79.6% 247430 ± 32% numa-vmstat.node1.nr_mapped
3083 ± 3% -61.4% 1188 ± 24% numa-vmstat.node1.nr_page_table_pages
2675300 ± 2% -79.7% 544388 ± 41% numa-vmstat.node1.nr_shmem
15383 ± 3% -31.4% 10547 ± 10% numa-vmstat.node1.nr_slab_reclaimable
22812 +1275.5% 313776 ± 42% numa-vmstat.node1.nr_zone_active_anon
2727947 ± 2% -90.3% 264439 ± 35% numa-vmstat.node1.nr_zone_inactive_anon
4540436 ± 2% -52.7% 2149540 ± 35% numa-vmstat.node1.numa_hit
4490961 ± 3% -53.1% 2105702 ± 36% numa-vmstat.node1.numa_local
1577 ± 15% +29859.5% 472661 ± 98% numa-meminfo.node0.Active
1532 ± 13% +30744.5% 472639 ± 98% numa-meminfo.node0.Active(anon)
177949 ± 32% +315.0% 738476 ± 82% numa-meminfo.node0.Inactive
177854 ± 32% +315.2% 738443 ± 82% numa-meminfo.node0.Inactive(anon)
176243 +211.6% 549130 ± 76% numa-meminfo.node0.Mapped
6399 ± 2% +15435.9% 994269 ±107% numa-meminfo.node0.Shmem
94498 ± 6% +1225.9% 1252939 ± 42% numa-meminfo.node1.Active
94453 ± 6% +1226.4% 1252867 ± 42% numa-meminfo.node1.Active(anon)
155852 ± 34% -68.9% 48404 ± 41% numa-meminfo.node1.AnonHugePages
301933 ± 19% -55.4% 134805 ± 22% numa-meminfo.node1.AnonPages
376400 ± 15% -43.1% 214145 ± 19% numa-meminfo.node1.AnonPages.max
10818908 ± 2% -78.5% 2330479 ± 37% numa-meminfo.node1.FilePages
10917966 ± 2% -90.3% 1056761 ± 35% numa-meminfo.node1.Inactive
10917868 ± 2% -90.3% 1056603 ± 35% numa-meminfo.node1.Inactive(anon)
61560 ± 3% -31.5% 42189 ± 10% numa-meminfo.node1.KReclaimable
4856202 ± 3% -79.7% 987910 ± 32% numa-meminfo.node1.Mapped
11816207 ± 2% -74.2% 3046680 ± 28% numa-meminfo.node1.MemUsed
12364 ± 2% -61.6% 4742 ± 24% numa-meminfo.node1.PageTables
61560 ± 3% -31.5% 42189 ± 10% numa-meminfo.node1.SReclaimable
10710570 ± 2% -79.7% 2174201 ± 41% numa-meminfo.node1.Shmem
147398 -13.8% 127057 ± 6% numa-meminfo.node1.Slab
23418 +1746.6% 432430 ± 4% proc-vmstat.nr_active_anon
118759 -26.0% 87887 proc-vmstat.nr_anon_pages
4488037 +4.4% 4684192 proc-vmstat.nr_dirty_background_threshold
8987049 +4.4% 9379838 proc-vmstat.nr_dirty_threshold
3533819 ± 2% -53.3% 1648861 ± 9% proc-vmstat.nr_file_pages
45164003 +4.3% 47128423 proc-vmstat.nr_free_pages
2773197 ± 2% -83.8% 448466 ± 16% proc-vmstat.nr_inactive_anon
1256312 ± 3% -69.5% 383482 ± 6% proc-vmstat.nr_mapped
3731 ± 2% -48.1% 1936 ± 3% proc-vmstat.nr_page_table_pages
2677956 ± 2% -70.4% 792860 ± 6% proc-vmstat.nr_shmem
36695 -11.5% 32488 proc-vmstat.nr_slab_reclaimable
23418 +1746.6% 432430 ± 4% proc-vmstat.nr_zone_active_anon
2773197 ± 2% -83.8% 448466 ± 16% proc-vmstat.nr_zone_inactive_anon
624422 ± 17% -60.7% 245678 ± 15% proc-vmstat.numa_hint_faults
539128 ± 21% -73.5% 142632 ± 19% proc-vmstat.numa_hint_faults_local
4916730 ± 2% -33.0% 3296362 proc-vmstat.numa_hit
2036 ± 18% -83.4% 337.33 ± 17% proc-vmstat.numa_huge_pte_updates
4822398 ± 2% -33.6% 3202031 proc-vmstat.numa_local
1802325 ± 4% -69.5% 550361 ± 9% proc-vmstat.numa_pte_updates
83499 ±162% +1435.7% 1282307 ± 12% proc-vmstat.pgactivate
5055756 ± 2% -32.8% 3395392 proc-vmstat.pgalloc_normal
1857025 ± 5% -26.5% 1365139 ± 2% proc-vmstat.pgfault
742457 ± 9% +350.4% 3344225 proc-vmstat.pgfree
39235 +8.1% 42423 ± 6% proc-vmstat.pgreuse
308870 ± 19% -99.3% 2179 ±115% sched_debug.cfs_rq:/.MIN_vruntime.avg
4280681 -94.7% 226660 ±115% sched_debug.cfs_rq:/.MIN_vruntime.max
1073798 ± 9% -97.9% 22118 ±115% sched_debug.cfs_rq:/.MIN_vruntime.stddev
0.68 ± 2% +13.8% 0.77 sched_debug.cfs_rq:/.h_nr_running.avg
0.42 ± 2% -21.5% 0.33 ± 3% sched_debug.cfs_rq:/.h_nr_running.stddev
0.67 ± 45% +183.3% 1.89 ± 8% sched_debug.cfs_rq:/.load_avg.min
308870 ± 19% -99.3% 2179 ±115% sched_debug.cfs_rq:/.max_vruntime.avg
4280681 -94.7% 226660 ±115% sched_debug.cfs_rq:/.max_vruntime.max
1073798 ± 9% -97.9% 22118 ±115% sched_debug.cfs_rq:/.max_vruntime.stddev
4201184 -19.5% 3381693 sched_debug.cfs_rq:/.min_vruntime.avg
4494117 -13.4% 3892039 ± 2% sched_debug.cfs_rq:/.min_vruntime.max
4124946 -19.2% 3331483 sched_debug.cfs_rq:/.min_vruntime.min
0.68 +12.2% 0.76 sched_debug.cfs_rq:/.nr_running.avg
0.39 ± 3% -20.5% 0.31 ± 4% sched_debug.cfs_rq:/.nr_running.stddev
451.34 -10.3% 404.65 sched_debug.cfs_rq:/.runnable_avg.avg
1319 -11.5% 1167 ± 3% sched_debug.cfs_rq:/.runnable_avg.max
154.04 -13.1% 133.83 ± 5% sched_debug.cfs_rq:/.runnable_avg.stddev
447.52 -10.2% 401.95 sched_debug.cfs_rq:/.util_avg.avg
135.46 -11.1% 120.41 ± 5% sched_debug.cfs_rq:/.util_avg.stddev
230.77 ± 3% -23.8% 175.94 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.avg
940.25 ± 3% -12.9% 818.86 ± 8% sched_debug.cfs_rq:/.util_est_enqueued.max
225.80 -10.3% 202.54 sched_debug.cfs_rq:/.util_est_enqueued.stddev
148237 ± 2% +20.6% 178708 sched_debug.cpu.avg_idle.avg
4546 ± 13% +579.3% 30886 ± 2% sched_debug.cpu.avg_idle.min
5.75 +33.2% 7.66 sched_debug.cpu.clock.stddev
2742 ± 3% -10.0% 2468 ± 3% sched_debug.cpu.curr->pid.avg
6610 ± 23% -31.1% 4557 ± 23% sched_debug.cpu.max_idle_balance_cost.stddev
0.00 ± 5% +42.1% 0.00 ± 29% sched_debug.cpu.next_balance.stddev
0.52 ± 4% -11.0% 0.47 ± 3% sched_debug.cpu.nr_running.avg
23781980 ± 2% -85.3% 3489556 sched_debug.cpu.nr_switches.avg
26139865 ± 3% -70.4% 7732937 ± 11% sched_debug.cpu.nr_switches.max
20830845 ± 5% -85.8% 2960756 ± 6% sched_debug.cpu.nr_switches.min
1902082 ± 27% -74.7% 480695 ± 16% sched_debug.cpu.nr_switches.stddev
1.76 ± 17% +206.4% 5.38 ± 3% perf-stat.i.MPKI
1.904e+10 -73.1% 5.12e+09 perf-stat.i.branch-instructions
1.64 -0.5 1.15 perf-stat.i.branch-miss-rate%
3.056e+08 ± 2% -80.9% 58311136 ± 2% perf-stat.i.branch-misses
20.15 ± 13% +6.3 26.45 ± 2% perf-stat.i.cache-miss-rate%
16636101 ± 2% -85.4% 2432393 perf-stat.i.context-switches
3.12 ± 2% +296.6% 12.37 perf-stat.i.cpi
1861 -33.6% 1236 perf-stat.i.cpu-migrations
0.08 ± 5% -0.0 0.04 ± 3% perf-stat.i.dTLB-load-miss-rate%
20981469 ± 5% -85.8% 2981601 ± 5% perf-stat.i.dTLB-load-misses
2.678e+10 -74.9% 6.71e+09 perf-stat.i.dTLB-loads
0.00 ± 18% +0.0 0.00 ± 23% perf-stat.i.dTLB-store-miss-rate%
233103 ± 18% -67.8% 74993 ± 22% perf-stat.i.dTLB-store-misses
1.604e+10 -82.2% 2.863e+09 ± 2% perf-stat.i.dTLB-stores
11175689 ± 3% -82.1% 2004383 ± 24% perf-stat.i.iTLB-load-misses
23053320 ± 6% -81.3% 4309745 ± 13% perf-stat.i.iTLB-loads
9.415e+10 -74.4% 2.406e+10 perf-stat.i.instructions
8923 ± 2% +47.9% 13200 ± 22% perf-stat.i.instructions-per-iTLB-miss
0.33 -74.4% 0.08 perf-stat.i.ipc
596.29 -76.2% 142.20 perf-stat.i.metric.M/sec
5512 ± 6% -29.0% 3912 ± 3% perf-stat.i.minor-faults
56.02 +41.6 97.64 perf-stat.i.node-load-miss-rate%
1970708 ± 4% +314.9% 8176399 ± 4% perf-stat.i.node-load-misses
2639467 ± 5% -89.9% 266188 ± 29% perf-stat.i.node-loads
85.83 +13.0 98.81 perf-stat.i.node-store-miss-rate%
1884736 ± 3% +152.2% 4752825 ± 2% perf-stat.i.node-store-misses
357304 ± 3% -74.0% 93012 ± 6% perf-stat.i.node-stores
5512 ± 6% -29.0% 3913 ± 3% perf-stat.i.page-faults
1.74 ± 16% +210.5% 5.41 ± 3% perf-stat.overall.MPKI
1.60 -0.5 1.14 perf-stat.overall.branch-miss-rate%
19.62 ± 13% +6.0 25.62 ± 2% perf-stat.overall.cache-miss-rate%
3.06 +290.5% 11.94 perf-stat.overall.cpi
0.08 ± 5% -0.0 0.04 ± 4% perf-stat.overall.dTLB-load-miss-rate%
0.00 ± 18% +0.0 0.00 ± 22% perf-stat.overall.dTLB-store-miss-rate%
8429 ± 2% +50.4% 12675 ± 22% perf-stat.overall.instructions-per-iTLB-miss
0.33 -74.4% 0.08 perf-stat.overall.ipc
42.42 ± 2% +54.4 96.80 perf-stat.overall.node-load-miss-rate%
84.10 +14.0 98.08 perf-stat.overall.node-store-miss-rate%
3374614 +71.0% 5770768 perf-stat.overall.path-length
1.898e+10 -73.1% 5.105e+09 perf-stat.ps.branch-instructions
3.046e+08 ± 2% -80.9% 58149017 ± 2% perf-stat.ps.branch-misses
16582299 ± 2% -85.4% 2424954 perf-stat.ps.context-switches
1861 -33.8% 1231 perf-stat.ps.cpu-migrations
20900401 ± 5% -85.7% 2979011 ± 5% perf-stat.ps.dTLB-load-misses
2.669e+10 -74.9% 6.691e+09 perf-stat.ps.dTLB-loads
231656 ± 18% -67.6% 74972 ± 22% perf-stat.ps.dTLB-store-misses
1.599e+10 -82.1% 2.855e+09 ± 2% perf-stat.ps.dTLB-stores
11141515 ± 3% -82.0% 2001300 ± 24% perf-stat.ps.iTLB-load-misses
22975410 ± 6% -81.3% 4295158 ± 13% perf-stat.ps.iTLB-loads
9.385e+10 -74.4% 2.4e+10 perf-stat.ps.instructions
5511 ± 6% -29.2% 3901 ± 3% perf-stat.ps.minor-faults
1964291 ± 4% +315.4% 8159316 ± 4% perf-stat.ps.node-load-misses
2667620 ± 5% -90.0% 268054 ± 29% perf-stat.ps.node-loads
1878586 ± 3% +152.2% 4737805 ± 2% perf-stat.ps.node-store-misses
354848 ± 3% -73.9% 92786 ± 6% perf-stat.ps.node-stores
5512 ± 6% -29.2% 3902 ± 3% perf-stat.ps.page-faults
2.839e+13 -74.4% 7.266e+12 perf-stat.total.instructions
26.70 -21.8 4.92 ± 2% perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event
26.52 -21.6 4.88 ± 2% perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
24.81 -20.3 4.54 ± 2% perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
19.22 -15.2 4.02 ± 3% perf-profile.calltrace.cycles-pp.__libc_write
16.54 -13.1 3.48 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
15.23 -12.4 2.79 perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
15.46 -12.2 3.28 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
13.72 -11.5 2.27 ± 2% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch
13.79 -10.7 3.09 ± 7% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
13.58 -10.7 2.90 ± 3% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
11.44 ± 2% -10.3 1.17 perf-profile.calltrace.cycles-pp.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
12.38 -9.7 2.69 ± 2% perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
12.12 -9.5 2.58 ± 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.vfs_read
9.83 ± 2% -9.2 0.60 ± 76% perf-profile.calltrace.cycles-pp.unwind_next_frame.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain
11.66 -9.2 2.44 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
9.47 ± 3% -9.0 0.44 ± 44% perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.pipe_read.vfs_read
10.60 -8.2 2.37 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
10.31 -8.1 2.18 ± 2% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
10.31 -8.0 2.32 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write.ksys_write
10.23 -7.9 2.30 ± 3% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write
10.04 -7.8 2.26 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
9.04 -7.6 1.42 ± 2% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.pipe_read.vfs_read
9.54 -7.5 2.04 ± 2% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
8.81 -7.4 1.36 ± 2% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.pipe_read
9.34 -7.4 1.98 ± 2% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
7.56 -7.2 0.34 ± 70% perf-profile.calltrace.cycles-pp.unwind_next_frame.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
8.54 -7.2 1.32 ± 2% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
9.07 -7.1 1.93 ± 2% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
9.00 -7.1 1.92 ± 2% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
8.84 -7.0 1.80 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
8.29 -6.6 1.66 ± 2% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime
8.19 -6.5 1.72 ± 4% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
7.98 -6.3 1.69 ± 3% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common
7.59 -6.0 1.62 ± 3% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function
7.54 -5.9 1.60 ± 3% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up
7.28 -5.7 1.60 perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
7.08 -5.5 1.54 perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.do_idle
6.89 -5.5 1.34 ± 4% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template
6.85 -5.4 1.49 perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle
5.32 -4.0 1.27 ± 6% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
5.26 -4.0 1.26 ± 6% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
4.42 -3.7 0.74 ± 5% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
52.88 -3.6 49.32 perf-profile.calltrace.cycles-pp.__libc_read
2.85 ± 2% -2.3 0.57 ± 2% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
2.38 -2.1 0.26 ±100% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
0.75 ± 14% -0.5 0.26 ±100% perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
0.75 ± 14% -0.5 0.27 ±100% perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
0.75 ± 14% -0.5 0.27 ±100% perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.75 ± 14% -0.4 0.37 ± 71% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen
0.76 ± 14% -0.4 0.39 ± 71% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn
0.76 ± 14% -0.4 0.40 ± 71% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn.perf_mmap__push
0.73 ± 6% +0.7 1.44 ± 8% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.vfs_read
0.00 +0.8 0.76 ± 10% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance
0.60 ± 6% +0.8 1.41 ± 8% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.pipe_read
0.00 +0.9 0.89 ± 9% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
0.00 +0.9 0.93 ± 9% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
0.00 +1.1 1.09 ± 10% perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
45.74 +2.2 47.91 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
45.24 +2.6 47.83 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
44.49 +3.2 47.69 perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
41.00 +6.1 47.15 perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
40.89 +6.2 47.13 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
25.48 +19.7 45.16 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
25.46 +19.7 45.16 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
25.73 +19.8 45.52 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
25.30 +19.8 45.14 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
11.35 +31.1 42.42 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
11.18 +31.2 42.39 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.00 +39.4 39.41 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle
0.00 +39.6 39.63 perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle.do_idle
0.00 +39.7 39.73 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
0.00 +40.2 40.19 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule
0.00 +40.4 40.41 perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule.pipe_read
0.00 +40.5 40.50 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.pipe_read.vfs_read
33.56 -26.7 6.85 ± 2% perf-profile.children.cycles-pp.perf_tp_event
32.38 -25.7 6.64 ± 2% perf-profile.children.cycles-pp.__perf_event_overflow
32.13 -25.5 6.59 ± 2% perf-profile.children.cycles-pp.perf_event_output_forward
29.34 -23.8 5.53 ± 2% perf-profile.children.cycles-pp.perf_prepare_sample
27.00 -21.9 5.14 ± 2% perf-profile.children.cycles-pp.perf_callchain
26.82 -21.7 5.10 ± 2% perf-profile.children.cycles-pp.get_perf_callchain
25.26 -20.5 4.79 ± 2% perf-profile.children.cycles-pp.perf_callchain_kernel
20.14 -15.5 4.66 perf-profile.children.cycles-pp.__libc_write
18.18 -14.7 3.44 perf-profile.children.cycles-pp.unwind_next_frame
66.41 -13.7 52.69 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
16.42 -13.4 3.05 perf-profile.children.cycles-pp.perf_trace_sched_switch
64.30 -12.1 52.24 perf-profile.children.cycles-pp.do_syscall_64
14.42 -11.0 3.45 perf-profile.children.cycles-pp.ksys_write
13.86 -10.5 3.34 perf-profile.children.cycles-pp.vfs_write
12.41 -9.7 2.70 ± 3% perf-profile.children.cycles-pp.pipe_write
10.68 ± 2% -9.7 1.02 ± 4% perf-profile.children.cycles-pp.finish_task_switch
11.71 ± 2% -9.6 2.14 perf-profile.children.cycles-pp.__unwind_start
12.13 -9.6 2.58 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair
11.72 -9.3 2.45 ± 2% perf-profile.children.cycles-pp.dequeue_entity
10.62 -8.2 2.38 ± 3% perf-profile.children.cycles-pp.__wake_up_common_lock
10.68 -8.2 2.48 ± 2% perf-profile.children.cycles-pp.update_curr
10.32 -8.0 2.32 ± 3% perf-profile.children.cycles-pp.__wake_up_common
10.25 -7.9 2.31 ± 3% perf-profile.children.cycles-pp.autoremove_wake_function
10.06 -7.8 2.27 ± 3% perf-profile.children.cycles-pp.try_to_wake_up
9.79 -7.5 2.29 ± 2% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
8.99 -7.2 1.82 ± 6% perf-profile.children.cycles-pp.cpuidle_idle_call
8.19 -6.5 1.73 ± 4% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
5.37 -4.1 1.28 ± 6% perf-profile.children.cycles-pp.cpuidle_enter
5.33 -4.1 1.27 ± 6% perf-profile.children.cycles-pp.cpuidle_enter_state
4.52 -3.8 0.76 ± 4% perf-profile.children.cycles-pp.poll_idle
52.99 -3.6 49.35 perf-profile.children.cycles-pp.__libc_read
3.85 -3.1 0.75 ± 3% perf-profile.children.cycles-pp.unwind_get_return_address
3.84 -3.1 0.75 ± 3% perf-profile.children.cycles-pp.__orc_find
3.53 -2.8 0.71 ± 5% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
3.18 -2.5 0.68 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret
3.05 -2.4 0.69 ± 4% perf-profile.children.cycles-pp.sched_ttwu_pending
2.92 -2.4 0.56 ± 3% perf-profile.children.cycles-pp.__kernel_text_address
2.89 ± 3% -2.3 0.58 perf-profile.children.cycles-pp.flush_smp_call_function_queue
2.66 ± 2% -2.2 0.46 perf-profile.children.cycles-pp.stack_access_ok
2.39 -1.9 0.48 ± 2% perf-profile.children.cycles-pp.orc_find
2.20 ± 5% -1.9 0.34 ± 7% perf-profile.children.cycles-pp.menu_select
2.62 -1.8 0.78 ± 3% perf-profile.children.cycles-pp.ttwu_do_activate
2.19 -1.8 0.44 ± 4% perf-profile.children.cycles-pp.kernel_text_address
2.50 ± 2% -1.7 0.76 ± 3% perf-profile.children.cycles-pp.enqueue_task_fair
2.04 ± 2% -1.4 0.61 ± 2% perf-profile.children.cycles-pp.enqueue_entity
1.58 ± 3% -1.1 0.44 ± 3% perf-profile.children.cycles-pp.update_load_avg
1.42 -1.1 0.29 ± 4% perf-profile.children.cycles-pp.__entry_text_start
1.48 ± 2% -1.1 0.35 ± 7% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
1.30 ± 3% -1.1 0.21 ± 4% perf-profile.children.cycles-pp.__perf_event_header__init_id
1.10 ± 3% -0.9 0.22 ± 2% perf-profile.children.cycles-pp.sched_clock_cpu
0.99 -0.9 0.12 ± 4% perf-profile.children.cycles-pp.perf_output_begin_forward
1.02 ± 4% -0.9 0.15 ± 9% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.93 ± 2% -0.8 0.12 ± 6% perf-profile.children.cycles-pp.ktime_get
0.93 ± 3% -0.8 0.17 ± 5% perf-profile.children.cycles-pp.set_next_entity
1.02 ± 2% -0.8 0.26 ± 8% perf-profile.children.cycles-pp.sysvec_call_function_single
0.91 ± 2% -0.7 0.18 ± 4% perf-profile.children.cycles-pp.native_sched_clock
0.82 ± 6% -0.7 0.12 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.86 -0.7 0.18 ± 3% perf-profile.children.cycles-pp.perf_callchain_user
0.80 ± 4% -0.7 0.13 ± 7% perf-profile.children.cycles-pp.mutex_lock
0.78 ± 2% -0.7 0.13 ± 10% perf-profile.children.cycles-pp.__switch_to_asm
0.79 ± 4% -0.6 0.15 ± 7% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.74 ± 3% -0.6 0.10 ± 11% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.71 ± 6% -0.6 0.11 ± 5% perf-profile.children.cycles-pp.prepare_to_wait_event
0.70 ± 3% -0.6 0.11 ± 6% perf-profile.children.cycles-pp.__switch_to
1.44 ± 4% -0.6 0.86 perf-profile.children.cycles-pp.perf_output_sample
2.24 ± 4% -0.5 1.70 ± 7% perf-profile.children.cycles-pp.pick_next_task_fair
0.72 ± 2% -0.5 0.20 ± 9% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.58 ± 4% -0.5 0.08 ± 10% perf-profile.children.cycles-pp.read_tsc
0.62 ± 2% -0.5 0.14 ± 5% perf-profile.children.cycles-pp.core_kernel_text
0.59 ± 11% -0.5 0.11 ± 8% perf-profile.children.cycles-pp.copy_page_from_iter
1.17 ± 3% -0.5 0.69 perf-profile.children.cycles-pp.__output_copy
0.57 ± 4% -0.4 0.12 ± 4% perf-profile.children.cycles-pp.__fdget_pos
0.54 ± 4% -0.4 0.11 ± 4% perf-profile.children.cycles-pp.__fget_light
0.47 ± 8% -0.4 0.05 ± 8% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.94 ± 4% -0.4 0.53 perf-profile.children.cycles-pp.perf_output_copy
0.52 ± 6% -0.4 0.11 ± 10% perf-profile.children.cycles-pp.security_file_permission
0.47 ± 6% -0.4 0.08 ± 4% perf-profile.children.cycles-pp.update_cfs_group
0.47 ± 14% -0.4 0.08 ± 20% perf-profile.children.cycles-pp.copy_page_to_iter
0.45 ± 4% -0.4 0.06 ± 7% perf-profile.children.cycles-pp.pick_next_task_idle
0.52 ± 3% -0.4 0.15 ± 2% perf-profile.children.cycles-pp.select_task_rq
0.47 -0.4 0.10 ± 3% perf-profile.children.cycles-pp.__get_user_nocheck_8
0.46 ± 10% -0.4 0.09 ± 9% perf-profile.children.cycles-pp._copy_from_iter
0.44 -0.4 0.08 ± 8% perf-profile.children.cycles-pp.ftrace_graph_ret_addr
0.45 ± 4% -0.4 0.10 ± 9% perf-profile.children.cycles-pp.apparmor_file_permission
0.45 ± 3% -0.3 0.11 ± 4% perf-profile.children.cycles-pp.__update_load_avg_se
0.41 ± 12% -0.3 0.07 ± 19% perf-profile.children.cycles-pp._copy_to_iter
0.41 ± 5% -0.3 0.08 ± 11% perf-profile.children.cycles-pp.tick_nohz_next_event
0.81 ± 3% -0.3 0.48 perf-profile.children.cycles-pp.memcpy_erms
0.41 ± 2% -0.3 0.08 ± 5% perf-profile.children.cycles-pp.__fget_files
0.38 ± 3% -0.3 0.06 ± 9% perf-profile.children.cycles-pp.__update_idle_core
0.38 ± 18% -0.3 0.07 ± 18% perf-profile.children.cycles-pp.touch_atime
0.80 ± 17% -0.3 0.50 ± 16% perf-profile.children.cycles-pp.generic_perform_write
0.38 ± 2% -0.3 0.08 ± 6% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.80 ± 17% -0.3 0.51 ± 16% perf-profile.children.cycles-pp.__generic_file_write_iter
0.42 -0.3 0.12 ± 3% perf-profile.children.cycles-pp.update_rq_clock
0.81 ± 17% -0.3 0.52 ± 16% perf-profile.children.cycles-pp.generic_file_write_iter
0.44 ± 9% -0.3 0.16 ± 10% perf-profile.children.cycles-pp.copyin
0.41 ± 3% -0.3 0.13 ± 5% perf-profile.children.cycles-pp.select_task_rq_fair
0.35 ± 6% -0.3 0.07 ± 6% perf-profile.children.cycles-pp.reweight_entity
0.30 ± 3% -0.3 0.02 ± 99% perf-profile.children.cycles-pp.put_prev_task_fair
0.30 ± 22% -0.3 0.03 ±100% perf-profile.children.cycles-pp.atime_needs_update
0.35 ± 7% -0.3 0.08 ± 24% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
0.32 ± 5% -0.3 0.06 ± 11% perf-profile.children.cycles-pp.os_xsave
0.32 ± 24% -0.3 0.05 ± 47% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.33 ± 3% -0.3 0.07 ± 7% perf-profile.children.cycles-pp.___perf_sw_event
0.29 ± 8% -0.3 0.02 ± 99% perf-profile.children.cycles-pp.perf_trace_buf_update
0.32 ± 7% -0.3 0.06 ± 8% perf-profile.children.cycles-pp.get_stack_info
0.40 ± 11% -0.3 0.15 ± 10% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.28 ± 9% -0.3 0.03 ±100% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.32 ± 3% -0.2 0.06 ± 7% perf-profile.children.cycles-pp.__might_resched
0.26 ± 2% -0.2 0.02 ± 99% perf-profile.children.cycles-pp.perf_instruction_pointer
0.42 ± 3% -0.2 0.19 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.27 ± 2% -0.2 0.05 ± 7% perf-profile.children.cycles-pp.__might_fault
0.27 ± 4% -0.2 0.06 ± 9% perf-profile.children.cycles-pp.kvm_guest_state
0.26 ± 3% -0.2 0.06 ± 8% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.25 ± 5% -0.2 0.05 ± 7% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.22 ± 7% -0.2 0.02 ± 99% perf-profile.children.cycles-pp.get_callchain_entry
0.22 ± 3% -0.2 0.02 ± 99% perf-profile.children.cycles-pp.__cond_resched
0.68 ± 3% -0.2 0.49 ± 4% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.81 ± 16% -0.2 0.63 ± 17% perf-profile.children.cycles-pp.writen
0.81 ± 16% -0.2 0.63 ± 17% perf-profile.children.cycles-pp.record__pushfn
0.60 ± 4% -0.2 0.42 ± 4% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.24 ± 5% -0.2 0.07 ± 7% perf-profile.children.cycles-pp.perf_trace_buf_alloc
0.30 ± 17% -0.2 0.14 ± 13% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
0.19 ± 7% -0.2 0.02 ± 99% perf-profile.children.cycles-pp.file_update_time
0.34 ± 18% -0.1 0.20 ± 17% perf-profile.children.cycles-pp.shmem_write_begin
0.33 ± 18% -0.1 0.19 ± 18% perf-profile.children.cycles-pp.shmem_get_folio_gfp
0.47 ± 4% -0.1 0.33 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.46 ± 4% -0.1 0.33 ± 4% perf-profile.children.cycles-pp.hrtimer_interrupt
0.14 ± 6% -0.1 0.05 perf-profile.children.cycles-pp.native_irq_return_iret
0.15 ± 5% -0.1 0.06 ± 7% perf-profile.children.cycles-pp.select_idle_sibling
0.17 ± 4% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.__irq_exit_rcu
0.12 ± 12% -0.1 0.04 ± 44% perf-profile.children.cycles-pp.place_entity
0.19 ± 12% -0.1 0.12 ± 29% perf-profile.children.cycles-pp.process_simple
0.18 ± 11% -0.1 0.11 ± 27% perf-profile.children.cycles-pp.ordered_events__queue
0.12 ± 16% -0.1 0.06 ± 45% perf-profile.children.cycles-pp.shmem_add_to_page_cache
0.13 ± 20% -0.1 0.07 ± 15% perf-profile.children.cycles-pp.shmem_alloc_and_acct_folio
0.31 ± 3% -0.1 0.26 ± 5% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.09 ± 21% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.vma_alloc_folio
0.10 ± 19% -0.1 0.05 ± 47% perf-profile.children.cycles-pp.shmem_alloc_folio
0.26 ± 4% -0.0 0.21 ± 5% perf-profile.children.cycles-pp.tick_sched_timer
0.10 ± 5% -0.0 0.05 ± 7% perf-profile.children.cycles-pp.__do_softirq
0.23 ± 5% -0.0 0.20 ± 4% perf-profile.children.cycles-pp.update_process_times
0.23 ± 5% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.tick_sched_handle
0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp._find_next_and_bit
0.00 +0.1 0.08 ± 25% perf-profile.children.cycles-pp.perf_session__deliver_event
0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.__update_blocked_fair
0.00 +0.1 0.10 ± 8% perf-profile.children.cycles-pp.idle_cpu
0.00 +0.1 0.11 ± 22% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.00 +0.2 0.18 ± 2% perf-profile.children.cycles-pp.update_blocked_averages
0.09 ± 14% +0.3 0.40 ± 13% perf-profile.children.cycles-pp.intel_idle_irq
0.09 ± 11% +0.3 0.40 ± 13% perf-profile.children.cycles-pp.mwait_idle_with_hints
0.00 +0.8 0.78 ± 10% perf-profile.children.cycles-pp.update_sg_lb_stats
0.61 ± 5% +0.8 1.41 ± 8% perf-profile.children.cycles-pp.newidle_balance
0.00 +0.9 0.90 ± 9% perf-profile.children.cycles-pp.update_sd_lb_stats
0.00 +0.9 0.94 ± 9% perf-profile.children.cycles-pp.find_busiest_group
0.00 +1.1 1.11 ± 10% perf-profile.children.cycles-pp.load_balance
45.78 +2.2 47.94 perf-profile.children.cycles-pp.ksys_read
45.26 +2.6 47.85 perf-profile.children.cycles-pp.vfs_read
44.56 +3.1 47.71 perf-profile.children.cycles-pp.pipe_read
41.02 +6.2 47.19 perf-profile.children.cycles-pp.schedule
25.48 +19.7 45.16 perf-profile.children.cycles-pp.start_secondary
25.73 +19.8 45.52 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
25.73 +19.8 45.52 perf-profile.children.cycles-pp.cpu_startup_entry
25.66 +19.9 45.51 perf-profile.children.cycles-pp.do_idle
11.49 +31.3 42.77 perf-profile.children.cycles-pp.schedule_idle
52.29 +37.6 89.92 perf-profile.children.cycles-pp.__schedule
0.63 ± 2% +79.9 80.56 perf-profile.children.cycles-pp.prepare_task_switch
0.00 +80.0 80.01 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.30 +80.3 80.56 perf-profile.children.cycles-pp._raw_spin_lock
9.60 ± 3% -9.2 0.40 ± 3% perf-profile.self.cycles-pp.finish_task_switch
8.30 ± 2% -7.8 0.46 ± 3% perf-profile.self.cycles-pp.__schedule
9.18 ± 2% -7.5 1.72 perf-profile.self.cycles-pp.unwind_next_frame
4.01 -3.3 0.68 ± 4% perf-profile.self.cycles-pp.poll_idle
3.81 -3.1 0.75 ± 3% perf-profile.self.cycles-pp.__orc_find
3.14 ± 2% -2.5 0.62 ± 3% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
3.18 -2.5 0.68 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret
2.39 -2.0 0.41 ± 2% perf-profile.self.cycles-pp.stack_access_ok
2.24 -1.8 0.45 ± 2% perf-profile.self.cycles-pp.orc_find
2.16 -1.7 0.45 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
1.91 -1.5 0.37 ± 3% perf-profile.self.cycles-pp.perf_callchain_kernel
1.30 -1.1 0.23 perf-profile.self.cycles-pp.__unwind_start
1.23 -1.0 0.26 ± 3% perf-profile.self.cycles-pp.__entry_text_start
1.14 -0.9 0.22 ± 3% perf-profile.self.cycles-pp.kernel_text_address
0.99 ± 5% -0.8 0.14 ± 9% perf-profile.self.cycles-pp.menu_select
0.96 ± 2% -0.8 0.11 ± 4% perf-profile.self.cycles-pp.perf_output_begin_forward
0.89 -0.7 0.18 ± 5% perf-profile.self.cycles-pp.unwind_get_return_address
0.89 ± 2% -0.7 0.18 ± 2% perf-profile.self.cycles-pp.native_sched_clock
0.81 ± 6% -0.7 0.12 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.85 ± 5% -0.7 0.16 ± 4% perf-profile.self.cycles-pp.perf_tp_event
0.78 ± 2% -0.6 0.13 ± 10% perf-profile.self.cycles-pp.__switch_to_asm
0.73 ± 6% -0.6 0.10 ± 14% perf-profile.self.cycles-pp.cpuidle_idle_call
0.73 ± 2% -0.6 0.12 ± 4% perf-profile.self.cycles-pp.__kernel_text_address
0.72 ± 4% -0.6 0.16 ± 6% perf-profile.self.cycles-pp.enqueue_entity
0.67 ± 7% -0.6 0.11 ± 9% perf-profile.self.cycles-pp.perf_prepare_sample
0.66 ± 4% -0.6 0.10 ± 7% perf-profile.self.cycles-pp.__switch_to
0.70 ± 4% -0.5 0.18 ± 4% perf-profile.self.cycles-pp.update_load_avg
0.55 ± 3% -0.5 0.08 ± 8% perf-profile.self.cycles-pp.read_tsc
0.58 ± 6% -0.5 0.11 ± 6% perf-profile.self.cycles-pp.newidle_balance
0.54 ± 4% -0.4 0.11 ± 4% perf-profile.self.cycles-pp.pipe_write
0.52 ± 7% -0.4 0.10 ± 5% perf-profile.self.cycles-pp.pipe_read
0.45 ± 11% -0.4 0.05 ± 8% perf-profile.self.cycles-pp.__perf_event_header__init_id
0.46 -0.4 0.09 ± 5% perf-profile.self.cycles-pp.get_perf_callchain
0.45 ± 5% -0.4 0.08 perf-profile.self.cycles-pp.update_cfs_group
0.46 ± 2% -0.4 0.10 ± 3% perf-profile.self.cycles-pp.__get_user_nocheck_8
0.42 ± 7% -0.4 0.07 ± 7% perf-profile.self.cycles-pp.dequeue_entity
0.40 ± 7% -0.3 0.06 ± 8% perf-profile.self.cycles-pp.mutex_lock
0.37 ± 4% -0.3 0.02 ± 99% perf-profile.self.cycles-pp.ktime_get
0.44 ± 5% -0.3 0.09 ± 5% perf-profile.self.cycles-pp.update_curr
0.40 ± 11% -0.3 0.06 ± 7% perf-profile.self.cycles-pp.pick_next_task_fair
0.44 ± 4% -0.3 0.10 ± 6% perf-profile.self.cycles-pp.__update_load_avg_se
0.79 ± 3% -0.3 0.47 ± 2% perf-profile.self.cycles-pp.memcpy_erms
0.40 ± 3% -0.3 0.08 ± 5% perf-profile.self.cycles-pp.__fget_files
0.40 ± 3% -0.3 0.09 ± 10% perf-profile.self.cycles-pp.apparmor_file_permission
0.37 ± 4% -0.3 0.06 ± 11% perf-profile.self.cycles-pp.do_idle
0.37 ± 3% -0.3 0.08 ± 6% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.43 ± 4% -0.3 0.14 ± 8% perf-profile.self.cycles-pp.enqueue_task_fair
0.39 ± 4% -0.3 0.10 perf-profile.self.cycles-pp.dequeue_task_fair
0.35 ± 2% -0.3 0.06 ± 7% perf-profile.self.cycles-pp.ftrace_graph_ret_addr
0.34 ± 6% -0.3 0.06 ± 6% perf-profile.self.cycles-pp.reweight_entity
0.34 ± 2% -0.3 0.07 ± 9% perf-profile.self.cycles-pp.perf_event_output_forward
0.32 ± 6% -0.3 0.06 ± 11% perf-profile.self.cycles-pp.os_xsave
0.32 ± 2% -0.3 0.06 perf-profile.self.cycles-pp.perf_callchain_user
0.31 ± 6% -0.3 0.05 ± 7% perf-profile.self.cycles-pp.vfs_write
0.33 ± 8% -0.3 0.07 ± 9% perf-profile.self.cycles-pp.core_kernel_text
0.31 ± 4% -0.3 0.06 ± 6% perf-profile.self.cycles-pp.set_next_entity
0.31 ± 6% -0.3 0.06 perf-profile.self.cycles-pp.prepare_to_wait_event
0.28 ± 26% -0.2 0.03 ±100% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.31 ± 2% -0.2 0.06 ± 7% perf-profile.self.cycles-pp.__might_resched
0.38 ± 11% -0.2 0.15 ± 10% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.29 ± 2% -0.2 0.06 perf-profile.self.cycles-pp.___perf_sw_event
0.28 ± 6% -0.2 0.05 ± 7% perf-profile.self.cycles-pp.vfs_read
0.27 ± 2% -0.2 0.05 ± 7% perf-profile.self.cycles-pp.__libc_write
0.26 ± 6% -0.2 0.04 ± 45% perf-profile.self.cycles-pp.__libc_read
0.38 ± 2% -0.2 0.16 ± 3% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.24 ± 3% -0.2 0.04 ± 45% perf-profile.self.cycles-pp.kvm_guest_state
0.24 ± 6% -0.2 0.04 ± 44% perf-profile.self.cycles-pp.try_to_wake_up
0.22 ± 4% -0.2 0.02 ± 99% perf-profile.self.cycles-pp.do_syscall_64
0.22 ± 5% -0.2 0.03 ± 70% perf-profile.self.cycles-pp.select_task_rq_fair
0.24 ± 9% -0.2 0.06 ± 9% perf-profile.self.cycles-pp.tick_nohz_idle_exit
0.29 ± 5% -0.2 0.11 ± 7% perf-profile.self.cycles-pp.prepare_task_switch
0.27 ± 5% -0.2 0.09 ± 9% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.33 ± 6% -0.1 0.20 ± 2% perf-profile.self.cycles-pp.__output_copy
0.18 ± 4% -0.1 0.05 ± 7% perf-profile.self.cycles-pp.update_rq_clock
0.14 ± 6% -0.1 0.05 perf-profile.self.cycles-pp.native_irq_return_iret
0.23 ± 9% -0.1 0.14 ± 5% perf-profile.self.cycles-pp.perf_output_sample
0.17 ± 10% -0.1 0.10 ± 27% perf-profile.self.cycles-pp.queue_event
0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.load_balance
0.00 +0.1 0.10 ± 8% perf-profile.self.cycles-pp.idle_cpu
0.00 +0.1 0.10 ± 9% perf-profile.self.cycles-pp.update_sd_lb_stats
0.30 +0.3 0.55 ± 2% perf-profile.self.cycles-pp._raw_spin_lock
0.09 ± 14% +0.3 0.40 ± 13% perf-profile.self.cycles-pp.mwait_idle_with_hints
0.00 +0.6 0.64 ± 10% perf-profile.self.cycles-pp.update_sg_lb_stats
0.00 +80.0 80.01 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath

***************************************************************************************************
lkp-csl-2sp3: 96 threads 2 sockets Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz (Cascade Lake) with 128G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-11/performance/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/300s/2T/lkp-csl-2sp3/shm-xread-seq-mt/vm-scalability

commit:
99babd04b2 ("selftests/rseq: Implement rseq numa node id field selftest")
af7f588d8f ("sched: Introduce per-memory-map concurrency ID")

99babd04b2505471 af7f588d8f7355bc4298dd1962d
---------------- ---------------------------
%stddev %change %stddev
\ | \
210861 -9.0% 191845 vm-scalability.median
20242720 -9.0% 18417158 vm-scalability.throughput
55019520 -10.2% 49390360 vm-scalability.time.maximum_resident_set_size
1.9e+08 -12.8% 1.657e+08 vm-scalability.time.minor_page_faults
2259 +7.5% 2428 vm-scalability.time.percent_of_cpu_this_job_got
4300 +19.2% 5126 vm-scalability.time.system_time
2642 ± 2% -11.8% 2329 vm-scalability.time.user_time
2.705e+08 -4.5% 2.583e+08 vm-scalability.time.voluntary_context_switches
6.075e+09 -8.9% 5.535e+09 vm-scalability.workload
14.58 +3.3 17.91 mpstat.cpu.all.sys%
11802313 ± 2% -15.5% 9968423 ± 8% numa-numastat.node1.local_node
11868695 ± 2% -15.5% 10031212 ± 7% numa-numastat.node1.numa_hit
31341412 -8.8% 28589584 vmstat.memory.cache
1739822 -4.6% 1659696 vmstat.system.cs
198273 +12.4% 222847 vmstat.system.in
29152003 -9.6% 26360019 meminfo.Committed_AS
10192230 ± 8% -23.6% 7787187 ± 10% meminfo.DirectMap2M
27542178 -9.8% 24843166 meminfo.Inactive
27541034 -9.8% 24842025 meminfo.Inactive(anon)
26690232 -9.7% 24102542 meminfo.Mapped
57049 -8.9% 51976 meminfo.PageTables
27457190 -9.9% 24751049 meminfo.Shmem
910.33 +15.6% 1052 turbostat.Avg_MHz
29.46 +4.6 34.05 turbostat.Busy%
1.992e+08 ± 2% -11.3% 1.767e+08 turbostat.C1
44.78 -5.7 39.12 turbostat.C1%
56047 ± 92% +1006.4% 620087 ± 67% turbostat.C6
0.24 -20.8% 0.19 turbostat.IPC
62645660 +12.3% 70378539 turbostat.IRQ
225.94 +2.1% 230.71 turbostat.PkgWatt
39.08 +1.0% 39.47 turbostat.RAMWatt
248641 ± 19% +35.2% 336049 ± 14% numa-meminfo.node0.AnonPages.max
17517837 -19.5% 14100517 ± 16% numa-meminfo.node0.FilePages
114473 ± 2% -35.4% 73942 ± 36% numa-meminfo.node0.KReclaimable
18308067 -18.7% 14882836 ± 15% numa-meminfo.node0.MemUsed
114473 ± 2% -35.4% 73942 ± 36% numa-meminfo.node0.SReclaimable
224848 ± 2% -17.6% 185243 ± 16% numa-meminfo.node0.Slab
3738197 -66.8% 1241460 ±140% numa-meminfo.node0.Unevictable
13816698 ± 3% -13.3% 11974436 ± 9% numa-meminfo.node1.Inactive
13816181 ± 3% -13.3% 11973743 ± 9% numa-meminfo.node1.Inactive(anon)
13327964 ± 3% -12.8% 11622819 ± 8% numa-meminfo.node1.Mapped
28651 ± 2% -13.3% 24845 ± 9% numa-meminfo.node1.PageTables
13689339 ± 2% -13.2% 11880742 ± 8% numa-meminfo.node1.Shmem
26826 ±122% +9309.0% 2524087 ± 69% numa-meminfo.node1.Unevictable
4378477 -19.5% 3524382 ± 16% numa-vmstat.node0.nr_file_pages
28614 ± 2% -35.4% 18480 ± 36% numa-vmstat.node0.nr_slab_reclaimable
934549 -66.8% 310364 ±140% numa-vmstat.node0.nr_unevictable
934549 -66.8% 310364 ±140% numa-vmstat.node0.nr_zone_unevictable
3452954 ± 3% -13.3% 2992588 ± 9% numa-vmstat.node1.nr_inactive_anon
3330866 ± 3% -12.8% 2904357 ± 8% numa-vmstat.node1.nr_mapped
7166 ± 2% -13.3% 6214 ± 9% numa-vmstat.node1.nr_page_table_pages
3421209 ± 2% -13.2% 2969320 ± 8% numa-vmstat.node1.nr_shmem
6706 ±122% +9309.8% 631021 ± 69% numa-vmstat.node1.nr_unevictable
3452953 ± 3% -13.3% 2992587 ± 9% numa-vmstat.node1.nr_zone_inactive_anon
6706 ±122% +9309.8% 631021 ± 69% numa-vmstat.node1.nr_zone_unevictable
11868617 ± 2% -15.5% 10031142 ± 7% numa-vmstat.node1.numa_hit
11802234 ± 2% -15.5% 9968353 ± 8% numa-vmstat.node1.numa_local
61039 -4.3% 58390 ± 2% proc-vmstat.nr_active_anon
2454405 +2.8% 2522406 proc-vmstat.nr_dirty_background_threshold
4914813 +2.8% 5050981 proc-vmstat.nr_dirty_threshold
7805675 -8.7% 7129014 proc-vmstat.nr_file_pages
24733767 +2.8% 25414774 proc-vmstat.nr_free_pages
6885305 -9.8% 6210359 proc-vmstat.nr_inactive_anon
6672510 -9.7% 6025638 proc-vmstat.nr_mapped
14262 -8.9% 12994 proc-vmstat.nr_page_table_pages
6864084 -9.9% 6187292 proc-vmstat.nr_shmem
44375 -3.4% 42867 proc-vmstat.nr_slab_reclaimable
61039 -4.3% 58390 ± 2% proc-vmstat.nr_zone_active_anon
6885305 -9.8% 6210359 proc-vmstat.nr_zone_inactive_anon
24681264 -12.8% 21531077 proc-vmstat.numa_hit
24592528 -12.8% 21443489 proc-vmstat.numa_local
24808648 -12.6% 21680796 proc-vmstat.pgalloc_normal
1.912e+08 -12.7% 1.669e+08 proc-vmstat.pgfault
24814673 -12.6% 21686002 proc-vmstat.pgfree
6448 ± 45% +115.5% 13893 ± 22% sched_debug.cfs_rq:/.MIN_vruntime.avg
531865 ± 56% +59.3% 847386 ± 2% sched_debug.cfs_rq:/.MIN_vruntime.max
57529 ± 51% +81.1% 104202 ± 10% sched_debug.cfs_rq:/.MIN_vruntime.stddev
0.22 ± 6% +11.1% 0.25 ± 9% sched_debug.cfs_rq:/.h_nr_running.avg
10123 ± 6% +15.6% 11702 ± 7% sched_debug.cfs_rq:/.load.avg
365.17 ± 36% -41.9% 212.22 sched_debug.cfs_rq:/.load_avg.max
6448 ± 45% +115.5% 13893 ± 22% sched_debug.cfs_rq:/.max_vruntime.avg
531865 ± 56% +59.3% 847386 ± 2% sched_debug.cfs_rq:/.max_vruntime.max
57529 ± 51% +81.1% 104202 ± 10% sched_debug.cfs_rq:/.max_vruntime.stddev
782070 +13.6% 888573 sched_debug.cfs_rq:/.min_vruntime.avg
821150 +13.6% 932625 sched_debug.cfs_rq:/.min_vruntime.max
753697 +12.3% 846465 sched_debug.cfs_rq:/.min_vruntime.min
0.22 ± 5% +11.4% 0.25 ± 9% sched_debug.cfs_rq:/.nr_running.avg
-5981 -259.2% 9521 ± 80% sched_debug.cfs_rq:/.spread0.avg
1179 +11.2% 1311 ± 3% sched_debug.cpu.curr->pid.avg
0.22 +13.5% 0.25 ± 2% sched_debug.cpu.nr_running.avg
18886 ± 16% +41.6% 26735 ± 12% sched_debug.cpu.nr_switches.stddev
4.18 +8.6% 4.54 perf-stat.i.MPKI
2.29e+10 -8.2% 2.101e+10 perf-stat.i.branch-instructions
0.19 +0.0 0.22 ± 2% perf-stat.i.branch-miss-rate%
40178596 +6.8% 42925617 perf-stat.i.branch-misses
21.94 +1.8 23.76 perf-stat.i.cache-miss-rate%
66452612 +8.9% 72391872 perf-stat.i.cache-misses
1760358 -4.6% 1678776 perf-stat.i.context-switches
1.20 +24.5% 1.49 perf-stat.i.cpi
8.839e+10 +15.2% 1.018e+11 perf-stat.i.cpu-cycles
6205 +7.3% 6660 perf-stat.i.cpu-migrations
1330 +5.9% 1408 perf-stat.i.cycles-between-cache-misses
4572848 -10.1% 4112764 perf-stat.i.dTLB-load-misses
2.122e+10 -7.9% 1.954e+10 perf-stat.i.dTLB-loads
0.02 -0.0 0.02 perf-stat.i.dTLB-store-miss-rate%
702307 -13.0% 611099 perf-stat.i.dTLB-store-misses
3.917e+09 -7.5% 3.622e+09 perf-stat.i.dTLB-stores
4714448 -5.7% 4444422 perf-stat.i.iTLB-load-misses
10445194 -7.0% 9716326 perf-stat.i.iTLB-loads
7.353e+10 -7.6% 6.797e+10 perf-stat.i.instructions
0.84 -19.4% 0.67 perf-stat.i.ipc
0.92 +15.2% 1.06 perf-stat.i.metric.GHz
473.64 +4.2% 493.63 perf-stat.i.metric.K/sec
503.52 -8.0% 463.30 perf-stat.i.metric.M/sec
620975 -12.9% 541054 perf-stat.i.minor-faults
19518092 +6.5% 20793294 perf-stat.i.node-load-misses
1039427 -8.9% 946875 ± 2% perf-stat.i.node-loads
10095749 +15.1% 11618998 perf-stat.i.node-store-misses
620979 -12.9% 541058 perf-stat.i.page-faults
4.22 +7.5% 4.54 perf-stat.overall.MPKI
0.18 +0.0 0.20 perf-stat.overall.branch-miss-rate%
21.43 +2.1 23.49 perf-stat.overall.cache-miss-rate%
1.20 +24.6% 1.50 perf-stat.overall.cpi
1330 +5.8% 1406 ± 2% perf-stat.overall.cycles-between-cache-misses
0.02 -0.0 0.02 perf-stat.overall.dTLB-load-miss-rate%
0.02 -0.0 0.02 perf-stat.overall.dTLB-store-miss-rate%
15600 -2.0% 15295 perf-stat.overall.instructions-per-iTLB-miss
0.83 -19.7% 0.67 perf-stat.overall.ipc
3724 +1.6% 3783 perf-stat.overall.path-length
2.282e+10 -8.2% 2.094e+10 perf-stat.ps.branch-instructions
40052157 +6.9% 42799990 perf-stat.ps.branch-misses
66240006 +9.0% 72170753 perf-stat.ps.cache-misses
1754492 -4.6% 1673329 perf-stat.ps.context-switches
8.81e+10 +15.2% 1.015e+11 perf-stat.ps.cpu-cycles
6191 +7.3% 6646 perf-stat.ps.cpu-migrations
4557705 -10.1% 4099312 perf-stat.ps.dTLB-load-misses
2.115e+10 -7.9% 1.947e+10 perf-stat.ps.dTLB-loads
699965 -13.0% 609113 perf-stat.ps.dTLB-store-misses
3.904e+09 -7.5% 3.611e+09 perf-stat.ps.dTLB-stores
4698565 -5.7% 4429361 perf-stat.ps.iTLB-load-misses
10410202 -7.0% 9684466 perf-stat.ps.iTLB-loads
7.329e+10 -7.6% 6.775e+10 perf-stat.ps.instructions
618900 -12.9% 539304 perf-stat.ps.minor-faults
19458339 +6.6% 20734296 perf-stat.ps.node-load-misses
1036377 -8.9% 944375 ± 2% perf-stat.ps.node-loads
10063664 +15.1% 11583746 perf-stat.ps.node-store-misses
618903 -12.9% 539307 perf-stat.ps.page-faults
2.263e+13 -7.4% 2.094e+13 perf-stat.total.instructions
23.77 -3.8 19.94 ± 3% perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
17.03 -3.4 13.62 perf-profile.calltrace.cycles-pp.do_rw_once
27.40 -2.9 24.55 perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
47.87 -2.6 45.25 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
46.90 -2.5 44.43 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
47.06 -2.4 44.67 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
12.10 -1.7 10.37 perf-profile.calltrace.cycles-pp.finish_fault.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
12.01 -1.7 10.30 perf-profile.calltrace.cycles-pp._raw_spin_lock.finish_fault.do_read_fault.do_fault.__handle_mm_fault
11.80 -1.6 10.15 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.finish_fault.do_read_fault.do_fault
10.69 -0.8 9.89 perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
0.94 -0.6 0.35 ± 70% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.folio_wait_bit_common.__filemap_get_folio.shmem_get_folio_gfp.shmem_fault
8.46 -0.5 7.98 perf-profile.calltrace.cycles-pp._raw_spin_lock.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
8.37 -0.5 7.89 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.filemap_map_pages.do_read_fault.do_fault
4.22 ± 5% -0.4 3.79 ± 2% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
1.20 -0.3 0.93 ± 3% perf-profile.calltrace.cycles-pp.folio_wake_bit.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
1.57 -0.3 1.30 perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.97 -0.2 0.77 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common.folio_wake_bit.do_read_fault.do_fault.__handle_mm_fault
0.76 -0.2 0.56 ± 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_page_function.__wake_up_common.folio_wake_bit.do_read_fault
1.03 -0.2 0.85 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
0.86 -0.2 0.68 ± 2% perf-profile.calltrace.cycles-pp.wake_page_function.__wake_up_common.folio_wake_bit.do_read_fault.do_fault
0.76 ± 2% -0.2 0.59 ± 4% perf-profile.calltrace.cycles-pp.next_uptodate_page.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
0.80 -0.1 0.69 ± 3% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
0.61 -0.1 0.55 perf-profile.calltrace.cycles-pp.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
0.97 -0.0 0.92 perf-profile.calltrace.cycles-pp.folio_wake_bit.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
0.58 ± 2% -0.0 0.54 perf-profile.calltrace.cycles-pp._raw_spin_lock.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
0.57 ± 2% -0.0 0.53 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.handle_pte_fault.__handle_mm_fault.handle_mm_fault
14.92 +0.9 15.86 perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
14.92 +0.9 15.86 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
30.86 +1.4 32.30 perf-profile.calltrace.cycles-pp.asm_exc_page_fault.do_access
29.74 +1.5 31.20 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.do_access
29.59 +1.5 31.05 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
28.77 +1.5 30.23 perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
28.76 +1.5 30.23 perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
30.41 +1.5 31.93 perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.do_access
30.42 +1.5 31.94 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.do_access
7.72 ± 4% +1.9 9.65 ± 6% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter
50.46 +2.1 52.56 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
50.47 +2.1 52.57 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
50.45 +2.1 52.55 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
50.93 +2.2 53.16 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
4.41 +4.3 8.75 ± 2% perf-profile.calltrace.cycles-pp.__do_fault.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
4.39 +4.4 8.74 ± 2% perf-profile.calltrace.cycles-pp.shmem_fault.__do_fault.do_read_fault.do_fault.__handle_mm_fault
0.00 +4.4 4.36 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle
4.28 +4.4 8.68 ± 2% perf-profile.calltrace.cycles-pp.shmem_get_folio_gfp.shmem_fault.__do_fault.do_read_fault.do_fault
3.86 +4.5 8.34 ± 2% perf-profile.calltrace.cycles-pp.__filemap_get_folio.shmem_get_folio_gfp.shmem_fault.__do_fault.do_read_fault
3.39 +4.6 8.01 ± 3% perf-profile.calltrace.cycles-pp.folio_wait_bit_common.__filemap_get_folio.shmem_get_folio_gfp.shmem_fault.__do_fault
0.00 +4.7 4.71 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule_idle.do_idle
0.00 +4.7 4.73 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prepare_task_switch.__schedule.schedule
0.00 +4.9 4.87 ± 5% perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
0.00 +5.1 5.05 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock.prepare_task_switch.__schedule.schedule.io_schedule
0.66 +5.1 5.73 ± 4% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.62 +5.1 5.70 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
1.77 +5.1 6.90 ± 4% perf-profile.calltrace.cycles-pp.schedule.io_schedule.folio_wait_bit_common.__filemap_get_folio.shmem_get_folio_gfp
1.78 +5.1 6.91 ± 4% perf-profile.calltrace.cycles-pp.io_schedule.folio_wait_bit_common.__filemap_get_folio.shmem_get_folio_gfp.shmem_fault
1.74 +5.1 6.88 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.folio_wait_bit_common.__filemap_get_folio
0.00 +5.2 5.17 ± 5% perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.io_schedule.folio_wait_bit_common
14.97 -3.1 11.87 perf-profile.children.cycles-pp.do_rw_once
27.65 -2.8 24.82 perf-profile.children.cycles-pp.intel_idle_irq
48.31 -2.6 45.76 perf-profile.children.cycles-pp.cpuidle_idle_call
47.33 -2.4 44.92 perf-profile.children.cycles-pp.cpuidle_enter
47.32 -2.4 44.92 perf-profile.children.cycles-pp.cpuidle_enter_state
42.69 -1.8 40.85 perf-profile.children.cycles-pp.mwait_idle_with_hints
12.10 -1.7 10.37 perf-profile.children.cycles-pp.finish_fault
10.69 -0.8 9.89 perf-profile.children.cycles-pp.filemap_map_pages
4.27 ± 4% -0.4 3.83 ± 2% perf-profile.children.cycles-pp.poll_idle
1.02 -0.4 0.58 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irq
2.18 -0.3 1.85 perf-profile.children.cycles-pp.folio_wake_bit
1.59 -0.3 1.32 perf-profile.children.cycles-pp.flush_smp_call_function_queue
39.78 -0.2 39.54 perf-profile.children.cycles-pp.do_access
1.73 -0.2 1.52 perf-profile.children.cycles-pp.__wake_up_common
1.35 ± 2% -0.2 1.14 perf-profile.children.cycles-pp.try_to_wake_up
1.53 -0.2 1.35 perf-profile.children.cycles-pp.wake_page_function
0.76 ± 2% -0.2 0.59 ± 4% perf-profile.children.cycles-pp.next_uptodate_page
1.05 -0.2 0.88 perf-profile.children.cycles-pp.sched_ttwu_pending
0.70 ± 2% -0.2 0.55 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.82 ± 2% -0.1 0.70 ± 3% perf-profile.children.cycles-pp.menu_select
0.89 ± 5% -0.1 0.78 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.56 -0.1 0.47 perf-profile.children.cycles-pp.ttwu_do_activate
0.64 ± 8% -0.1 0.55 perf-profile.children.cycles-pp.hrtimer_interrupt
0.50 -0.1 0.41 perf-profile.children.cycles-pp.enqueue_task_fair
0.65 ± 8% -0.1 0.56 perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.49 -0.1 0.42 perf-profile.children.cycles-pp.dequeue_task_fair
0.52 -0.1 0.45 ± 2% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.32 ± 2% -0.1 0.25 ± 4% perf-profile.children.cycles-pp.select_task_rq
0.42 ± 2% -0.1 0.35 ± 3% perf-profile.children.cycles-pp.enqueue_entity
0.44 -0.1 0.38 ± 2% perf-profile.children.cycles-pp.dequeue_entity
0.26 -0.1 0.21 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair
0.27 ± 3% -0.1 0.21 ± 2% perf-profile.children.cycles-pp.__smp_call_single_queue
0.27 -0.1 0.21 ± 2% perf-profile.children.cycles-pp.llist_add_batch
0.34 -0.1 0.28 perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.32 ± 6% -0.1 0.27 ± 3% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.61 -0.1 0.55 perf-profile.children.cycles-pp.handle_pte_fault
0.21 ± 2% -0.1 0.16 ± 2% perf-profile.children.cycles-pp.folio_unlock
0.26 ± 8% -0.0 0.21 ± 2% perf-profile.children.cycles-pp.tick_sched_timer
0.31 ± 6% -0.0 0.26 ± 7% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.34 ± 8% -0.0 0.30 ± 3% perf-profile.children.cycles-pp.ktime_get
0.22 ± 2% -0.0 0.18 perf-profile.children.cycles-pp.xas_load
0.19 ± 2% -0.0 0.14 ± 6% perf-profile.children.cycles-pp.wake_affine
0.20 ± 6% -0.0 0.15 ± 6% perf-profile.children.cycles-pp.tick_sched_handle
0.33 ± 2% -0.0 0.28 perf-profile.children.cycles-pp.update_load_avg
0.28 -0.0 0.24 perf-profile.children.cycles-pp.update_rq_clock
0.18 -0.0 0.14 ± 5% perf-profile.children.cycles-pp.available_idle_cpu
0.20 ± 2% -0.0 0.17 ± 2% perf-profile.children.cycles-pp.llist_reverse_order
0.19 ± 8% -0.0 0.15 ± 6% perf-profile.children.cycles-pp.update_process_times
0.13 ± 6% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.scheduler_tick
0.12 ± 6% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.xas_find
0.20 ± 2% -0.0 0.17 ± 2% perf-profile.children.cycles-pp.native_irq_return_iret
0.06 ± 7% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.21 ± 9% -0.0 0.18 ± 6% perf-profile.children.cycles-pp.tick_nohz_next_event
0.15 ± 3% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.native_sched_clock
0.35 -0.0 0.32 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair
0.17 ± 2% -0.0 0.15 ± 3% perf-profile.children.cycles-pp.__switch_to_asm
0.18 ± 2% -0.0 0.15 ± 5% perf-profile.children.cycles-pp.sched_clock_cpu
0.21 -0.0 0.18 ± 5% perf-profile.children.cycles-pp.update_curr
0.08 ± 10% -0.0 0.06 ± 8% perf-profile.children.cycles-pp.read_tsc
0.09 ± 5% -0.0 0.07 perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.12 -0.0 0.10 ± 4% perf-profile.children.cycles-pp.set_next_entity
0.22 ± 2% -0.0 0.20 ± 4% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.12 ± 3% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.10 ± 9% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.__list_add_valid
0.07 ± 6% -0.0 0.06 ± 8% perf-profile.children.cycles-pp.__perf_sw_event
0.07 -0.0 0.05 ± 8% perf-profile.children.cycles-pp.reweight_entity
0.15 ± 3% -0.0 0.13 perf-profile.children.cycles-pp.__switch_to
0.12 -0.0 0.10 ± 4% perf-profile.children.cycles-pp.___perf_sw_event
0.12 -0.0 0.10 ± 4% perf-profile.children.cycles-pp.sync_regs
0.07 ± 7% -0.0 0.05 perf-profile.children.cycles-pp.shmem_alloc_and_acct_folio
0.11 ± 4% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.perf_tp_event
0.10 ± 4% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.put_page
0.07 -0.0 0.06 ± 8% perf-profile.children.cycles-pp.__list_del_entry_valid
0.14 ± 5% +0.0 0.16 ± 3% perf-profile.children.cycles-pp.load_balance
0.16 ± 5% +0.0 0.18 ± 2% perf-profile.children.cycles-pp.newidle_balance
0.10 ± 4% +0.0 0.12 perf-profile.children.cycles-pp.update_sg_lb_stats
0.12 ± 8% +0.0 0.14 perf-profile.children.cycles-pp.find_busiest_group
0.21 ± 6% +0.0 0.25 ± 3% perf-profile.children.cycles-pp.up_read
0.15 ± 5% +0.1 0.21 ± 6% perf-profile.children.cycles-pp.down_read_trylock
0.55 ± 5% +0.1 0.61 ± 2% perf-profile.children.cycles-pp.finish_task_switch
0.00 +0.1 0.10 ± 12% perf-profile.children.cycles-pp._find_first_zero_bit
0.09 +0.3 0.38 ± 3% perf-profile.children.cycles-pp.switch_mm_irqs_off
4.61 ± 4% +0.9 5.48 ± 5% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
15.05 +1.0 16.04 perf-profile.children.cycles-pp.intel_idle
30.89 +1.4 32.33 perf-profile.children.cycles-pp.asm_exc_page_fault
29.75 +1.5 31.21 perf-profile.children.cycles-pp.handle_mm_fault
29.60 +1.5 31.06 perf-profile.children.cycles-pp.__handle_mm_fault
28.77 +1.5 30.23 perf-profile.children.cycles-pp.do_read_fault
28.77 +1.5 30.24 perf-profile.children.cycles-pp.do_fault
30.42 +1.5 31.94 perf-profile.children.cycles-pp.do_user_addr_fault
30.43 +1.5 31.95 perf-profile.children.cycles-pp.exc_page_fault
50.47 +2.1 52.57 perf-profile.children.cycles-pp.start_secondary
50.93 +2.2 53.16 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
50.93 +2.2 53.16 perf-profile.children.cycles-pp.cpu_startup_entry
50.91 +2.2 53.14 perf-profile.children.cycles-pp.do_idle
4.41 +4.3 8.75 ± 2% perf-profile.children.cycles-pp.__do_fault
4.39 +4.4 8.74 ± 2% perf-profile.children.cycles-pp.shmem_fault
4.29 +4.4 8.69 ± 2% perf-profile.children.cycles-pp.shmem_get_folio_gfp
3.86 +4.5 8.34 ± 2% perf-profile.children.cycles-pp.__filemap_get_folio
3.40 +4.6 8.01 ± 3% perf-profile.children.cycles-pp.folio_wait_bit_common
0.67 +5.1 5.79 ± 4% perf-profile.children.cycles-pp.schedule_idle
1.78 +5.1 6.91 ± 4% perf-profile.children.cycles-pp.schedule
1.78 +5.1 6.91 ± 4% perf-profile.children.cycles-pp.io_schedule
21.60 +6.6 28.16 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
21.40 +7.5 28.92 perf-profile.children.cycles-pp._raw_spin_lock
0.11 ± 4% +10.0 10.10 ± 5% perf-profile.children.cycles-pp.prepare_task_switch
2.40 +10.3 12.66 ± 4% perf-profile.children.cycles-pp.__schedule
10.91 -2.4 8.54 perf-profile.self.cycles-pp.do_rw_once
42.19 -1.8 40.35 perf-profile.self.cycles-pp.mwait_idle_with_hints
6.82 -1.2 5.58 perf-profile.self.cycles-pp.do_access
4.18 ± 4% -0.4 3.76 ± 2% perf-profile.self.cycles-pp.poll_idle
0.75 ± 2% -0.2 0.58 ± 5% perf-profile.self.cycles-pp.next_uptodate_page
0.32 ± 4% -0.1 0.20 ± 6% perf-profile.self.cycles-pp.__filemap_get_folio
0.57 ± 3% -0.1 0.46 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.57 -0.1 0.49 perf-profile.self.cycles-pp.folio_wait_bit_common
0.29 -0.1 0.22 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.44 -0.1 0.38 ± 2% perf-profile.self.cycles-pp.menu_select
0.09 ± 5% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.shmem_fault
0.26 ± 3% -0.1 0.21 ± 2% perf-profile.self.cycles-pp.llist_add_batch
0.21 -0.0 0.16 ± 2% perf-profile.self.cycles-pp.folio_unlock
0.34 ± 2% -0.0 0.30 ± 4% perf-profile.self.cycles-pp.filemap_map_pages
0.18 ± 2% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.available_idle_cpu
0.20 ± 2% -0.0 0.17 ± 2% perf-profile.self.cycles-pp.llist_reverse_order
0.20 ± 2% -0.0 0.17 ± 2% perf-profile.self.cycles-pp.native_irq_return_iret
0.17 ± 2% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.__switch_to_asm
0.15 ± 5% -0.0 0.12 ± 6% perf-profile.self.cycles-pp.native_sched_clock
0.15 ± 3% -0.0 0.12 ± 3% perf-profile.self.cycles-pp.enqueue_entity
0.17 ± 2% -0.0 0.15 ± 3% perf-profile.self.cycles-pp.xas_load
0.21 -0.0 0.18 ± 2% perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.20 ± 4% -0.0 0.17 ± 2% perf-profile.self.cycles-pp.__wake_up_common
0.10 ± 9% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.__list_add_valid
0.08 ± 5% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.enqueue_task_fair
0.07 ± 6% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.read_tsc
0.13 -0.0 0.11 ± 4% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.10 ± 4% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.cpuidle_idle_call
0.07 -0.0 0.05 ± 8% perf-profile.self.cycles-pp.reweight_entity
0.14 ± 3% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.__switch_to
0.12 -0.0 0.10 ± 4% perf-profile.self.cycles-pp.sync_regs
0.12 ± 4% -0.0 0.10 ± 8% perf-profile.self.cycles-pp.update_load_avg
0.20 ± 4% -0.0 0.18 ± 2% perf-profile.self.cycles-pp.update_rq_clock
0.16 ± 3% -0.0 0.14 ± 5% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.12 ± 4% -0.0 0.10 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.10 -0.0 0.09 ± 5% perf-profile.self.cycles-pp.___perf_sw_event
0.07 -0.0 0.06 ± 8% perf-profile.self.cycles-pp.__list_del_entry_valid
0.06 ± 7% -0.0 0.05 perf-profile.self.cycles-pp.select_task_rq_fair
0.09 ± 5% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.__update_load_avg_se
0.07 -0.0 0.06 perf-profile.self.cycles-pp.sched_ttwu_pending
0.08 ± 6% +0.0 0.09 perf-profile.self.cycles-pp.update_sg_lb_stats
0.14 ± 5% +0.0 0.17 ± 5% perf-profile.self.cycles-pp.wake_page_function
0.21 ± 6% +0.0 0.25 ± 3% perf-profile.self.cycles-pp.up_read
0.21 +0.1 0.26 perf-profile.self.cycles-pp.__handle_mm_fault
0.15 ± 5% +0.1 0.21 ± 6% perf-profile.self.cycles-pp.down_read_trylock
0.42 ± 2% +0.1 0.50 ± 3% perf-profile.self.cycles-pp.__schedule
0.00 +0.1 0.09 ± 10% perf-profile.self.cycles-pp._find_first_zero_bit
0.00 +0.1 0.12 ± 8% perf-profile.self.cycles-pp.prepare_task_switch
0.09 +0.3 0.38 ± 3% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.65 +0.6 1.21 ± 2% perf-profile.self.cycles-pp._raw_spin_lock
21.50 +6.6 28.08 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#!/bin/sh

export_top_env()
{
export suite='will-it-scale'
export testcase='will-it-scale'
export category='benchmark'
export job_origin='will-it-scale-fpu-test.yaml'
export queue='bisect'
export testbox='lkp-csl-2sp9'
export commit='af7f588d8f7355bc4298dd1962d7826358fc95f0'
export branch='linus/master'
export name='/cephfs/tmp/d20230424-8447-1jrb2t2/will-it-scale-fpu-test.yaml'
export kernel_cmdline=
export tbox_group='lkp-csl-2sp9'
export submit_id='645bc5376f560d86a15a1974'
export job_file='/lkp/jobs/scheduled/lkp-csl-2sp9/will-it-scale-performance-context_switch1-debian-11.1-x86_64-20220510.cgz-af7f588d8f7355bc4298dd1962d7826358fc95f0-20230511-34465-pn9jw1-1.yaml'
export id='8b52890475f8222c91c7963430fa959e6783865e'
export queuer_version='/zday/lkp'
export model='Cascade Lake'
export nr_node=2
export nr_cpu=88
export memory='128G'
export nr_hdd_partitions=4
export nr_ssd_partitions=1
export hdd_partitions='/dev/disk/by-id/ata-ST4000NM0035-1V4107_ZC13Q1RD-part*'
export ssd_partitions='/dev/disk/by-id/ata-INTEL_SSDSC2BB480G7_PHDV723200JX480BGN-part1'
export rootfs_partition='/dev/disk/by-id/ata-INTEL_SSDSC2BB480G7_PHDV723200JX480BGN-part2'
export brand='Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz'
export ucode='0x5003303'
export need_kconfig_hw='{"I40E"=>"y"}
SATA_AHCI
INTEL_TH
INTEL_TH_ACPI'
export disable_latency_stats=1
export rootfs='debian-11.1-x86_64-20220510.cgz'
export kconfig='x86_64-rhel-8.3'
export enqueue_time='2023-05-11 00:24:23 +0800'
export compiler='gcc-11'
export _id='645bc54d6f560d86a15a1975'
export _rt='/result/will-it-scale/performance-context_switch1/lkp-csl-2sp9/debian-11.1-x86_64-20220510.cgz/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0'
export user='lkp'
export LKP_SERVER='internal-lkp-server'
export result_root='/result/will-it-scale/performance-context_switch1/lkp-csl-2sp9/debian-11.1-x86_64-20220510.cgz/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/0'
export scheduler_version='/lkp/lkp/src'
export arch='x86_64'
export max_uptime=2100
export initrd='/osimage/debian/debian-11.1-x86_64-20220510.cgz'
export bootloader_append='root=/dev/ram0
RESULT_ROOT=/result/will-it-scale/performance-context_switch1/lkp-csl-2sp9/debian-11.1-x86_64-20220510.cgz/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/0
BOOT_IMAGE=/pkg/linux/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/vmlinuz-6.2.0-rc1-00009-gaf7f588d8f73
branch=linus/master
job=/lkp/jobs/scheduled/lkp-csl-2sp9/will-it-scale-performance-context_switch1-debian-11.1-x86_64-20220510.cgz-af7f588d8f7355bc4298dd1962d7826358fc95f0-20230511-34465-pn9jw1-1.yaml
user=lkp
ARCH=x86_64
kconfig=x86_64-rhel-8.3
commit=af7f588d8f7355bc4298dd1962d7826358fc95f0
initcall_debug
nmi_watchdog=0
max_uptime=2100
LKP_SERVER=internal-lkp-server
nokaslr
selinux=0
debug
apic=debug
sysrq_always_enabled
rcupdate.rcu_cpu_stall_timeout=100
net.ifnames=0
printk.devkmsg=on
panic=-1
softlockup_panic=1
nmi_watchdog=panic
oops=panic
load_ramdisk=2
prompt_ramdisk=0
drbd.minor_count=8
systemd.log_level=err
ignore_loglevel
console=tty0
earlyprintk=ttyS0,115200
console=ttyS0,115200
vga=normal
rw'
export modules_initrd='/pkg/linux/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/modules.cgz'
export bm_initrd='/osimage/deps/debian-11.1-x86_64-20220510.cgz/lkp_20220513.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/run-ipconfig_20220515.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/rsync-rootfs_20220515.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/will-it-scale_20220823.cgz,/osimage/pkg/debian-11.1-x86_64-20220510.cgz/will-it-scale-x86_64-a34a85c-1_20220601.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/mpstat_20220516.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/turbostat_20220514.cgz,/osimage/pkg/debian-11.1-x86_64-20220510.cgz/turbostat-x86_64-210e04ff7681-1_20220518.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/perf_20230402.cgz,/osimage/pkg/debian-11.1-x86_64-20220510.cgz/perf-x86_64-00c7b5f4ddc5-1_20230402.cgz,/osimage/pkg/debian-11.1-x86_64-20220510.cgz/sar-x86_64-c5bb321-1_20220518.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/hw_20220526.cgz'
export ucode_initrd='/osimage/ucode/intel-ucode-20230406.cgz'
export lkp_initrd='/osimage/user/lkp/lkp-x86_64.cgz'
export site='lkp-wsx01'
export LKP_CGI_PORT=80
export LKP_CIFS_PORT=139
export last_kernel='6.3.0'
export queue_cmdline_keys='branch
commit
kbuild_queue_analysis'
export repeat_to=3
export kbuild_queue_analysis=1
export kernel='/pkg/linux/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/vmlinuz-6.2.0-rc1-00009-gaf7f588d8f73'
export dequeue_time='2023-05-11 00:31:59 +0800'
export job_initrd='/lkp/jobs/scheduled/lkp-csl-2sp9/will-it-scale-performance-context_switch1-debian-11.1-x86_64-20220510.cgz-af7f588d8f7355bc4298dd1962d7826358fc95f0-20230511-34465-pn9jw1-1.cgz'

[ -n "$LKP_SRC" ] ||
export LKP_SRC=/lkp/${user:-lkp}/src
}

run_job()
{
echo $$ > $TMP/run-job.pid

. $LKP_SRC/lib/http.sh
. $LKP_SRC/lib/job.sh
. $LKP_SRC/lib/env.sh

export_top_env

run_setup $LKP_SRC/setup/cpufreq_governor 'performance'

run_monitor $LKP_SRC/monitors/wrapper kmsg
run_monitor $LKP_SRC/monitors/no-stdout/wrapper boot-time
run_monitor $LKP_SRC/monitors/wrapper uptime
run_monitor $LKP_SRC/monitors/wrapper iostat
run_monitor $LKP_SRC/monitors/wrapper heartbeat
run_monitor $LKP_SRC/monitors/wrapper vmstat
run_monitor $LKP_SRC/monitors/wrapper numa-numastat
run_monitor $LKP_SRC/monitors/wrapper numa-vmstat
run_monitor $LKP_SRC/monitors/wrapper numa-meminfo
run_monitor $LKP_SRC/monitors/wrapper proc-vmstat
run_monitor $LKP_SRC/monitors/wrapper proc-stat
run_monitor $LKP_SRC/monitors/wrapper meminfo
run_monitor $LKP_SRC/monitors/wrapper slabinfo
run_monitor $LKP_SRC/monitors/wrapper interrupts
run_monitor $LKP_SRC/monitors/wrapper lock_stat
run_monitor lite_mode=1 $LKP_SRC/monitors/wrapper perf-sched
run_monitor $LKP_SRC/monitors/wrapper softirqs
run_monitor $LKP_SRC/monitors/one-shot/wrapper bdi_dev_mapping
run_monitor $LKP_SRC/monitors/wrapper diskstats
run_monitor $LKP_SRC/monitors/wrapper nfsstat
run_monitor $LKP_SRC/monitors/wrapper cpuidle
run_monitor $LKP_SRC/monitors/wrapper cpufreq-stats
run_monitor $LKP_SRC/monitors/wrapper turbostat
run_monitor $LKP_SRC/monitors/wrapper sched_debug
run_monitor $LKP_SRC/monitors/wrapper perf-stat
run_monitor $LKP_SRC/monitors/wrapper mpstat
run_monitor debug_mode=0 $LKP_SRC/monitors/no-stdout/wrapper perf-profile
run_monitor $LKP_SRC/monitors/wrapper oom-killer
run_monitor $LKP_SRC/monitors/plain/watchdog

run_test test='context_switch1' $LKP_SRC/tests/wrapper will-it-scale
}

extract_stats()
{
export stats_part_begin=
export stats_part_end=

env test='context_switch1' $LKP_SRC/stats/wrapper will-it-scale
$LKP_SRC/stats/wrapper kmsg
$LKP_SRC/stats/wrapper boot-time
$LKP_SRC/stats/wrapper uptime
$LKP_SRC/stats/wrapper iostat
$LKP_SRC/stats/wrapper vmstat
$LKP_SRC/stats/wrapper numa-numastat
$LKP_SRC/stats/wrapper numa-vmstat
$LKP_SRC/stats/wrapper numa-meminfo
$LKP_SRC/stats/wrapper proc-vmstat
$LKP_SRC/stats/wrapper meminfo
$LKP_SRC/stats/wrapper slabinfo
$LKP_SRC/stats/wrapper interrupts
$LKP_SRC/stats/wrapper lock_stat
env lite_mode=1 $LKP_SRC/stats/wrapper perf-sched
$LKP_SRC/stats/wrapper softirqs
$LKP_SRC/stats/wrapper diskstats
$LKP_SRC/stats/wrapper nfsstat
$LKP_SRC/stats/wrapper cpuidle
$LKP_SRC/stats/wrapper turbostat
$LKP_SRC/stats/wrapper sched_debug
$LKP_SRC/stats/wrapper perf-stat
$LKP_SRC/stats/wrapper mpstat
env debug_mode=0 $LKP_SRC/stats/wrapper perf-profile

$LKP_SRC/stats/wrapper time will-it-scale.time
$LKP_SRC/stats/wrapper dmesg
$LKP_SRC/stats/wrapper kmsg
$LKP_SRC/stats/wrapper last_state
$LKP_SRC/stats/wrapper stderr
$LKP_SRC/stats/wrapper time
}

"$@"
---

#! /cephfs/tmp/d20230424-8447-1jrb2t2/will-it-scale-fpu-test.yaml
suite: will-it-scale
testcase: will-it-scale
category: benchmark
will-it-scale:
test: context_switch1
job_origin: will-it-scale-fpu-test.yaml
queue: bisect
testbox: lkp-csl-2sp9
commit: af7f588d8f7355bc4298dd1962d7826358fc95f0
branch: linus/master
name: "/cephfs/tmp/d20230424-8447-1jrb2t2/will-it-scale-fpu-test.yaml"
kernel_cmdline:
tbox_group: lkp-csl-2sp9
submit_id: 645bc5376f560d86a15a1974
job_file: "/lkp/jobs/scheduled/lkp-csl-2sp9/will-it-scale-performance-context_switch1-debian-11.1-x86_64-20220510.cgz-af7f588d8f7355bc4298dd1962d7826358fc95f0-20230511-34465-pn9jw1-1.yaml"
id: 8b52890475f8222c91c7963430fa959e6783865e
queuer_version: "/zday/lkp"

#! hosts/lkp-csl-2sp9
model: Cascade Lake
nr_node: 2
nr_cpu: 88
memory: 128G
nr_hdd_partitions: 4
nr_ssd_partitions: 1
hdd_partitions: "/dev/disk/by-id/ata-ST4000NM0035-1V4107_ZC13Q1RD-part*"
ssd_partitions: "/dev/disk/by-id/ata-INTEL_SSDSC2BB480G7_PHDV723200JX480BGN-part1"
rootfs_partition: "/dev/disk/by-id/ata-INTEL_SSDSC2BB480G7_PHDV723200JX480BGN-part2"
brand: Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz

#! include/category/benchmark
kmsg:
boot-time:
uptime:
iostat:
heartbeat:
vmstat:
numa-numastat:
numa-vmstat:
numa-meminfo:
proc-vmstat:
proc-stat:
meminfo:
slabinfo:
interrupts:
lock_stat:
perf-sched:
lite_mode: 1
softirqs:
bdi_dev_mapping:
diskstats:
nfsstat:
cpuidle:
cpufreq-stats:
turbostat:
sched_debug:
perf-stat:
mpstat:
perf-profile:
debug_mode: 0

#! include/category/ALL
cpufreq_governor: performance

#! include/testbox/lkp-csl-2sp9
ucode: '0x5003303'
need_kconfig_hw:
- I40E: y
- SATA_AHCI
- INTEL_TH
- INTEL_TH_ACPI
disable_latency_stats: 1
rootfs: debian-11.1-x86_64-20220510.cgz
kconfig: x86_64-rhel-8.3
enqueue_time: 2023-05-11 00:24:23.655011677 +08:00
compiler: gcc-11
_id: 645bc54d6f560d86a15a1975
_rt: "/result/will-it-scale/performance-context_switch1/lkp-csl-2sp9/debian-11.1-x86_64-20220510.cgz/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0"

#! schedule options
user: lkp
LKP_SERVER: internal-lkp-server
result_root: "/result/will-it-scale/performance-context_switch1/lkp-csl-2sp9/debian-11.1-x86_64-20220510.cgz/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/0"
scheduler_version: "/lkp/lkp/src"
arch: x86_64
max_uptime: 2100
initrd: "/osimage/debian/debian-11.1-x86_64-20220510.cgz"
bootloader_append:
- root=/dev/ram0
- RESULT_ROOT=/result/will-it-scale/performance-context_switch1/lkp-csl-2sp9/debian-11.1-x86_64-20220510.cgz/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/0
- BOOT_IMAGE=/pkg/linux/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/vmlinuz-6.2.0-rc1-00009-gaf7f588d8f73
- branch=linus/master
- job=/lkp/jobs/scheduled/lkp-csl-2sp9/will-it-scale-performance-context_switch1-debian-11.1-x86_64-20220510.cgz-af7f588d8f7355bc4298dd1962d7826358fc95f0-20230511-34465-pn9jw1-1.yaml
- user=lkp
- ARCH=x86_64
- kconfig=x86_64-rhel-8.3
- commit=af7f588d8f7355bc4298dd1962d7826358fc95f0
- initcall_debug
- nmi_watchdog=0
- max_uptime=2100
- LKP_SERVER=internal-lkp-server
- nokaslr
- selinux=0
- debug
- apic=debug
- sysrq_always_enabled
- rcupdate.rcu_cpu_stall_timeout=100
- net.ifnames=0
- printk.devkmsg=on
- panic=-1
- softlockup_panic=1
- nmi_watchdog=panic
- oops=panic
- load_ramdisk=2
- prompt_ramdisk=0
- drbd.minor_count=8
- systemd.log_level=err
- ignore_loglevel
- console=tty0
- earlyprintk=ttyS0,115200
- console=ttyS0,115200
- vga=normal
- rw

#! runtime status
modules_initrd: "/pkg/linux/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/modules.cgz"
bm_initrd: "/osimage/deps/debian-11.1-x86_64-20220510.cgz/lkp_20220513.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/run-ipconfig_20220515.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/rsync-rootfs_20220515.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/will-it-scale_20220823.cgz,/osimage/pkg/debian-11.1-x86_64-20220510.cgz/will-it-scale-x86_64-a34a85c-1_20220601.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/mpstat_20220516.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/turbostat_20220514.cgz,/osimage/pkg/debian-11.1-x86_64-20220510.cgz/turbostat-x86_64-210e04ff7681-1_20220518.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/perf_20230402.cgz,/osimage/pkg/debian-11.1-x86_64-20220510.cgz/perf-x86_64-00c7b5f4ddc5-1_20230402.cgz,/osimage/pkg/debian-11.1-x86_64-20220510.cgz/sar-x86_64-c5bb321-1_20220518.cgz,/osimage/deps/debian-11.1-x86_64-20220510.cgz/hw_20220526.cgz"
ucode_initrd: "/osimage/ucode/intel-ucode-20230406.cgz"
lkp_initrd: "/osimage/user/lkp/lkp-x86_64.cgz"
site: lkp-wsx01

#! /db/releases/20230424231518/lkp-src/include/site/lkp-wsx01
LKP_CGI_PORT: 80
LKP_CIFS_PORT: 139
oom-killer:
watchdog:
last_kernel: 6.4.0-rc1-2023-05-09-intel-next-01765-gcc072b5075bd

#! queue options
queue_cmdline_keys:
- branch
- commit
- kbuild_queue_analysis

#! /db/releases/20230506144350/lkp-src/include/site/lkp-wsx01
repeat_to: 3

#! user overrides
kbuild_queue_analysis: 1
kernel: "/pkg/linux/x86_64-rhel-8.3/gcc-11/af7f588d8f7355bc4298dd1962d7826358fc95f0/vmlinuz-6.2.0-rc1-00009-gaf7f588d8f73"
dequeue_time: 2023-05-11 00:31:59.152337560 +08:00

#! /db/releases/20230510204317/lkp-src/include/site/lkp-wsx01
job_state: finished
loadavg: 84.95 41.93 16.63 1/752 14738
start_time: '1683736397'
end_time: '1683736713'
version: "/lkp/lkp/.src-20230508-150358:ad1384b06da5:c6711f306d8a"

for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
do
online_file="$cpu_dir"/online
[ -f "$online_file" ] && [ "$(cat "$online_file")" -eq 0 ] && continue

file="$cpu_dir"/cpufreq/scaling_governor
[ -f "$file" ] && echo "performance" > "$file"
done

"/lkp/benchmarks/python3/bin/python3" "./runtest.py" "context_switch1" "25" "both" "0" "0" "1" "22" "44" "66" "88"

Next message: Marc Kleine-Budde: "Re: [PATCH V7] can: usb: f81604: add Fintek F81604 support"
Previous message: Abel Wu: "Re: [PATCH] sock: Fix misuse of sk_under_memory_pressure()"
Next in thread: Peter Zijlstra: "Re: [linus:master] [sched] af7f588d8f: will-it-scale.per_thread_ops -13.9% regression"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]