Re: [btrfs] e86bb85b1f: stress-ng.utime.ops_per_sec -70.1% regression

[Qu Wenruo]

On 2021/1/12 下午11:24, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a -70.1% regression of stress-ng.utime.ops_per_sec due to commit:
>
>
> commit: e86bb85b1fec48bcb8dfb79ec9f104d1a38fda78 ("[PATCH] btrfs: make btrfs_dirty_inode() to always reserve metadata space")
> url: https://github.com/0day-ci/linux/commits/Qu-Wenruo/btrfs-make-btrfs_dirty_inode-to-always-reserve-metadata-space/20210108-134133
> base: https://git.kernel.org/cgit/linux/kernel/git/kdave/linux.git for-next
>
> in testcase: stress-ng
> on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
> with following parameters:
>
> 	nr_threads: 10%
> 	disk: 1HDD
> 	testtime: 30s
> 	class: filesystem
> 	cpufreq_governor: performance
> 	ucode: 0x5003003
> 	fs: btrfs
>
>
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <oliver.sang@xxxxxxxxx>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
>          git clone https://github.com/intel/lkp-tests.git
>          cd lkp-tests
>          bin/lkp install job.yaml  # job file is attached in this email
>          bin/lkp run     job.yaml
>
> =========================================================================================
> class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
>    filesystem/gcc-9/performance/1HDD/btrfs/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/stress-ng/30s/0x5003003
>
> commit:
>    97847e0652 ("Merge branch 'for-next-next-v5.10-20201211' into for-next-20201211")
>    e86bb85b1f ("btrfs: make btrfs_dirty_inode() to always reserve metadata space")
>
> 97847e06525b51ea e86bb85b1fec48bcb8dfb79ec9f
> ---------------- ---------------------------
>           %stddev     %change         %stddev
>               \          |                \
>     1098218           -40.4%     654054        stress-ng.access.ops
>       36607           -40.4%      21801        stress-ng.access.ops_per_sec

This is a little interesting.
Although accessing an inode will update its atime, but don't we have
lazy_atime mount option?

>       92962 ±  2%     -44.1%      51992 ±  3%  stress-ng.chmod.ops
>        3098 ±  2%     -44.1%       1733 ±  3%  stress-ng.chmod.ops_per_sec
>      936128 ±  6%     -41.0%     552284        stress-ng.chown.ops
>       31204 ±  6%     -41.0%      18409        stress-ng.chown.ops_per_sec
>     1939514           -18.5%    1580533        stress-ng.fcntl.ops
>       64650           -18.5%      52684        stress-ng.fcntl.ops_per_sec
>     3705607 ±  2%     -70.1%    1109769        stress-ng.utime.ops
>      123519 ±  2%     -70.1%      36992        stress-ng.utime.ops_per_sec

Another interesting part is, only stress-ng is reporting such
regressioin on the commit?
No other report on the commit with different test env? E.g. NVME SSD?

Above operations is affected by such commit, but I'm a little surprised
only one report here.

Just because flushing on HDD is more expensive? If no other test suite
is fine, I would prefer to accept the drop, as it really streamline the
operations.

Thanks,
Qu

>      381.20 ±  6%     +12.3%     428.27 ±  9%  sched_debug.cfs_rq:/.load_avg.avg
>        6316 ± 57%     -79.8%       1278 ± 68%  softirqs.CPU77.BLOCK
>       10488 ±101%     -89.5%       1100 ±124%  softirqs.CPU78.BLOCK
>        5605 ± 92%     -82.3%     990.50 ± 32%  softirqs.CPU80.BLOCK
>        6094 ±128%     -89.9%     614.50 ± 44%  softirqs.CPU92.BLOCK
>        4921 ±  9%     +20.5%       5931 ±  5%  slabinfo.dmaengine-unmap-16.active_objs
>        4922 ±  9%     +20.5%       5933 ±  5%  slabinfo.dmaengine-unmap-16.num_objs
>        9818 ±  5%      -6.9%       9139 ±  3%  slabinfo.kmalloc-rcl-256.active_objs
>       49223 ±  3%     -18.4%      40177 ±  3%  slabinfo.radix_tree_node.active_objs
>      903.25 ±  3%     -18.0%     740.50 ±  3%  slabinfo.radix_tree_node.active_slabs
>       50620 ±  3%     -18.0%      41505 ±  3%  slabinfo.radix_tree_node.num_objs
>      903.25 ±  3%     -18.0%     740.50 ±  3%  slabinfo.radix_tree_node.num_slabs
>        9927 ±  3%      +5.8%      10504        proc-vmstat.nr_active_anon
>     6043459 ±  2%      -2.2%    5911900        proc-vmstat.nr_dirtied
>        1125            -6.1%       1056 ±  4%  proc-vmstat.nr_dirty
>       20361 ±  2%      +4.7%      21309        proc-vmstat.nr_shmem
>       66221            -4.3%      63404 ±  2%  proc-vmstat.nr_slab_reclaimable
>        9927 ±  3%      +5.8%      10504        proc-vmstat.nr_zone_active_anon
>        1225            -5.8%       1154 ±  3%  proc-vmstat.nr_zone_write_pending
>    11313111            -2.1%   11072335        proc-vmstat.pgfault
>        0.00          +125.0%       0.00 ± 19%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.start_transaction.btrfs_dirty_inode
>        0.01 ± 13%     -24.5%       0.01 ± 15%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
>        3.26 ± 41%     -98.8%       0.04        perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.chmod_common.do_fchmodat.__x64_sys_chmod
>        0.00 ± 50%    +123.5%       0.01 ± 24%  perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.path_openat
>        4.77 ±  6%     -25.7%       3.54        perf-sched.total_sch_delay.max.ms
>        0.02 ±  7%     +19.5%       0.02 ±  5%  perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.chmod_common.do_fchmodat.__x64_sys_fchmodat
>        1711 ± 27%     +60.7%       2750 ± 22%  perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.chmod_common.__x64_sys_fchmod.do_syscall_64
>        0.03 ±132%   +2497.6%       0.81 ± 66%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.dput.path_put.do_fchmodat
>        0.25 ± 67%    +243.4%       0.86 ± 49%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.start_transaction.btrfs_dirty_inode
>        0.02 ± 11%     +28.6%       0.02 ± 13%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.chmod_common.do_fchmodat.__x64_sys_chmod
>        0.27 ±160%   +1248.5%       3.67 ± 57%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.dput.path_put.do_fchmodat
>        0.04 ±165%   +4851.4%       2.17 ± 99%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.mnt_want_write.chmod_common.do_fchmodat
>    36969238            -5.8%   34829162 ±  2%  perf-stat.i.cache-misses
>   2.009e+09            -2.7%  1.954e+09        perf-stat.i.dTLB-stores
>    12297175 ±  2%      -6.5%   11500340        perf-stat.i.iTLB-load-misses
>        1843            +7.0%       1972 ±  2%  perf-stat.i.instructions-per-iTLB-miss
>        8936            -2.2%       8740        perf-stat.i.minor-faults
>      952962            -3.6%     919068        perf-stat.i.node-loads
>     5348157            -7.4%    4950006 ±  2%  perf-stat.i.node-store-misses
>        8942            -2.2%       8748        perf-stat.i.page-faults
>        0.00 ± 20%      +0.0        0.00 ± 18%  perf-stat.overall.dTLB-store-miss-rate%
>        1263            +6.1%       1339        perf-stat.overall.instructions-per-iTLB-miss
>    36924576            -5.8%   34784593 ±  2%  perf-stat.ps.cache-misses
>   2.003e+09            -2.7%  1.949e+09        perf-stat.ps.dTLB-stores
>    12274047 ±  2%      -6.5%   11479296        perf-stat.ps.iTLB-load-misses
>        8880            -2.1%       8694        perf-stat.ps.minor-faults
>      950813            -3.6%     916929        perf-stat.ps.node-loads
>     5343222            -7.5%    4944422 ±  2%  perf-stat.ps.node-store-misses
>        8887            -2.1%       8702        perf-stat.ps.page-faults
>       57.00 ±166%    -100.0%       0.00        interrupts.92:PCI-MSI.31981625-edge.i40e-eth0-TxRx-56
>       19339 ±  7%     +31.3%      25385 ± 16%  interrupts.CPU13.CAL:Function_call_interrupts
>       19917 ±  5%      +9.7%      21856        interrupts.CPU23.TLB:TLB_shootdowns
>       20940 ±  5%      +9.5%      22935 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
>        3470 ± 65%     -96.1%     136.75 ±  5%  interrupts.CPU29.NMI:Non-maskable_interrupts
>        3470 ± 65%     -96.1%     136.75 ±  5%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
>        2061 ±160%     -93.5%     134.00 ±  3%  interrupts.CPU33.NMI:Non-maskable_interrupts
>        2061 ±160%     -93.5%     134.00 ±  3%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
>        3041 ± 47%     +75.6%       5342 ± 36%  interrupts.CPU35.RES:Rescheduling_interrupts
>        3748 ± 96%     -96.5%     133.00 ±  5%  interrupts.CPU40.NMI:Non-maskable_interrupts
>        3748 ± 96%     -96.5%     133.00 ±  5%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
>       21172 ±  5%     -12.7%      18476 ±  6%  interrupts.CPU45.CAL:Function_call_interrupts
>       56.25 ±169%    -100.0%       0.00        interrupts.CPU56.92:PCI-MSI.31981625-edge.i40e-eth0-TxRx-56
>      122.50 ± 25%     -34.5%      80.25 ± 32%  interrupts.CPU56.NMI:Non-maskable_interrupts
>      122.50 ± 25%     -34.5%      80.25 ± 32%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
>       21585 ±  5%     +13.4%      24479 ±  5%  interrupts.CPU58.TLB:TLB_shootdowns
>        3340 ± 22%     +76.3%       5888 ± 48%  interrupts.CPU62.RES:Rescheduling_interrupts
>       23969 ±  5%     +12.5%      26976 ± 14%  interrupts.CPU66.CAL:Function_call_interrupts
>       22832 ±  5%     +26.5%      28873 ± 14%  interrupts.CPU69.CAL:Function_call_interrupts
>        2329 ±106%     -66.3%     784.50 ±150%  interrupts.CPU73.NMI:Non-maskable_interrupts
>        2329 ±106%     -66.3%     784.50 ±150%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
>        3520 ± 65%     -96.7%     116.00 ± 27%  interrupts.CPU77.NMI:Non-maskable_interrupts
>        3520 ± 65%     -96.7%     116.00 ± 27%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
>        1570 ±156%     -93.6%     101.00 ± 37%  interrupts.CPU81.NMI:Non-maskable_interrupts
>        1570 ±156%     -93.6%     101.00 ± 37%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
>        2501 ±109%     -95.9%     102.75 ± 41%  interrupts.CPU88.NMI:Non-maskable_interrupts
>        2501 ±109%     -95.9%     102.75 ± 41%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
>        5208 ± 44%     -48.6%       2674 ± 46%  interrupts.CPU9.RES:Rescheduling_interrupts
>       20273 ± 10%     +22.3%      24803 ± 11%  interrupts.CPU91.CAL:Function_call_interrupts
>        1833 ±158%     -93.6%     117.75 ± 24%  interrupts.CPU94.NMI:Non-maskable_interrupts
>        1833 ±158%     -93.6%     117.75 ± 24%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
>       47.69 ± 11%     -27.3       20.39 ±  3%  perf-profile.calltrace.cycles-pp.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       20.56 ± 11%     -12.1        8.50 ±  3%  perf-profile.calltrace.cycles-pp.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       18.78 ± 11%     -11.9        6.92 ±  3%  perf-profile.calltrace.cycles-pp.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        5.42 ± 12%      -3.8        1.60 ±  3%  perf-profile.calltrace.cycles-pp.free_uid.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        5.38 ± 12%      -3.8        1.57 ±  3%  perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat.do_syscall_64
>        5.36 ± 12%      -3.8        1.56 ±  3%  perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat
>        5.32 ± 11%      -3.1        2.19 ±  3%  perf-profile.calltrace.cycles-pp.key_put.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        5.17 ± 11%      -2.4        2.74 ±  4%  perf-profile.calltrace.cycles-pp.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        4.77 ±  8%      -2.3        2.50 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_update_inode.btrfs_dirty_inode.btrfs_setattr.notify_change.chmod_common
>        3.79 ±  7%      -2.0        1.84 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_delayed_update_inode.btrfs_update_inode.btrfs_dirty_inode.btrfs_setattr.notify_change
>        3.54 ± 12%      -1.8        1.74 ±  4%  perf-profile.calltrace.cycles-pp.apparmor_cred_prepare.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
>        3.49 ±  9%      -1.8        1.72 ±  6%  perf-profile.calltrace.cycles-pp.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        2.49 ±  9%      -1.4        1.08 ±  8%  perf-profile.calltrace.cycles-pp.apparmor_cred_free.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
>        3.38 ± 11%      -1.3        2.04 ±  6%  perf-profile.calltrace.cycles-pp.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        3.25 ± 11%      -1.3        1.96 ±  3%  perf-profile.calltrace.cycles-pp.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        2.99 ± 11%      -1.2        1.82 ±  6%  perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        1.55 ± 11%      -1.2        0.39 ± 57%  perf-profile.calltrace.cycles-pp.join_transaction.start_transaction.btrfs_dirty_inode.btrfs_setattr.notify_change
>        2.89 ± 10%      -1.1        1.75 ±  3%  perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        2.46 ±  7%      -1.1        1.33 ±  2%  perf-profile.calltrace.cycles-pp.__btrfs_release_delayed_node.btrfs_delayed_update_inode.btrfs_update_inode.btrfs_dirty_inode.btrfs_setattr
>        1.79 ± 11%      -0.7        1.11 ±  6%  perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.do_faccessat.do_syscall_64
>        1.57 ± 11%      -0.7        0.91 ±  3%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        1.64 ±  9%      -0.6        1.00 ±  4%  perf-profile.calltrace.cycles-pp.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        1.60 ±  9%      -0.6        0.98 ±  4%  perf-profile.calltrace.cycles-pp.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        1.54 ±  8%      -0.6        0.94 ±  3%  perf-profile.calltrace.cycles-pp.__kmalloc.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
>        1.21 ± 10%      -0.4        0.77 ±  5%  perf-profile.calltrace.cycles-pp.strncpy_from_user.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64
>        1.00 ± 10%      -0.4        0.63 ±  3%  perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        0.95 ± 14%      -0.4        0.58 ±  4%  perf-profile.calltrace.cycles-pp.kmem_cache_free.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        0.95 ±  7%      -0.3        0.61 ±  4%  perf-profile.calltrace.cycles-pp.kfree.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
>        0.91 ± 10%      -0.3        0.57 ±  3%  perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.vfs_statx.__do_sys_newfstatat.do_syscall_64
>        0.00            +1.0        0.98 ±  4%  perf-profile.calltrace.cycles-pp._find_next_bit.cpumask_next.__percpu_counter_sum.__reserve_bytes.btrfs_reserve_metadata_bytes
>        0.00            +1.6        1.56 ±  4%  perf-profile.calltrace.cycles-pp.cpumask_next.__percpu_counter_sum.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_block_rsv_add
>        0.00            +4.2        4.23 ±  4%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_block_rsv_add.start_transaction
>        0.00           +12.6       12.55 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_block_rsv_add
>        0.00           +13.0       13.01 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_block_rsv_add.start_transaction
>        0.00           +15.3       15.29 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.btrfs_block_rsv_release.btrfs_trans_release_metadata.__btrfs_end_transaction
>        0.00           +15.8       15.81 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.btrfs_block_rsv_release.btrfs_trans_release_metadata.__btrfs_end_transaction.btrfs_dirty_inode
>        0.00           +16.0       16.04 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_block_rsv_release.btrfs_trans_release_metadata.__btrfs_end_transaction.btrfs_dirty_inode.btrfs_setattr
>        0.00           +16.0       16.05 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_trans_release_metadata.__btrfs_end_transaction.btrfs_dirty_inode.btrfs_setattr.notify_change
>        0.63 ± 10%     +16.7       17.34 ±  3%  perf-profile.calltrace.cycles-pp.__btrfs_end_transaction.btrfs_dirty_inode.btrfs_setattr.notify_change.chmod_common
>        0.00           +18.3       18.27 ±  4%  perf-profile.calltrace.cycles-pp.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_block_rsv_add.start_transaction.btrfs_dirty_inode
>        0.00           +18.4       18.39 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_reserve_metadata_bytes.btrfs_block_rsv_add.start_transaction.btrfs_dirty_inode.btrfs_setattr
>        2.18 ± 12%     +18.4       20.61 ±  4%  perf-profile.calltrace.cycles-pp.start_transaction.btrfs_dirty_inode.btrfs_setattr.notify_change.chmod_common
>        0.00           +18.9       18.88 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_block_rsv_add.start_transaction.btrfs_dirty_inode.btrfs_setattr.notify_change
>        9.24 ± 10%     +32.2       41.40 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        9.16 ± 10%     +32.2       41.36 ±  3%  perf-profile.calltrace.cycles-pp.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        8.30 ±  9%     +32.6       40.94 ±  3%  perf-profile.calltrace.cycles-pp.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
>        8.09 ± 10%     +32.7       40.80 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_setattr.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64
>        7.74 ±  9%     +32.8       40.56 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_dirty_inode.btrfs_setattr.notify_change.chmod_common.__x64_sys_fchmod
>       47.72 ± 11%     -27.3       20.41 ±  3%  perf-profile.children.cycles-pp.do_faccessat
>       20.57 ± 11%     -12.1        8.51 ±  3%  perf-profile.children.cycles-pp.prepare_creds
>       18.81 ± 11%     -11.9        6.93 ±  3%  perf-profile.children.cycles-pp.put_cred_rcu
>        5.42 ± 12%      -3.8        1.60 ±  3%  perf-profile.children.cycles-pp.free_uid
>        5.39 ± 12%      -3.8        1.58 ±  3%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
>        5.36 ± 12%      -3.8        1.56 ±  3%  perf-profile.children.cycles-pp.refcount_dec_not_one
>        5.34 ± 11%      -3.1        2.21 ±  3%  perf-profile.children.cycles-pp.key_put
>        5.18 ± 11%      -2.4        2.75 ±  4%  perf-profile.children.cycles-pp.security_prepare_creds
>        4.77 ±  8%      -2.3        2.50 ±  4%  perf-profile.children.cycles-pp.btrfs_update_inode
>        3.80 ±  7%      -2.0        1.84 ±  4%  perf-profile.children.cycles-pp.btrfs_delayed_update_inode
>        3.54 ± 12%      -1.8        1.74 ±  4%  perf-profile.children.cycles-pp.apparmor_cred_prepare
>        3.49 ±  9%      -1.8        1.72 ±  6%  perf-profile.children.cycles-pp.security_cred_free
>        4.39 ± 11%      -1.7        2.68 ±  5%  perf-profile.children.cycles-pp.filename_lookup
>        3.93 ± 11%      -1.5        2.40 ±  5%  perf-profile.children.cycles-pp.path_lookupat
>        2.50 ±  9%      -1.4        1.09 ±  9%  perf-profile.children.cycles-pp.apparmor_cred_free
>        3.26 ± 11%      -1.3        1.96 ±  3%  perf-profile.children.cycles-pp.__do_sys_newfstatat
>        2.90 ± 10%      -1.1        1.75 ±  3%  perf-profile.children.cycles-pp.vfs_statx
>        2.47 ±  7%      -1.1        1.34 ±  3%  perf-profile.children.cycles-pp.__btrfs_release_delayed_node
>        1.55 ± 11%      -1.1        0.49 ±  7%  perf-profile.children.cycles-pp.join_transaction
>        2.37 ± 11%      -0.9        1.49 ±  6%  perf-profile.children.cycles-pp.link_path_walk
>        2.11 ± 10%      -0.9        1.23 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
>        2.13 ±  9%      -0.8        1.30 ±  4%  perf-profile.children.cycles-pp.user_path_at_empty
>        2.08 ±  9%      -0.8        1.29 ±  4%  perf-profile.children.cycles-pp.getname_flags
>        1.17 ± 12%      -0.7        0.50 ±  6%  perf-profile.children.cycles-pp.common_perm_cond
>        1.58 ± 14%      -0.7        0.92 ±  7%  perf-profile.children.cycles-pp.inode_permission
>        1.10 ± 12%      -0.6        0.46 ±  6%  perf-profile.children.cycles-pp.common_perm
>        1.58 ±  9%      -0.6        0.97 ±  3%  perf-profile.children.cycles-pp.__kmalloc
>        1.56 ± 11%      -0.6        0.99 ±  4%  perf-profile.children.cycles-pp.strncpy_from_user
>        1.29 ± 14%      -0.5        0.79 ±  5%  perf-profile.children.cycles-pp.kmem_cache_free
>        1.10 ± 13%      -0.5        0.62 ± 13%  perf-profile.children.cycles-pp.mutex_lock
>        0.59 ± 10%      -0.4        0.14 ± 16%  perf-profile.children.cycles-pp.btrfs_get_or_create_delayed_node
>        0.57 ± 10%      -0.4        0.14 ± 18%  perf-profile.children.cycles-pp.btrfs_get_delayed_node
>        0.83 ± 13%      -0.4        0.46 ±  4%  perf-profile.children.cycles-pp.generic_permission
>        0.91 ± 10%      -0.4        0.56 ±  6%  perf-profile.children.cycles-pp.walk_component
>        0.96 ±  7%      -0.4        0.61 ±  4%  perf-profile.children.cycles-pp.kfree
>        0.60 ± 14%      -0.3        0.26 ±  3%  perf-profile.children.cycles-pp.security_path_chmod
>        0.65 ± 10%      -0.3        0.31 ±  7%  perf-profile.children.cycles-pp.vfs_getattr
>        0.64 ± 10%      -0.3        0.30 ±  7%  perf-profile.children.cycles-pp.security_inode_getattr
>        0.77 ± 12%      -0.3        0.46 ±  6%  perf-profile.children.cycles-pp.btrfs_update_root_times
>        0.69 ±  5%      -0.3        0.39 ±  5%  perf-profile.children.cycles-pp.mutex_unlock
>        0.65 ± 12%      -0.3        0.39 ±  9%  perf-profile.children.cycles-pp.syscall_return_via_sysret
>        0.70 ±  9%      -0.3        0.44 ±  4%  perf-profile.children.cycles-pp.__check_object_size
>        0.62 ± 12%      -0.3        0.37 ±  3%  perf-profile.children.cycles-pp.complete_walk
>        0.51 ± 12%      -0.3        0.26 ±  5%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
>        0.46 ± 31%      -0.2        0.22 ± 16%  perf-profile.children.cycles-pp.revert_creds
>        0.59 ± 12%      -0.2        0.36 ±  3%  perf-profile.children.cycles-pp.unlazy_walk
>        0.48 ± 11%      -0.2        0.28 ±  5%  perf-profile.children.cycles-pp.obj_cgroup_charge
>        0.58 ± 10%      -0.2        0.38 ±  3%  perf-profile.children.cycles-pp.btrfs_getattr
>        0.52 ± 10%      -0.2        0.33 ±  9%  perf-profile.children.cycles-pp.lookup_fast
>        0.47 ± 20%      -0.2        0.29 ± 15%  perf-profile.children.cycles-pp.btrfs_permission
>        0.33 ± 12%      -0.2        0.15 ±  8%  perf-profile.children.cycles-pp.capable_wrt_inode_uidgid
>        0.46 ± 12%      -0.2        0.29        perf-profile.children.cycles-pp.__legitimize_path
>        0.32 ± 12%      -0.2        0.14 ± 10%  perf-profile.children.cycles-pp.security_capable
>        0.31 ± 12%      -0.2        0.13 ± 11%  perf-profile.children.cycles-pp.apparmor_capable
>        0.32 ± 12%      -0.2        0.15 ±  8%  perf-profile.children.cycles-pp.ns_capable_common
>        0.45 ±  9%      -0.2        0.28 ± 10%  perf-profile.children.cycles-pp.refill_obj_stock
>        0.28 ± 63%      -0.2        0.12 ± 27%  perf-profile.children.cycles-pp.map_id_up
>        0.35 ± 28%      -0.1        0.20 ± 10%  perf-profile.children.cycles-pp.cp_new_stat
>        0.42 ± 10%      -0.1        0.28 ±  6%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
>        0.36 ± 10%      -0.1        0.23 ±  8%  perf-profile.children.cycles-pp.path_put
>        0.35 ± 10%      -0.1        0.22 ± 11%  perf-profile.children.cycles-pp.dput
>        0.35 ±  9%      -0.1        0.22 ±  9%  perf-profile.children.cycles-pp.__d_lookup_rcu
>        0.32 ± 10%      -0.1        0.19 ± 12%  perf-profile.children.cycles-pp.path_init
>        0.36 ± 14%      -0.1        0.24 ±  5%  perf-profile.children.cycles-pp.__entry_text_start
>        0.35 ± 11%      -0.1        0.23 ±  9%  perf-profile.children.cycles-pp.___might_sleep
>        0.32 ±  9%      -0.1        0.20 ±  4%  perf-profile.children.cycles-pp.__check_heap_object
>        0.27 ± 10%      -0.1        0.16 ±  7%  perf-profile.children.cycles-pp.__list_add_valid
>        0.26 ±  7%      -0.1        0.16 ± 11%  perf-profile.children.cycles-pp.__might_sleep
>        0.23 ± 15%      -0.1        0.13 ±  5%  perf-profile.children.cycles-pp._cond_resched
>        0.19 ± 39%      -0.1        0.09 ±  7%  perf-profile.children.cycles-pp.fill_stack_inode_item
>        0.23 ± 14%      -0.1        0.14 ± 12%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
>        0.23 ± 14%      -0.1        0.15 ±  7%  perf-profile.children.cycles-pp.__legitimize_mnt
>        0.18 ± 14%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.override_creds
>        0.17 ± 15%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.__list_del_entry_valid
>        0.18 ± 13%      -0.1        0.10 ± 12%  perf-profile.children.cycles-pp.step_into
>        0.19 ± 14%      -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.lockref_get_not_dead
>        0.20 ± 10%      -0.1        0.13 ± 12%  perf-profile.children.cycles-pp.__mod_memcg_state
>        0.19 ± 16%      -0.1        0.12 ± 11%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
>        0.15 ± 14%      -0.1        0.09 ± 13%  perf-profile.children.cycles-pp.lockref_put_or_lock
>        0.17 ± 11%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.memset_erms
>        0.17 ± 17%      -0.1        0.12 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
>        0.16 ±  7%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp._copy_to_user
>        0.10 ± 21%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.mnt_want_write
>        0.11 ±  9%      -0.0        0.07 ± 13%  perf-profile.children.cycles-pp.inode_get_bytes
>        0.11 ± 18%      -0.0        0.07 ± 13%  perf-profile.children.cycles-pp.rcu_all_qs
>        0.11 ± 16%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.security_inode_permission
>        0.10            -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__fget_light
>        0.11 ± 13%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
>        0.12 ± 12%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
>        0.10 ± 19%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__put_cred
>        0.11 ± 14%      -0.0        0.07 ± 14%  perf-profile.children.cycles-pp.btrfs_balance_delayed_items
>        0.09 ± 14%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
>        0.10 ± 14%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
>        0.08 ± 10%      -0.0        0.06 ± 15%  perf-profile.children.cycles-pp.mntput_no_expire
>        0.21 ±  6%      +0.1        0.28 ± 12%  perf-profile.children.cycles-pp.btrfs_put_transaction
>        0.05 ± 61%      +0.1        0.15 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
>        0.00            +0.2        0.21 ± 10%  perf-profile.children.cycles-pp.find_next_bit
>        0.06 ± 17%      +0.3        0.38 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
>        0.00            +0.4        0.43 ± 14%  perf-profile.children.cycles-pp.btrfs_get_alloc_profile
>        0.00            +0.5        0.46 ± 14%  perf-profile.children.cycles-pp.calc_available_free_space
>        0.00            +0.5        0.48 ±  2%  perf-profile.children.cycles-pp.btrfs_block_rsv_add_bytes
>        0.00            +0.5        0.48 ±  8%  perf-profile.children.cycles-pp.wait_current_trans
>        0.00            +1.0        1.00 ±  4%  perf-profile.children.cycles-pp._find_next_bit
>        0.00            +1.8        1.77 ±  4%  perf-profile.children.cycles-pp.cpumask_next
>        0.00            +4.2        4.24 ±  4%  perf-profile.children.cycles-pp.__percpu_counter_sum
>        0.00           +16.0       16.04 ±  3%  perf-profile.children.cycles-pp.btrfs_block_rsv_release
>        0.00           +16.0       16.05 ±  3%  perf-profile.children.cycles-pp.btrfs_trans_release_metadata
>        0.64 ±  9%     +16.7       17.35 ±  3%  perf-profile.children.cycles-pp.__btrfs_end_transaction
>        0.00           +18.3       18.27 ±  4%  perf-profile.children.cycles-pp.__reserve_bytes
>        0.00           +18.4       18.39 ±  4%  perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes
>        2.19 ± 12%     +18.4       20.61 ±  4%  perf-profile.children.cycles-pp.start_transaction
>        0.00           +18.9       18.88 ±  4%  perf-profile.children.cycles-pp.btrfs_block_rsv_add
>        0.75 ±  8%     +27.1       27.88 ±  3%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
>        2.32 ± 10%     +28.3       30.66 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
>        9.25 ± 10%     +32.2       41.41 ±  3%  perf-profile.children.cycles-pp.__x64_sys_fchmod
>        9.16 ± 10%     +32.2       41.36 ±  3%  perf-profile.children.cycles-pp.chmod_common
>        8.31 ±  9%     +32.6       40.95 ±  3%  perf-profile.children.cycles-pp.notify_change
>        8.10 ± 10%     +32.7       40.80 ±  3%  perf-profile.children.cycles-pp.btrfs_setattr
>        7.75 ±  9%     +32.8       40.56 ±  4%  perf-profile.children.cycles-pp.btrfs_dirty_inode
>       13.73 ± 11%      -8.9        4.81 ±  4%  perf-profile.self.cycles-pp.prepare_creds
>        5.33 ± 12%      -3.8        1.55 ±  3%  perf-profile.self.cycles-pp.refcount_dec_not_one
>        4.57 ± 11%      -3.1        1.42 ±  4%  perf-profile.self.cycles-pp.put_cred_rcu
>        5.30 ± 11%      -3.1        2.19 ±  3%  perf-profile.self.cycles-pp.key_put
>        3.53 ± 12%      -1.8        1.73 ±  4%  perf-profile.self.cycles-pp.apparmor_cred_prepare
>        2.48 ±  9%      -1.4        1.08 ±  9%  perf-profile.self.cycles-pp.apparmor_cred_free
>        1.10 ± 12%      -0.6        0.45 ±  5%  perf-profile.self.cycles-pp.common_perm
>        1.04 ± 14%      -0.5        0.58 ± 13%  perf-profile.self.cycles-pp.mutex_lock
>        0.57 ± 11%      -0.4        0.13 ± 20%  perf-profile.self.cycles-pp.btrfs_get_delayed_node
>        1.06 ±  9%      -0.4        0.63 ±  5%  perf-profile.self.cycles-pp.kmem_cache_alloc
>        1.02 ± 10%      -0.4        0.64 ±  4%  perf-profile.self.cycles-pp.link_path_walk
>        0.93 ± 14%      -0.4        0.55 ±  5%  perf-profile.self.cycles-pp.kmem_cache_free
>        0.79 ±  8%      -0.3        0.48 ±  4%  perf-profile.self.cycles-pp.__kmalloc
>        0.69 ±  5%      -0.3        0.39 ±  6%  perf-profile.self.cycles-pp.mutex_unlock
>        0.72 ± 11%      -0.3        0.46 ±  5%  perf-profile.self.cycles-pp.strncpy_from_user
>        0.65 ± 12%      -0.3        0.39 ±  9%  perf-profile.self.cycles-pp.syscall_return_via_sysret
>        0.46 ± 31%      -0.2        0.22 ± 16%  perf-profile.self.cycles-pp.revert_creds
>        0.64 ±  9%      -0.2        0.40 ±  3%  perf-profile.self.cycles-pp.kfree
>        0.60 ±  9%      -0.2        0.38 ±  7%  perf-profile.self.cycles-pp.join_transaction
>        0.41 ± 11%      -0.2        0.22 ±  5%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
>        0.45 ± 10%      -0.2        0.27 ±  5%  perf-profile.self.cycles-pp.obj_cgroup_charge
>        0.30 ± 13%      -0.2        0.13 ± 11%  perf-profile.self.cycles-pp.apparmor_capable
>        0.44 ± 20%      -0.2        0.28 ± 16%  perf-profile.self.cycles-pp.btrfs_permission
>        0.42 ± 11%      -0.2        0.27 ±  8%  perf-profile.self.cycles-pp.generic_permission
>        0.42 ±  8%      -0.2        0.27 ± 10%  perf-profile.self.cycles-pp.refill_obj_stock
>        0.47 ± 10%      -0.2        0.32 ±  2%  perf-profile.self.cycles-pp.btrfs_getattr
>        0.27 ± 65%      -0.2        0.12 ± 25%  perf-profile.self.cycles-pp.map_id_up
>        0.36 ± 14%      -0.1        0.23 ±  4%  perf-profile.self.cycles-pp.__entry_text_start
>        0.34 ± 10%      -0.1        0.22 ±  9%  perf-profile.self.cycles-pp.__d_lookup_rcu
>        0.38 ± 11%      -0.1        0.25 ±  5%  perf-profile.self.cycles-pp.do_faccessat
>        0.32 ± 19%      -0.1        0.20 ± 13%  perf-profile.self.cycles-pp.inode_permission
>        0.34 ± 13%      -0.1        0.22 ±  9%  perf-profile.self.cycles-pp.___might_sleep
>        0.31 ±  9%      -0.1        0.20 ±  4%  perf-profile.self.cycles-pp.__check_heap_object
>        0.27 ± 10%      -0.1        0.16 ±  7%  perf-profile.self.cycles-pp.__list_add_valid
>        0.28 ± 11%      -0.1        0.17 ± 11%  perf-profile.self.cycles-pp.path_init
>        0.25 ±  8%      -0.1        0.15 ± 10%  perf-profile.self.cycles-pp.__might_sleep
>        0.26 ±  8%      -0.1        0.16 ±  7%  perf-profile.self.cycles-pp.__check_object_size
>        0.23 ± 16%      -0.1        0.13 ± 13%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
>        0.17 ± 16%      -0.1        0.09 ±  7%  perf-profile.self.cycles-pp.override_creds
>        0.22 ± 15%      -0.1        0.14 ±  8%  perf-profile.self.cycles-pp.__legitimize_mnt
>        0.20 ± 13%      -0.1        0.12 ± 12%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
>        0.14 ± 18%      -0.1        0.06        perf-profile.self.cycles-pp.btrfs_update_inode
>        0.21 ± 13%      -0.1        0.13 ±  5%  perf-profile.self.cycles-pp.walk_component
>        0.17 ± 11%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.step_into
>        0.17 ± 17%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.__list_del_entry_valid
>        0.19 ± 14%      -0.1        0.12 ± 12%  perf-profile.self.cycles-pp.lockref_get_not_dead
>        0.20 ± 10%      -0.1        0.13 ± 12%  perf-profile.self.cycles-pp.__mod_memcg_state
>        0.16 ± 10%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.lookup_fast
>        0.15 ± 14%      -0.1        0.08 ± 13%  perf-profile.self.cycles-pp.lockref_put_or_lock
>        0.20 ± 10%      -0.1        0.14 ±  5%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
>        0.15 ± 14%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.filename_lookup
>        0.17 ±  9%      -0.1        0.11 ±  3%  perf-profile.self.cycles-pp.memset_erms
>        0.14 ± 12%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.getname_flags
>        0.07 ± 11%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.dput
>        0.11 ± 16%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.security_inode_permission
>        0.15 ± 15%      -0.0        0.10 ± 14%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
>        0.12 ± 12%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
>        0.11 ± 11%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp._cond_resched
>        0.11 ± 13%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
>        0.10 ±  5%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.__fget_light
>        0.10 ± 19%      -0.0        0.07 ± 13%  perf-profile.self.cycles-pp.path_lookupat
>        0.11 ± 14%      -0.0        0.07 ± 20%  perf-profile.self.cycles-pp.btrfs_balance_delayed_items
>        0.10 ± 14%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.__virt_addr_valid
>        0.10 ± 21%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.__put_cred
>        0.07 ± 11%      -0.0        0.04 ± 60%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
>        0.09 ± 20%      -0.0        0.06 ± 17%  perf-profile.self.cycles-pp.notify_change
>        0.07 ± 12%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.mntput_no_expire
>        0.21 ±  6%      +0.1        0.28 ± 12%  perf-profile.self.cycles-pp.btrfs_put_transaction
>        0.05 ± 62%      +0.1        0.13 ± 11%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
>        0.00            +0.1        0.12 ±  5%  perf-profile.self.cycles-pp.btrfs_reserve_metadata_bytes
>        0.00            +0.1        0.14 ± 13%  perf-profile.self.cycles-pp.btrfs_block_rsv_release
>        0.00            +0.2        0.16 ± 18%  perf-profile.self.cycles-pp.btrfs_get_alloc_profile
>        0.00            +0.2        0.18 ±  7%  perf-profile.self.cycles-pp.wait_current_trans
>        0.46 ± 15%      +0.2        0.64 ±  6%  perf-profile.self.cycles-pp.start_transaction
>        0.00            +0.2        0.21 ± 10%  perf-profile.self.cycles-pp.find_next_bit
>        0.06 ± 20%      +0.3        0.38 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
>        0.00            +0.6        0.57 ± 14%  perf-profile.self.cycles-pp.__reserve_bytes
>        0.00            +0.6        0.57 ±  7%  perf-profile.self.cycles-pp.cpumask_next
>        0.36 ± 12%      +0.6        0.98 ± 11%  perf-profile.self.cycles-pp.__btrfs_end_transaction
>        0.00            +1.0        0.99 ±  4%  perf-profile.self.cycles-pp._find_next_bit
>        1.57 ± 11%      +1.2        2.77 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock
>        0.00            +2.1        2.08 ±  4%  perf-profile.self.cycles-pp.__percpu_counter_sum
>        0.73 ±  8%     +27.1       27.79 ±  3%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>
>                                   stress-ng.access.ops
>
>    1.2e+06 +-----------------------------------------------------------------+
>            |     +..+..+..+..  .+..+..+.         .+..  .+..+..+..+..+..+..+  |
>      1e+06 |-+   :           +.                +.    +.                      |
>            |     :                                                           |
>            |    :                                                            |
>     800000 |-+  :                                                            |
>            |    :               O  O  O           O  O  O  O  O  O  O  O  O  |
>     600000 |-+O :O  O  O  O  O           O  O  O                             |
>            |   :                                                             |
>     400000 |-+ :                                                             |
>            |   :                                                             |
>            |   :                                                             |
>     200000 |-+:                                                              |
>            |  :                                                              |
>          0 +-----------------------------------------------------------------+
>
>
>                              stress-ng.access.ops_per_sec
>
>    40000 +-------------------------------------------------------------------+
>          |     +..+..+..+...  .+..+..+.         .+..  .+..+...+..+..+..+..+  |
>    35000 |-+   :            +.                +.    +.                       |
>    30000 |-+   :                                                             |
>          |    :                                                              |
>    25000 |-+  :                                                              |
>          |    :                O  O  O           O  O  O  O   O  O  O  O  O  |
>    20000 |-+O :O  O  O  O   O           O  O  O                              |
>          |   :                                                               |
>    15000 |-+ :                                                               |
>    10000 |-+ :                                                               |
>          |   :                                                               |
>     5000 |-+:                                                                |
>          |  :                                                                |
>        0 +-------------------------------------------------------------------+
>
>
>                                   stress-ng.chmod.ops
>
>    140000 +------------------------------------------------------------------+
>           |     +..                                                          |
>    120000 |-+   :        +..+..  .+..                                        |
>           |     :  +.. ..      +.    +..          +..   +..   +..            |
>    100000 |-+  :      +                   ..+.. ..    ..    ..   +..         |
>           |    :                        +.     +     +     +        +..+..+  |
>     80000 |-+  :                                                             |
>           |    :                                                             |
>     60000 |-+ :                                      O  O           O        |
>           |  O: O  O  O  O  O  O  O  O  O   O  O  O        O  O  O     O  O  |
>     40000 |-+ :                                                              |
>           |   :                                                              |
>     20000 |-+:                                                               |
>           |  :                                                               |
>         0 +------------------------------------------------------------------+
>
>
>                              stress-ng.chmod.ops_per_sec
>
>    4500 +--------------------------------------------------------------------+
>         |     +..       +..     .+..                                         |
>    4000 |-+   :        +   +..+.    +..                      +..             |
>    3500 |-+   :  +... +                         .+..  .+..  +                |
>         |    :       +                 +...+..+.    +.     +    +...  .+..+  |
>    3000 |-+  :                                            +         +.       |
>    2500 |-+  :                                                               |
>         |    :                                                               |
>    2000 |-+ :                 O  O  O            O  O  O            O        |
>    1500 |-+O: O  O   O  O  O           O   O  O           O  O  O      O  O  |
>         |   :                                                                |
>    1000 |-+ :                                                                |
>     500 |-+:                                                                 |
>         |  :                                                                 |
>       0 +--------------------------------------------------------------------+
>
>
>                                    stress-ng.chown.ops
>
>    1.2e+06 +-----------------------------------------------------------------+
>            |     +..+..+..  .+..            +..                              |
>      1e+06 |-+   :        +.    +..  .+.. ..              .+..+..  .+..      |
>            |     :                 +.    +     +..+..+..+.       +.      .+  |
>            |    :                                                      +.    |
>     800000 |-+  :                                                            |
>            |    :                                                            |
>     600000 |-+O :O              O  O  O           O  O     O           O     |
>            |   :    O  O  O  O           O  O  O        O     O  O  O     O  |
>     400000 |-+ :                                                             |
>            |   :                                                             |
>            |   :                                                             |
>     200000 |-+:                                                              |
>            |  :                                                              |
>          0 +-----------------------------------------------------------------+
>
>
>                               stress-ng.chown.ops_per_sec
>
>    40000 +-------------------------------------------------------------------+
>          |     +..+..+..  ..+..            +..                               |
>    35000 |-+   :        +.     +..  .+.. ..              .+...+..  .+..      |
>    30000 |-+   :                  +.    +     +..+..+..+.        +.      .+  |
>          |    :                                                        +.    |
>    25000 |-+  :                                                              |
>          |    :                                                              |
>    20000 |-+O :O               O  O  O           O  O     O            O     |
>          |   :    O  O  O   O           O  O  O        O      O  O  O     O  |
>    15000 |-+ :                                                               |
>    10000 |-+ :                                                               |
>          |   :                                                               |
>     5000 |-+:                                                                |
>          |  :                                                                |
>        0 +-------------------------------------------------------------------+
>
>
>                                    stress-ng.utime.ops
>
>    4.5e+06 +-----------------------------------------------------------------+
>            |          .+..+..  .+..  .+..  .+..                              |
>      4e+06 |-+   +..+.       +.    +.    +.               .+..+..+..+..  .+  |
>    3.5e+06 |-+   :                             +..+..+..+.             +.    |
>            |     :                                                           |
>      3e+06 |-+  :                                                            |
>    2.5e+06 |-+  :                                                            |
>            |    :                                                            |
>      2e+06 |-+  :                                                            |
>    1.5e+06 |-+ :                                                             |
>            |   :                O  O  O           O  O                       |
>      1e+06 |-+O: O  O  O  O  O           O  O  O        O  O  O  O  O  O  O  |
>     500000 |-+ :                                                             |
>            |  :                                                              |
>          0 +-----------------------------------------------------------------+
>
>
>                               stress-ng.utime.ops_per_sec
>
>    160000 +------------------------------------------------------------------+
>           |          .+..      +..                                           |
>    140000 |-+     .+.    +.. ..     .+..  ..+..                              |
>    120000 |-+   +.          +     +.    +.       .+..  .+..+..+..+..+..  .+  |
>           |     :                              +.    +.                +.    |
>    100000 |-+  :                                                             |
>           |    :                                                             |
>     80000 |-+  :                                                             |
>           |    :                                                             |
>     60000 |-+ :                                                              |
>     40000 |-+ :                                                              |
>           |  O: O  O  O  O  O  O  O  O  O   O  O  O  O  O  O  O  O  O  O  O  |
>     20000 |-+ :                                                              |
>           |  :                                                               |
>         0 +------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad  sample
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Oliver Sang