Re: Re: [PATCHSET wq/for-6.8] workqueue: Implement system-wide max_active for unbound workqueues
From: Naohiro Aota
Date: Mon Jan 15 2024 - 00:46:33 EST
On Thu, Jan 11, 2024 at 02:49:21PM -1000, Tejun Heo wrote:
> Hello,
>
> On Fri, Jan 05, 2024 at 02:44:08AM +0000, Naohiro Aota wrote:
> > Thank you for the series. I applied the patches on btrfs's development tree
> > below, and ran the benchmark.
> >
> > https://gitlab.com/kdave/btrfs-devel.git misc-next
> >
> > - misc-next, numa=off (baseline)
> > WRITE: bw=1117MiB/s (1171MB/s), 1117MiB/s-1117MiB/s (1171MB/s-1171MB/s), io=332GiB (356GB), run=304322-304322msec
> > - misc-next + wq patches, numa=off
> > WRITE: bw=1866MiB/s (1957MB/s), 1866MiB/s-1866MiB/s (1957MB/s-1957MB/s), io=684GiB (735GB), run=375472-375472msec
> >
> > So, the patches surely improved the performance. However, as show below, it
> > is still lower than reverting previous workqueue patches. The reverting is
> > done by reverse applying output of "git diff 4cbfd3de737b
> > kernel/workqueue.c kernel/workqueue_internal.h include/linux/workqueue*
> > init/main.c"
> >
> > - misc-next + wq reverted, numa=off
> > WRITE: bw=2472MiB/s (2592MB/s), 2472MiB/s-2472MiB/s (2592MB/s-2592MB/s), io=732GiB (786GB), run=303257-303257msec
>
> Can you describe the test setup in detail? What kind of machine is it? What
> do you mean by `numa=off`? Can you report tools/workqueue/wq_dump.py output?
The testing machines' hardware is as follows:
CPU: Intel(R) Xeon(R) Platinum 8260 CPU, 96 cores
NUMA nodes: 2
RAM: 1024 GB
However, for another benchmark experiment I'm doing, I booted the machine
with "numa=off mem=16G" kernel command-line. I admit this is an unusual
setup...
On that machine, I create a fresh btrfs with "mkfs.btrfs -d raid0 -m raid0
<devices>" with 6 SSD devices. And, I run the following command on the FS.
fio --group_reporting --eta=always --eta-interval=30s --eta-newline=30s \
--rw=write --fallocate=none \
--direct=1 --ioengine=libaio --iodepth=32 \
--filesize=100G \
--blocksize=64k \
--time_based --runtime=300s \
--end_fsync=1 \
--directory=${MNT} \
--name=writer --numjobs=32
tools/workqueue/wq_dump.py output is pasted at the
bottom. "btrfs-endio-write" is the workqueue, which had many workers on the
unpatched kernel.
FYI, without the kernel command-line (i.e, numa=on and all RAM available as
usual), as shown below, your patch series (v1) improved the performance
significantly. It is even better than the reverted case.
- misc-next, numa=on
WRITE: bw=1121MiB/s (1175MB/s), 1121MiB/s-1121MiB/s (1175MB/s-1175MB/s), io=332GiB (356GB), run=303030-303030msec
- misc-next+wq patches, numa=on
WRITE: bw=2185MiB/s (2291MB/s), 2185MiB/s-2185MiB/s (2291MB/s-2291MB/s), io=667GiB (717GB), run=312806-312806msec
- misc-next+wq reverted, numa=on
WRITE: bw=1557MiB/s (1633MB/s), 1557MiB/s-1557MiB/s (1633MB/s-1633MB/s), io=659GiB (708GB), run=433426-433426msec
> Thanks.
>
> --
> tejun
Affinity Scopes
===============
wq_unbound_cpumask=0xffffffff 0xffffffff ffffffff
CPU
nr_pods 96
pod_cpus [0]=00000001 [1]=00000002 [2]=00000004 [3]=00000008 [4]=00000010 [5]=00000020 [6]=00000040 [7]=00000080 [8]=00000100 [9]=00000200 [10]=00000400 [11]=00000800 [12]=00001000 [13]=00002000 [14]=00004000 [15]=00008000 [16]=00010000 [17]=00020000 [18]=00040000 [19]=00080000 [20]=00100000 [21]=00200000 [22]=00400000 [23]=00800000 [24]=01000000 [25]=02000000 [26]=04000000 [27]=08000000 [28]=10000000 [29]=20000000 [30]=40000000 [31]=80000000 [32]=0x0 00000001 [33]=0x0 00000002 [34]=0x0 00000004 [35]=0x0 00000008 [36]=0x0 00000010 [37]=0x0 00000020 [38]=0x0 00000040 [39]=0x0 00000080 [40]=0x0 00000100 [41]=0x0 00000200 [42]=0x0 00000400 [43]=0x0 00000800 [44]=0x0 00001000 [45]=0x0 00002000 [46]=0x0 00004000 [47]=0x0 00008000 [48]=0x0 00010000 [49]=0x0 00020000 [50]=0x0 00040000 [51]=0x0 00080000 [52]=0x0 00100000 [53]=0x0 00200000 [54]=0x0 00400000 [55]=0x0 00800000 [56]=0x0 01000000 [57]=0x0 02000000 [58]=0x0 04000000 [59]=0x0 08000000 [60]=0x0 10000000 [61]=0x0 20000000 [62]=0x0 40000000 [63]=0x0 80000000 [64]=0x0 0x0 00000001 [65]=0x0 0x0 00000002 [66]=0x0 0x0 00000004 [67]=0x0 0x0 00000008 [68]=0x0 0x0 00000010 [69]=0x0 0x0 00000020 [70]=0x0 0x0 00000040 [71]=0x0 0x0 00000080 [72]=0x0 0x0 00000100 [73]=0x0 0x0 00000200 [74]=0x0 0x0 00000400 [75]=0x0 0x0 00000800 [76]=0x0 0x0 00001000 [77]=0x0 0x0 00002000 [78]=0x0 0x0 00004000 [79]=0x0 0x0 00008000 [80]=0x0 0x0 00010000 [81]=0x0 0x0 00020000 [82]=0x0 0x0 00040000 [83]=0x0 0x0 00080000 [84]=0x0 0x0 00100000 [85]=0x0 0x0 00200000 [86]=0x0 0x0 00400000 [87]=0x0 0x0 00800000 [88]=0x0 0x0 01000000 [89]=0x0 0x0 02000000 [90]=0x0 0x0 04000000 [91]=0x0 0x0 08000000 [92]=0x0 0x0 10000000 [93]=0x0 0x0 20000000 [94]=0x0 0x0 40000000 [95]=0x0 0x0 80000000
pod_node [0]=0 [1]=0 [2]=0 [3]=0 [4]=0 [5]=0 [6]=0 [7]=0 [8]=0 [9]=0 [10]=0 [11]=0 [12]=0 [13]=0 [14]=0 [15]=0 [16]=0 [17]=0 [18]=0 [19]=0 [20]=0 [21]=0 [22]=0 [23]=0 [24]=0 [25]=0 [26]=0 [27]=0 [28]=0 [29]=0 [30]=0 [31]=0 [32]=0 [33]=0 [34]=0 [35]=0 [36]=0 [37]=0 [38]=0 [39]=0 [40]=0 [41]=0 [42]=0 [43]=0 [44]=0 [45]=0 [46]=0 [47]=0 [48]=0 [49]=0 [50]=0 [51]=0 [52]=0 [53]=0 [54]=0 [55]=0 [56]=0 [57]=0 [58]=0 [59]=0 [60]=0 [61]=0 [62]=0 [63]=0 [64]=0 [65]=0 [66]=0 [67]=0 [68]=0 [69]=0 [70]=0 [71]=0 [72]=0 [73]=0 [74]=0 [75]=0 [76]=0 [77]=0 [78]=0 [79]=0 [80]=0 [81]=0 [82]=0 [83]=0 [84]=0 [85]=0 [86]=0 [87]=0 [88]=0 [89]=0 [90]=0 [91]=0 [92]=0 [93]=0 [94]=0 [95]=0
cpu_pod [0]=0 [1]=1 [2]=2 [3]=3 [4]=4 [5]=5 [6]=6 [7]=7 [8]=8 [9]=9 [10]=10 [11]=11 [12]=12 [13]=13 [14]=14 [15]=15 [16]=16 [17]=17 [18]=18 [19]=19 [20]=20 [21]=21 [22]=22 [23]=23 [24]=24 [25]=25 [26]=26 [27]=27 [28]=28 [29]=29 [30]=30 [31]=31 [32]=32 [33]=33 [34]=34 [35]=35 [36]=36 [37]=37 [38]=38 [39]=39 [40]=40 [41]=41 [42]=42 [43]=43 [44]=44 [45]=45 [46]=46 [47]=47 [48]=48 [49]=49 [50]=50 [51]=51 [52]=52 [53]=53 [54]=54 [55]=55 [56]=56 [57]=57 [58]=58 [59]=59 [60]=60 [61]=61 [62]=62 [63]=63 [64]=64 [65]=65 [66]=66 [67]=67 [68]=68 [69]=69 [70]=70 [71]=71 [72]=72 [73]=73 [74]=74 [75]=75 [76]=76 [77]=77 [78]=78 [79]=79 [80]=80 [81]=81 [82]=82 [83]=83 [84]=84 [85]=85 [86]=86 [87]=87 [88]=88 [89]=89 [90]=90 [91]=91 [92]=92 [93]=93 [94]=94 [95]=95
SMT
nr_pods 48
pod_cpus [0]=0x1 00010000 [1]=0x2 00020000 [2]=0x4 00040000 [3]=0x8 00080000 [4]=0x10 00100000 [5]=0x20 00200000 [6]=0x40 00400000 [7]=0x80 00800000 [8]=0x100 01000000 [9]=0x200 02000000 [10]=0x400 04000000 [11]=0x800 08000000 [12]=0x1000 10000000 [13]=0x2000 20000000 [14]=0x4000 40000000 [15]=0x8000 80000000 [16]=0x10000 0x0 00000001 [17]=0x20000 0x0 00000002 [18]=0x40000 0x0 00000004 [19]=0x80000 0x0 00000008 [20]=0x100000 0x0 00000010 [21]=0x200000 0x0 00000020 [22]=0x400000 0x0 00000040 [23]=0x800000 0x0 00000080 [24]=0x1000000 0x0 00000100 [25]=0x2000000 0x0 00000200 [26]=0x4000000 0x0 00000400 [27]=0x8000000 0x0 00000800 [28]=0x10000000 0x0 00001000 [29]=0x20000000 0x0 00002000 [30]=0x40000000 0x0 00004000 [31]=0x80000000 0x0 00008000 [32]=0x0 0x1 00010000 [33]=0x0 0x2 00020000 [34]=0x0 0x4 00040000 [35]=0x0 0x8 00080000 [36]=0x0 0x10 00100000 [37]=0x0 0x20 00200000 [38]=0x0 0x40 00400000 [39]=0x0 0x80 00800000 [40]=0x0 0x100 01000000 [41]=0x0 0x200 02000000 [42]=0x0 0x400 04000000 [43]=0x0 0x800 08000000 [44]=0x0 0x1000 10000000 [45]=0x0 0x2000 20000000 [46]=0x0 0x4000 40000000 [47]=0x0 0x8000 80000000
pod_node [0]=0 [1]=0 [2]=0 [3]=0 [4]=0 [5]=0 [6]=0 [7]=0 [8]=0 [9]=0 [10]=0 [11]=0 [12]=0 [13]=0 [14]=0 [15]=0 [16]=0 [17]=0 [18]=0 [19]=0 [20]=0 [21]=0 [22]=0 [23]=0 [24]=0 [25]=0 [26]=0 [27]=0 [28]=0 [29]=0 [30]=0 [31]=0 [32]=0 [33]=0 [34]=0 [35]=0 [36]=0 [37]=0 [38]=0 [39]=0 [40]=0 [41]=0 [42]=0 [43]=0 [44]=0 [45]=0 [46]=0 [47]=0
cpu_pod [0]=0 [1]=1 [2]=2 [3]=3 [4]=4 [5]=5 [6]=6 [7]=7 [8]=8 [9]=9 [10]=10 [11]=11 [12]=12 [13]=13 [14]=14 [15]=15 [16]=16 [17]=17 [18]=18 [19]=19 [20]=20 [21]=21 [22]=22 [23]=23 [24]=24 [25]=25 [26]=26 [27]=27 [28]=28 [29]=29 [30]=30 [31]=31 [32]=32 [33]=33 [34]=34 [35]=35 [36]=36 [37]=37 [38]=38 [39]=39 [40]=40 [41]=41 [42]=42 [43]=43 [44]=44 [45]=45 [46]=46 [47]=47 [48]=0 [49]=1 [50]=2 [51]=3 [52]=4 [53]=5 [54]=6 [55]=7 [56]=8 [57]=9 [58]=10 [59]=11 [60]=12 [61]=13 [62]=14 [63]=15 [64]=16 [65]=17 [66]=18 [67]=19 [68]=20 [69]=21 [70]=22 [71]=23 [72]=24 [73]=25 [74]=26 [75]=27 [76]=28 [77]=29 [78]=30 [79]=31 [80]=32 [81]=33 [82]=34 [83]=35 [84]=36 [85]=37 [86]=38 [87]=39 [88]=40 [89]=41 [90]=42 [91]=43 [92]=44 [93]=45 [94]=46 [95]=47
CACHE (default)
nr_pods 2
pod_cpus [0]=0xffffff 0xffff0000 000000ff [1]=0xff000000 0xffff ffffff00
pod_node [0]=0 [1]=0
cpu_pod [0]=0 [1]=0 [2]=0 [3]=0 [4]=0 [5]=0 [6]=0 [7]=0 [8]=0 [9]=0 [10]=0 [11]=0 [12]=0 [13]=0 [14]=0 [15]=0 [16]=0 [17]=0 [18]=0 [19]=0 [20]=0 [21]=0 [22]=0 [23]=0 [24]=1 [25]=1 [26]=1 [27]=1 [28]=1 [29]=1 [30]=1 [31]=1 [32]=1 [33]=1 [34]=1 [35]=1 [36]=1 [37]=1 [38]=1 [39]=1 [40]=1 [41]=1 [42]=1 [43]=1 [44]=1 [45]=1 [46]=1 [47]=1 [48]=0 [49]=0 [50]=0 [51]=0 [52]=0 [53]=0 [54]=0 [55]=0 [56]=0 [57]=0 [58]=0 [59]=0 [60]=0 [61]=0 [62]=0 [63]=0 [64]=0 [65]=0 [66]=0 [67]=0 [68]=0 [69]=0 [70]=0 [71]=0 [72]=1 [73]=1 [74]=1 [75]=1 [76]=1 [77]=1 [78]=1 [79]=1 [80]=1 [81]=1 [82]=1 [83]=1 [84]=1 [85]=1 [86]=1 [87]=1 [88]=1 [89]=1 [90]=1 [91]=1 [92]=1 [93]=1 [94]=1 [95]=1
NUMA
nr_pods 1
pod_cpus [0]=0xffffffff 0xffffffff ffffffff
pod_node [0]=0
cpu_pod [0]=0 [1]=0 [2]=0 [3]=0 [4]=0 [5]=0 [6]=0 [7]=0 [8]=0 [9]=0 [10]=0 [11]=0 [12]=0 [13]=0 [14]=0 [15]=0 [16]=0 [17]=0 [18]=0 [19]=0 [20]=0 [21]=0 [22]=0 [23]=0 [24]=0 [25]=0 [26]=0 [27]=0 [28]=0 [29]=0 [30]=0 [31]=0 [32]=0 [33]=0 [34]=0 [35]=0 [36]=0 [37]=0 [38]=0 [39]=0 [40]=0 [41]=0 [42]=0 [43]=0 [44]=0 [45]=0 [46]=0 [47]=0 [48]=0 [49]=0 [50]=0 [51]=0 [52]=0 [53]=0 [54]=0 [55]=0 [56]=0 [57]=0 [58]=0 [59]=0 [60]=0 [61]=0 [62]=0 [63]=0 [64]=0 [65]=0 [66]=0 [67]=0 [68]=0 [69]=0 [70]=0 [71]=0 [72]=0 [73]=0 [74]=0 [75]=0 [76]=0 [77]=0 [78]=0 [79]=0 [80]=0 [81]=0 [82]=0 [83]=0 [84]=0 [85]=0 [86]=0 [87]=0 [88]=0 [89]=0 [90]=0 [91]=0 [92]=0 [93]=0 [94]=0 [95]=0
SYSTEM
nr_pods 1
pod_cpus [0]=0xffffffff 0xffffffff ffffffff
pod_node [0]=-1
cpu_pod [0]=0 [1]=0 [2]=0 [3]=0 [4]=0 [5]=0 [6]=0 [7]=0 [8]=0 [9]=0 [10]=0 [11]=0 [12]=0 [13]=0 [14]=0 [15]=0 [16]=0 [17]=0 [18]=0 [19]=0 [20]=0 [21]=0 [22]=0 [23]=0 [24]=0 [25]=0 [26]=0 [27]=0 [28]=0 [29]=0 [30]=0 [31]=0 [32]=0 [33]=0 [34]=0 [35]=0 [36]=0 [37]=0 [38]=0 [39]=0 [40]=0 [41]=0 [42]=0 [43]=0 [44]=0 [45]=0 [46]=0 [47]=0 [48]=0 [49]=0 [50]=0 [51]=0 [52]=0 [53]=0 [54]=0 [55]=0 [56]=0 [57]=0 [58]=0 [59]=0 [60]=0 [61]=0 [62]=0 [63]=0 [64]=0 [65]=0 [66]=0 [67]=0 [68]=0 [69]=0 [70]=0 [71]=0 [72]=0 [73]=0 [74]=0 [75]=0 [76]=0 [77]=0 [78]=0 [79]=0 [80]=0 [81]=0 [82]=0 [83]=0 [84]=0 [85]=0 [86]=0 [87]=0 [88]=0 [89]=0 [90]=0 [91]=0 [92]=0 [93]=0 [94]=0 [95]=0
Worker Pools
============
pool[000] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 0
pool[001] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 0
pool[002] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 1
pool[003] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 1
pool[004] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 2
pool[005] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 2
pool[006] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 3
pool[007] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 3
pool[008] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 4
pool[009] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 4
pool[010] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 5
pool[011] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 5
pool[012] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 6
pool[013] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 6
pool[014] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 7
pool[015] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 7
pool[016] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 8
pool[017] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 8
pool[018] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 9
pool[019] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 9
pool[020] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 10
pool[021] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 10
pool[022] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 11
pool[023] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 11
pool[024] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 12
pool[025] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 12
pool[026] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 13
pool[027] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 13
pool[028] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 14
pool[029] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 14
pool[030] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 15
pool[031] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 15
pool[032] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 16
pool[033] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 16
pool[034] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 17
pool[035] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 17
pool[036] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 18
pool[037] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 18
pool[038] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 19
pool[039] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 19
pool[040] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 20
pool[041] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 20
pool[042] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 21
pool[043] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 21
pool[044] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 22
pool[045] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 22
pool[046] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 23
pool[047] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 23
pool[048] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 24
pool[049] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 24
pool[050] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 25
pool[051] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 25
pool[052] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 26
pool[053] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 26
pool[054] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 27
pool[055] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 27
pool[056] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 28
pool[057] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 28
pool[058] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 29
pool[059] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 29
pool[060] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 30
pool[061] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 30
pool[062] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 31
pool[063] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 31
pool[064] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 32
pool[065] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 32
pool[066] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 33
pool[067] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 33
pool[068] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 34
pool[069] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 34
pool[070] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 35
pool[071] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 35
pool[072] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 36
pool[073] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 36
pool[074] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 37
pool[075] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 37
pool[076] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 38
pool[077] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 38
pool[078] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 39
pool[079] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 39
pool[080] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 40
pool[081] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 40
pool[082] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 41
pool[083] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 41
pool[084] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 42
pool[085] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 42
pool[086] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 43
pool[087] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 43
pool[088] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 44
pool[089] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 44
pool[090] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 45
pool[091] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 45
pool[092] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 46
pool[093] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 46
pool[094] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 47
pool[095] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 47
pool[096] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 48
pool[097] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 48
pool[098] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 49
pool[099] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 49
pool[100] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 50
pool[101] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 50
pool[102] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 51
pool[103] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 51
pool[104] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 52
pool[105] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 52
pool[106] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 53
pool[107] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 53
pool[108] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 54
pool[109] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 54
pool[110] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 55
pool[111] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 55
pool[112] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 56
pool[113] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 56
pool[114] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 57
pool[115] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 57
pool[116] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 58
pool[117] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 58
pool[118] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 59
pool[119] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 59
pool[120] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 60
pool[121] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 60
pool[122] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 61
pool[123] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 61
pool[124] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 62
pool[125] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 62
pool[126] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 63
pool[127] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 63
pool[128] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 64
pool[129] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 64
pool[130] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 65
pool[131] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 65
pool[132] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 66
pool[133] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 66
pool[134] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 67
pool[135] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 67
pool[136] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 68
pool[137] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 68
pool[138] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 69
pool[139] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 69
pool[140] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 70
pool[141] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 70
pool[142] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 71
pool[143] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 71
pool[144] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 72
pool[145] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 72
pool[146] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 73
pool[147] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 73
pool[148] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 74
pool[149] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 74
pool[150] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 75
pool[151] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 75
pool[152] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 76
pool[153] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 76
pool[154] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 77
pool[155] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 77
pool[156] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 78
pool[157] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 78
pool[158] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 79
pool[159] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 79
pool[160] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 80
pool[161] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 80
pool[162] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 81
pool[163] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 81
pool[164] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 82
pool[165] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 82
pool[166] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 83
pool[167] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 83
pool[168] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 84
pool[169] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 84
pool[170] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 85
pool[171] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 85
pool[172] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 86
pool[173] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 86
pool[174] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 87
pool[175] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 87
pool[176] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 88
pool[177] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 88
pool[178] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 89
pool[179] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 89
pool[180] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 90
pool[181] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 90
pool[182] ref= 1 nice= 0 idle/workers= 4/ 4 cpu= 91
pool[183] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 91
pool[184] ref= 1 nice= 0 idle/workers= 4/ 4 cpu= 92
pool[185] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 92
pool[186] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 93
pool[187] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 93
pool[188] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 94
pool[189] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 94
pool[190] ref= 1 nice= 0 idle/workers= 2/ 2 cpu= 95
pool[191] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 95
pool[192] ref= 78 nice= 0 idle/workers= 2/ 2 cpus=0xffffffff 0xffffffff ffffffff pod_cpus=0xffffffff 0xffffffff ffffffff
pool[193] ref=3744 nice= 0 idle/workers= 8/ 10 cpus=0xffffffff 0xffffffff ffffffff pod_cpus=0xffffff 0xffff0000 000000ff
pool[194] ref=3744 nice= 0 idle/workers= 5/ 9 cpus=0xffffffff 0xffffffff ffffffff pod_cpus=0xff000000 0xffff ffffff00
pool[195] ref= 7 nice=-20 idle/workers= 1/ 1 cpus=0xffffffff 0xffffffff ffffffff pod_cpus=0xffffffff 0xffffffff ffffffff
pool[196] ref= 336 nice=-20 idle/workers= 1/ 1 cpus=0xffffffff 0xffffffff ffffffff pod_cpus=0xffffff 0xffff0000 000000ff
pool[197] ref= 336 nice=-20 idle/workers= 2/ 2 cpus=0xffffffff 0xffffffff ffffffff pod_cpus=0xff000000 0xffff ffffff00
Workqueue CPU -> pool
=====================
[ workqueue \ type CPU 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 dfl]
events percpu 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 146 148 150 152 154 156 158 160 162 164 166 168 170 172 174 176 178 180 182 184 186 188 190
events_highpri percpu 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99 101 103 105 107 109 111 113 115 117 119 121 123 125 127 129 131 133 135 137 139 141 143 145 147 149 151 153 155 157 159 161 163 165 167 169 171 173 175 177 179 181 183 185 187 189 191
events_long percpu 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 146 148 150 152 154 156 158 160 162 164 166 168 170 172 174 176 178 180 182 184 186 188 190
events_unbound unbound 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
events_freezable percpu 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 146 148 150 152 154 156 158 160 162 164 166 168 170 172 174 176 178 180 182 184 186 188 190
events_power_efficient percpu 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 146 148 150 152 154 156 158 160 162 164 166 168 170 172 174 176 178 180 182 184 186 188 190
events_freezable_power_ percpu 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 146 148 150 152 154 156 158 160 162 164 166 168 170 172 174 176 178 180 182 184 186 188 190
(snip)
btrfs-worker unbound 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-delalloc unbound 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-flush_delalloc ordered 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-cache ordered 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-fixup ordered 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-endio unbound 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-endio-meta unbound 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-rmw unbound 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-endio-write unbound 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-compressed-write unbound 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-freespace-write ordered 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-delayed-meta ordered 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs-qgroup-rescan ordered 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
btrfs_discard ordered 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 193 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 194 192
dio/sdbj percpu 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 146 148 150 152 154 156 158 160 162 164 166 168 170 172 174 176 178 180 182 184 186 188 190