Re: [PATCH v1 1/4] perf vendor events intel: Alderlake/rocketlake metric fixes
From: Ian Rogers
Date: Thu Jan 04 2024 - 12:52:06 EST
On Thu, Jan 4, 2024 at 6:30 AM Liang, Kan <kan.liang@xxxxxxxxxxxxxxx> wrote:
>
>
>
> On 2024-01-04 8:56 a.m., Ian Rogers wrote:
> >> Testing tma_slow_pause
> >> Metric 'tma_slow_pause' not printed in:
> >> # Running 'internals/synthesize' benchmark:
> >> Computing performance of single threaded perf event synthesis by
> >> synthesizing events on the perf process itself:
> >> Average synthesis took: 49.987 usec (+- 0.049 usec)
> >> Average num. events: 47.000 (+- 0.000)
> >> Average time per event 1.064 usec
> >> Average data synthesis took: 53.490 usec (+- 0.033 usec)
> >> Average num. events: 245.000 (+- 0.000)
> >> Average time per event 0.218 usec
> >>
> >> Performance counter stats for 'perf bench internals synthesize':
> >>
> >> <not counted> cpu_core/TOPDOWN.SLOTS/ (0.00%)
> >> <not counted> cpu_core/topdown-retiring/ (0.00%)
> >> <not counted> cpu_core/topdown-mem-bound/ (0.00%)
> >> <not counted> cpu_core/topdown-bad-spec/ (0.00%)
> >> <not counted> cpu_core/topdown-fe-bound/ (0.00%)
> >> <not counted> cpu_core/topdown-be-bound/ (0.00%)
> >> <not counted> cpu_core/RESOURCE_STALLS.SCOREBOARD/ (0.00%)
> >> <not counted> cpu_core/EXE_ACTIVITY.1_PORTS_UTIL/ (0.00%)
> >> <not counted> cpu_core/EXE_ACTIVITY.BOUND_ON_LOADS/ (0.00%)
> >> <not counted> cpu_core/CPU_CLK_UNHALTED.PAUSE/ (0.00%)
> >> <not counted> cpu_core/CYCLE_ACTIVITY.STALLS_TOTAL/ (0.00%)
> >> <not counted> cpu_core/CPU_CLK_UNHALTED.THREAD/ (0.00%)
> >> <not counted> cpu_core/ARITH.DIV_ACTIVE/ (0.00%)
> >> <not counted> cpu_core/EXE_ACTIVITY.2_PORTS_UTIL,umask=0xc/ (0.00%)
> >> <not counted> cpu_core/EXE_ACTIVITY.3_PORTS_UTIL,umask=0x80/ (0.00%)
> >>
> >> 1.186254766 seconds time elapsed
> >>
> >> 0.427220000 seconds user
> >> 0.752217000 seconds sys
> >> Testing smi_cycles
> >> Testing smi_num
> >> Testing tsx_aborted_cycles
> >> Testing tsx_cycles_per_elision
> >> Testing tsx_cycles_per_transaction
> >> Testing tsx_transactional_cycles
> >> test child finished with -1
> >> ---- end ----
> >> perf all metrics test: FAILED!
> >> root@number:~#
> > Have a try disabling the NMI watchdog. Agreed that there is more to
> > fix here but I think the PMU driver is in part to blame because
> > manually breaking the weak group of events is a fix.
>
> I think we have a NO_GROUP_EVENTS_NMI metric constraint to mark a group
> which require disabling of the NMI watchdog.
> Maybe we should mark the group a NO_GROUP_EVENTS_NMI metric.
+Weilin due to the affects of event grouping.
Thanks Kan, NO_GROUP_EVENTS_NMI would be good. Something I see for
tma_ports_utilized_1 that may be worsening things is:
```
Testing tma_ports_utilized_1
Metric 'tma_ports_utilized_1' not printed in:
# Running 'internals/synthesize' benchmark:
Computing performance of single threaded perf event synthesis by
synthesizing events on the perf process itself:
Average synthesis took: 49.581 usec (+- 0.030 usec)
Average num. events: 47.000 (+- 0.000)
Average time per event 1.055 usec
Average data synthesis took: 53.367 usec (+- 0.032 usec)
Average num. events: 246.000 (+- 0.000)
Average time per event 0.217 usec
Performance counter stats for 'perf bench internals synthesize':
<not counted> cpu_core/TOPDOWN.SLOTS/
(0.00%)
<not counted> cpu_core/topdown-retiring/
(0.00%)
<not counted> cpu_core/topdown-mem-bound/
(0.00%)
<not counted> cpu_core/topdown-bad-spec/
(0.00%)
<not counted> cpu_core/topdown-fe-bound/
(0.00%)
<not counted> cpu_core/topdown-be-bound/
(0.00%)
<not counted> cpu_core/RESOURCE_STALLS.SCOREBOARD/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.1_PORTS_UTIL/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.BOUND_ON_LOADS/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.1_PORTS_UTIL/
(0.00%)
<not counted> cpu_core/CYCLE_ACTIVITY.STALLS_TOTAL/
(0.00%)
<not counted> cpu_core/CPU_CLK_UNHALTED.THREAD/
(0.00%)
<not counted> cpu_core/ARITH.DIV_ACTIVE/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.2_PORTS_UTIL,umask=0xc/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.3_PORTS_UTIL,umask=0x80/
(0.00%)
1.180394056 seconds time elapsed
0.409881000 seconds user
0.764134000 seconds sys
```
The event EXE_ACTIVITY.1_PORTS_UTIL is repeated, this is because the
metric code deduplicates events based purely on their name and so
doesn't realize EXE_ACTIVITY.1_PORTS_UTIL is the same as
cpu_core@EXE_ACTIVITY.1_PORTS_UTIL@. This is a hybrid only glitch as
we only prefix with a PMU for hybrid metrics, and I should find and
remove why there's no PMU for the 1 case of EXE_ACTIVITY.1_PORTS_UTIL.
This problem doesn't occur for tma_slow_pause and I wondered if you
could give insight. That metric has the counters below:
```
$ perf stat -M tma_slow_pause -a sleep 0.1
Performance counter stats for 'system wide':
<not counted> cpu_core/TOPDOWN.SLOTS/
(0.00%)
<not counted> cpu_core/topdown-retiring/
(0.00%)
<not counted> cpu_core/topdown-mem-bound/
(0.00%)
<not counted> cpu_core/topdown-bad-spec/
(0.00%)
<not counted> cpu_core/topdown-fe-bound/
(0.00%)
<not counted> cpu_core/topdown-be-bound/
(0.00%)
<not counted> cpu_core/RESOURCE_STALLS.SCOREBOARD/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.1_PORTS_UTIL/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.BOUND_ON_LOADS/
(0.00%)
<not counted> cpu_core/CPU_CLK_UNHALTED.PAUSE/
(0.00%)
<not counted> cpu_core/CYCLE_ACTIVITY.STALLS_TOTAL/
(0.00%)
<not counted> cpu_core/CPU_CLK_UNHALTED.THREAD/
(0.00%)
<not counted> cpu_core/ARITH.DIV_ACTIVE/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.2_PORTS_UTIL,umask=0xc/
(0.00%)
<not counted> cpu_core/EXE_ACTIVITY.3_PORTS_UTIL,umask=0x80/
(0.00%)
0.102074888 seconds time elapsed
```
With -vv I see the event string is:
'{RESOURCE_STALLS.SCOREBOARD/metric-id=RESOURCE_STALLS.SCOREBOARD/,cpu_core/EXE_ACTIVITY.1_PORTS_UTIL,metric-id=cpu_core!3EXE_ACTIVITY.1_PORTS_UTIL!3/,cpu_core/TOPDOWN.SLOTS,metric-id=cpu_core!3TOPDOWN.SLOTS!3/,cpu_core/EXE_ACTIVITY.BOUND_ON_LOADS,metric-id=cpu_core!3EXE_ACTIVITY.BOUND_ON_LOADS!3/,cpu_core/topdown-retiring,metric-id=cpu_core!3topdown!1retiring!3/,cpu_core/topdown-mem-bound,metric-id=cpu_core!3topdown!1mem!1bound!3/,cpu_core/topdown-bad-spec,metric-id=cpu_core!3topdown!1bad!1spec!3/,CPU_CLK_UNHALTED.PAUSE/metric-id=CPU_CLK_UNHALTED.PAUSE/,cpu_core/CYCLE_ACTIVITY.STALLS_TOTAL,metric-id=cpu_core!3CYCLE_ACTIVITY.STALLS_TOTAL!3/,cpu_core/CPU_CLK_UNHALTED.THREAD,metric-id=cpu_core!3CPU_CLK_UNHALTED.THREAD!3/,cpu_core/ARITH.DIV_ACTIVE,metric-id=cpu_core!3ARITH.DIV_ACTIVE!3/,cpu_core/topdown-fe-bound,metric-id=cpu_core!3topdown!1fe!1bound!3/,cpu_core/EXE_ACTIVITY.2_PORTS_UTIL,umask=0xc,metric-id=cpu_core!3EXE_ACTIVITY.2_PORTS_UTIL!0umask!20xc!3/,cpu_core/EXE_ACTIVITY.3_PORTS_UTIL,umask=0x80,metric-id=cpu_core!3EXE_ACTIVITY.3_PORTS_UTIL!0umask!20x80!3/,cpu_core/topdown-be-bound,metric-id=cpu_core!3topdown!1be!1bound!3/}:W'
which without the metric-ids becomes:
'{RESOURCE_STALLS.SCOREBOARD,cpu_core/EXE_ACTIVITY.1_PORTS_UTIL/,cpu_core/TOPDOWN.SLOTS/,cpu_core/EXE_ACTIVITY.BOUND_ON_LOADS/,cpu_core/topdown-retiring/,cpu_core/topdown-mem-bound/,cpu_core/topdown-bad-spec/,CPU_CLK_UNHALTED.PAUSE,cpu_core/CYCLE_ACTIVITY.STALLS_TOTAL/,cpu_core/CPU_CLK_UNHALTED.THREAD/,cpu_core/ARITH.DIV_ACTIVE/,cpu_core/topdown-fe-bound/,cpu_core/EXE_ACTIVITY.2_PORTS_UTIL,umask=0xc/,cpu_core/EXE_ACTIVITY.3_PORTS_UTIL,umask=0x80/,cpu_core/topdown-be-bound/}:W'
I count 9 none slots/top-down counters there, but I see
CPU_CLK_UNHALTED.THREAD can use fixed counter 1. Should
perf_event_open fail for a CPU that has a pinned use of a fixed
counter and the group needs the fixed counter? I'm guessing you don't
want this as CPU_CLK_UNHALTED.THREAD can also go on a generic counter
and the driver doesn't want to count counter usage, it seems feasible
to add it though. I guess we need a NO_GROUP_EVENTS_NMI whenever
CPU_CLK_UNHALTED.THREAD is an event and 8 generic counters are in use.
Checking on Tigerlake I see:
```
$ perf stat -M tma_slow_pause -a sleep 0.1
Performance counter stats for 'system wide':
105,210,913 TOPDOWN.SLOTS # 0.1 %
tma_slow_pause (72.65%)
6,701,129 topdown-retiring
(72.65%)
52,359,712 topdown-fe-bound
(72.65%)
32,904,532 topdown-be-bound
(72.65%)
14,117,814 topdown-bad-spec
(72.65%)
6,602,391 RESOURCE_STALLS.SCOREBOARD
(76.17%)
4,220,773 cpu/EXE_ACTIVITY.3_PORTS_UTIL,umask=0x80/
(76.73%)
421,812 EXE_ACTIVITY.BOUND_ON_STORES
(76.69%)
5,164,088 EXE_ACTIVITY.1_PORTS_UTIL
(76.70%)
299,681 cpu/INT_MISC.RECOVERY_CYCLES,cmask=1,edge/
(76.69%)
245 MISC_RETIRED.PAUSE_INST
(76.67%)
58,403,687 CPU_CLK_UNHALTED.THREAD
(76.72%)
25,297,841 CYCLE_ACTIVITY.STALLS_MEM_ANY
(76.67%)
3,788,772 EXE_ACTIVITY.2_PORTS_UTIL
(62.69%)
20,973,875 CYCLE_ACTIVITY.STALLS_TOTAL
(62.16%)
68,053 ARITH.DIVIDER_ACTIVE
(62.18%)
0.102624327 seconds time elapsed
```
so 10 generic counters which would never fit and the weak group is
broken - the difference in the metric explaining why I've not been
seeing the issue. I think I need to add alderlake/sapphirerapids
constraints here:
https://github.com/captain5050/perfmon/blob/main/scripts/create_perf_json.py#L1382
Ideally we'd automate the constraint generation (or the PMU driver
would help us out by failing to open the weak group).
Thanks,
Ian
> Thanks,
> Kan
>
> > Fwiw, if we
> > switch to the buddy watchdog mechanism then we'll no longer need to
> > disable the NMI watchdog:
> > https://lore.kernel.org/lkml/20230421155255.1.I6bf789d21d0c3d75d382e7e51a804a7a51315f2c@changeid/
>