Re: [PATCH v4 0/5] powerpc/perf: IMC trace-mode support

From: Anju T Sudhakar
Date: Tue Apr 16 2019 - 06:08:10 EST



On 4/16/19 3:14 PM, Anju T Sudhakar wrote:
Hi,

Kindly ignore this series, since patch 5/5 in this series doesn't incorporate the event-format change

that I've done in v4 of this series.


Apologies for the inconvenience. I will post the updated v5 soon.


s/v5/v4


Thanks,

Anju

On 4/15/19 3:41 PM, Anju T Sudhakar wrote:
IMC (In-Memory collection counters) is a hardware monitoring facility
that collects large number of hardware performance events.
POWER9 support two modes for IMC which are the Accumulation mode and
Trace mode. In Accumulation mode, event counts are accumulated in system
Memory. Hypervisor then reads the posted counts periodically or when
requested. In IMC Trace mode, the 64 bit trace scom value is initialized
with the event information. The CPMC*SEL and CPMC_LOAD in the trace scom, specifies
the event to be monitored and the sampling duration. On each overflow in the
CPMC*SEL, hardware snapshots the program counter along with event counts
and writes into memory pointed by LDBAR. LDBAR has bits to indicate whether
hardware is configured for accumulation or trace mode.
Currently the event monitored for trace-mode is fixed as cycle.

Trace-IMC Implementation:
--------------------------
To enable trace-imc, we need to

* Add trace node in the DTS file for power9, so that the new trace node can
be discovered by the kernel.

Information included in the DTS file are as follows, (a snippet from
the ima-catalog)

TRACE_IMC: trace-events {
ÂÂÂÂÂ #address-cells = <0x1>;
ÂÂÂÂÂ #size-cells = <0x1>;
ÂÂÂÂÂ event at 10200000 {
ÂÂÂÂ event-name = "cycles" ;
ÂÂÂÂ reg = <0x10200000 0x8>;
ÂÂÂÂ desc = "Reference cycles" ;
ÂÂÂÂÂ };
 };
 trace@0 {
ÂÂÂÂ compatible = "ibm,imc-counters";
ÂÂÂÂ events-prefix = "trace_";
ÂÂÂÂ reg = <0x0 0x8>;
ÂÂÂÂ events = < &TRACE_IMC >;
ÂÂÂÂ type = <0x2>;
ÂÂÂÂ size = <0x40000>;
 };

OP-BUILD changes needed to include the "trace node" is already pulled in
to the ima-catalog repo.

ps://github.com/open-power/op-build/commit/d3e75dc26d1283d7d5eb444bff1ec9e40d5dfc07

* Enchance the opal_imc_counters_* calls to support this new trace mode
in imc. Add support to initialize the trace-mode scom.

TRACE_IMC_SCOM bit representation:

0:1ÂÂÂÂ : SAMPSEL
2:33ÂÂÂ : CPMC_LOAD
34:40ÂÂ : CPMC1SEL
41:47ÂÂ : CPMC2SEL
48:50ÂÂ : BUFFERSIZE
51:63ÂÂ : RESERVED

CPMC_LOAD contains the sampling duration. SAMPSEL and CPMC*SEL determines
the event to count. BUFFRSIZE indicates the memory range. On each overflow,
hardware snapshots program counter along with event counts and update the
memory and reloads the CMPC_LOAD value for the next sampling duration.
IMC hardware does not support exceptions, so it quietly wraps around if
memory buffer reaches the end.

OPAL support for IMC trace mode is already upstream.

* Set LDBAR spr to enable imc-trace mode.
ÂÂ LDBAR Layout:
ÂÂ 0ÂÂÂÂ : Enable/Disable
ÂÂ 1ÂÂÂÂ : 0 -> Accumulation Mode
ÂÂÂÂÂÂÂÂÂÂ 1 -> Trace Mode
ÂÂ 2:3ÂÂ : Reserved
ÂÂ 4-6ÂÂ : PB scope
ÂÂ 7ÂÂÂÂ : Reserved
ÂÂ 8:50Â : Counter Address
ÂÂ 51:63 : Reserved

----------------------

PMI interrupt handling is avoided, since IMC trace mode snapshots the
program counter and update to the memory. And this also provide a way for
the operating system to do instruction sampling in real time without
PMI(Performance Monitoring Interrupts) processing overhead.
Performance data using 'perf top' with and without trace-imc event:

PMI interrupts count when `perf top` command is executed without trace-imc event.

# cat /proc/interrupts (a snippet from the output)
9944ÂÂÂÂÂ 1072ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂ 1644ÂÂÂÂÂÂÂ 804 1306
804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804 804ÂÂÂÂÂÂÂ 804
804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂ 1961ÂÂÂÂÂÂ 1602ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804 1258
[-----------------------------------------------------------------]
803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803 803ÂÂÂÂÂÂÂ 803
803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 804 804ÂÂÂÂÂÂÂ 804
804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804 804ÂÂÂÂÂÂÂ 803
803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803 1306ÂÂÂÂÂÂÂ 803
803ÂÂ Performance monitoring interrupts


`perf top` with trace-imc (executed right after 'perf top' without trace-imc event):

# perf top -e trace_imc/trace_cycles/
12.50%Â [kernel]ÂÂÂÂÂÂÂÂÂ [k] arch_cpu_idle
11.81%Â [kernel]ÂÂÂÂÂÂÂÂÂ [k] __next_timer_interrupt
11.22%Â [kernel]ÂÂÂÂÂÂÂÂÂ [k] rcu_idle_enter
10.25%Â [kernel]ÂÂÂÂÂÂÂÂÂ [k] find_next_bit
 7.91% [kernel] [k] do_idle
 7.69% [kernel] [k] rcu_dynticks_eqs_exit
 5.20% [kernel] [k] tick_nohz_idle_stop_tick
ÂÂÂÂÂ [-----------------------]

# cat /proc/interrupts (a snippet from the output)

9944ÂÂÂÂÂ 1072ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂ 1644ÂÂÂÂÂÂÂ 804 1306
804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804 804ÂÂÂÂÂÂÂ 804
804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂ 1961ÂÂÂÂÂÂ 1602ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804 1258
[-----------------------------------------------------------------]
803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803 803ÂÂÂÂÂÂÂ 803
803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804 804ÂÂÂÂÂÂÂ 804
804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804ÂÂÂÂÂÂÂ 804 804ÂÂÂÂÂÂÂ 803
803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803ÂÂÂÂÂÂÂ 803 1306ÂÂÂÂÂÂÂ 803
803ÂÂ Performance monitoring interrupts

The PMI interrupts count remains the same.


Changelog:
----------
ÂFrom v3 -> v4:

* trace_imc_refc is introduced. So that even if, core-imc
is disabled, trace-imc can be used.

* trace_imc_pmu_sched_task is removed and opal start/stop
is invoked in trace_imc_event_add/del function.


Suggestions/comments are welcome.

Anju T Sudhakar (4):
ÂÂ powerpc/include: Add data structures and macros for IMC trace mode
ÂÂ powerpc/perf: Rearrange setting of ldbar for thread-imc
ÂÂ powerpc/perf: Trace imc events detection and cpuhotplug
ÂÂ powerpc/perf: Trace imc PMU functions

Madhavan Srinivasan (1):
ÂÂ powerpc/perf: Add privileged access check for thread_imc

 arch/powerpc/include/asm/imc-pmu.h | 39 +++
 arch/powerpc/include/asm/opal-api.h | 1 +
 arch/powerpc/perf/imc-pmu.c | 318 +++++++++++++++++++++-
 arch/powerpc/platforms/powernv/opal-imc.c | 3 +
 include/linux/cpuhotplug.h | 1 +
 5 files changed, 351 insertions(+), 11 deletions(-)