Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Suzuki K Poulose]

Hi Steve

On 04/12/2023 21:41, Steve Clevenger wrote:
> Hi Suzuki,
>
> On 9/7/2023 2:31 PM, Steve Clevenger wrote:
> > Hi Suzuki,
> >
> > On 9/1/2023 5:22 AM, Suzuki K Poulose wrote:
> > > Hi Steve
> > >
> > > On 30/08/2023 17:04, Suzuki K Poulose wrote:
> > > > Cc: Sudeep
> > > >
> > > > Hi Steve
> > > >
> > > > On 28/08/2023 17:35, Steve Clevenger wrote:
> > > > > Hi Suzuki,
> > > > >
> > > > > On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
> > > > > > Hi Steve
> > > > > >
> > > > > > On 26/08/2023 01:14, Steve Clevenger wrote:
> > > > > > > Unfortunately, I tested with the original patch not [PATCH V2]. I've
> > > > > > > remedied this. My results below:
> > > > > > >
> > > > > > > [root@sut01sys-b212 linux]# cat
> > > > > > > /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
> > > > > > > auto flat catu
> > > > > > > [root@sut01sys-b212 linux]# cat
> > > > > > > /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
> > > > > > > auto
> > > > > > > [root@sut01sys-b212 linux]# echo "catu" >
> > > > > > > /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
> > > > > > > [root@sut01sys-b212 linux]# cat
> > > > > > > /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
> > > > > > > catu
> > > > > > >
> > > > > > > As with the V1 patch, auto defaults to catu.
> > > > > > >
> > > > > > > I expected to see tmc-sg (former default) as an available mode, but do
> > > > > > > not. As I recall, the buffer mode defaulted to ETR scatter-gather
> > > > > > > prior
> > > > > > > to this patch. Must this capability now be explicitly advertised? I've
> > > > > > > seen this done as "arm,scatter-gather" in device trees, but not
> > > > > > > used by
> > > > > > > Ampere. Perhaps someone can enlighten me.
> > > > > >
> > > > > > Yes, you must add that property to the TMC-ETR node (for both DT and
> > > > > > ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
> > > > > > locked up the system while using the SG mode (due to the interconnect
> > > > > > issues, something to do with the transaction). Thus, we decided to
> > > > > > add a property explicitly enabling this for a given platform.
> > > > > >
> > > > > > When you mentioned, it was using TMC-ETR SG mode, how did you verify
> > > > > > this ? Please be aware that the table allocation code etc are shared
> > > > > > by both TMC-SG and CATU.
> > > > >
> > > > > You might recall how this started. I had no way to test the CATU due to
> > > > > the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
> > > > > purposes, I programmatically swapped the ETR-SG/CATU order and could
> > > > > then verify CATU operation by the driver calling into CATU code. This
> > > >
> > > > So, were you using the DT based boot for the above runs ?
> > > >
> > > > > suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
> > > > > prior to this hack. This didn't offer the user any control, hence my
> > > > > feature request. Note that most of the early Ampere self-hosted trace
> > > > > collection used ETR-SG. Now I can't select it.
> > > >
> > > >
> > > > > How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
> > > > > Platform Design Document" (DEN0067) doesn't describe this.
> > > >
> > > > This is not specified in the ACPI platform design document. I can get
> > > > it fixed. Ideally we need a property describing that the scatter-gather
> > > > mode is safe to use.
> > >
> > > Looks like this is not straight forward copying of DT property. We are
> > > investigating this on our side and will get back to you.
>
> I noticed this work is queued for 6.7 (coresight-next-6.7). Do you have
> an update to the ACPI platform design document Ampere can use to base an
> update so the scatter-gather mode can be used?

I have chased this with the specification team, and we should have
something published soon. Apologies for the delay.

Suzuki

> Thanks,
> Steve
>
> > >
> >
> > The intent behind my request was to have a way to SysFS configure the
> > (available) ETR mode. Unless there's a change to the ACPI, the CATU is
> > the only SG option for Ampere in the near term.
> >
> > Thanks,
> > Steve
> >
> >
> > > Suzuki
> > >
> > >
> > > > DT uses "arm,scatter-gather" property [0] and this is what we now expect
> > > > in the ACPI based systems too.
> > > >
> > > > https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73
> > > >
> > > > Does it sound fine ?
> > > >
> > > > Suzuki
> > > >
> > > > > Thanks,
> > > > > Steve
> > > > >
> > > > >
> > > > > > Kind regards
> > > > > > Suzuki
> > > > > >
> > > > > > > Steve C.
> > > > > > >
> > > > > > > On 8/23/2023 4:10 PM, Steve Clevenger wrote:
> > > > > > > > Here's some quick feedback. My system shows two modes available; auto
> > > > > > > > catu
> > > > > > > >
> > > > > > > > etr_buf_mode_current is writable. I expected to see tmc-sg (former
> > > > > > > > default) listed in etr_buf_modes_available but it doesn't show up.
> > > > > > > >
> > > > > > > > Note that both the auto and catu etr_buf_mode_current settings
> > > > > > > > default
> > > > > > > > to catu. My understanding is auto should revert to the default
> > > > > > > > behavior.
> > > > > > > > On my system the default was tmc-sg.
> > > > > > > >
> > > > > > > > More later.
> > > > > > > >
> > > > > > > > [root@sut01sys-b212 kernel]# cat
> > > > > > > > /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
> > > > > > > >
> > > > > > > > auto catu
> > > > > > > > [root@sut01sys-b212 kernel]# cat
> > > > > > > > /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
> > > > > > > > catu
> > > > > > > > [root@sut01sys-b212 kernel]# echo "catu" >
> > > > > > > > /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
> > > > > > > > [root@sut01sys-b212 kernel]# cat
> > > > > > > > /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
> > > > > > > > catu
> > > > > > > >
> > > > > > > > Steve C.
> > > > > > > >
> > > > > > > >
> > > > > > > > On 8/21/2023 12:40 PM, Steve Clevenger wrote:
> > > > > > > > > Hi Suzuki,
> > > > > > > > >
> > > > > > > > > I may be able to test it this week. You've already pointed me at the
> > > > > > > > > patch thread(s). The main holdup is I need to merge the 6.6 pending
> > > > > > > > > platform work in order to use the Ampere ACPI. I couldn't get these
> > > > > > > > > patches to apply directly to 6.4 last I tried.
> > > > > > > > >
> > > > > > > > > Steve C.
> > > > > > > > >
> > > > > > > > > On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
> > > > > > > > > > Cc: Steve
> > > > > > > > > >
> > > > > > > > > > Steve,
> > > > > > > > > >
> > > > > > > > > > Are you able to test this with CATU ?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 18/08/2023 09:21, Anshuman Khandual wrote:
> > > > > > > > > > > Currently TMC-ETR automatically selects the buffer mode from all
> > > > > > > > > > > available
> > > > > > > > > > > methods in the following sequentially fallback manner - also in
> > > > > > > > > > > that
> > > > > > > > > > > order.
> > > > > > > > > > >
> > > > > > > > > > > 1. FLAT mode with or without IOMMU
> > > > > > > > > > > 2. TMC-ETR-SG (scatter gather) mode when available
> > > > > > > > > > > 3. CATU mode when available
> > > > > > > > > > >
> > > > > > > > > > > But this order might not be ideal for all situations. For
> > > > > > > > > > > example if
> > > > > > > > > > > there
> > > > > > > > > > > is a CATU connected to ETR, it may be better to use TMC-ETR
> > > > > > > > > > > scatter
> > > > > > > > > > > gather
> > > > > > > > > > > method, rather than CATU. But hard coding such order changes will
> > > > > > > > > > > prevent
> > > > > > > > > > > us from testing or using a particular mode. This change provides
> > > > > > > > > > > following
> > > > > > > > > > > new sysfs tunables for the user to control TMC-ETR buffer mode
> > > > > > > > > > > explicitly,
> > > > > > > > > > > if required. This adds following new sysfs files for buffer mode
> > > > > > > > > > > selection
> > > > > > > > > > > purpose explicitly in the user space.
> > > > > > > > > > >
> > > > > > > > > > > /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
> > > > > > > > > > > /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
> > > > > > > > > > >
> > > > > > > > > > > $ cat buf_modes_available
> > > > > > > > > > > auto flat tmc-sg catu    ------------------> Supported TMC-ETR
> > > > > > > > > > > buffer
> > > > > > > > > > > modes
> > > > > > > > > > >
> > > > > > > > > > > $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
> > > > > > > > > > > request
> > > > > > > > > > >
> > > > > > > > > > > But explicit user request has to be within supported ETR buffer
> > > > > > > > > > > modes
> > > > > > > > > > > only.
> > > > > > > > > > > These sysfs interface files are exclussive to ETR, and hence these
> > > > > > > > > > > are
> > > > > > > > > > > not
> > > > > > > > > > > available for other TMC devices such as ETB or ETF etc.
> > > > > > > > > > >
> > > > > > > > > > > A new auto' mode (i.e ETR_MODE_AUTO) has been added to help
> > > > > > > > > > > fallback
> > > > > > > > > > > to the
> > > > > > > > > > > existing default behaviour, when user provided preferred buffer
> > > > > > > > > > > mode
> > > > > > > > > > > fails.
> > > > > > > > > > > ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
> > > > > > > > > > > modes.
> > > > > > > > > > >
> > > > > > > > > > > Cc: Suzuki K Poulose <suzuki.poulose@xxxxxxx>
> > > > > > > > > > > Cc: Mike Leach <mike.leach@xxxxxxxxxx>
> > > > > > > > > > > Cc: James Clark <james.clark@xxxxxxx>
> > > > > > > > > > > Cc: Leo Yan <leo.yan@xxxxxxxxxx>
> > > > > > > > > > > Cc: Alexander Shishkin <alexander.shishkin@xxxxxxxxxxxxxxx>
> > > > > > > > > > > Cc: coresight@xxxxxxxxxxxxxxxx
> > > > > > > > > > > Cc: linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
> > > > > > > > > > > Cc: linux-kernel@xxxxxxxxxxxxxxx
> > > > > > > > > > > Signed-off-by: Anshuman Khandual <anshuman.khandual@xxxxxxx>
> > > > > > > > > > > ---
> > > > > > > > > > > This applies on v6.5-rc6
> > > > > > > > > > >
> > > > > > > > > > > Changes in V2:
> > > > > > > > > > >
> > > > > > > > > > > - Renamed sysfs file etr_buf_modes_available as
> > > > > > > > > > > buf_modes_available
> > > > > > > > > > > - Renamed sysfs file buf_mode_current as buf_mode_preferred
> > > > > > > > > > > - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
> > > > > > > > > > > - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
> > > > > > > > > > > - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
> > > > > > > > > > > - Fallback trying ETR_MODE_AUTO when user preferred mode fails
> > > > > > > > > > > - Moved ETR sysfs details into coresight-tmc-etr.c
> > > > > > > > > > > - Dropped etr_can_use_flat_mode() check while offering
> > > > > > > > > > > ETR_MODE_FLAT
> > > > > > > > > > > in sysfs
> > > > > > > > > > > - Moved struct etr_buf_hw inside coresight-tmc-etr.c
> > > > > > > > > > > - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
> > > > > > > > > > > coresight-tmc-etr.c
> > > > > > > > > > > - Updated month in
> > > > > > > > > > > Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
> > > > > > > > > > >
> > > > > > > > > > > Changes in V1:
> > > > > > > > > > >
> > > > > > > > > > > https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@xxxxxxx/
> > > > > > > > > > >
> > > > > > > > > > >      .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
> > > > > > > > > > >      .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
> > > > > > > > > > >      .../hwtracing/coresight/coresight-tmc-etr.c   | 111
> > > > > > > > > > > ++++++++++++++++--
> > > > > > > > > > >      drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
> > > > > > > > > > >      4 files changed, 131 insertions(+), 14 deletions(-)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Looks good to me.
> > > > > > > > > >
> > > > > > > > > > Suzuki