Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Steve Clevenger]

Hi Suzuki,

On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
> Hi Steve
> 
> On 26/08/2023 01:14, Steve Clevenger wrote:
>>
>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>> remedied this. My results below:
>>
>> [root@sut01sys-b212 linux]# cat
>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>> auto flat catu
>> [root@sut01sys-b212 linux]# cat
>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>> auto
>> [root@sut01sys-b212 linux]# echo "catu" >
>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>> [root@sut01sys-b212 linux]# cat
>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>> catu
>>
>> As with the V1 patch, auto defaults to catu.
>>
>> I expected to see tmc-sg (former default) as an available mode, but do
>> not. As I recall, the buffer mode defaulted to ETR scatter-gather prior
>> to this patch. Must this capability now be explicitly advertised? I've
>> seen this done as "arm,scatter-gather" in device trees, but not used by
>> Ampere. Perhaps someone can enlighten me.
> 
> Yes, you must add that property to the TMC-ETR node (for both DT and
> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
> locked up the system while using the SG mode (due to the interconnect
> issues, something to do with the transaction). Thus, we decided to
> add a property explicitly enabling this for a given platform.
> 
> When you mentioned, it was using TMC-ETR SG mode, how did you verify
> this ? Please be aware that the table allocation code etc are shared
> by both TMC-SG and CATU.
> 

You might recall how this started. I had no way to test the CATU due to
the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
purposes, I programmatically swapped the ETR-SG/CATU order and could
then verify CATU operation by the driver calling into CATU code. This
suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
prior to this hack. This didn't offer the user any control, hence my
feature request. Note that most of the early Ampere self-hosted trace
collection used ETR-SG. Now I can't select it.

How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
Platform Design Document" (DEN0067) doesn't describe this.

Thanks,
Steve


> Kind regards
> Suzuki
> 
>>
>> Steve C.
>>
>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>
>>> Here's some quick feedback. My system shows two modes available; auto
>>> catu
>>>
>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>
>>> Note that both the auto and catu etr_buf_mode_current settings default
>>> to catu. My understanding is auto should revert to the default behavior.
>>> On my system the default was tmc-sg.
>>>
>>> More later.
>>>
>>> [root@sut01sys-b212 kernel]# cat
>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>
>>> auto catu
>>> [root@sut01sys-b212 kernel]# cat
>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>> catu
>>> [root@sut01sys-b212 kernel]# echo "catu" >
>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>> [root@sut01sys-b212 kernel]# cat
>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>> catu
>>>
>>> Steve C.
>>>
>>>
>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>
>>>> Hi Suzuki,
>>>>
>>>> I may be able to test it this week. You've already pointed me at the
>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>> patches to apply directly to 6.4 last I tried.
>>>>
>>>> Steve C.
>>>>
>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>> Cc: Steve
>>>>>
>>>>> Steve,
>>>>>
>>>>> Are you able to test this with CATU ?
>>>>>
>>>>>
>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>> available
>>>>>> methods in the following sequentially fallback manner - also in that
>>>>>> order.
>>>>>>
>>>>>> 1. FLAT mode with or without IOMMU
>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>> 3. CATU mode when available
>>>>>>
>>>>>> But this order might not be ideal for all situations. For example if
>>>>>> there
>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>>>>>> gather
>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>> prevent
>>>>>> us from testing or using a particular mode. This change provides
>>>>>> following
>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>> explicitly,
>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>> selection
>>>>>> purpose explicitly in the user space.
>>>>>>
>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>
>>>>>> $ cat buf_modes_available
>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR buffer
>>>>>> modes
>>>>>>
>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>> request
>>>>>>
>>>>>> But explicit user request has to be within supported ETR buffer modes
>>>>>> only.
>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>> are
>>>>>> not
>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>
>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback
>>>>>> to the
>>>>>> existing default behaviour, when user provided preferred buffer mode
>>>>>> fails.
>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>> modes.
>>>>>>
>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@xxxxxxx>
>>>>>> Cc: Mike Leach <mike.leach@xxxxxxxxxx>
>>>>>> Cc: James Clark <james.clark@xxxxxxx>
>>>>>> Cc: Leo Yan <leo.yan@xxxxxxxxxx>
>>>>>> Cc: Alexander Shishkin <alexander.shishkin@xxxxxxxxxxxxxxx>
>>>>>> Cc: coresight@xxxxxxxxxxxxxxxx
>>>>>> Cc: linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
>>>>>> Cc: linux-kernel@xxxxxxxxxxxxxxx
>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@xxxxxxx>
>>>>>> ---
>>>>>> This applies on v6.5-rc6
>>>>>>
>>>>>> Changes in V2:
>>>>>>
>>>>>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT
>>>>>> in sysfs
>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>> coresight-tmc-etr.c
>>>>>> - Updated month in
>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>
>>>>>> Changes in V1:
>>>>>>
>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@xxxxxxx/
>>>>>>
>>>>>>    .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>    .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>    .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>> ++++++++++++++++--
>>>>>>    drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>    4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>
>>>>>
>>>>> Looks good to me.
>>>>>
>>>>> Suzuki
>>>>>
>>>>>
>