Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Steve Clevenger]

Hi Suzuki,

On 9/1/2023 5:22 AM, Suzuki K Poulose wrote:
> Hi Steve
> 
> On 30/08/2023 17:04, Suzuki K Poulose wrote:
>> Cc: Sudeep
>>
>> Hi Steve
>>
>> On 28/08/2023 17:35, Steve Clevenger wrote:
>>>
>>> Hi Suzuki,
>>>
>>> On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
>>>> Hi Steve
>>>>
>>>> On 26/08/2023 01:14, Steve Clevenger wrote:
>>>>>
>>>>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>>>>> remedied this. My results below:
>>>>>
>>>>> [root@sut01sys-b212 linux]# cat
>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>>>>> auto flat catu
>>>>> [root@sut01sys-b212 linux]# cat
>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>> auto
>>>>> [root@sut01sys-b212 linux]# echo "catu" >
>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>> [root@sut01sys-b212 linux]# cat
>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>> catu
>>>>>
>>>>> As with the V1 patch, auto defaults to catu.
>>>>>
>>>>> I expected to see tmc-sg (former default) as an available mode, but do
>>>>> not. As I recall, the buffer mode defaulted to ETR scatter-gather
>>>>> prior
>>>>> to this patch. Must this capability now be explicitly advertised? I've
>>>>> seen this done as "arm,scatter-gather" in device trees, but not
>>>>> used by
>>>>> Ampere. Perhaps someone can enlighten me.
>>>>
>>>> Yes, you must add that property to the TMC-ETR node (for both DT and
>>>> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
>>>> locked up the system while using the SG mode (due to the interconnect
>>>> issues, something to do with the transaction). Thus, we decided to
>>>> add a property explicitly enabling this for a given platform.
>>>>
>>>> When you mentioned, it was using TMC-ETR SG mode, how did you verify
>>>> this ? Please be aware that the table allocation code etc are shared
>>>> by both TMC-SG and CATU.
>>>>
>>>
>>> You might recall how this started. I had no way to test the CATU due to
>>> the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
>>> purposes, I programmatically swapped the ETR-SG/CATU order and could
>>> then verify CATU operation by the driver calling into CATU code. This
>>
>> So, were you using the DT based boot for the above runs ?
>>
>>> suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
>>> prior to this hack. This didn't offer the user any control, hence my
>>> feature request. Note that most of the early Ampere self-hosted trace
>>> collection used ETR-SG. Now I can't select it.
>>
>>
>>>
>>> How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
>>> Platform Design Document" (DEN0067) doesn't describe this.
>>
>> This is not specified in the ACPI platform design document. I can get
>> it fixed. Ideally we need a property describing that the scatter-gather
>> mode is safe to use.
> 
> Looks like this is not straight forward copying of DT property. We are
> investigating this on our side and will get back to you.
>

The intent behind my request was to have a way to SysFS configure the
(available) ETR mode. Unless there's a change to the ACPI, the CATU is
the only SG option for Ampere in the near term.

Thanks,
Steve


> Suzuki
> 
> 
>>
>> DT uses "arm,scatter-gather" property [0] and this is what we now expect
>> in the ACPI based systems too.
>>
>> https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73
>>
>> Does it sound fine ?
>>
>> Suzuki
>>
>>>
>>> Thanks,
>>> Steve
>>>
>>>
>>>> Kind regards
>>>> Suzuki
>>>>
>>>>>
>>>>> Steve C.
>>>>>
>>>>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>>>>
>>>>>> Here's some quick feedback. My system shows two modes available; auto
>>>>>> catu
>>>>>>
>>>>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>>>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>>>>
>>>>>> Note that both the auto and catu etr_buf_mode_current settings
>>>>>> default
>>>>>> to catu. My understanding is auto should revert to the default
>>>>>> behavior.
>>>>>> On my system the default was tmc-sg.
>>>>>>
>>>>>> More later.
>>>>>>
>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>>>>
>>>>>> auto catu
>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>> catu
>>>>>> [root@sut01sys-b212 kernel]# echo "catu" >
>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>> catu
>>>>>>
>>>>>> Steve C.
>>>>>>
>>>>>>
>>>>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>>>>
>>>>>>> Hi Suzuki,
>>>>>>>
>>>>>>> I may be able to test it this week. You've already pointed me at the
>>>>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>>>>> patches to apply directly to 6.4 last I tried.
>>>>>>>
>>>>>>> Steve C.
>>>>>>>
>>>>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>>>>> Cc: Steve
>>>>>>>>
>>>>>>>> Steve,
>>>>>>>>
>>>>>>>> Are you able to test this with CATU ?
>>>>>>>>
>>>>>>>>
>>>>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>>>>> available
>>>>>>>>> methods in the following sequentially fallback manner - also in
>>>>>>>>> that
>>>>>>>>> order.
>>>>>>>>>
>>>>>>>>> 1. FLAT mode with or without IOMMU
>>>>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>>>>> 3. CATU mode when available
>>>>>>>>>
>>>>>>>>> But this order might not be ideal for all situations. For
>>>>>>>>> example if
>>>>>>>>> there
>>>>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR
>>>>>>>>> scatter
>>>>>>>>> gather
>>>>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>>>>> prevent
>>>>>>>>> us from testing or using a particular mode. This change provides
>>>>>>>>> following
>>>>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>>>>> explicitly,
>>>>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>>>>> selection
>>>>>>>>> purpose explicitly in the user space.
>>>>>>>>>
>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>>>>
>>>>>>>>> $ cat buf_modes_available
>>>>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR
>>>>>>>>> buffer
>>>>>>>>> modes
>>>>>>>>>
>>>>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>>>>> request
>>>>>>>>>
>>>>>>>>> But explicit user request has to be within supported ETR buffer
>>>>>>>>> modes
>>>>>>>>> only.
>>>>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>>>>> are
>>>>>>>>> not
>>>>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>>>>
>>>>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help
>>>>>>>>> fallback
>>>>>>>>> to the
>>>>>>>>> existing default behaviour, when user provided preferred buffer
>>>>>>>>> mode
>>>>>>>>> fails.
>>>>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>>>>> modes.
>>>>>>>>>
>>>>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@xxxxxxx>
>>>>>>>>> Cc: Mike Leach <mike.leach@xxxxxxxxxx>
>>>>>>>>> Cc: James Clark <james.clark@xxxxxxx>
>>>>>>>>> Cc: Leo Yan <leo.yan@xxxxxxxxxx>
>>>>>>>>> Cc: Alexander Shishkin <alexander.shishkin@xxxxxxxxxxxxxxx>
>>>>>>>>> Cc: coresight@xxxxxxxxxxxxxxxx
>>>>>>>>> Cc: linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Cc: linux-kernel@xxxxxxxxxxxxxxx
>>>>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@xxxxxxx>
>>>>>>>>> ---
>>>>>>>>> This applies on v6.5-rc6
>>>>>>>>>
>>>>>>>>> Changes in V2:
>>>>>>>>>
>>>>>>>>> - Renamed sysfs file etr_buf_modes_available as
>>>>>>>>> buf_modes_available
>>>>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>>>>> - Dropped etr_can_use_flat_mode() check while offering
>>>>>>>>> ETR_MODE_FLAT
>>>>>>>>> in sysfs
>>>>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>>>>> coresight-tmc-etr.c
>>>>>>>>> - Updated month in
>>>>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>>>>
>>>>>>>>> Changes in V1:
>>>>>>>>>
>>>>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@xxxxxxx/
>>>>>>>>>
>>>>>>>>>     .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>>>>     .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>>>>     .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>>>>> ++++++++++++++++--
>>>>>>>>>     drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>>>>     4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>>>>
>>>>>>>>
>>>>>>>> Looks good to me.
>>>>>>>>
>>>>>>>> Suzuki
>>>>>>>>
>>>>>>>>
>>>>
>>
>