Re: [PATCH net-next v7 1/3] net: macb: cache ENST config across link down/up
From: Théo Lebrun
Date: Thu Nov 19 2026 - 11:10:52 EST
Hello Théo,
On Thu Nov 19, 2026 at 4:40 PM CET, Théo Lebrun wrote:
> Hello Vineeth,
>
> On Thu Nov 19, 2026 at 1:36 PM CET, Vineeth Karumanchi wrote:
>> Hi Théo,
>>
>> On 11/19/2026 2:23 AM, Théo Lebrun wrote:
>>> Currently, ENST is not explicitly restored across interface close/open.
>>> If HW keeps its register state then ENST is kept, otherwise it is lost;
>>> but we do not write the ENST schedule again on open.
>>>
>>> This causes trouble if the config is not compatible with the new open
>>> and link-up. The two issues that can occur are a link speed change (as
>>> all timings are configured in a unit relative to the link speed) or a
>>> change in the # of queues.
>>>
>>
>> Thanks for the patch.
>>
>> We are working on the same problem for CBS, covering the four scenarios
>> listed in:
>> https://lore.kernel.org/netdev/a04aa4d5-335b-47c1-b969-e89b18c9b142@xxxxxxx/
>>
>> Since this patch reworks how ENST state is kept and restored, it would
>> be good to introduce a generic top-level tc structure here, which ENST,
>> CBS and any later TSN offloads can share, instead of an ENST-specific one.
>
> Sorry I forgot replying on that thread. It is still marked as unread in
> my mailbox!
>
> Let me quote it here and reply:
>
>
>
> On Thu Oct 8, 2026 at 5:27 PM CEST, Karumanchi, Vineeth wrote:
>> On 9/17/2026 3:02 PM, Théo Lebrun wrote:
>>> So we write cbs_control/0x04BC only on ndo_setup_tc(TC_SETUP_QDISC_CBS)
>>> callback. As Sashiko pointed how I think that causes an issue with HW
>>> resets & suspend:
>>> - what if configured then link down then link up?
>>> - what if configured before link up?
>>>
>>> Also when not using CBS we hope tx_sched_ctrl/0x0580 is at its reset
>>> value (0b00, fixed priority) for all queues. I would expect an
>>> unconditional writel tx_sched_ctrl/0x0580 in macb_init_hw(). This
>>> solves the down/up issue, the speed change and also ensures we don't
>>> inherit CBS or DWRR from the boot stages (super unlikely though).
>>>
>>> Also, it might be simpler to always use tx_sched_ctrl/0x0580 rather than
>>> sometimes tx_sched_ctrl/0x0580 and sometimes cbs_control/0x04BC, which
>>> are aliases.
>>>
>>> What do you think?
>>
>> Sorry for the delayed response.
>>
>> I tested the current implementation in the following four scenarios:
>>
>> 1. Configuring CBS while the link is down: CBS cannot be programmed
>> because the link speed is unknown.
>>
>> 2. Link down followed by link up: The hardware retains the CBS register
>> values, and CBS is effective again after the link comes up at the same
>> speed.
>
> Here you talk about link-down followed by link-up.
>
> But about a close then open: this behavior might be board specific.
> Runtime PM does suspend and this might shut off the IP on some
> platforms and not others. That would in turn mean the IP loses its
> state.
>
> There is no easy way of knowing if that occurs on any platform that uses
> MACB. It sounds probable though.
>
>> 3. Suspend/resume: The CBS register contents are lost after resume.
>> However, the CBS child qdisc remains installed in the kernel and is
>> still shown by tc as offloaded, leaving the software configuration
>> inconsistent with the hardware state.
>
> Again, this is platform specific. Some boards + suspend combo keep the
> SoC and its peripherals alive enough for them to keep their state.
>
> A board with little runtime PM doing suspend-to-idle probably won't lose
> CBS register state. A board doing proper suspend-to-RAM will surely
> lose its state.
>
>> 4. Link-speed change: The CBS register values are retained and shaping
>> remains enabled. However, those values were calculated for the previous
>> speed, so the resulting shaping bandwidth is incorrect.
>>
>> I see two possible approaches:
>>
>> - Retain the requested CBS parameters in the driver and restore the
>> hardware configuration, following the saved-state approach used by IGB.
>> For MACB, we would also recalculate the hardware idleslope on link-up
>> using the negotiated speed.
>> If the saved request cannot be supported at the new speed, retain the
>> request, leave CBS disabled for that queue, and warn the user that the
>> configuration is invalid for the new link speed.
>>
>> - Explicitly reset the scheduling configuration in macb_init_hw() and
>> require users to remove and recreate the CBS qdisc. This establishes a
>> known hardware state, but leaves the installed qdisc inconsistent until
>> users reconfigure it. It would also need handling for speed changes that
>> do not invoke macb_init_hw().
>>
>> Please let me know your thoughts on these approaches.
>
> Option 2 is not super doable. There is no API for the driver to
> say to the network stack and userspace "I'm dropping your qdisc, please
> re-enable one if you care".
>
> So ideally we would do option 1, but dealing with link speed changes
> sounds complex. I'm super afraid of introducing bugs with code that
> recomputes the timings on the fly. I'm sure I will create rounding bugs
> in the edge cases.
>
> That's why I made the cached state be in number of words and store the
> link speed alongside. This means we cannot introduce such bugs AND is
> less code overall AND is the common case.
>
> It feels like easier and smaller incremental changes.
To expand with some numbers on this.
- For link speed increases, our trouble is the max on/off time interval
that can be described. We might be asked for windows that we cannot
express with our hardware registers.
- For link speed decreases, our trouble is the on/off time interval
quantum. We risk introducing too much overlap when the desire was to
have none (pretty common with ENST).
link speed │ 10M │ 100M │ 1G │ 2.5G │ 5G │ 10G
───────────┼────────┼───────┼──────┼────────┼────────┼───────
max on/off │ 105 ms │ 10 ms │ 1 ms │ 419 µs │ 210 µs │ 105 µs
───────────┼────────┼───────┼──────┼────────┼────────┼───────
quantum │ 800 ns │ 80 ns │ 8 ns │ 3.2 ns │ 1.6 ns │ 0.8 ns
So only ENST configs that have all timings in that .8µs - 105µs can
safely-ish be described. But at 10M with the max on-time of 105µs, you
cannot even transmit a full 1500 byte frame.
That is talking about the extremes; it is true that it could make more
sense for less extreme link speed changes (eg from 100M to 1G).
Overall, this still conforts me in the current proposal: cache a
link-speed specific config. Userspace must have had looked at the link
speed to pick a sensible ENST schedule anyway.
Thanks,
--
Théo Lebrun, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com