RE: [PATCH net v4] net: mana: Avoid waiting on dead HWC after device reset

From: Haiyang Zhang

Date: Thu Nov 19 2026 - 16:33:45 EST




> -----Original Message-----
> From: Sahil Chandna <sahilchandna@xxxxxxxxxxxxxxxxxxx>
> Sent: Wednesday, November 18, 2026 4:14 AM
> To: Haiyang Zhang <haiyangz@xxxxxxxxxxxxx>; wei.liu@xxxxxxxxxx; Dexuan
> Cui <DECUI@xxxxxxxxxxxxx>; andrew+netdev@xxxxxxx; davem@xxxxxxxxxxxxx;
> edumazet@xxxxxxxxxx; kuba@xxxxxxxxxx; pabeni@xxxxxxxxxx; Konstantin
> Taranov <kotaranov@xxxxxxxxxxxxx>; horms@xxxxxxxxxx;
> ernis@xxxxxxxxxxxxxxxxxxx; gargaditya@xxxxxxxxxxxxxxxxxxx;
> mawasthi@xxxxxxxxxxxxxxxxxxx; kees+treewide@xxxxxxxxxx;
> schakrabarti@xxxxxxxxxxxxxxxxxxx; leitao@xxxxxxxxxx; linux-
> hyperv@xxxxxxxxxxxxxxx; netdev@xxxxxxxxxxxxxxx; linux-
> kernel@xxxxxxxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx
> Cc: stable@xxxxxxxxxxxxxxx
> Subject: [PATCH net v4] net: mana: Avoid waiting on dead HWC after device
> reset
>
> After device reset is issued due to TX drain timeout in
> mana_dealloc_queues(), hardware channel (HWC) state is non-functional.
> However, the driver continues to issue HWC commands
> (mana_disable_vport_rx,
> mana_fence_rqs, mana_destroy_wq_obj) which will each wait for a
> completion that never arrives.
>
> The first such command (MANA_CONFIG_VPORT_RX) blocks for the full
> hwc_timeout (60s as set by the firmware on the tested system), after
> which the timeout is reduced to 1 ms for subsequent commands. This
> causes the calling thread to hold the RTNL lock for over 180 seconds
> (120s TX drain + 60s HWC timeout), triggering hung task warnings and
> blocking all other networking operations on the system.
>
> Fix this by:
> 1. Replacing the sleep-based TX drain with a CQE polling approach:
> disable NAPI per queue and poll its CQ synchronously, with up to 3
> more polls 100 ms apart to collect late completions. Reset the
> device only if sends remain unacknowledged after that, then free
> uncompleted SKBs. The device is reset at most once per call, and
> not at all if an earlier reset already marked the HWC dead. Once a
> queue is found stuck or the HWC is dead, the remaining queues are
> polled only once. This replaces the 120s drain timeout with at most
> 300 ms of sleep per queue. If queue is still stuck,
> mana_gd_dev_reset() waits for device to return. The 300 ms budget
> is not based on a hardware limit but in testing under load, the
> completions still outstanding on a queue had all arrived by its
> second poll.
> 2. Setting the hwc_dead flag after a successful device reset. If the
> reset fails, the device may not have been reset and may still access
> memory, so teardown commands are still posted.
> 3. Checking the hwc_dead flag in mana_hwc_send_request() before taking
> an HWC message slot and again before posting the WQE, and returning
> -ETIMEDOUT immediately without submitting commands to hardware that
> cannot respond. A request that was already posted, or that passed
> these checks just before hwc_dead was set (for example from the
> gf_stats worker or mana_ib, which do not hold RTNL), still waits for
> its own timeout, as before this patch.
> 4. Skipping mana_fence_rqs() and the SHM channel teardown handshake
> once hwc_dead is set.
>
> Fixes: a7dfeda6fdec ("net: mana: Fix MANA VF unload when hardware is
> unresponsive")
> Cc: stable@xxxxxxxxxxxxxxx
> Co-developed-by: Dipayaan Roy <dipayanroy@xxxxxxxxxxxxxxxxxxx>
> Signed-off-by: Dipayaan Roy <dipayanroy@xxxxxxxxxxxxxxxxxxx>
> Signed-off-by: Sahil Chandna <sahilchandna@xxxxxxxxxxxxxxxxxxx>
> ---

Reviewed-by: Haiyang Zhang <haiyangz@xxxxxxxxxxxxx>