[PATCH v6 5/6] fs/resctrl: Release the limbo RMIDs when the CDP state changes
From: Jinjie Ruan
Date: Thu Nov 19 2026 - 04:49:56 EST
On MPAM the counter of an RMID is selected by the PARTIDs of the CLOSID
that owned it. Without CDP, CLOSID c is PARTID c; with CDP it is
PARTIDs 2c and 2c+1. An RMID that stays in limbo across an unmount and
a mount that changes the CDP state is read through the PARTIDs of the
new state, which are not the PARTIDs its cache lines are tagged with.
With 5 PARTIDs:
mount CDP off, CLOSID c -> PARTID c
mkdir g1..g4 CLOSIDs 1..4
umount RMIDs enter limbo with CLOSIDs 1..4
mount -o cdp CDP on, CLOSID c -> PARTIDs 2c, 2c+1
limbo handler reads CLOSID 2 as PARTID 5 > PARTID_MAX 4
MSC error interrupt, mpam_disable()
The other direction is worse because nothing reports it. If resctrl is
mounted with CDP and mounted again without it, CLOSID c is read as
PARTID c, which is in range, so the limbo handler silently tests the
occupancy of the PARTID of a different CLOSID and frees or keeps the
RMIDs based on it.
Where the RMID depends on the CLOSID, as on MPAM, release the RMIDs in
limbo without reading them when a mount changes the CDP state, instead
of restarting the limbo handler. Cache lines that still carry those
RMIDs are accounted to the next user of each RMID. Where the RMID does
not depend on the CLOSID, as on x86, the CDP state does not change how
an RMID is read and the limbo handler is restarted as before.
Compare the CDP state of every resource, not just the L3: on MPAM, CDP
enabled on any resource gives every CLOSID two PARTIDs in all MSCs, so
mounting with -o cdpl2 alone also changes how the L3 monitors read a
CLOSID, and cache levels beyond the L3 would too.
Reproduced on QEMU with the sequence above:
mpam:__mpam_irq_handler: error irq from msc:1 'MSMONCFG_ID_RANGE', partid:5, pmg: 0, ris: 0
mpam:mpam_disable: MPAM disabled due to hardware error interrupt
With the change the sequence completes without an error.
Fixes: 6789fb99282c ("arm_mpam: resctrl: Add CDP emulation")
Suggested-by: Ben Horgan <ben.horgan@xxxxxxx>
Suggested-by: Reinette Chatre <reinette.chatre@xxxxxxxxx>
Signed-off-by: Jinjie Ruan <ruanjinjie@xxxxxxxxxx>
Cc: <stable@xxxxxxxxxxxxxxx>
---
v6:
- Drop resctrl_arch_mon_cdp_enabled() and compare the CDP state of
every resouce's cdp state, with a comment on why they are
checked (Reinette, Ben).
- Restrict the release to the systems where the RMID depends on the
CLOSID, in the changelog and the comments, and keep the hardware IDs
out of the file system comments (Reinette).
- Reword the limbo_cdp_enabled comment (Reinette).
- Drop the stable version range, the Fixes tag already gives it.
v5:
- Split the change of the limbo handler lifetime into the previous
patch. The limbo handler now stops while resctrl is unmounted on every
architecture, and only the release of the RMIDs depends on
CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID.
- Drop the cqm_work_cpu changes. resctrl_limbo_resume() cancels the
worker before it queues it again or releases the RMIDs.
- Move the rdt_disable_ctx() reordering to a separate patch.
- Add resctrl_arch_mon_cdp_enabled() instead of comparing the CDP state
of L2 in the file system.
- Rename limbo_cdp to limbo_cdp_enabled and document that it is only set
with CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID.
- Check QOS_L3_OCCUP_EVENT_ID first and drop the rmid_busy_llc check.
- Use lockdep_assert_cpus_held() at the start of resctrl_limbo_resume().
- Describe the reverse sequence, from CDP on to CDP off.
- Link to v4: https://lore.kernel.org/all/20261105070934.2258613-4-ruanjinjie@xxxxxxxxxx/
v4:
- Use resctrl_mounted instead of a separate limbo_paused flag.
- Record the CDP state once when resctrl is mounted.
- Add resctrl_limbo_resume(), called once resctrl_mounted is set.
- Previous Link: https://lore.kernel.org/all/20261031094616.1691026-1-ruanjinjie@xxxxxxxxxx/
---
fs/resctrl/monitor.c | 35 +++++++++++++++++++++++++++++++++--
1 file changed, 33 insertions(+), 2 deletions(-)
diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c
index ede1708b920e..2c8c60138768 100644
--- a/fs/resctrl/monitor.c
+++ b/fs/resctrl/monitor.c
@@ -70,6 +70,13 @@ static u32 *closid_num_dirty_rmid;
*/
static unsigned int rmid_limbo_count;
+/*
+ * @limbo_cdp_enabled - True if the RMIDs in limbo were placed there while CDP
+ * was enabled. Only used when CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is
+ * defined.
+ */
+static bool limbo_cdp_enabled;
+
/*
* @rmid_entry - The entry in the limbo and free lists.
*/
@@ -1044,12 +1051,15 @@ int setup_rmid_lru_list(void)
* cqm_resume_limbo_handler() - Restart the limbo handler after a mount.
*
* The limbo handler stops while resctrl is unmounted. Restart it on every
- * domain that still has RMIDs in limbo.
+ * domain that still has RMIDs in limbo. Where the RMID depends on the CLOSID
+ * and the mount changed the RMID index a CLOSID is monitored with, the RMIDs
+ * in limbo can no longer be read: release them instead.
*/
void cqm_resume_limbo_handler(void)
{
struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
struct rdt_l3_mon_domain *d;
+ bool release = false;
lockdep_assert_cpus_held();
lockdep_assert_held(&rdtgroup_mutex);
@@ -1057,6 +1067,24 @@ void cqm_resume_limbo_handler(void)
if (!r->mon_capable || !resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID))
return;
+ /*
+ * Where the RMID depends on the CLOSID, the RMIDs in limbo can only
+ * be read with the CDP state they were freed under. Check every
+ * resource because CDP enabled on any resource changes how every
+ * CLOSID is monitored, so CDP on another resource alone also changes
+ * how the L3 monitors are read.
+ */
+ if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
+ struct rdt_resource *res;
+ bool cdp_enabled = false;
+
+ for_each_rdt_resource(res)
+ cdp_enabled |= resctrl_arch_get_cdp_enabled(res->rid);
+
+ release = cdp_enabled != limbo_cdp_enabled;
+ limbo_cdp_enabled = cdp_enabled;
+ }
+
list_for_each_entry_rcu(d, &r->mon_domains, hdr.list,
lockdep_is_cpus_held()) {
if (!has_busy_rmid(d))
@@ -1067,7 +1095,10 @@ void cqm_resume_limbo_handler(void)
* it is queued again on a CPU picked now.
*/
cancel_delayed_work(&d->cqm_limbo);
- cqm_setup_limbo_handler(d, 0, RESCTRL_PICK_ANY_CPU);
+ if (release)
+ release_limbo_rmids(d);
+ else
+ cqm_setup_limbo_handler(d, 0, RESCTRL_PICK_ANY_CPU);
}
}
--
2.34.1