Re: [Patch v4 17/22] sched/cache: Avoid cache-aware scheduling for memory-heavy processes

From: Jirka Hladky

Date: Tue Nov 17 2026 - 22:07:14 EST


Hi Tim, Chen Yu,

Thank you both for the patches and the detailed design discussion.
Two updates: the trace Tim asked for, and test results for the two
patches.

PART 1: is NUMA balancing preventing the re-spread?

> Wonder if NUMA balancing is preventing the re-spread?
> If your traces have numa_preferred_nid and task_numa_migrate activity
> after the trip, that would confirm or rule out out.

I traced it on the Fix-2 test kernel (mg.D 1x64 CAS-on, two runs that
aggregated on different nodes, same structure in both): per-thread
numa_preferred_nid sampled at every context switch, sched_setnuma()
and migrate_task_to() in the task's own context, every
sched_migrate_task with origin and destination, and wakeup target
CPUs, alongside the group state per epoch.

The short answer: yes, but only late, and it is not the main
obstacle. The run decomposes into three phases:

1. 0 - 4.9 s: consolidation. About 30 cross-node migrations toward
the aggregated node; the footprint crosses the threshold at
4.2 - 4.9 s and grp->cpu is cleared.

2. Trip to sweep completion (4.9 - 35.8 s in the traced run): the
re-spread runs at about 0.5 net cross-node migrations per second
(thread placement 2/63 across the two nodes at t=4 s, 18/47 at
t=28 s), and numa_preferred_nid is -1 for every thread during this
whole window: sched_setnuma() first fires only when the sweep
completes and placement runs, at t=35.8 s. So the NUMA preference
cannot be what holds the re-spread back here. What the trace shows
instead: the wakeup placement keeps directing threads to the
aggregated node at roughly 2:1 to 3:1 throughout the window
(1264 vs 87 wakeups per 2 s at t=8 s, still 639 vs 380 at t=32 s;
these are OMP barrier wakeups landing near their wakers), while
the periodic balancer moves tasks out at only about 2 per 4 s.
The net drift is the difference.

3. Sweep completion to end (35.8 - 42 s): placement sets
numa_preferred_nid to the aggregated node for 53 of 66 threads
(the pages have migrated there by then, so the fault statistics
point there, exactly your reinforcing loop). From this moment the
pull-back engages: migrate_task_to() toward the preferred node
appears, and the cross-node churn in the last 6 s is 8 moves out
vs 10 moves back, i.e. the spread stalls.

So NUMA balancing does prevent the re-spread, but only in the last
15 percent of the run. For the first 85 percent the preferred node is
simply not set, and what holds it back is wake placement plus the
conservative pace of cross-node load balancing. Resetting NUMA
preferences at withdrawal would therefore not help this workload:
the sweep outlives the run, so there is nothing to reset during the
window that matters. And the wake-placement observation suggests the
re-spread would stay slow even with preferences reset; an active
re-spread may need to move the tasks and let the wake placement
follow.

PART 2: test results for the two patches

I built a kernel from the same 7.4.0-0.rc2.19.eln159 source, with
Chen Yu's sketch (as tested before) and both patches applied. Both
applied cleanly. Same test battery as the earlier rounds, n=5
medians for the NAS runs:

workload stock OFF stock ON patched OFF patched ON
NAS MG D, 1x64thr 26.45 s 41.78 s 26.38 s 26.47 s (+0.3%)
(+58%)
NAS MG C, 1x4thr 9.34 s 11.99 s 9.38 s 9.36 s (-0.2%)
(+28%)

Both reported regressions are gone. The mechanism trace shows why:
grp->cpu = -1 in all 42,566 task_cache_work() samples over one
CAS-on MG run. The aggregation never settles, not even during the
warm-up: the RSS bound refuses within ~0.1 s of the allocation phase,
before first-touch can skew anything. The hysteresis from PART 1
never develops because there is nothing to unwind.

The VM reproducer [3], the workload that benefits from packing,
shows one cost of the patches and one way around it:

VM runtime: stock off 22.12 s, stock on 17.66 s (-20%)
patched off 21.81 s
patched on, tol=1: 22.47 s (+3.0%, win lost)
patched on, tol=4: 17.70 s (-18.9%, win back)

cache-locality: tol=1: -4.6%, tol=4: +24.6%
mem-bandwidth: tol=1: +2.5%, tol=4: +30.6%

The win is lost at the default tolerance: qemu's RSS (guest memory,
all anon) far exceeds the 108 MiB LLC, and the first sweep of its
~25 GB address space takes ~100 s while a reproducer pass runs ~20 s,
so the bound governs throughout and aggregation is refused. At
aggr_tolerance=4 the bound is (1 + 3*256) x 108 MiB = 83 GiB, qemu
fits, and the win returns. The tolerance knob thus doubles as the
opt-in for large-RSS workloads that benefit from aggregation.
Splitting the two, so the tolerance scales only the estimate and
not the bound, may be worth considering; either way the coupling
deserves a line in the documentation.

On Chen Yu's question, whether Fix 1 is still needed if Fix 2
accumulates the RSS bound: the data says yes for the regression
class. The bound is what refuses aggregation for MG D and MG C,
because their fault-based estimate never exists within a run (the
sweep is too slow), and Fix 2 alone recovered only one sixth of the
regression in the earlier test.

Fix 2 correcting the initial placement does happen for a
long-running VM, and I traced it. I ran the VM reproducer with the
in-VM workloads scaled 10x (about 100 s per phase, 4 minutes per
pass). The group state over the CAS-on pass at the default
tolerance:

t=0 VM boots; rss_bound is 0 for the first epoch, CAS
aggregates for ~0.1 s until the first RSS sample lands
(435k pages), then refuses.
t=16 s the first scan sweep folds: the history becomes 27,298
pages, the VM's actual hot set, within 1.3 percent of
the 27,648-page LLC. The estimate takes over from the
RSS bound, the check passes, and the aggregation is
granted: grp->cpu is set for the first time since boot.
t=19 s the bandwidth phase grows the current window to 23,576
pages; max(history, history/2 + window) crosses the
threshold and the aggregation is withdrawn.
rest stably refused; the estimate stays above the threshold.

So the mechanism works exactly as designed end to end: the bound
guards the window before the first sweep, the estimate corrects the
decision afterward, and the flat combination catches the growth
within the sweep. This VM is a boundary case by accident rather
than design: its workload data is tiny, but the guest kernel and
initramfs dominate the hot set, which was never sized against the
LLC. Either way it is precisely the flip territory your max()
smoothing addresses.

The aggregate numbers reflect how little of the run the granted
window covers: the long pass ends at -2.9% runtime, +2.4%
cache-locality, +6.0% bandwidth, against -20% for the stock kernel
and +3.0% for the same patched kernel with the short (~20 s)
workload. A persistent VM partially recovers the benefit.

One race worth noting: rss_bound is sampled at the first
task_cache_work() epoch after group creation, so for about 0.1 s
the bound is 0 and CAS can aggregate. In the traced run that blip
was enough to set the initial vCPU placement (the threads clustered
and never moved afterward; the box is idle and nothing pulls 8
threads back). Sampling rss_bound at group creation in
mm_init_sched(), or withholding aggregation until the first sample,
would close it.

Putting the three rounds of testing together, my recommendation:

I vote for keeping both fixes. They do different jobs: the RSS
bound is the only thing that protects the window before the first
sweep, which is where the memory-bound workloads take their damage,
and the within-sweep accumulation is what makes the decision
correct afterward and recovers the beneficial cases. The cost
asymmetry settles the default: a wrongly granted aggregation cost
us 58 percent, a wrongly refused one forfeits a 20 percent gain.

Two refinements seem worth considering on top:

- Hysteresis on the grant/withdraw decision. The VM trace showed a
boundary case as it sits close to the threshold, so the aggregation
flips as the window grows and the history decays. Withdrawing above
the threshold and re-granting below half of it, the pattern imb_pct
and overaggr_pct already use, would stabilize it.

- The tolerance splitting. aggr_tolerance was designed to scale the
footprint threshold, and it now scales the RSS bound too, so
tolerance 2 jumps the bound from 108 MiB to 27.5 GiB. That works
as an opt-in but is a blunt one; the per-group control from the
prctl RFC seems the better tool for "this large-RSS workload
wants aggregation", with the tolerance kept for the estimate.

Happy to share the probe scripts or re-run anything.

Thanks,
Jirka

[3] The VM test boots a small VM directly off the host's own kernel
and initramfs, with no guest OS install: 1 VM, 8 vCPUs, 25 GB of
guest memory. Two workloads run inside it, in this order: a
cache-locality test (process pairs exchanging data through FIFO
pipes, reported as ops/s) and a memory-bandwidth test (cp over a
216 MB buffer, twice this host's LLC, reported as MB/s). The
test records which physical CPUs the vCPU threads land on, which
is how the packing effect is measured. The long variant scales
the in-VM iterations 10x, to about 100 s per workload phase.

On Mon, Nov 17, 2026 at 12:39 AM Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx> wrote: