Re: [PATCH 02/27] sched/isolation: Introduce housekeeping per-cpu rwsem

[Waiman Long]

On 6/25/25 11:50 AM, Phil Auld wrote:
> On Wed, Jun 25, 2025 at 04:34:18PM +0200 Frederic Weisbecker wrote:
> > Le Wed, Jun 25, 2025 at 08:18:50AM -0400, Phil Auld a écrit :
> > > Hi Waiman,
> > >
> > > On Mon, Jun 23, 2025 at 01:34:58PM -0400 Waiman Long wrote:
> > > > On 6/20/25 11:22 AM, Frederic Weisbecker wrote:
> > > > > The HK_TYPE_DOMAIN isolation cpumask, and further the
> > > > > HK_TYPE_KERNEL_NOISE cpumask will be made modifiable at runtime in the
> > > > > future.
> > > > >
> > > > > The affected subsystems will need to synchronize against those cpumask
> > > > > changes so that:
> > > > >
> > > > > * The reader get a coherent snapshot
> > > > > * The housekeeping subsystem can safely propagate a cpumask update to
> > > > >     the susbsytems after it has been published.
> > > > >
> > > > > Protect against readsides that can sleep with per-cpu rwsem. Updates are
> > > > > expected to be very rare given that CPU isolation is a niche usecase and
> > > > > related cpuset setup happen only in preparation work. On the other hand
> > > > > read sides can occur in more frequent paths.
> > > > >
> > > > > Signed-off-by: Frederic Weisbecker <frederic@xxxxxxxxxx>
> > > > Thanks for the patch series and it certainly has some good ideas. However I
> > > > am a bit concern about the overhead of using percpu-rwsem for
> > > > synchronization especially when the readers have to wait for the completion
> > > > on the writer side. From my point of view, during the transition period when
> > > > new isolated CPUs are being added or old ones being removed, the reader will
> > > > either get the old CPU data or the new one depending on the exact timing.
> > > > The effect the CPU selection may persist for a while after the end of the
> > > > critical section.
> > > >
> > > > Can we just rely on RCU to make sure that it either get the new one or the
> > > > old one but nothing in between without the additional overhead?
> > > >
> > > > My current thinking is to make use CPU hotplug to enable better CPU
> > > > isolation. IOW, I would shut down the affected CPUs, change the housekeeping
> > > > masks and then bring them back online again. That means the writer side will
> > > > take a while to complete.
> > > The problem with this approach is that offlining a cpu effects all the other
> > > cpus and causes latency spikes on other low latency tasks which may already be
> > > running on other parts of the system.
> > >
> > > I just don't want us to finally get to dynamic isolation and have it not
> > > usable for the usecases asking for it.
> > We'll have to discuss that eventually because that's the plan for nohz_full.
> > We can work around the stop machine rendez-vous on nohz_full if that's the
> > problem. If the issue is not to interrupt common RT-tasks, then that's a
> > different problem for which I don't have a solution.
> My understanding is that it's the stop machine issue. If you have a way
> around that then great!

My current thinking is to just run a selected set of CPUHP teardown and 
startup methods relevant to housekeeping cpumasks usage without calling 
the full set from CPUHP_ONLINE to CPUHP_OFFLINE. I don't know if it is 
possible or not or how much additional changes will be needed to make 
that possible. That will skip the CPUHP_TEARDOWN_CPU teardown method 
that is likely the cause of most the latency spike experienced by other 
CPUs.

Cheers,
Longman