Re: [PATCH -mm] mm, swap: Fix race between swapoff and some swap operations

From: Paul E. McKenney
Date: Tue Dec 12 2017 - 22:27:47 EST


On Wed, Dec 13, 2017 at 10:17:41AM +0800, Huang, Ying wrote:
> "Paul E. McKenney" <paulmck@xxxxxxxxxxxxxxxxxx> writes:
>
> > On Tue, Dec 12, 2017 at 09:12:20AM +0800, Huang, Ying wrote:
> >> Hi, Pual,
> >>
> >> "Paul E. McKenney" <paulmck@xxxxxxxxxxxxxxxxxx> writes:
> >>
> >> > On Mon, Dec 11, 2017 at 01:30:03PM +0800, Huang, Ying wrote:
> >> >> Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> writes:
> >> >>
> >> >> > On Fri, 08 Dec 2017 16:41:38 +0800 "Huang\, Ying" <ying.huang@xxxxxxxxx> wrote:
> >> >> >
> >> >> >> > Why do we need srcu here? Is it enough with rcu like below?
> >> >> >> >
> >> >> >> > It might have a bug/room to be optimized about performance/naming.
> >> >> >> > I just wanted to show my intention.
> >> >> >>
> >> >> >> Yes. rcu should work too. But if we use rcu, it may need to be called
> >> >> >> several times to make sure the swap device under us doesn't go away, for
> >> >> >> example, when checking si->max in __swp_swapcount() and
> >> >> >> add_swap_count_continuation(). And I found we need rcu to protect swap
> >> >> >> cache radix tree array too. So I think it may be better to use one
> >> >> >> calling to srcu_read_lock/unlock() instead of multiple callings to
> >> >> >> rcu_read_lock/unlock().
> >> >> >
> >> >> > Or use stop_machine() ;) It's very crude but it sure is simple. Does
> >> >> > anyone have a swapoff-intensive workload?
> >> >>
> >> >> Sorry, I don't know how to solve the problem with stop_machine().
> >> >>
> >> >> The problem we try to resolved is that, we have a swap entry, but that
> >> >> swap entry can become invalid because of swappoff between we check it
> >> >> and we use it. So we need to prevent swapoff to be run between checking
> >> >> and using.
> >> >>
> >> >> I don't know how to use stop_machine() in swapoff to wait for all users
> >> >> of swap entry to finish. Anyone can help me on this?
> >> >
> >> > You can think of stop_machine() as being sort of like a reader-writer
> >> > lock. The readers can be any section of code with preemption disabled,
> >> > and the writer is the function passed to stop_machine().
> >> >
> >> > Users running real-time applications on Linux don't tend to like
> >> > stop_machine() much, but perhaps it is nevertheless the right tool
> >> > for this particular job.
> >>
> >> Thanks a lot for explanation! Now I understand this.
> >>
> >> Another question, for this specific problem, I think both stop_machine()
> >> based solution and rcu_read_lock/unlock() + synchronize_rcu() based
> >> solution work. If so, what is the difference between them? I guess rcu
> >> based solution will be a little better for real-time applications? So
> >> what is the advantage of stop_machine() based solution?
> >
> > The stop_machine() solution places similar restrictions on readers as
> > does rcu_read_lock/unlock() + synchronize_rcu(), if that is what you
> > are asking.
> >
> > More precisely, the stop_machine() solution places exactly the
> > same restrictions on readers as does preempt_disable/enable() and
> > synchronize_sched().
> >
> > I would expect stop_machine() to be faster than either synchronize_rcu()
> > synchronize_sched(), or synchronize_srcu(), but stop_machine() operates
> > by making each CPU spin with interrupts until all the other CPUs arrive.
> > This normally does not make real-time people happy.
> >
> > An compromise position is available in the form of
> > synchronize_rcu_expedited() and synchronize_sched_expedited(). These
> > are faster than their non-expedited counterparts, and only momentarily
> > disturb each CPU, rather than spinning with interrupts disabled. However,
> > stop_machine() is probably a bit faster.
> >
> > Finally, syncrhonize_srcu_expedited() is reasonably fast, but
> > avoids disturbing other CPUs. Last I checked, not quite as fast as
> > synchronize_rcu_expedited() and synchronize_sched_expedited(), though.
> >
> > You asked! ;-)
>
> Thanks a lot Paul! That exceeds my expectation!
>
> The performance of swapoff() isn't very important, probably it's not
> necessary to accelerate it at the cost of realtime. I think it is
> better to use a rcu or srcu based solution. I think the cost at reader
> side should be almost same between rcu and srcu? To use srcu, we need
> to select CONFIG_SRCU when CONFIG_SWAP is enabled in Kconfig. I think
> that should be OK?

The thing to do is to try SRCU and see if you can see significant
performance degradation. Given that there is swapping involved, I
would be surprised if the added read-side overhead of SRCU was even
measurable, but then again I have been surprised before.

And yes, just select CONFIG_SRCU when you need it.

Thanx, Paul