Re: [RFC-PROTOTYPE 1/1] mm: Add __GFP_FAST_TRY flag

From: Uladzislau Rezki
Date: Tue Aug 04 2020 - 17:04:55 EST


On Tue, Aug 04, 2020 at 07:34:18PM +0200, Vlastimil Babka wrote:
> On 8/4/20 7:12 PM, Matthew Wilcox wrote:
> > On Tue, Aug 04, 2020 at 07:02:14PM +0200, Vlastimil Babka wrote:
> >> > 2) There was a proposal from Matthew Wilcox: https://lkml.org/lkml/2020/7/31/1015
> >> >
> >> > <snip>
> >> > On non-RT, we could make that lock a raw spinlock. On RT, we could
> >> > decline to take the lock. We'd need to abstract the spin_lock() away
> >> > behind zone_lock(zone), but that should be OK.
> >> > <snip>
> >> >
> >> > It would be great to use any existing flag, say GFP_NOWAIT. Suppose we
> >> > decline to take the lock across the page allocator for RT. But there is
> >> > at least one path that does it outside of the page allocator. GFP_NOWAIT
> >> > can wakeup the kswapd, whereas a "wake-up path" uses sleepable lock:
> >> >
> >> > wakeup_kswapd() -> wake_up_interruptible(&pgdat->kswapd_wait).
> >> >
> >> > Probably it can be fixed by the excluding of waking of the kswapd process
> >> > defining something like below:
> >>
> >> Is something missing here?
> >>
> >> > what is equal to zero and i am not sure if __get_free_page(0) handles
> >> > all that correctly, though it allocates and seems working on my test
> >> > machine! Please note it is related to "if we can reuse existing flags".
> >> >
> >> > In the meantime, please see below for a patch that adds a __GFP_FAST_TRY,
> >> > which can at least serve as a baseline against which other proposals can
> >> > be compared. The patch is based on the 5.8.0-rc3.
> >> >
> >> > Please RFC.
> >>
> >> At first glance __GFP_FAST_TRY (more descriptive name? __GFP_NO_LOCKS?) seems
> >> better than doing weird things with GFP_NOWAIT, but depends on the real benefits
> >> (hence my first questions).
> >
> > I think what Vlad is trying to say is that even GFP_NOWAIT will wake
> > kswapd, which involves taking a spinlock. If you specify 0 in your GFP
> > flags, then we won't wake kswapd. So a simple:
> >
> > #define GFP_NOLOCKS 0
> >
> > should do the trick (modulo various casting, blah blah blah)
>
> Ah, you're right, waking up kswapd is is only done with __GFP_KSWAPD_RECLAIM and
> GFP_NOWAIT equals to that. So that's easy to avoid for the rcu allocation.
>
> But still IIUC option 2) would mean that even with "#define GFP_NOLOCKS 0" would
> mean we need to abstract away the zone lock, and behave differently depending on
> the kernel being RT, and inadvertedly changing other users that happen to
> specify gfp where "gfp & GFP_RECLAIM_MASK == 0" (or however we would exactly
> check if we can take the lock on RT kernel). That sounds too complicated to me.
>
I think a different behaviour, i mean RT/non-rt, is not a way forward, because the
things will be over complicated. Please note, the proposed variant is common. It
provides a fast access to pcp-cache, what can be done lock-less. If we could extend
the "fast path" even do the lock-less prefetch(make fast path fully lock-less) from
the body would be fantastic, but that is a bit out of the question.

For example implement removing/inserting pages from "zone->free_area" as lock-less:
llist_add()/llist_del(). But that is theory and on the high level. During investigation
the things might become complicated.

--
Vlad Rezki