Re: [PATCH v5 15/17] mm/mglru: folio LRU refs based active/inactive number accounting
From: KunWu Chan
Date: Fri Nov 20 2026 - 04:30:19 EST
On Thu, Nov 19, 2026 at 5:04 AM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@xxxxxxxxxx> wrote:
>
> From: Kairui Song <kasong@xxxxxxxxxxx>
>
> Currently the active/inactive LRU sizes exported through /proc/vmstat
> and memory.stat are derived from the generation window: folios in the
> two newest generations are accounted as active, the rest as inactive.
> This causes several problems:
>
> - Unmapped file folios can stay in older gens despite repeated use,
> so active files can be under-reported, unlike classical LRU.
>
> - The reading is jumpy: on aging, the second-newest gen, previously
> active, becomes inactive all at once, causing a huge drop of active
> folios without any change in use.
>
> - The reading is reversed on swapless machines. When swapping is
> disabled or unavailable, MGLRU ages anon by moving the gens directly
> without walking or touching any folio, to reduce the overhead. So
> inactive anon folios can suddenly count as active, or vice versa,
> despite no change in their hotness.
>
> With frequency guided promotion, hotness is already tracked per folio by
> its referenced count, so we can do folio level accounting instead: a
> folio is active once its refs reach LRU_REFS_ACTIVATED (currently
> LRU_REFS_PROTECTED).
>
> On reaching this threshold, folios are accounted to the active part.
> On forced per-folio aging or PID protection, refs are capped at
> LRU_REFS_WORKINGSET, dropping active folios to the inactive part.
> Note that on swapless machines anon refs are not reset by aging (the
> shortcut skips the per-folio walk), but the counts no longer oscillate
> as the generation indices wrap.
>
> The accounting is now per folio and lruvec independent, so no size
> adjustment is needed when reparenting a dying memcg: the hierarchical
> stats keep the counts in the parent automatically.
>
> folio_migrate_lru_refs() also gains off-LRU VM_WARN_ON_ONCE checks and
> __folio_set_lru_refs() warns when the folio is still on an LRU list.
>
> The latter uses lru_get_gen_flags(), so that helper now has to move out of
> the CONFIG_LRU_GEN block and is built with the option off, where
> LRU_GEN_WIDTH is 0 and LRU_GEN_MASK is zero. Keep the decoded value in
> int so the subtraction of the one-off encoding cannot fold to ULONG_MAX
> before it happens: that is -Woverflow, and fatal with CONFIG_WERROR.
>
> Signed-off-by: Kairui Song <kasong@xxxxxxxxxxx>
> ---
> Documentation/admin-guide/mm/multigen_lru.rst | 4 +-
> include/linux/mm_inline.h | 118 +++++++-------
> include/linux/mmzone.h | 23 ++-
> mm/folio.c | 39 +----
> mm/madvise.c | 2 +-
> mm/vmscan.c | 218 +++++++++++++++++---------
> mm/workingset.c | 2 +-
> 7 files changed, 230 insertions(+), 176 deletions(-)
>
> diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst
> index 9cb54b4ff5d9..808d2e97b3f3 100644
> --- a/Documentation/admin-guide/mm/multigen_lru.rst
> +++ b/Documentation/admin-guide/mm/multigen_lru.rst
> @@ -149,8 +149,8 @@ generations less than or equal to ``min_gen_nr``.
> ``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]``
>
> ``min_gen_nr`` should be less than ``max_gen_nr-1``, since
> -``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to
> -the active list) and therefore cannot be evicted. ``swappiness``
> +``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged and therefore
> +cannot be evicted. ``swappiness``
> overrides the default value in ``/proc/sys/vm/swappiness`` and the valid
> range is [0-200, max], with max being exclusively used for the reclamation
> of anonymous memory. ``nr_to_reclaim`` limits the number of pages to evict.
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index 14aab2b1ed08..77f514143266 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -105,6 +105,22 @@ static __always_inline enum lru_list folio_lru_list(const struct folio *folio)
> return lru;
> }
>
> +/**
> + * lru_get_gen_flags - Return the LRU generation number from folio flags.
> + * @flags: folio flags
> + *
> + * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
> + * -1 if the flags indicate the folio is off the list (e.g., isolated).
> + */
> +static inline int lru_get_gen_flags(unsigned long flags)
> +{
> + int gen = (int)((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
> +
> + /* The gen is stored offset by one, so zero means off-list */
> + VM_WARN_ON_ONCE(gen > MAX_NR_GENS);
> + return gen - 1;
> +}
> +
> #ifdef CONFIG_LRU_GEN
>
> static inline bool lru_gen_switching(void)
> @@ -168,22 +184,6 @@ static inline void lru_set_gen_flags(unsigned long *flags, int gen)
> *flags |= (gen + 1UL) << LRU_GEN_PGOFF;
> }
>
> -/**
> - * lru_get_gen_flags - Return the LRU generation number from folio flags.
> - * @flags: folio flags
> - *
> - * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
> - * -1 if the flags indicate the folio is off the list (e.g., isolated).
> - */
> -static inline int lru_get_gen_flags(unsigned long flags)
> -{
> - int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> -
> - /* Exclude the legal -1 from the unsigned MAX_NR_GENS comparison */
> - VM_WARN_ON_ONCE(gen != -1 && gen >= MAX_NR_GENS);
> - return gen;
> -}
> -
> static inline int folio_lru_gen(const struct folio *folio)
> {
> return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> @@ -247,12 +247,21 @@ static inline bool lru_refs_is_workingset(unsigned int refs)
> return refs >= LRU_REFS_WORKINGSET;
> }
>
> -static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> +/**
> + * __folio_set_lru_refs - Set a folio's LRU refs.
> + * @folio: the folio
> + * @refs: the new referenced count (0 .. LRU_REFS_MAX)
> + *
> + * Set the folio's LRU refs. The folio must be off the LRU list (e.g.,
> + * isolated), or use folio_inc_lru_refs or folio_reset_lru_refs instead.
> + */
> +static inline void __folio_set_lru_refs(struct folio *folio, unsigned int refs)
> {
> unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>
> do {
> new_flags = old_flags;
> + VM_WARN_ON_ONCE(lru_get_gen_flags(old_flags) != -1);
> lru_set_refs_flags(&new_flags, refs);
> if (new_flags == old_flags)
> break;
> @@ -261,12 +270,18 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
>
> #ifdef CONFIG_LRU_GEN
> void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
> +bool folio_reset_lru_refs(struct folio *folio);
> #else
> static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> {
> /* Should not be called with !CONFIG_LRU_GEN */
> WARN_ON_ONCE(1);
> }
> +
> +static inline bool folio_reset_lru_refs(struct folio *folio)
> +{
> + return false;
> +}
> #endif
>
> /**
> @@ -280,10 +295,18 @@ static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> */
> static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
> {
> - folio_set_lru_refs(new, folio_lru_refs(old));
> + VM_WARN_ON_ONCE_FOLIO(folio_test_lru(old), old);
> + VM_WARN_ON_ONCE_FOLIO(folio_test_lru(new), new);
> + __folio_set_lru_refs(new, folio_lru_refs(old));
> }
>
> #ifdef CONFIG_LRU_GEN
> +static inline bool lru_refs_is_active(unsigned int refs)
> +{
> + VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> + return refs >= LRU_REFS_ACTIVATED;
> +}
> +
> static inline bool lru_gen_is_old(struct lruvec *lruvec, int type, int gen)
> {
> unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
> @@ -294,23 +317,12 @@ static inline bool lru_gen_is_old(struct lruvec *lruvec, int type, int gen)
> gen == lru_gen_from_seq(min_seq + 1);
> }
>
> -static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
> -{
> - unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
> -
> - VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
> -
> - /* see the comment on MIN_NR_GENS */
> - return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
> -}
> -
> static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *folio,
> int old_gen, int new_gen)
> {
> int type = folio_is_file_lru(folio);
> int zone = folio_zonenum(folio);
> int delta = folio_nr_pages(folio);
> - enum lru_list lru = type * LRU_INACTIVE_FILE;
> struct lru_gen_folio *lrugen = &lruvec->lrugen;
>
> VM_WARN_ON_ONCE(old_gen != -1 && old_gen >= MAX_NR_GENS);
> @@ -321,19 +333,6 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
> atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
> if (new_gen >= 0)
> atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
> -
> - /* return now if not a promotion */
> - if (old_gen < 0 || new_gen < 0)
> - return;
> -
> - /* promotion */
> - if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) {
> - __update_lru_size(lruvec, lru, zone, -delta);
> - __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta);
> - }
> -
> - /* demotion requires isolation, e.g., lru_deactivate_fn() */
> - VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen));
> }
>
> static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
> @@ -376,7 +375,8 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> {
> unsigned long seq;
> unsigned long flags;
> - int gen = folio_lru_gen(folio);
> + unsigned int refs;
> + int gen;
> int type = folio_is_file_lru(folio);
> int zone = folio_zonenum(folio);
> int delta = folio_nr_pages(folio);
> @@ -384,7 +384,7 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> struct lru_gen_folio *lrugen = &lruvec->lrugen;
>
> BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
> - VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
> + VM_WARN_ON_ONCE_FOLIO(folio_lru_gen(folio) != -1, folio);
>
> if (folio_test_unevictable(folio) || !lrugen->enabled)
> return false;
> @@ -395,10 +395,12 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> gen = lru_gen_from_seq(seq);
> flags = (gen + 1UL) << LRU_GEN_PGOFF;
> /* see the comment on MIN_NR_GENS about PG_active */
> - set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
> + flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
> + /* use the refs from the atomic snapshot to avoid raced update */
> + refs = lru_get_refs_flags(flags);
>
> lru_gen_update_size(lruvec, folio, -1, gen);
> - if (lru_gen_is_active(lruvec, gen))
> + if (lru_refs_is_active(refs))
> lru += LRU_ACTIVE;
> __update_lru_size(lruvec, lru, zone, delta);
>
> @@ -425,24 +427,33 @@ static inline bool lru_gen_add_rejected_folio(struct lruvec *lruvec, struct foli
> static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
> {
> unsigned long flags;
> - int gen = folio_lru_gen(folio);
> + unsigned int refs;
> + int gen;
> int zone = folio_zonenum(folio);
> int delta = folio_nr_pages(folio);
> enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
> + unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
>
> + /* Keep LRU_REFS_FLAGS: refs carry access history across isolation */
> + flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, 0);
> + gen = lru_get_gen_flags(flags);
> + refs = lru_get_refs_flags(flags);
> if (gen < 0)
> return false;
>
> VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
> VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
>
> - /* for folio_migrate_flags() */
> - flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
> - flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
> - gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> + /*
> + * See the comment in lru_gen_folio_seq. For migration, compaction,
> + * or any other isolation of a hot folio, try best to retain its gen
> + * info. Ideally we would keep the full gen info.
> + */
> + if (!reclaiming && ((max_seq - gen) % MAX_NR_GENS) < MIN_NR_GENS)
> + folio_set_active(folio);
>
> lru_gen_update_size(lruvec, folio, gen, -1);
> - if (lru_gen_is_active(lruvec, gen))
> + if (lru_refs_is_active(refs))
> lru += LRU_ACTIVE;
> __update_lru_size(lruvec, lru, zone, -delta);
> list_del(&folio->lru);
> @@ -499,6 +510,7 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
> return;
> }
>
> + BUILD_BUG_ON(LRU_REFS_REFERENCED >= LRU_REFS_ACTIVATED);
> old_flags = READ_ONCE(*folio_flags(folio, 0));
> do {
> if (lru_get_refs_flags(old_flags))
> @@ -534,7 +546,7 @@ static inline bool folio_check_lru_refs(const struct folio *folio)
> static inline void folio_clear_lru_refs(struct folio *folio)
> {
> if (lru_gen_enabled())
> - folio_set_lru_refs(folio, 0);
> + folio_reset_lru_refs(folio);
> else
> folio_clear_referenced(folio);
> }
Can clear_refs race with disabling MGLRU? Once the static key is
off, it can drop refs from 3 to 2 on a folio still on a gen list.
The drain then subtracts it from inactive, though it was counted
as active.
Thanks,
Kunwu