Re: [PATCH 2/3] mm/memory: do not report OOM if insert_pfn() races a huge fault

From: Lorenzo Stoakes (ARM)

Date: Mon Nov 16 2026 - 12:22:02 EST


On Mon, Nov 16, 2026 at 04:59:35PM +0100, David Hildenbrand (Arm) wrote:
> On 11/15/26 21:58, Lorenzo Stoakes (ARM) wrote:
> > From: Eliav Farber <farbere@xxxxxxxxxx>
> >
> > A driver that implements both ->fault and ->huge_fault must anticipate
> > concurrent use of both.
> >
> > This can happen, for example, if a user calls prctl(PR_SET_THP_DISABLE)
> > concurrent with faulting, as that only holds the mmap write lock and thus
> > does not exclude faulting threads stabilised by VMA lock.
> >
> > This is not currently safely handled by insert_pfn(), however, which
> > obtains its PTE entry via get_locked_pte(), and treats a NULL return value
> > as an allocation failure, returning VM_FAULT_OOM.
> >
> > The issue is that get_locked_pte() also returns NULL if the PMD entry is
> > not a page table which can happen if a concurrent huge fault installs a
> > huge PMD entry between __handle_mm_fault() observing the PMD entry to be
> > empty and insert_pfn() being invoked.
> >
> > This can result in VM_FAULT_OOM being reported incorrectly.
> >
> > Address this by proactively checking whether a page table allocation
> > actually failed, and returning VM_FAULT_NOPAGE if the NULL indicates that
> > we were raced instead.
> >
> > The same assumption is made in insert_page(), however only DAX implements
> > ->huge_fault and can reach it, and it serialises faults of different orders
> > with its own mapping entry lock, so it does not require a change.
> >
> > Target the commit introducing the first driver able to do this.
> >
> > Link: https://lore.kernel.org/all/20261112051440.16140-3-farbere@xxxxxxxxxx/
> > Fixes: f9e54c3a2f5b ("vfio/pci: implement huge_fault support")
> > Cc: stable@xxxxxxxxxxxxxxx
> > Suggested-by: Jose A. Perez de Azpillaga <azpijr@xxxxxxxxx>
> > Signed-off-by: Eliav Farber <farbere@xxxxxxxxxx>
> > [ ljs: rewrote commit message, trimmed comment ]
> > Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
> > ---
> > mm/memory.c | 11 +++++++++--
> > 1 file changed, 9 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/memory.c b/mm/memory.c
> > index e03110906aa8..cff39eccfd42 100644
> > --- a/mm/memory.c
> > +++ b/mm/memory.c
> > @@ -2866,10 +2866,17 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr,
> > struct mm_struct *mm = vma->vm_mm;
> > pte_t *pte, entry;
> > spinlock_t *ptl;
> > + pmd_t *pmd;
> >
> > - pte = get_locked_pte(mm, addr, &ptl);
> > - if (!pte)
> > + pmd = walk_to_pmd(mm, addr);
> > + if (!pmd)
> > return VM_FAULT_OOM;
> > + if (pte_alloc(mm, pmd))
> > + return VM_FAULT_OOM;
> > + /* We may have raced a huge fault, if so let the fault be retried. */
> > + pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
> > + if (!pte)
> > + return VM_FAULT_NOPAGE;
>
> Same question, what if we would stumble over a PUD entry, how would we handle
> that today? I'd assume we'd also want to return VM_FAULT_NOPAGE instead of
> VM_FAULT_OOM.

It can't for similar reasons.

Actually changing this would require changing walk_to_pmd() a fair bit so it can
distinguish a failure to allocate from that, and I'm not sure it's worth doing
that at this point, better as follow-up?

>
>
> --
> Cheers,
>
> David

--
Cheers, Lorenzo