[PATCH v3 2/2] mm/shmem: restore SHMEM_HUGE_DENY precedence over MADV_COLLAPSE

From: Zach O'Keefe
Date: Sat Dec 24 2022 - 03:20:51 EST


SHMEM_HUGE_DENY is for emergency use by the admin, to disable allocation
of shmem huge pages if, for example, a dangerous bug is found in their
usage: see "deny" in Documentation/mm/transhuge.rst. An app using
madvise(,,MADV_COLLAPSE) should not be allowed to override it: restore
its precedence over shmem_huge_force.

Restore SHMEM_HUGE_DENY precedence over MADV_COLLAPSE.

Fixes: 7c6c6cc4d3a2 ("mm/shmem: add flag to enforce shmem THP in hugepage_vma_check()")
Suggested-by: Hugh Dickins <hughd@xxxxxxxxxx>
Signed-off-by: Zach O'Keefe <zokeefe@xxxxxxxxxx>
Cc: Yang Shi <shy828301@xxxxxxxxx>
Cc: stable@xxxxxxxxxxxxxxx
---
v2->v3: Add 'Cc: stable@xxxxxxxxxxxxxxx' as per stable-kernel-rules.
v1->v2: Update changelog, and add note explaining rationale for
backporting (Andrew Morton).

Request to backport this to 6.1.X stable. We'd like SHMEM_HUGE_DENY to
take precedence over MADV_COLLAPSE. If we make this change later, it
will be a userspace API change. As such, 6.1 cannot be allowed to
continue as-is, and we should fix up the code there.
---
mm/shmem.c | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index c301487be5fb..0005ab2c29af 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -478,12 +478,10 @@ bool shmem_is_huge(struct vm_area_struct *vma, struct inode *inode,
if (vma && ((vma->vm_flags & VM_NOHUGEPAGE) ||
test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags)))
return false;
- if (shmem_huge_force)
- return true;
- if (shmem_huge == SHMEM_HUGE_FORCE)
- return true;
if (shmem_huge == SHMEM_HUGE_DENY)
return false;
+ if (shmem_huge_force || shmem_huge == SHMEM_HUGE_FORCE)
+ return true;

switch (SHMEM_SB(inode->i_sb)->huge) {
case SHMEM_HUGE_ALWAYS:
--
2.39.0.314.g84b9a713c41-goog