Re: [PATCH 0/4] mm: ksm: prevent KSM from entirely breaking VMA merging

[Lorenzo Stoakes]

On Mon, May 19, 2025 at 01:53:43PM +0200, David Hildenbrand wrote:
> On 19.05.25 10:51, Lorenzo Stoakes wrote:
> > When KSM-by-default is established using prctl(PR_SET_MEMORY_MERGE), this
> > defaults all newly mapped VMAs to having VM_MERGEABLE set, and thus makes
> > them available to KSM for samepage merging. It also sets VM_MERGEABLE in
> > all existing VMAs.
> >
> > However this causes an issue upon mapping of new VMAs - the initial flags
> > will never have VM_MERGEABLE set when attempting a merge with adjacent VMAs
> > (this is set later in the mmap() logic), and adjacent VMAs will ALWAYS have
> > VM_MERGEABLE set.
>
> Just to clarify, you mean that VM_MERGEABLE is set later, during
> __mmap_new_vma()->ksm_add_vma()->__ksm_add_vma(), and we are already past
> vma_merge_new_range(), correct?

Yes.

The self test asserts this is in fact the behaviour, and if you run it in a
kernel without this patchset you can observe it in action.

>
> --
> Cheers,
>
> David / dhildenb
>