Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
From: David Laight
Date: Fri Nov 20 2026 - 09:02:09 EST
On Fri, 20 Nov 2026 11:45:34 +0100
Peter Zijlstra <peterz@xxxxxxxxxxxxx> wrote:
> On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:
> > +#ifdef CONFIG_X86_64
> > +#define __stackleak_poison __stackleak_poison
> > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > + unsigned long erase_high,
> > + unsigned long poison)
> > +{
> > + unsigned long count;
> > +
> > + /* Short spans do not amortize the startup cost of REP STOSQ. */
> > + if (erase_high <= erase_low + 512) {
> > + while (erase_low < erase_high) {
> > + *(unsigned long *)erase_low = poison;
> > + erase_low += sizeof(unsigned long);
> > + }
> > + return;
> > + }
> > +
> > + count = (erase_high - erase_low) / sizeof(unsigned long);
> > + asm volatile("rep stosq"
> > + : "+D" (erase_low), "+c" (count)
> > + : "a" (poison)
> > + : "memory");
> > +}
> > +#endif
>
> Gemini seems to suggest 16-32 words would be sufficient to amortize the
> REP STOS setup.
That really does sound like wishful thinking.
Especially if you go back a few generations on the cpu architecture.
> It was also adamant I try and unroll that earlier loop.
>
> How about something like so, does that make sense? Also, I suppose we
> should make memset64() __always_inline if we use it here. I'm not at all
> sold on the unroll here, the asm is pretty horrid for what it does.
>
> ---
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -28,6 +28,7 @@ struct vm86;
> #include <asm/vmxfeatures.h>
> #include <asm/vdso/processor.h>
> #include <asm/shstk.h>
> +#include <asm/string.h>
>
> #include <linux/personality.h>
> #include <linux/cache.h>
> @@ -580,6 +581,33 @@ static __always_inline bool on_thread_st
> current_stack_pointer) < THREAD_SIZE;
> }
>
> +#ifdef CONFIG_X86_64
> +#define __stackleak_poison __stackleak_poison
> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> + unsigned long erase_high,
> + unsigned long poison)
> +{
> + unsigned long count = (erase_high - erase_low) / sizeof(unsigned long);
> +
> + /* Amortize the startup cost of REP STOSQ. */
> + if (count < 32) {
> + unsigned long *ptr = (unsigned long *)erase_low;
> +
> + while (count) {
> + switch ((count-1) % 4) {
> + case 3: ptr[--count] = poison; fallthrough;
> + case 2: ptr[--count] = poison; fallthrough;
> + case 1: ptr[--count] = poison; fallthrough;
> + case 0: ptr[--count] = poison; break;
> + }
> + }
That looks horrid to me.
You want as few instructions (especially conditionals) inside the
loop as possible.
The Intel cpu will execute a two clock loop and the later ones will
do two writes per clock
(zen5 will execute a one clock loop and do two writes/clock).
So you want a loop with just 4 writes and the loop control code.
The 'pointer add' just needs to execute at the same time as some
of the stores - to avoid lengthening the register dependency chain.
For this code, can the length be a multiple of 32?
Also on some cpu the 'rep stos' might be twice as fast (per byte) if
the destination in 64 byte aligned - true for 'rep movsb'.
David
> + return;
> + }
> +
> + memset64((void *)erase_low, poison, count);
> +}
> +#endif
> +
> #ifdef CONFIG_PARAVIRT_XXL
> #include <asm/paravirt.h>
> #else
>