Re: [PATCH v8 0/6] Introduce CET supervisor state support

[Chang S. Bae]

On 5/27/2025 4:01 AM, Chao Gao wrote:
> *: https://lore.kernel.org/all/88cb75d3-01b9-38ea-e29f-b8fefb548573@xxxxxxxxx/
>
> The issue arises because the XFD MSR retains the value (i.e., 0, indicating
> AMX enabled) from the previous process, while both the passed-in fpstate
> (init_fpstate) and the current fpstate have AMX disabled.
>
> To reproduce this issue, compile the kernel with CONFIG_PREEMPT=y, apply the
> attached diff to the amx selftest and run:
>
> # numactl -C 1 ./tools/testing/selftests/x86/amx_64
>
> diff --git a/tools/testing/selftests/x86/amx.c b/tools/testing/selftests/x86/amx.c
> index 40769c16de1b..4d533d1a530d 100644
> --- a/tools/testing/selftests/x86/amx.c
> +++ b/tools/testing/selftests/x86/amx.c
> @@ -430,6 +430,10 @@ static inline void validate_tiledata_regs_changed(struct xsave_buffer *xbuf)
> 		fatal_error("TILEDATA registers did not change");
>   }
>   
> +static void dummy_handler(int sig)
> +{
> +}
> +
>   /* tiledata inheritance test */
>   
>   static void test_fork(void)
> @@ -444,6 +448,10 @@ static void test_fork(void)
> 		/* fork() succeeded.  Now in the parent. */
> 		int status;
>   
> +		req_xtiledata_perm();
> +		load_rand_tiledata(stashed_xsave);
> +		while(1);
> +
> 		wait(&status);
> 		if (!WIFEXITED(status) || WEXITSTATUS(status))
> 			fatal_error("fork test child");
> @@ -452,7 +460,9 @@ static void test_fork(void)
> 	/* fork() succeeded.  Now in the child. */
> 	printf("[RUN]\tCheck tile data inheritance.\n\tBefore fork(), load tiledata\n");
>   
> -	load_rand_tiledata(stashed_xsave);
> +	signal(SIGSEGV, dummy_handler);
> +	while(1)
> +		raise(SIGSEGV);
>   
> 	grandchild = fork();
> 	if (grandchild < 0) {
> @@ -500,9 +510,6 @@ int main(void)
>   
> 	test_dynamic_state();
>   
> -	/* Request permission for the following tests */
> -	req_xtiledata_perm();
> -
> 	test_fork();
>   
> 	/*

The test case creates two processes -- the first uses AMX (task #1), and 
the other continuously sends signals without using AMX (task #2).

This leads to task #2 being preempted by task #1.

This behavior aligns with Sean’s report. From my investigation, the 
issue appears to have existed for quite some time and is not related to 
the changes in this series.

Here’s a summary of my findings:

== Preempt Case ==

To illustrate how the XFD MSR state becomes incorrect in this scenario:

 task #1 (fpstate->xfd=0)  task #2 (fpstate->xfd=0x80000)
 ========================  ==============================
                           handle_signal()
                           -> setup_rt_frame()
                              -> get_siframe()
                                 -> copy_fpstate_to_sigframe()
                                    -> fpregs_unlock()
                                 ...
  ...
  switch_fpu_return()
  -> fpregs_restore_userregs()
     -> restore_fpregs_from_fpstate()
        -> xfd_write_state()
           ^ IA32_XFD_MSR = 0
  ...
                                 ...
                              -> fpu__clear_user_states()
                                 -> fpregs_lock()
                                 -> restore_fpregs_from_init_fpstate()
                                    -> os_rstor()
                                       -> xfd_validate_state()
                                          ^ IA32_XFD_MSR != fpstate->xfd
                                 -> fpregs_mark_active()
                                 -> fpregs_unlock()

Since fpu__clear_user_states() marks the FPU state as valid in the end, 
an XFD MSR sync-up was clearly missing.

== Return-to-Userspace Path ==

Both tasks at that moment are on the return-to-userspace path, but at 
different points in IRQ state:

  * task #2 is inside handle_signal() and already re-enabled IRQs.
  * task #1 is after IRQ is disabled again when calling
    switch_fpu_return().

  local_irq_disable_exit_to_user()
  exit_to_user_mode_prepare()
  -> exit_to_user_mode_loop()
     -> local_irq_enable_exit_to_user()
        -> arch_do_signal_or_restart()
           -> handle_signal()
     -> local_irq_disable_exit_to_user()
  -> arch_exit_user_mode_prepare()
     -> arch_exit_work()
        -> switch_fpu_return()

This implies that fpregs_lock()/fpregs_unlock() is necessary inside 
handle_signal() when XSAVE instructions are invoked.

But, it should be okay for switch_fpu_return() to call 
fpregs_restore_userregs() without fpregs_lock().

== XFD Sanity Checker ==

The XFD sanity checker -- xfd_op_valid() -- correctly caught this issue 
in the test case. However, it may have a false negative when AMX usage 
was flipped between the two tasks.

Despite that, I don't think extending its coverage is worthwhile, as it 
would complicate the logic. The current logic and documentation seem 
sound.

== Fix Consideration ==

I think the fix is straightforward: resynchronize the IA32_XFD MSR in 
fpu__clear_user_states().

The existing xfd_update_state() function is self-contained and already 
performs feature checks and conditional MSR updates. Thus, it is not 
necessary to check the TIF_NEED_FPU_LOAD flag for this.

On the other hand, the sigreturn path already performs the XFD resync 
introduced by this commit:

  672365477ae8a ("x86/fpu: Update XFD state where required")

But I think that change was supposed to cover _both_ signal return and 
signal delivery paths. Sorry, it seems I had overlooked the latter
before.

Thanks,
Chang