Re: [BUG] Kernel panic in __migrate_swap_task() on 6.16-rc2 (NULL pointer dereference)

[Jirka Hladky]

Hi Chenyu,

I have the first

if (unlikely(!p->mm)) {
trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm, p->flags);

log from dual socket AMD Milanx 7573X server:

[74615.841401] # entries-in-buffer/entries-written: 1/1   #P:128
[74615.841401] #
[74615.841401] #                                _-----=> irqs-off/BH-disabled
[74615.841401] #                               / _----=> need-resched
[74615.841401] #                              | / _---=> hardirq/softirq
[74615.841401] #                              || / _--=> preempt-depth
[74615.841401] #                              ||| / _-=> migrate-disable
[74615.841401] #                              |||| /     delay
[74615.841401] #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
[74615.841401] #              | |         |   |||||     |         |
[74615.841401]      migration/6-54      [006] d..5. 58590.368364:
__migrate_swap_task: !! (2939199 stress-ng-sem) flags=40004c

debug code caught a task with p->mm == NULL during __migrate_swap_task().

Trace shows:
  * It was a stress-ng-sem process (PID 2939199),
  * That process was in the process of exiting (PF_EXITING | PF_EXITPIDONE),
  * And was a workqueue worker (PF_WQ_WORKER),
  * It explains why p->mm had already been freed

I'm curious as to why the command is stress-ng-sem, and the flag
indicates it is a workqueue worker.

More jobs are still running. I will keep an eye on them and report the
other cases as they finish.

Jirka

On Tue, Jul 1, 2025 at 5:10 PM Jirka Hladky <jhladky@xxxxxxxxxx> wrote:
>
> Hi Chenyu and Libo,
>
> I agree, let's wait for the results. Hopefully, we can fix the root
> cause based on the debug messages.
>
> There was a problem with the infrastructure over the weekend, so I had
> to restart the jobs yesterday. I should have results tomorrow.
>
> Stay tuned.
> Jirka
>
>
> On Tue, Jul 1, 2025 at 4:59 PM Chen, Yu C <yu.c.chen@xxxxxxxxx> wrote:
> >
> > Hi Libo,
> >
> > On 7/1/2025 3:32 PM, Libo Chen wrote:
> > > Hi Chenyu,
> > >
> > > On 6/27/25 00:33, Chen, Yu C wrote:
> > >> On 6/27/2025 3:16 PM, Chen, Yu C wrote:
> > >>> Hi Jirka,
> > >>>
> > >>> On 6/27/2025 5:46 AM, Jirka Hladky wrote:
> > >>>> Hi Chen and all,
> > >>>>
> > >>>> we have now verified that the following commit causes a kernel panic
> > >>>> discussed in this thread:
> > >>>>
> > >>>> ad6b26b6a0a79 sched/numa: add statistics of numa balance task
> > >>>>
> > >>>> Reverting this commit fixes the issue.
> > >>>>
> > >>>> I'm happy to help debug this further or test a proposed fix.
> > >>>>
> > >>>
> > >>> Thanks very much for your report, it seems that there is a
> > >>> race condition that when the swap task candidate was chosen,
> > >>> but its mm_struct get released due to task exit, then later
> > >>> when doing the task swaping, the p->mm is NULL which caused
> > >>> the problem:
> > >>>
> > >>> CPU0                                   CPU1
> > >>> :
> > >>> ...
> > >>> task_numa_migrate
> > >>>     task_numa_find_cpu
> > >>>      task_numa_compare
> > >>>        # a normal task p is chosen
> > >>>        env->best_task = p
> > >>>
> > >>>                                          # p exit:
> > >>>                                          exit_signals(p);
> > >>>                                             p->flags |= PF_EXITING
> > >>>                                          exit_mm
> > >>>                                             p->mm = NULL;
> > >>>
> > >>>      migrate_swap_stop
> > >>>        __migrate_swap_task((arg->src_task, arg->dst_cpu)
> > >>>         count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULL
> > >>>
> > >>> Could you please help check if the following debug patch works,
> > >>
> > >> Attached the patch:
> > >>
> > >> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > >> index 8988d38d46a3..82fc966b390c 100644
> > >> --- a/kernel/sched/core.c
> > >> +++ b/kernel/sched/core.c
> > >> @@ -3364,7 +3364,12 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)
> > >>   {
> > >>       __schedstat_inc(p->stats.numa_task_swapped);
> > >>       count_vm_numa_event(NUMA_TASK_SWAP);
> > >> -    count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> > >> +    if (unlikely(!p->mm)) {
> > >
> > > I am starting to wonder if we should keep this check and add a big fat warning
> > > like  "there is a bug here, please report it!" rather than brick the kernel.
> > > A kernel panic, for sure, helps catch bugs like this more than a line of dmesg,
> > > so it's a tradeoff I guess. What do you think?
> > >
> >
> > I agree with you that adding something like WARN_ON_ONCE()
> > could help, and it seems that the kernel panic is not expected
> > in this scenario because this feature is a statistic calculation
> > rather than the critical logic. Maybe waiting for Jirka's feedback
> > to decide the next step.
> >
> > thanks,
> > Chenyu
> >
> >
> >
> > >
> > > Thanks,
> > > Libo
> > >
> > >> +        trace_printk("!! (%d %s) flags=%lx\n", p->pid, p->comm,
> > >> +                p->flags);
> > >> +    } else {
> > >> +        count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
> > >> +    }
> > >>
> > >>       if (task_on_rq_queued(p)) {
> > >>           struct rq *src_rq, *dst_rq;
> > >
> >
>
>
> --
> -Jirka



-- 
-Jirka