Re: NOHZ: WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule,round 2

From: Frederic Weisbecker
Date: Mon May 20 2013 - 04:10:21 EST

2013/5/20 Borislav Petkov <bp@xxxxxxxxx>:
> On Mon, May 20, 2013 at 11:16:33AM +0800, Michael Wang wrote:
>> I suppose the reason is that the cpu we passed to
>> mod_delayed_work_on() has a chance to become offline before we
>> disabled irq, what about check it before send resched ipi? like:
> I think this is only addressing the symptoms - what we should be doing
> instead is asking ourselves why are we even scheduling work on a cpu if
> the machine goes offline?
> I don't know though who should be responsible for killing all that
> work - the workqueue itself or the guy who created it, i.e. cpufreq
> governor...
> Hmmm.

Let's look at this portion of cpu_down():

err = __stop_machine(take_cpu_down, &tcd_param, cpumask_of(cpu));
if (err) {
/* CPU didn't die: tell everyone. Can't complain. */
cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
goto out_release;

* The migration_call() CPU_DYING callback will have removed all
* runnable tasks from the cpu, there's only the idle task left now
* that the migration thread is done doing the stop_machine thing.
* Wait for the stop thread to go away.
while (!idle_cpu(cpu))
/* This actually kills the CPU. */

/* CPU is completely dead: tell everyone. Too late to complain. */
cpu_notify_nofail(CPU_DEAD | mod, hcpu);


The CPU is considered offline after the take_cpu_down stop machine job
completes. But the struct timer_list timers are migrated later through
CPU_DEAD notification. Only once that's completed we check for illegal
residual tasks in the CPU. So there is a little window between the
stop machine thing and __cpu_die() where a timer can fire with
cpu_online(cpu) == 1.

Now concerning the workqueue I don't know. I guess the per cpu ones
are not migrated due to their affinity. Apparently they can still wake
up and execute works due to the timers...
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at