Re: cpu stopper threads and setaffinity leads to deadlock

From: Peter Zijlstra
Date: Thu Aug 02 2018 - 12:03:15 EST


On Wed, Aug 01, 2018 at 06:34:40PM -0700, Sodagudi Prasad wrote:
> Due to cross migration of tasks between cpu7 and cpu3, migration/7 has
> started executing and waits for the migration/3 task, so that they can
> proceed within the multi cpu stop state machine together.
> Unfortunately stress-ng-affin is affine to cpu7, and since migration 7 has
> started running, and has monopolized cpu7âs execution, stress-ng will never
> run on cpu7, and cpu3âs migration task is never woken up.

> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index e190d1e..f932e1e 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -87,9 +87,9 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct
> cpu_stop_work *work)
> __cpu_stop_queue_work(stopper, work, &wakeq);
> else if (work->done)
> cpu_stop_signal_done(work->done);
> - raw_spin_unlock_irqrestore(&stopper->lock, flags);
>
> wake_up_q(&wakeq);
> + raw_spin_unlock_irqrestore(&stopper->lock, flags);
>

So why didn't you do the 'obvious' parallel to what you did for
cpu_stop_queue_two_works(), namely:

--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -81,6 +81,7 @@ static bool cpu_stop_queue_work(unsigned
unsigned long flags;
bool enabled;

+ preempt_disable();
raw_spin_lock_irqsave(&stopper->lock, flags);
enabled = stopper->enabled;
if (enabled)
@@ -90,6 +91,7 @@ static bool cpu_stop_queue_work(unsigned
raw_spin_unlock_irqrestore(&stopper->lock, flags);

wake_up_q(&wakeq);
+ preempt_enable();

return enabled;
}