Re: [PATCH][v2] PM / s2idle: Clear _TIF_POLLING_NRFLAG before suspend to idle

From: Rafael J. Wysocki
Date: Mon Jun 22 2020 - 12:19:49 EST


On Tue, Jun 16, 2020 at 6:03 AM Chen Yu <yu.c.chen@xxxxxxxxx> wrote:
>
> Suspend to idle was found to not work on Goldmont CPU recently.
> And the issue was triggered due to:
>
> 1. On Goldmont the CPU in idle can only be woken up via IPIs,
> not POLLING mode:
> Commit 08e237fa56a1 ("x86/cpu: Add workaround for MONITOR
> instruction erratum on Goldmont based CPUs")
> 2. When the CPU is entering suspend to idle process, the
> _TIF_POLLING_NRFLAG is kept on, due to cpuidle_enter_s2idle()
> doesn't properly match call_cpuidle().
> 3. Commit b2a02fc43a1f ("smp: Optimize send_call_function_single_ipi()")
> makes use of _TIF_POLLING_NRFLAG to avoid sending IPIs to
> idle CPUs.
> 4. As a result, some IPIs related functions might not work
> well during suspend to idle on Goldmont. For example, one
> suspected victim:
> tick_unfreeze() -> timekeeping_resume() -> hrtimers_resume()
> -> clock_was_set() -> on_each_cpu() might wait forever,
> because the IPIs will not be sent to the CPUs which are
> sleeping with _TIF_POLLING_NRFLAG set, and Goldmont CPU
> could not be woken up by only setting _TIF_NEED_RESCHED
> on the monitor address.
>
> I don't find a way in Ubuntu to update the firmware of Goldmont
> and check if the issue was gone, a fix patch would do no harm.
> Clear the _TIF_POLLING_NRFLAG flag before entering suspend to idle,
> and let the driver's enter_s2idle() to decide whether to set
> _TIF_POLLING_NRFLAG or not. So that to avoid the scenario described
> above and keep the context consistent with before. Also adjust
> the naming to be consistent with call_cpuidle().
>
> Fixes: b2a02fc43a1f ("smp: Optimize send_call_function_single_ipi()")
> Suggested-by: Peter Zijlstra (Intel) <peterz@xxxxxxxxxxxxx>
> Reported-by: kbuild test robot <lkp@xxxxxxxxx>
> Cc: "Rafael J. Wysocki" <rafael@xxxxxxxxxx>
> Cc: Len Brown <lenb@xxxxxxxxxx>
> Cc: Peter Zijlstra (Intel) <peterz@xxxxxxxxxxxxx>
> Signed-off-by: Chen Yu <yu.c.chen@xxxxxxxxx>

Peter, any more comments here?

> ---
> v2: According to Peter's review, v1 is racy, if someone already
> set TIF_NEED_RESCHED this patch just clear POLLING and go to sleep.
> Check TIF_NEED_RESCHED before entering suspend to idle and
> adjust the naming to be consistent with call_cpuidle().
> --
> drivers/cpuidle/cpuidle.c | 16 +++++++++++++---
> 1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/cpuidle/cpuidle.c b/drivers/cpuidle/cpuidle.c
> index c149d9e20dfd..b003767abebd 100644
> --- a/drivers/cpuidle/cpuidle.c
> +++ b/drivers/cpuidle/cpuidle.c
> @@ -13,6 +13,7 @@
> #include <linux/mutex.h>
> #include <linux/sched.h>
> #include <linux/sched/clock.h>
> +#include <linux/sched/idle.h>
> #include <linux/notifier.h>
> #include <linux/pm_qos.h>
> #include <linux/cpu.h>
> @@ -133,8 +134,8 @@ int cpuidle_find_deepest_state(struct cpuidle_driver *drv,
> }
>
> #ifdef CONFIG_SUSPEND
> -static void enter_s2idle_proper(struct cpuidle_driver *drv,
> - struct cpuidle_device *dev, int index)
> +static void s2idle_enter(struct cpuidle_driver *drv,
> + struct cpuidle_device *dev, int index)
> {
> ktime_t time_start, time_end;
>
> @@ -168,6 +169,15 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
> dev->states_usage[index].s2idle_usage++;
> }
>
> +static int call_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev,
> + int index)
> +{
> + if (!current_clr_polling_and_test())
> + s2idle_enter(drv, dev, index);
> +
> + return index;

Is the value returned here used at all?

> +}
> +
> /**
> * cpuidle_enter_s2idle - Enter an idle state suitable for suspend-to-idle.
> * @drv: cpuidle driver for the given CPU.
> @@ -187,7 +197,7 @@ int cpuidle_enter_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev)
> */
> index = find_deepest_state(drv, dev, U64_MAX, 0, true);
> if (index > 0)
> - enter_s2idle_proper(drv, dev, index);
> + call_s2idle(drv, dev, index);

I'm wondering why this can't be

if (index > 0 && !current_clr_polling_and_test())
enter_s2idle_proper(drv, dev, index);

> return index;
> }
> --