[PATCH 28/35] x86/idle: Control Indirect Branch Speculation in idle

From: Peter Zijlstra
Date: Thu Jan 18 2018 - 10:12:14 EST


From: Thomas Gleixner <tglx@xxxxxxxxxxxxx>

Indirect Branch Speculation (IBS) is controlled per physical core. If one
thread disables it then it's disabled for the core. If a thread enters idle
it makes sense to reenable IBS so the sibling thread can run with full
speculation enabled in user space.

This makes only sense in mwait_idle_with_hints() because mwait_idle() can
serve an interrupt immediately before speculation can be stopped again. SKL
which requires IBRS should use mwait_idle_with_hints() so this is a non
issue and in the worst case a missed optimization.

[peterz: fixed stop_indirect_branch_speculation placement]

Originally-by: Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx>
Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
Signed-off-by: Peter Zijlstra (Intel) <peterz@xxxxxxxxxxxxx>
---
arch/x86/include/asm/mwait.h | 14 ++++++++++++++
arch/x86/kernel/process.c | 14 ++++++++++++++
2 files changed, 28 insertions(+)

--- a/arch/x86/include/asm/mwait.h
+++ b/arch/x86/include/asm/mwait.h
@@ -6,6 +6,7 @@
#include <linux/sched/idle.h>

#include <asm/cpufeature.h>
+#include <asm/nospec-branch.h>

#define MWAIT_SUBSTATE_MASK 0xf
#define MWAIT_CSTATE_MASK 0xf
@@ -106,9 +107,22 @@ static inline void mwait_idle_with_hints
mb();
}

+ /*
+ * Indirect Branch Speculation (IBS) is controlled per
+ * physical core. If one thread disables it, then it's
+ * disabled on all threads of the core. The kernel disables
+ * it on entry from user space. Reenable it on the thread
+ * which goes idle so the other thread has a chance to run
+ * with full speculation enabled in userspace.
+ */
+ restart_indirect_branch_speculation();
__monitor((void *)&current_thread_info()->flags, 0, 0);
if (!need_resched())
__mwait(eax, ecx);
+ /*
+ * Stop IBS again to protect kernel execution.
+ */
+ stop_indirect_branch_speculation();
}
current_clr_polling();
}
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -461,6 +461,20 @@ static __cpuidle void mwait_idle(void)
mb(); /* quirk */
}

+ /*
+ * Indirect Branch Speculation (IBS) is controlled per
+ * physical core. If one thread disables it, then it's
+ * disabled on all threads of the core. The kernel disables
+ * it on entry from user space. For __sti_mwait() it's
+ * wrong to reenable it because an interrupt can be served
+ * before speculation can be stopped again.
+ *
+ * To plug that hole the interrupt entry code would need to
+ * save current state and restore. Not worth the trouble as
+ * SKL should not use mwait_idle(). It should use
+ * mwait_idle_with_hints() which can do speculation control
+ * safely.
+ */
__monitor((void *)&current_thread_info()->flags, 0, 0);
if (!need_resched())
__sti_mwait(0, 0);