[PATCH] arm64: fix per-cpu offset initialization

[Mark Rutland]

Qian sees a boot-time hang introduced by commit:

  353e228eb355be5a ("arm64: initialize per-cpu offsets earlier")

... which happens because task_cpu() can be instrumented by KASAN, and
we call init_this_cpu_offset() before we've performed the early KASAN
initialization.

We don't need to initialize the per-cpu offset before the early KASAN
initialization runs (and we didn't prior to the patch in question), so
we can avoid bothering with that.

However, were task_cpu() instrumented with something else, this could
cause similar issues, so let's also open-code that within
init_this_cpu_offset() to avoid that possibility.

It's also possible that set_my_cpu_offset() gets instrumented in
future, so let's avoid that by marking it __always_inline. It's only
used by init_this_cpu_offset(), so this doesn't matter for any other
code.

Finally, per_cpu_offset(x) is a macro expanding to __per_cpu_offset[x],
which is inlined and not instrumented.

Fixes: 353e228eb355be5a ("arm64: initialize per-cpu offsets earlier")
Signed-off-by: Mark Rutland <mark.rutland@xxxxxxx>
Reported-by:  Qian Cai <cai@xxxxxxxxxx>
Cc: Catalin Marinas <catalin.marinas@xxxxxxx>
Cc: Will Deacon <will@xxxxxxxxxx>
---
 arch/arm64/include/asm/percpu.h | 2 +-
 arch/arm64/kernel/head.S        | 2 --
 arch/arm64/kernel/setup.c       | 2 +-
 3 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 0b6409b89e5e0..0c347d3faf55c 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -11,7 +11,7 @@
 #include <asm/cmpxchg.h>
 #include <asm/stack_pointer.h>
 
-static inline void set_my_cpu_offset(unsigned long off)
+static __always_inline void set_my_cpu_offset(unsigned long off)
 {
 	asm volatile(ALTERNATIVE("msr tpidr_el1, %0",
 				 "msr tpidr_el2, %0",
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index e28c9d4e5278c..9bbea14a9ca3f 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -448,8 +448,6 @@ SYM_FUNC_START_LOCAL(__primary_switched)
 	bl	__pi_memset
 	dsb	ishst				// Make zero page visible to PTW
 
-	bl	init_this_cpu_offset
-
 #ifdef CONFIG_KASAN
 	bl	kasan_early_init
 #endif
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 005171972764b..161eaa83264ea 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -278,7 +278,7 @@ EXPORT_SYMBOL_GPL(cpu_logical_map);
 
 void noinstr init_this_cpu_offset(void)
 {
-	unsigned int cpu = task_cpu(current);
+	unsigned int cpu = current->cpu;
 	set_my_cpu_offset(per_cpu_offset(cpu));
 }
 
-- 
2.11.0