Re: [External] Re: [PATCH v7 0/9] Parallel CPU bringup for x86_64

[Usama Arif]

On 09/02/2023 10:06, David Woodhouse wrote:
> On Thu, 2023-02-09 at 09:49 +0000, Usama Arif wrote:
> > --- a/arch/x86/kernel/smpboot.c
> > +++ b/arch/x86/kernel/smpboot.c
> > @@ -1080,6 +1080,7 @@ wakeup_cpu_via_init_nmi(int cpu, unsigned long
> > start_ip, int apicid,
> >                                             wakeup_cpu0_nmi, 0, "wake_cpu0");
> >
> >           if (!boot_error) {
> > +               initial_gs = per_cpu_offset(cpu);
>
> That's for 64-bit.
>
> >                   enable_start_cpu0 = 1;
> >                   *cpu0_nmi_registered = 1;
> >                   id = apic->dest_mode_logical ? cpu0_logical_apicid :
> > apicid;
> > @@ -1188,10 +1189,14 @@ static int do_boot_cpu(int apicid, int cpu,
> > struct task_struct *idle,
> >                   boot_error = apic->wakeup_secondary_cpu_64(apicid,
> > start_ip);
> >           else if (apic->wakeup_secondary_cpu)
> >                   boot_error = apic->wakeup_secondary_cpu(apicid, start_ip);
> > -       else
> > +       else {
> > +               if(!cpu) {
> > +                       early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu);
> > +                       initial_stack  = idle->thread.sp;
> > +               }
>
> And that's 32-bit.
>
> These were previously done in common_cpu_up() or do_boot_cpu(), which
> means they were done not only for the wakeup_cpu_via_init_nmi() code
> path, but also when we call apic->wakeup_secondary_cpu() (or _64()) for
> the esoteric UV/NumaConnect machines.
>
> Thanks for diagnosing it so quickly; I'll work up a subtly different
> fix.

Yeah, was just a hacky fix while I was trying to figure out the issue.

do_boot_cpu is only called in cpu0 in hotplug case, so I think this a 
much better diff:

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 3ec5182d9698..f7ae82969ee4 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1136,13 +1136,16 @@ static int do_boot_cpu(int apicid, int cpu, 
struct task_struct *idle,
        idle->thread.sp = (unsigned long)task_pt_regs(idle);
        initial_code = (unsigned long)start_secondary;

-       if (IS_ENABLED(CONFIG_X86_32)) {
+       if (IS_ENABLED(CONFIG_X86_32) || !cpu) {
                early_gdt_descr.address = (unsigned 
long)get_cpu_gdt_rw(cpu);
                initial_stack  = idle->thread.sp;
        } else if (!do_parallel_bringup) {
                smpboot_control = STARTUP_SECONDARY | apicid;
        }

+       if (!cpu)
+               initial_gs = per_cpu_offset(cpu);
+
        /* Enable the espfix hack for this CPU */
        init_espfix_ap(cpu);



> >                   boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid,
> >                                                        cpu0_nmi_registered);
> > -
> > +       }
> >           return boot_error;
> >    }