RE: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world might require ARM spec change?

[Salil Mehta]

Hello,

> From: bibo mao <maobibo@xxxxxxxxxxx>
> Sent: Tuesday, July 25, 2023 2:14 AM
> To: Salil Mehta <salil.mehta@xxxxxxxxxx>


[...]


> 在 2023/7/25 08:56, Salil Mehta 写道:
> > Hi Bibo,
> >
> >> From: bibo mao <maobibo@xxxxxxxxxxx>
> >> Sent: Tuesday, July 25, 2023 1:29 AM
> >> To: Salil Mehta <salil.mehta@xxxxxxxxxx>
> >> Cc: Catalin Marinas <catalin.marinas@xxxxxxx>; Jonathan Cameron
> >> <jonathan.cameron@xxxxxxxxxx>; Marc Zyngier <maz@xxxxxxxxxx>; Will Deacon
> >> <will@xxxxxxxxxx>; christoffer.dall@xxxxxxx; oliver.upton@xxxxxxxxx;
> >> mark.rutland@xxxxxxx; pbonzini@xxxxxxxxxx; Salil Mehta
> >> <salil.mehta@xxxxxxxxxx>; andrew.jones@xxxxxxxxx; yuzenghui
> >> <yuzenghui@xxxxxxxxxx>; kvmarm@xxxxxxxxxxxxxxxxxxxxx; linux-arm-
> >> kernel@xxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> >> kvm@xxxxxxxxxxxxxxx; qemu-devel@xxxxxxxxxx; james.morse@xxxxxxx;
> >> steven.price@xxxxxxx; Suzuki K Poulose <suzuki.poulose@xxxxxxx>; Jean-
> >> Philippe Brucker <jean-philippe@xxxxxxxxxx>; kvmarm@xxxxxxxxxxxxxxx; linux-coco@xxxxxxxxxxxxxxx
> >> Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world might require ARM spec change?
> >>
> >> Is vcpu hotplug supported in arm virt-machine now?
> >
> > Not yet. We are working on it. Please check the RFCs being tested.
> >
> >
> > [1] Pre-RFC V2 Changes: Support of Virtual CPU Hotplug for ARMv8 Arch (WIP)
> >     https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> > [2] [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug
> >     https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
> >
> >
> >> There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
> >> I do not know why it is not merged.
> >
> >
> > I think you are referring to patches [3], [4]? Please follow the discussion
> > for details.
>
> yeap, we reference the patch [3], [4] and benefit from them greatly -:)


I am glad that our current work is useful to more than one architecture and it
was one of the aim of our work as well but...

> The patch for LoongArch vcpu hotplug link is:
> https://lore.kernel.org/qemu-devel/cover.1689837093.git.lixianglai@xxxxxxxxxxx/T/#t


I quickly went through above patches and it looks like this patch-set is mostly
based on our latest patches which are at [1], [2] and not just at [3], [4]. As I
could see most of the functions which you have ported to your architecture are
part of our Qemu repositories [2] which we have yet to push to community. As I
am working towards RFC V2 patches and which shall be floated soon. It does not
makes sense for you to duplicate the GED/ACPI changes which are common across
architectures and which have been derived from the ARM64 vCPU Hotplug support
original patches. 

This will create merge conflicts, will break large part of our original patch-set.

Hence, I would request you to drop patches 1-4 from your patch-set or rebase
it over ARM64 original patches in a week or 2 week of time. This is to avoid
spoiling our previous years of hard work for the topic we have been persistently
making efforts as you can see through the code and our detailed presentations.

Hope you will agree we all need to respect others efforts and time in this
mode of open-source collaboration. 

Rest assured I will help you in review of your architecture specific patch-set
as it is a work of mutual interest.

Thanks for understanding!


Best Wishes,
Salil.



> Regards
> Bibo Mao
> 
> >
> >
> > [3] [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
> >     https://lore.kernel.org/all/20200613213629.21984-1-salil.mehta@xxxxxxxxxx/
> > [4] [PATCH RFC 0/4] Changes to Support *Virtual* CPU Hotplug for ARM64
> >     https://lore.kernel.org/all/20200625133757.22332-1-salil.mehta@xxxxxxxxxx/#r
> >
> >
> > In summary, there were some ARM64 Architecture constraints which were being
> > violated in the earlier patches of the kernel [4] so we had to re-think of the
> > kernel changes. The Qemu part mostly remains same with some new introductions
> > of Guest HVC/SMC hyper call exit handling in user space etc. for policy checks
> > in VMM/Qemu.
> >
> >
> > You can follow the KVMForum conference presentations [5], [6] delivered in the
> > year 2020 and 2023 to get hold of more details related to this.
> >
> >
> > [5] KVMForum 2023: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don't Support CPU Hotplug (like ARM64)
> >     https://kvm-forum.qemu.org/2023/talk/9SMPDQ/
> > [6] KVMForum 2020: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64)
> >     https://kvmforum2020.sched.com/event/eE4m
> >
> >
> >
> >> I ask this question because we propose
> >> similar patch about LoongArch system in qemu mailing list, and kernel need not be
> >> modified for vcpu hotplug.
> >
> >
> > Could you please share the link of your patches so that we can have a look and
> > draw a comparison?
> >
> >
> > Thanks
> > Salil.
> >
> >>
> >> Regards
> >> Bibo, mao
> >>
> >> 在 2023/7/19 10:35, Salil Mehta 写道:
> >>> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
> >>>
> >>> Hello,
> >>> I have recently started to dabble with ARM CCA stuff and check if our
> >>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> >>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> >>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> >>> from the host. This might be required to support vCPU Hotplug feature
> >>> in the realm world in future. vCPU Hotplug is an important feature to
> >>> support kata-containers in realm world as it reduces the VM boot time
> >>> and facilitates dynamic adjustment of vCPUs (which I think should be
> >>> true even with Realm world as current implementation only makes use
> >>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> >>>
> >>>
> >>> As per our recent changes [2], [3] related to support vCPU Hotplug on
> >>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> >>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> >>> PSCI_CPU_ON should undergo similar policy checks and I think,
> >>>
> >>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> >>> 2. This means target REC should be denied by host. Can host call
> >>>    RMI_PSCI_COMPETE in such s case?
> >>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> >>> 4. Failure condition (B5.3.3.2) should be amended with
> >>>    runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> >>>             post: result == PSCI_DENIED (?)
> >>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> >>>    PSCI_CPU_ON flow (D1.4.1)
> >>>
> >>>
> >>> I do understand that ARM CCA support is in its infancy stage and
> >>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> >>> idea right now. But specification changes require lot of time and if
> >>> this change is really required then it should be further discussed
> >>> within ARM.
> >>>
> >>> Many thanks!
> >>>
> >>>
> >>> Bes regards
> >>> Salil
> >>>
> >>>
> >>> References:
> >>>
> >>> [1] https://developer.arm.com/documentation/den0137/latest/
> >>> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> >>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2