Re: [PATCH v8 0/6] LoongArch: Add pv ipi support on LoongArch VM

[Huacai Chen]

Hi, Bibo,

I have done an off-list discussion with some KVM experts, and they
think user-space have its right to know PV features, so cpucfg
solution is acceptable.

And I applied this series with some modifications at
https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson.git/log/?h=loongarch-kvm
You can test it now. But it seems the upstream qemu cannot enable PV IPI now.

I will reply to other patches about my modifications.

Huacai

On Sun, Apr 28, 2024 at 6:05 PM Bibo Mao <maobibo@xxxxxxxxxxx> wrote:
>
> On physical machine, ipi HW uses IOCSR registers, however there is trap
> into hypervisor when vcpu accesses IOCSR registers if system is in VM
> mode. SWI is a interrupt mechanism like SGI on ARM, software can send
> interrupt to CPU, only that on LoongArch SWI can only be sent to local CPU
> now. So SWI can not used for IPI on real HW system, however it can be used
> on VM when combined with hypercall method. IPI can be sent with hypercall
> method and SWI interrupt is injected to vcpu, vcpu can treat SWI
> interrupt as IPI.
>
> With PV IPI supported, there is one trap with IPI sending, however with IPI
> receiving there is no trap. with IOCSR HW ipi method, there will be one
> trap with IPI sending and two trap with ipi receiving.
>
> Also IPI multicast support is added for VM, the idea comes from x86 PV ipi.
> IPI can be sent to 128 vcpus in one time. With IPI multicast support, trap
> will be reduced greatly.
>
> Here is the microbenchmarck data with "perf bench futex wake" testcase on
> 3C5000 single-way machine, there are 16 cpus on 3C5000 single-way machine,
> VM has 16 vcpus also. The benchmark data is ms time unit to wakeup 16
> threads, the performance is better if data is smaller.
>
> physical machine                     0.0176 ms
> VM original                          0.1140 ms
> VM with pv ipi patch                 0.0481 ms
>
> It passes to boot with 128/256 vcpus, and passes to run runltp command
> with package ltp-20230516.
>
> ---
> v7 --- v8:
>  1. Remove kernel PLV mode checking with cpucfg emulation for hypervisor
> feature inquiry.
>  2. Remove document about loongarch hypercall ABI per request of huacai,
> will add English/Chinese doc at the same time in later.
>
> v6 --- v7:
>   1. Refine LoongArch virt document by review comments.
>   2. Add function kvm_read_reg()/kvm_write_reg() in hypercall emulation,
> and later it can be used for other trap emulations.
>
> v5 --- v6:
>   1. Add privilege checking when emulating cpucfg at index 0x4000000 --
> 0x400000FF, return 0 if not executed at kernel mode.
>   2. Add document about LoongArch pv ipi with new creatly directory
> Documentation/virt/kvm/loongarch/
>   3. Fix pv ipi handling in kvm backend function kvm_pv_send_ipi(),
> where min should plus BITS_PER_LONG with second bitmap, otherwise
> VM with more than 64 vpus fails to boot.
>   4. Adjust patch order and code refine with review comments.
>
> v4 --- v5:
>   1. Refresh function/macro name from review comments.
>
> v3 --- v4:
>   1. Modfiy pv ipi hook function name call_func_ipi() and
> call_func_single_ipi() with send_ipi_mask()/send_ipi_single(), since pv
> ipi is used for both remote function call and reschedule notification.
>   2. Refresh changelog.
>
> v2 --- v3:
>   1. Add 128 vcpu ipi multicast support like x86
>   2. Change cpucfg base address from 0x10000000 to 0x40000000, in order
> to avoid confliction with future hw usage
>   3. Adjust patch order in this patchset, move patch
> Refine-ipi-ops-on-LoongArch-platform to the first one.
>
> v1 --- v2:
>   1. Add hw cpuid map support since ipi routing uses hw cpuid
>   2. Refine changelog description
>   3. Add hypercall statistic support for vcpu
>   4. Set percpu pv ipi message buffer aligned with cacheline
>   5. Refine pv ipi send logic, do not send ipi message with if there is
> pending ipi message.
> ---
> Bibo Mao (6):
>   LoongArch/smp: Refine some ipi functions on LoongArch platform
>   LoongArch: KVM: Add hypercall instruction emulation support
>   LoongArch: KVM: Add cpucfg area for kvm hypervisor
>   LoongArch: KVM: Add vcpu search support from physical cpuid
>   LoongArch: KVM: Add pv ipi support on kvm side
>   LoongArch: Add pv ipi support on guest kernel side
>
>  arch/loongarch/Kconfig                        |   9 +
>  arch/loongarch/include/asm/Kbuild             |   1 -
>  arch/loongarch/include/asm/hardirq.h          |   5 +
>  arch/loongarch/include/asm/inst.h             |   1 +
>  arch/loongarch/include/asm/irq.h              |  10 +-
>  arch/loongarch/include/asm/kvm_host.h         |  27 +++
>  arch/loongarch/include/asm/kvm_para.h         | 155 ++++++++++++++++++
>  arch/loongarch/include/asm/kvm_vcpu.h         |  11 ++
>  arch/loongarch/include/asm/loongarch.h        |  11 ++
>  arch/loongarch/include/asm/paravirt.h         |  27 +++
>  .../include/asm/paravirt_api_clock.h          |   1 +
>  arch/loongarch/include/asm/smp.h              |  31 ++--
>  arch/loongarch/include/uapi/asm/Kbuild        |   2 -
>  arch/loongarch/kernel/Makefile                |   1 +
>  arch/loongarch/kernel/irq.c                   |  24 +--
>  arch/loongarch/kernel/paravirt.c              | 151 +++++++++++++++++
>  arch/loongarch/kernel/perf_event.c            |  14 +-
>  arch/loongarch/kernel/smp.c                   |  62 ++++---
>  arch/loongarch/kernel/time.c                  |  12 +-
>  arch/loongarch/kvm/exit.c                     | 132 +++++++++++++--
>  arch/loongarch/kvm/vcpu.c                     |  94 ++++++++++-
>  arch/loongarch/kvm/vm.c                       |  11 ++
>  22 files changed, 690 insertions(+), 102 deletions(-)
>  create mode 100644 arch/loongarch/include/asm/kvm_para.h
>  create mode 100644 arch/loongarch/include/asm/paravirt.h
>  create mode 100644 arch/loongarch/include/asm/paravirt_api_clock.h
>  delete mode 100644 arch/loongarch/include/uapi/asm/Kbuild
>  create mode 100644 arch/loongarch/kernel/paravirt.c
>
>
> base-commit: 5eb4573ea63d0c83bf58fb7c243fc2c2b6966c02
> --
> 2.39.3
>
>