Re: [PATCH v8 0/6] LoongArch: Add pv ipi support on LoongArch VM

From: Huacai Chen
Date: Sun May 05 2024 - 21:45:32 EST


Hi, Bibo,

I have done an off-list discussion with some KVM experts, and they
think user-space have its right to know PV features, so cpucfg
solution is acceptable.

And I applied this series with some modifications at
https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson.git/log/?h=loongarch-kvm
You can test it now. But it seems the upstream qemu cannot enable PV IPI now.

I will reply to other patches about my modifications.

Huacai

On Sun, Apr 28, 2024 at 6:05 PM Bibo Mao <maobibo@xxxxxxxxxxx> wrote:
>
> On physical machine, ipi HW uses IOCSR registers, however there is trap
> into hypervisor when vcpu accesses IOCSR registers if system is in VM
> mode. SWI is a interrupt mechanism like SGI on ARM, software can send
> interrupt to CPU, only that on LoongArch SWI can only be sent to local CPU
> now. So SWI can not used for IPI on real HW system, however it can be used
> on VM when combined with hypercall method. IPI can be sent with hypercall
> method and SWI interrupt is injected to vcpu, vcpu can treat SWI
> interrupt as IPI.
>
> With PV IPI supported, there is one trap with IPI sending, however with IPI
> receiving there is no trap. with IOCSR HW ipi method, there will be one
> trap with IPI sending and two trap with ipi receiving.
>
> Also IPI multicast support is added for VM, the idea comes from x86 PV ipi.
> IPI can be sent to 128 vcpus in one time. With IPI multicast support, trap
> will be reduced greatly.
>
> Here is the microbenchmarck data with "perf bench futex wake" testcase on
> 3C5000 single-way machine, there are 16 cpus on 3C5000 single-way machine,
> VM has 16 vcpus also. The benchmark data is ms time unit to wakeup 16
> threads, the performance is better if data is smaller.
>
> physical machine 0.0176 ms
> VM original 0.1140 ms
> VM with pv ipi patch 0.0481 ms
>
> It passes to boot with 128/256 vcpus, and passes to run runltp command
> with package ltp-20230516.
>
> ---
> v7 --- v8:
> 1. Remove kernel PLV mode checking with cpucfg emulation for hypervisor
> feature inquiry.
> 2. Remove document about loongarch hypercall ABI per request of huacai,
> will add English/Chinese doc at the same time in later.
>
> v6 --- v7:
> 1. Refine LoongArch virt document by review comments.
> 2. Add function kvm_read_reg()/kvm_write_reg() in hypercall emulation,
> and later it can be used for other trap emulations.
>
> v5 --- v6:
> 1. Add privilege checking when emulating cpucfg at index 0x4000000 --
> 0x400000FF, return 0 if not executed at kernel mode.
> 2. Add document about LoongArch pv ipi with new creatly directory
> Documentation/virt/kvm/loongarch/
> 3. Fix pv ipi handling in kvm backend function kvm_pv_send_ipi(),
> where min should plus BITS_PER_LONG with second bitmap, otherwise
> VM with more than 64 vpus fails to boot.
> 4. Adjust patch order and code refine with review comments.
>
> v4 --- v5:
> 1. Refresh function/macro name from review comments.
>
> v3 --- v4:
> 1. Modfiy pv ipi hook function name call_func_ipi() and
> call_func_single_ipi() with send_ipi_mask()/send_ipi_single(), since pv
> ipi is used for both remote function call and reschedule notification.
> 2. Refresh changelog.
>
> v2 --- v3:
> 1. Add 128 vcpu ipi multicast support like x86
> 2. Change cpucfg base address from 0x10000000 to 0x40000000, in order
> to avoid confliction with future hw usage
> 3. Adjust patch order in this patchset, move patch
> Refine-ipi-ops-on-LoongArch-platform to the first one.
>
> v1 --- v2:
> 1. Add hw cpuid map support since ipi routing uses hw cpuid
> 2. Refine changelog description
> 3. Add hypercall statistic support for vcpu
> 4. Set percpu pv ipi message buffer aligned with cacheline
> 5. Refine pv ipi send logic, do not send ipi message with if there is
> pending ipi message.
> ---
> Bibo Mao (6):
> LoongArch/smp: Refine some ipi functions on LoongArch platform
> LoongArch: KVM: Add hypercall instruction emulation support
> LoongArch: KVM: Add cpucfg area for kvm hypervisor
> LoongArch: KVM: Add vcpu search support from physical cpuid
> LoongArch: KVM: Add pv ipi support on kvm side
> LoongArch: Add pv ipi support on guest kernel side
>
> arch/loongarch/Kconfig | 9 +
> arch/loongarch/include/asm/Kbuild | 1 -
> arch/loongarch/include/asm/hardirq.h | 5 +
> arch/loongarch/include/asm/inst.h | 1 +
> arch/loongarch/include/asm/irq.h | 10 +-
> arch/loongarch/include/asm/kvm_host.h | 27 +++
> arch/loongarch/include/asm/kvm_para.h | 155 ++++++++++++++++++
> arch/loongarch/include/asm/kvm_vcpu.h | 11 ++
> arch/loongarch/include/asm/loongarch.h | 11 ++
> arch/loongarch/include/asm/paravirt.h | 27 +++
> .../include/asm/paravirt_api_clock.h | 1 +
> arch/loongarch/include/asm/smp.h | 31 ++--
> arch/loongarch/include/uapi/asm/Kbuild | 2 -
> arch/loongarch/kernel/Makefile | 1 +
> arch/loongarch/kernel/irq.c | 24 +--
> arch/loongarch/kernel/paravirt.c | 151 +++++++++++++++++
> arch/loongarch/kernel/perf_event.c | 14 +-
> arch/loongarch/kernel/smp.c | 62 ++++---
> arch/loongarch/kernel/time.c | 12 +-
> arch/loongarch/kvm/exit.c | 132 +++++++++++++--
> arch/loongarch/kvm/vcpu.c | 94 ++++++++++-
> arch/loongarch/kvm/vm.c | 11 ++
> 22 files changed, 690 insertions(+), 102 deletions(-)
> create mode 100644 arch/loongarch/include/asm/kvm_para.h
> create mode 100644 arch/loongarch/include/asm/paravirt.h
> create mode 100644 arch/loongarch/include/asm/paravirt_api_clock.h
> delete mode 100644 arch/loongarch/include/uapi/asm/Kbuild
> create mode 100644 arch/loongarch/kernel/paravirt.c
>
>
> base-commit: 5eb4573ea63d0c83bf58fb7c243fc2c2b6966c02
> --
> 2.39.3
>
>