Re: [PATCH 00/12] KVM: arm64: Support stage2 hardware DBM

From: zhukeqian
Date: Thu Jun 18 2020 - 00:15:12 EST


Hi,

On 2020/6/16 17:35, Keqian Zhu wrote:
> This patch series add support for stage2 hardware DBM, and it is only
> used for dirty log for now.
>
> It works well under some migration test cases, including VM with 4K
> pages or 2M THP. I checked the SHA256 hash digest of all memory and
> they keep same for source VM and destination VM, which means no dirty
> pages is missed under hardware DBM.
>
> Some key points:
>
> 1. Only support hardware updates of dirty status for PTEs. PMDs and PUDs
> are not involved for now.
>
> 2. About *performance*: In RFC patch, I have mentioned that for every 64GB
> memory, KVM consumes about 40ms to scan all PTEs to collect dirty log.
>
> Initially, I plan to solve this problem using parallel CPUs. However
> I faced two problems.
>
> The first is bottleneck of memory bandwith. Single thread will occupy
> bandwidth about 500GB/s, we can support about 4 parallel threads at
> most, so the ideal speedup ratio is low.
Aha, I make it wrong here. I test it again, and find that speedup ratio can
be about 23x when I use 32 CPUs to scan PTs (takes about 5ms when scanning PTs
of 200GB RAM).

>
> The second is huge impact on other CPUs. To scan PTs quickly, I use
> smp_call_function_many, which is based on IPI, to dispatch workload
> on other CPUs. Though it can complete work in time, the interrupt is
> disabled during scaning PTs, which has huge impact on other CPUs.
I think we can divide scanning workload into smaller ones, which can disable
and enable interrupts periodly.

>
> Now, I make hardware dirty log can be dynamic enabled and disabled.
> Userspace can enable it before VM migration and disable it when
> remaining dirty pages is little. Thus VM downtime is not affected.
BTW, we can reserve this interface for userspace if CPU computing resources
are not enough.

Thanks,
Keqian
>
>
> 3. About correctness: Only add DBM bit when PTE is already writable, so
> we still have readonly PTE and some mechanisms which rely on readonly
> PTs are not broken.
>
> 4. About PTs modification races: There are two kinds of PTs modification.
>
> The first is adding or clearing specific bit, such as AF or RW. All
> these operations have been converted to be atomic, avoid covering
> dirty status set by hardware.
>
> The second is replacement, such as PTEs unmapping or changement. All
> these operations will invoke kvm_set_pte finally. kvm_set_pte have
> been converted to be atomic and we save the dirty status to underlying
> bitmap if dirty status is coverred.
>
>
> Keqian Zhu (12):
> KVM: arm64: Add some basic functions to support hw DBM
> KVM: arm64: Modify stage2 young mechanism to support hw DBM
> KVM: arm64: Report hardware dirty status of stage2 PTE if coverred
> KVM: arm64: Support clear DBM bit for PTEs
> KVM: arm64: Add KVM_CAP_ARM_HW_DIRTY_LOG capability
> KVM: arm64: Set DBM bit of PTEs during write protecting
> KVM: arm64: Scan PTEs to sync dirty log
> KVM: Omit dirty log sync in log clear if initially all set
> KVM: arm64: Steply write protect page table by mask bit
> KVM: arm64: Save stage2 PTE dirty status if it is coverred
> KVM: arm64: Support disable hw dirty log after enable
> KVM: arm64: Enable stage2 hardware DBM
>
> arch/arm64/include/asm/kvm_host.h | 11 +
> arch/arm64/include/asm/kvm_mmu.h | 56 +++-
> arch/arm64/include/asm/sysreg.h | 2 +
> arch/arm64/kvm/arm.c | 22 +-
> arch/arm64/kvm/mmu.c | 411 ++++++++++++++++++++++++++++--
> arch/arm64/kvm/reset.c | 14 +-
> include/uapi/linux/kvm.h | 1 +
> tools/include/uapi/linux/kvm.h | 1 +
> virt/kvm/kvm_main.c | 7 +-
> 9 files changed, 499 insertions(+), 26 deletions(-)
>