Re: KVM: x86/vPMU/AMD: Can we detect PMU is off for a VM?

From: Denis V. Lunev
Date: Mon Nov 13 2023 - 11:33:45 EST

On 11/13/23 17:17, Dongli Zhang wrote:

On 11/13/23 06:42, Denis V. Lunev wrote:
On 11/13/23 15:14, Dongli Zhang wrote:
Hi Denis,

On 11/13/23 01:31, Denis V. Lunev wrote:
On 11/10/23 01:01, Dongli Zhang wrote:
On 11/9/23 3:46 PM, Denis V. Lunev wrote:
On 11/9/23 23:52, Jim Mattson wrote:
On Thu, Nov 9, 2023 at 10:18 AM Konstantin Khorenko
<khorenko@xxxxxxxxxxxxx> wrote:
Hi All,

as a followup for my patch: i have noticed that
currently Intel kernel code provides an ability to detect if PMU is totally
disabled for a VM
(pmu->version == 0 in this case), but for AMD code pmu->version is never 0,
no matter if PMU is enabled or disabled for a VM (i mean <pmu state='off'/>
in the VM config which
results in "-cpu pmu=off" qemu option).

So the question is - is it possible to enhance the code for AMD to also
honor
PMU VM setting or it is
impossible by design?
The AMD architectural specification prior to AMD PMU v2 does not allow
one to describe a CPU (via CPUID or MSRs) that has fewer than 4
general purpose PMU counters. While AMD PMU v2 does allow one to
describe such a CPU, legacy software that knows nothing of AMD PMU v2
can expect four counters regardless.

Having said that, KVM does provide a per-VM capability for disabling
the virtual PMU: KVM_CAP_PMU_CAPABILITY(KVM_PMU_CAP_DISABLE). See
section 8.35 in Documentation/virt/kvm/api.rst.
But this means in particular that QEMU should immediately
use this KVM_PMU_CAP_DISABLE if this capability is supported and PMU=off. I am
not seeing this code thus I believe that we have missed this. I think that
this
change worth adding. We will measure the impact :-) Den

I used to have a patch to use KVM_PMU_CAP_DISABLE in QEMU, but that did not
draw
many developers' attention.

https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/20230621013821.6874-2-dongli.zhang@xxxxxxxxxx/__;!!ACWV5N9M2RV99hQ!McSH2M-kuHmzAwTuXKxrjLkrdJoPqML6cY_Ndc-8k9LRQ7D1V9bSBRQPwHqtx9XCVLK3uzdsMaxyfwve$
It is time to first re-send that again.

Dongli Zhang
We have checked that setting KVM_PMU_CAP_DISABLE really helps. Konstantin has
done this and this is good. On the other hand, looking into these patches I
disagree with them. We should not introduce new option for QEMU. If PMU is
disabled, i.e. we assume that pmu=off passed in the command line, we should set
KVM_PMU_CAP_DISABLE for that virtual machine. Den
Can I assume you meant pmu=off, that is, cpu->enable_pmu in QEMU?

In my opinion, cpu->enable_pmu indicates the option to control the cpu features.
It may be used by any accelerators, and it is orthogonal to the KVM cap.


The KVM_PMU_CAP_DISABLE is only specific to the KVM accelerator.


That's why I had introduced a new option, to allow to configure the VM in my
dimensions.

It means one dimension to AMD, but two for Intel: to disable PMU via cpuid, or
KVM cap.

Anyway, this is KVM mailing list, and I may initiate the discussion in QEMU list.

Thank you very much!

Dongli Zhang
with the option pmu='off' it is expected that PMU should be
off for the guest. At the moment (without this KVM capability)
we can disable PMU for Intel only and thus have performance
degradation on AMD.

This option disables PMU and thus normally when we are
running KVM guest and wanting PMU to be off it would
be required to
* disable CPUID leaf for Intel
* set KVM_PMU_CAP_DISABLE for both processors This would be quite natural and
transparent for the libvirt. Alexander will prepare the patch today or tomorrow
for the discussion. Den
That is what I had implemented in the v1 of patch.

https://lore.kernel.org/all/20221119122901.2469-3-dongli.zhang@xxxxxxxxxx/

However, I changed that after people suggested introduce a new property.

Dongli Zhang
That would save a bit of our work :)

For me this patch looks absolutely awesome and is doing exactly
what I want to do in our downstream. This would get us required
15+% benefit for each VMexit.

Den