Re: KVM: x86/vPMU/AMD: Can we detect PMU is off for a VM?

[Denis V. Lunev]

On 11/13/23 15:14, Dongli Zhang wrote:
> Hi Denis,
>
> On 11/13/23 01:31, Denis V. Lunev wrote:
> > On 11/10/23 01:01, Dongli Zhang wrote:
> > > On 11/9/23 3:46 PM, Denis V. Lunev wrote:
> > > > On 11/9/23 23:52, Jim Mattson wrote:
> > > > > On Thu, Nov 9, 2023 at 10:18 AM Konstantin Khorenko
> > > > > <khorenko@xxxxxxxxxxxxx> wrote:
> > > > > > Hi All,
> > > > > >
> > > > > > as a followup for my patch: i have noticed that
> > > > > > currently Intel kernel code provides an ability to detect if PMU is totally
> > > > > > disabled for a VM
> > > > > > (pmu->version == 0 in this case), but for AMD code pmu->version is never 0,
> > > > > > no matter if PMU is enabled or disabled for a VM (i mean <pmu state='off'/>
> > > > > > in the VM config which
> > > > > > results in "-cpu pmu=off" qemu option).
> > > > > >
> > > > > > So the question is - is it possible to enhance the code for AMD to also honor
> > > > > > PMU VM setting or it is
> > > > > > impossible by design?
> > > > > The AMD architectural specification prior to AMD PMU v2 does not allow
> > > > > one to describe a CPU (via CPUID or MSRs) that has fewer than 4
> > > > > general purpose PMU counters. While AMD PMU v2 does allow one to
> > > > > describe such a CPU, legacy software that knows nothing of AMD PMU v2
> > > > > can expect four counters regardless.
> > > > >
> > > > > Having said that, KVM does provide a per-VM capability for disabling
> > > > > the virtual PMU: KVM_CAP_PMU_CAPABILITY(KVM_PMU_CAP_DISABLE). See
> > > > > section 8.35 in Documentation/virt/kvm/api.rst.
> > > > But this means in particular that QEMU should immediately
> > > > use this KVM_PMU_CAP_DISABLE if this capability is supported and PMU=off. I am
> > > > not seeing this code thus I believe that we have missed this. I think that this
> > > > change worth adding. We will measure the impact :-) Den
> > > I used to have a patch to use KVM_PMU_CAP_DISABLE in QEMU, but that did not draw
> > > many developers' attention.
> > >
> > > https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/20230621013821.6874-2-dongli.zhang@xxxxxxxxxx/__;!!ACWV5N9M2RV99hQ!McSH2M-kuHmzAwTuXKxrjLkrdJoPqML6cY_Ndc-8k9LRQ7D1V9bSBRQPwHqtx9XCVLK3uzdsMaxyfwve$
> > > It is time to first re-send that again.
> > >
> > > Dongli Zhang
> > We have checked that setting KVM_PMU_CAP_DISABLE really helps. Konstantin has
> > done this and this is good. On the other hand, looking into these patches I
> > disagree with them. We should not introduce new option for QEMU. If PMU is
> > disabled, i.e. we assume that pmu=off passed in the command line, we should set
> > KVM_PMU_CAP_DISABLE for that virtual machine. Den
> Can I assume you meant pmu=off, that is, cpu->enable_pmu in QEMU?
>
> In my opinion, cpu->enable_pmu indicates the option to control the cpu features.
> It may be used by any accelerators, and it is orthogonal to the KVM cap.
>
>
> The KVM_PMU_CAP_DISABLE is only specific to the KVM accelerator.
>
>
> That's why I had introduced a new option, to allow to configure the VM in my
> dimensions.
>
> It means one dimension to AMD, but two for Intel: to disable PMU via cpuid, or
> KVM cap.
>
> Anyway, this is KVM mailing list, and I may initiate the discussion in QEMU list.
>
> Thank you very much!
>
> Dongli Zhang
with the option pmu='off' it is expected that PMU should be
off for the guest. At the moment (without this KVM capability)
we can disable PMU for Intel only and thus have performance
degradation on AMD.

This option disables PMU and thus normally when we are
running KVM guest and wanting PMU to be off it would
be required to
* disable CPUID leaf for Intel
* set KVM_PMU_CAP_DISABLE for both processors This would be quite 
natural and transparent for the libvirt. Alexander will prepare the 
patch today or tomorrow for the discussion. Den