Re: [PATCH v11 023/113] KVM: TDX: allocate/free TDX vcpu structure

[Huang, Kai]

On Tue, 2023-02-28 at 16:35 -0800, Isaku Yamahata wrote:
> On Tue, Feb 28, 2023 at 09:49:10PM +0000,
> "Huang, Kai" <kai.huang@xxxxxxxxx> wrote:
> 
> > On Tue, 2023-02-28 at 12:18 -0800, Isaku Yamahata wrote:
> > > On Tue, Feb 28, 2023 at 11:52:59AM +0000,
> > > "Huang, Kai" <kai.huang@xxxxxxxxx> wrote:
> > > 
> > > > On Tue, 2023-02-28 at 03:06 -0800, Isaku Yamahata wrote:
> > > > > > > +	if (!e)
> > > > > > > +		return -ENOMEM;
> > > > > > > +	*e  = (struct kvm_cpuid_entry2) {
> > > > > > > +		.function = 1,	/* Features for X2APIC */
> > > > > > > +		.index = 0,
> > > > > > > +		.eax = 0,
> > > > > > > +		.ebx = 0,
> > > > > > > +		.ecx = 1ULL << 21,	/* X2APIC */
> > > > > > > +		.edx = 0,
> > > > > > > +	};
> > > > > > > +	vcpu->arch.cpuid_entries = e;
> > > > > > > +	vcpu->arch.cpuid_nent = 1;
> > > > > > 
> > > > > > As mentioned above, why doing it here? Won't be this be overwritten later in
> > > > > > KVM_SET_CPUID2?
> > > > > 
> > > > > Yes, user space VMM can overwrite cpuid[0x1] and APIC base MSR.  But it
> > > > > doesn't
> > > > > matter because it's a bug of user space VMM. user space VMM has to keep the
> > > > > consistency of cpuid and MSRs.
> > > > > Because TDX module virtualizes cpuid[0x1].x2apic to fixed 1, KVM value doesn't
> > > > > matter after vcpu creation.
> > > > > Because KVM virtualizes APIC base as read only to guest, cpuid[0x1].x2apic
> > > > > doesn't matter after vcpu creation as long as user space VMM keeps KVM APIC
> > > > > BASE
> > > > > value.
> > > > > 
> > > > 
> > > > Contrary, can we depend on userspace VMM to set x2APIC in CPUID, but not do this
> > > > in KVM?  If userspace doesn't do it, we treat it as userspace's bug.
> > > > 
> > > > Plus, userspace anyway needs to set x2APIC in CPUID regardless whether you have
> > > > done above here, correct?
> > > > 
> > > > I don't see the point of doing above in KVM because you are neither enforcing
> > > > anything in KVM, nor you are reducing effort of userspace.
> > > 
> > > Good idea. I can drop cpuid part from tdx_vcpu_create() and apic base part from
> > > tdx_vcpu_reset(). It needs to modify tdx_has_emulated_msr() to allow user space
> > > VMM to update APIC BASE MSR.
> > 
> > My personal preference would be:
> > 
> > 1) In KVM_SET_CPUID2, we do sanity check of CPUIDs provided by userspace, and
> > return error if not met (i.e X2APIC isn't advertised).  We already have cases
> > that KVM_SET_CPUID2 can fail, so extending to do TDX-specific check seems
> > reasonable to me too.
> 
> This is moot. The current check does only check maxphys address bit size and
> specified xfeatures are supported by host.  It's bare minimum for kvm to work.
> It doesn't try to check consistency.
> 
> 
> > 2) For APIC_BASE, you can just initialize the MSR in tdx_vcpu_reset() and ignore
> > any update (+pr_warn()?) to MSR_IA32_APIC_BASE.
> 
> The x86 common code for KVM_CREATE_VCPU, kvm_arch_vcpu_create(), calls vcpu_create,
> creates lapic, and calls vcpu_reset(). 
> 
> Setting ACPI BASE MSR with X2APIC enabled, checks if cpuid x2apic bit is set.
> Please notice guest_cpuid_has(vcpu, X86_FEATURE_X2APIC) in kvm_set_apic_base().
> To work around it, one way is set cpuid artificially in create method as this
> patch does.  Other way would be to introduce another version of
> kvm_set_apic_base() that doesn't check cpuid dedicated for this purpose.
> The third option is to make it user space responsibility to set initial reset
> value of APIC BASE MSR.
> 
> Which option do you prefer?
> 

I just recall you have already set all CPUIDs via tdx_td_init().  I would do
below:

1) keep all CPUIDs in tdx_td_init(), and make vcpu->cpuid point to that.
2) Ignore KVM_SET_CPUID2 for TDX guest (+ pr_warn(), etc).
3) Set TDX-fixed CPU registers/msrs, etc in reset_vcpu().