Re: [PATCH] KVM: x86/mmu: Do not create SPTEs for GFNs that exceed host.MAXPHYADDR

[Maxim Levitsky]

On Mon, 2022-05-02 at 16:51 +0000, Sean Christopherson wrote:
> On Mon, May 02, 2022, Maxim Levitsky wrote:
> > On Mon, 2022-05-02 at 10:59 +0300, Maxim Levitsky wrote:
> > > > > Also I can reproduce it all the way to 5.14 kernel (last kernel I have installed in this VM).
> > > > > 
> > > > > I tested kvm/queue as of today, sadly I still see the warning.
> > > > 
> > > > Due to a race, the above statements are out of order ;-)
> > > 
> > > So futher investigation shows that the trigger for this *is* cpu_pm=on :(
> > > 
> > > So this is enough to trigger the warning when run in the guest:
> > > 
> > > qemu-system-x86_64  -nodefaults  -vnc none -serial stdio -machine accel=kvm
> > > -kernel x86/dummy.flat -machine kernel-irqchip=on -smp 8 -m 1g -cpu host
> > > -overcommit cpu-pm=on
> > > 
> > > 
> > > '-smp 8' is needed, and the more vCPUs the more often the warning appears.
> > > 
> > > 
> > > Due to non atomic memslot update bug, I use patched qemu version, with an
> > > attached hack, to pause/resume vcpus around the memslot update it does, but
> > > even without this hack, you can just ctrl+c the test after it gets the KVM
> > > internal error, and then tdp mmu memory leak warning shows up (not always
> > > but very often).
> > > 
> > > 
> > > Oh, and if I run the above command on the bare metal, it  never terminates.
> > > Must be due to preemption, qemu shows beeing stuck in kvm_vcpu_block. AVIC
> > > disabled, kvm/queue.  Bugs, bugs, and features :)
> > 
> > All right, at least that was because I removed the '-device isa-debug-exit,iobase=0xf4,iosize=0x4',
> > which is apparently used by KVM unit tests to signal exit from the VM.
> 
> Can you provide your QEMU command line for running your L1 VM?  And your L0 and L1
> Kconfigs too?  I've tried both the dummy and ipi_stress tests on a variety of hardware,
> kernels, QEMUs, etc..., with no luck.
> 

So now both L0 and L1 run almost pure kvm/queue)
(commit 2764011106d0436cb44702cfb0981339d68c3509)

I have some local patches but they are not relevant to KVM at all, more
like various tweaks to sensors, a sad hack for yet another regression
in AMDGPU, etc.

The config and qemu command line attached.

AVIC disabled in L0, L0 qemu is from master upstream.
Bug reproduces too well IMHO, almost always.

For reference the warning is printed in L1's dmesg.



Best regards,
	Maxim Levitsky

Attachment: .config (L0)
Description: application/config

Attachment: .config (L1)
Description: application/config

/home/mlevitsk/.build/home/mlevitsk/Qemu/master/unstable/output/bin/qemu-system-x86_64
-smp 8
-name debug-threads=on
-pidfile /run/vmspawn/fedora30_m72p75h8//qemu.pid
-accel kvm
-nodefaults
-display none
-name guest=/home/mlevitsk/.build/home/mlevitsk/Qemu/master@unstable@Fedora34,debug-threads=on
-uuid c4aa9ea6-942a-11ea-ae1e-8c16456df72f
-qmp tcp:0:3001,server,nowait
-machine kernel-irqchip=on
-machine q35,sata=off,usb=off,vmport=on,smbus=off,smm=off
-rtc base=utc,clock=host
-no-hpet
-device pcie-root-port,slot=0,id=rport.0,bus=pcie.0,addr=0x1c.0x0,multifunction=on
-device pcie-root-port,slot=1,id=rport.1,bus=pcie.0,addr=0x1c.0x1
-device pcie-root-port,slot=2,id=rport.2,bus=pcie.0,addr=0x1c.0x2
-device pcie-root-port,slot=3,id=rport.3,bus=pcie.0,addr=0x1c.0x3
-device pcie-root-port,slot=4,id=rport.4,bus=pcie.0,addr=0x1c.0x4
-device pcie-root-port,slot=5,id=rport.5,bus=pcie.0,addr=0x1c.0x5
-device pcie-root-port,slot=6,id=rport.6,bus=pcie.0,addr=0x1c.0x6
-device pcie-root-port,slot=7,id=rport.7,bus=pcie.0,addr=0x1c.0x7
-machine smm=on
-blockdev node-name=flash0,driver=file,filename=/home/mlevitsk/FIRMWARE/ovmf/build-unstable/smm3264/OVMF_CODE.fd,read-only=on
-blockdev node-name=flash1,driver=file,filename=.fw/.ovmf_vars.fd
-machine pflash0=flash0,pflash1=flash1
-global driver=cfi.pflash01,property=secure,value=on
-boot menu=on,strict=on,splash-time=1000
-global kvm-pit.lost_tick_policy=discard
-smp maxcpus=64,sockets=1,cores=32,threads=2
-cpu host,host-cache-info,invtsc,topoext,x2apic=off,-hv-vapic,kvm-pv-ipi=off,+hv-avic
-overcommit mem-lock=on,cpu-pm=off
-m 16G,maxmem=64G
-object iothread,id=iothread0
-drive if=none,id=os_image,file=./disk_s1.qcow2,aio=native,discard=unmap,cache=none
-device virtio-blk,id=scsi-ctrl,bus=rport.0,drive=os_image,iothread=iothread0,bootindex=1,num-queues=8
-device virtio-vga,max_outputs=1,id=gpu1,bus=rport.1
-display gtk,window-close=off
-netdev tap,id=tap0,vhost=off,ifname=tap0_Fedora34,script=no,downscript=no
-device virtio-net-pci,id=net0,mac=02:00:00:21:CB:01,netdev=tap0,bus=rport.2,disable-legacy=on
-audiodev pa,id=pulseaudio0,server=/run/user/103992/pulse/native,timer-period=2000,out.mixing-engine=off,out.fixed-settings=off,out.buffer-length=50000
-device ich9-intel-hda,id=sound0,msi=on,bus=pcie.0,addr=0x1f.0x4
-device hda-micro,id=sound0-codec0,bus=sound0.0,cad=0,audiodev=pulseaudio0
-device qemu-xhci,id=usb0,bus=pcie.0,addr=0x1e.0x0,p3=16,p2=16
-device usb-tablet,id=auto_id22
-device virtio-keyboard-pci,disable-legacy=on,bus=rport.3,addr=00.0,id=auto_id23
-device virtio-serial-pci,id=virtio-serial0,bus=rport.4,disable-legacy=on
-chardev socket,id=chr_qga,path=/run/vmspawn/fedora30_m72p75h8//guest_agent.socket,server=on,wait=no
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=chr_qga,name=org.qemu.guest_agent.0,id=auto_id24
-chardev qemu-vdagent,id=ch1,name=vdagent,clipboard=on,mouse=off
-device virtserialport,bus=virtio-serial0.0,nr=4,chardev=ch1,name=com.redhat.spice.0,id=auto_id25
-chardev socket,path=/run/vmspawn/fedora30_m72p75h8//hmp_monitor.socket,id=internal_hmp_monitor_socket_chardev,server=on,wait=off
-mon chardev=internal_hmp_monitor_socket_chardev,mode=readline
-chardev socket,path=/run/vmspawn/fedora30_m72p75h8//qmp_monitor.socket,id=internal_qmp_monitor_socket_chardev,server=on,wait=off
-mon chardev=internal_qmp_monitor_socket_chardev,mode=control
-chardev socket,path=/run/vmspawn/fedora30_m72p75h8//serial.socket,id=internal_serial0_chardev,server=on,logfile=/home/mlevitsk/.shared/VM/fedora34/.logs/serial.log,wait=off
-device isa-serial,chardev=internal_serial0_chardev,index=0,id=auto_id28
-chardev file,path=/home/mlevitsk/.shared/VM/fedora34/.logs/firmware.log,id=internal_debugcon0_chardev
-device isa-debugcon,chardev=internal_debugcon0_chardev,iobase=1026,id=auto_id29