Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Jinpu Wang]

Hi Sean,

On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@xxxxxxxxxx> wrote:
>
> Trimmed Cc: to drop people from the original thread.  In the future, just start
> a new bug report.  Piggybacking a 10 year old bug just because the symptoms are
> similar does more harm than good.  Whatever the old thread was chasing was already
> fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> the problem.  I.e. whatever they were chasing _can't_ be the same root cause,
> because even if it's literally the same code bug, it would require a code change
> and thus a regression between v4.0 and v6.1.
Thx for the reply,  it makes sense. I will remember this next time.
>
> On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@xxxxxxxxx> wrote:
> > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>
> Are there any host kernels that _do_ work?  E.g. have you tried a bleeding edge
> host kernel?
I will try linus/master today.
>
> > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > >
> > > **Summary:**
> > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
>
> Does Sierra Forest have IPI virtualization?  If so, you could try running with
> APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
Yes, it does:
$  grep . /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
/sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:Y
/sys/module/kvm_intel/parameters/enable_ipiv:Y
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:Y
/sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:Y
/sys/module/kvm_intel/parameters/nested_early_check:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/ple_window_grow:2
/sys/module/kvm_intel/parameters/ple_window_max:4294967295
/sys/module/kvm_intel/parameters/ple_window_shrink:0
/sys/module/kvm_intel/parameters/pml:Y
/sys/module/kvm_intel/parameters/preemption_timer:Y
/sys/module/kvm_intel/parameters/sgx:N
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
/sys/module/kvm_intel/parameters/vnmi:Y
/sys/module/kvm_intel/parameters/vpid:Y

I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
sudo modprobe -r kvm_intel
sudo modprobe  kvm_intel enable_ipiv=N
/sys/module/kvm_intel/parameters/enable_ipiv:N

Thx!