Re: [PATCH v4 4/4] x86/vmware: Report guest crashes after kmsg dumpers

From: Zack Rusin

Date: Fri Nov 20 2026 - 11:50:27 EST


On Fri, Nov 20, 2026 at 5:27 AM Petr Mladek <pmladek@xxxxxxxx> wrote:
>
> Hi,
>
> this patch looks a bit messy, But maybe, I just do not know enough
> about VMvare.

It's my fault for not explaining it better. I'm trying to find a
balance between being overly verbose and explaining only what's
necessary for review, while omitting details irrelevant outside of
VMware, and I'm failing.

> On Wed 2026-11-18 01:01:51, Zack Rusin wrote:
> > VMware accepts a structured guest-crash event, but a host configured with
> > backdoor.coredumpOnCrash may terminate the VM before the hypercall
> > returns. Keep this event separate from the log sender, which runs
> > earlier in panic.
>
> I am a bit lost here ;-)
>
> What exactly means the "structured guest-crash event"?
> Which hypercall are we talking about?

VMWARE_CMD_REPORTGUESTCRASH, the vmware_hypercall1() in
vmware_report_guest_crash(). It takes no arguments and carries no
data. It tells the host that the guest OS has crashed. The host
records that, and vSphere can raise a "Guest OS crashed" event for the
VM that management tools and alarms act on.

"Structured" was a poor way of saying "a crash state the host
understands". Patch 2, by contrast, only copies log text into the VM's
log file on the host. I can drop the word "structured".

> Is this primary about synchronization of vmware_panic_log_notify()
> against termination of the crashing guest by the host?
> Or is this about something else?

It's about ordering rather than synchronization. Patch 2 sends the log
from the platform list, before kdump, so it is done before the report
can happen. Patch 4 is about the other side: it skips the report when
a crash kernel is loaded, and otherwise tries to give the guest's own
log dumpers (pstore etc.) a chance to finish first.

> What exactly does vmware_hypercall1(VMWARE_CMD_REPORTGUESTCRASH, 0)?
> Does it _block_ or _trigger_ backdoor.coredumpOnCrash?

It triggers it. Normally the call records the guest OS crash and
returns, and Linux continues its panic path. backdoor.coredumpOnCrash
is a debugging option in the VM's configuration, off by default. With
it enabled, the host attempts a dump of the guest and stops the VM
during the call, so the hypercall never returns.

So with that option set, anything panic() still has to do after the
report never happens. That always includes the kmsg dumpers that come
after the report, and, without the crash-kernel check, kdump too when
crash_kexec_post_notifiers is set. That is also why the report can't
share the platform list with the log sender.

> > Register a panic-only kmsg dumper with late_initcall_sync(), after built-in
> > dumpers registered by then.
>
> Is this ordering important?

Yes. kmsg_dump_desc() calls the dumpers in registration order, since
kmsg_dump_register() appends with list_add_tail_rcu(). Registering
from late_initcall_sync() puts the report after the panic dumpers that
are already registered at that point, giving them a chance to save
their records before a coredumpOnCrash host stops the VM. Dumpers
registered later, by modules or late-probing drivers, end up behind
it. It only makes a difference with that option set.

> > Skip the event while a crash kernel is loaded
> > so the host cannot terminate the guest before kdump saves its vmcore.
>
> This suggests that vmware_hypercall1(VMWARE_CMD_REPORTGUESTCRASH, 0)
> actually triggers the external crash dump procedure made by the host.

Yes, when the option is set. With crash_kexec_post_notifiers, kdump
runs after kmsg_dump_desc(), so the report would stop the VM before
the capture kernel started. A guest with a crash kernel loaded
therefore leaves the crash to kdump and does not report it.

> > Issue the event at most once. With CONFIG_PRINTK=n, use a late INT_MIN
> > panic notifier. An unexpected dumper registration failure with printk
> > enabled leaves reporting disabled rather than overtaking working dumpers.
>
> This is another mystery. What do you want to achieve when
> CONFIG_PRINTK is not defined, please?
>
> Note that printk() is a NOP when CONFIG_PRINTK is not defined.
> There is nothing to dump in this case.
>
> Honestly, I would not complicate the code by this.
> I doubt that anyone is using kernel without CONFIG_PRINTK.
>
> > --- a/arch/x86/kernel/cpu/vmware.c
> > +++ b/arch/x86/kernel/cpu/vmware.c
> > +static int __init vmware_crash_report_init(void)
> > +{
> > + int ret;
> > +
> > + if (!hypervisor_is_type(X86_HYPER_VMWARE))
> > + return 0;
> > +
> > + ret = kmsg_dump_register(&vmware_crash_report_dumper);
> > + if (!ret)
> > + return 0;
> > + if (IS_ENABLED(CONFIG_PRINTK)) {
> > + pr_err("failed to register crash report dumper: %d\n", ret);
> > + return 0;
>
> This is weird. As I said, printk() is a NOP when CONFIG_PRINTK
> is not defined. So, you would achieve the same with:
>
> ret = kmsg_dump_register(&vmware_crash_report_dumper);
> if (ret)
> pr_err("failed to register crash report dumper: %d\n", ret);
> return 0;
>
> But wait, this means that the code below is never reached.
>
> Ah, I see, you wanted to run the code below when CONFIG_PRINTK
> was disabled. It was far from obvious. And I do not understand
> the motivation much.
>
> Anyway, it won't work because kmsg_dump_register() returns -EINVAL
> when CONFIG_PRINTK is not defined. err != 0 ...

That -EINVAL is what the fallback relies on. With CONFIG_PRINTK=n the
stub returns -EINVAL and IS_ENABLED(CONFIG_PRINTK) is false, so init
falls through and registers the INT_MIN panic notifier instead. The
pr_err() branch is only for a real registration failure with printk
enabled.

The motivation was that the report doesn't need the log, so a kernel
without printk could still report its crash. Now having said that, is
CONFIG_PRINTK=n actively used inside vmware? Not to my knowledge, I
did it for completness more than a very specific need so if you really
don't want it in, I can remove all the CONFIG_PRINTK=n handling, at
least until I have a specific need for it.

> Let's take a step back.
>
> Why do you want to use kmsg_dump infrastructure for notifying
> host about the crash, please?
>
> The kmsg_dump API should be used to dump the printk messages.
> But this just notifies the host about the crash. It looks like
> an API misuse.

The only reason is ordering. In vpanic() the platform list runs before
kdump, and panic_notifier_list, even at INT_MIN, runs before
kmsg_dump_desc(). Registering a late dumper is the only way to run
after the dumpers registered before it. I agree that a dumper which
never reads the log is a bit hacky :)

> > + }
> > +
> > + ret = atomic_notifier_chain_register(&panic_notifier_list,
> > + &vmware_crash_report_nb);
> > + if (ret)
> > + pr_err("failed to register crash report notifier: %d\n", ret);
> > +
> > + return 0;
>
> IMHO, you either want to use the panic_notifier list.
>
> Or you might even use the new panic_notifiers_platform_list
> if you make sure that this is called as the last event
> and you really want to stop the guest at this point.

I don't think that the platform can be last: by design it runs before
kdump, and it already holds the log sender from patch 2.

panic_notifier_list at INT_MIN would work, and it would cover
CONFIG_PRINTK=n without a special case. The cost applies only when
coredumpOnCrash is set and no crash kernel is loaded: the VM would
stop before panic_print's sys_info() output and before all of the kmsg
dumpers, pstore included. The host's dump would win over pstore. On a
default host the dumpers still run; the host just hears about the
crash a little earlier.

I'm fine with either. If you don't like this approach for v5 I can
switch to a panic_notifier_list entry at INT_MIN (still skipped while
a crash kernel is loaded), or keep the dumper and add a comment
explaining why it ignores the log. Which would you prefer?

> My assumption is that the "host" will stop the "crashing guest"
> when it gets notified about the crash.

Not by default. Normally the host records the crash and the call
returns, so the guest continues its panic path. It's stopped during
the call only when backdoor.coredumpOnCrash is enabled.

> PS: It looks to me like Sashiko AI completely failed here. It did not
> mind if the commit message made sense. It did not see the big
> picture. And it even missed that the code did not work with
> CONFIG_PRINTK undefined.

I don't know if sashiko reads commit messages, that one is on me. I'm
trying to get better at finding the right balance between oversharing
and not sharing enough and it's not easy.

The CONFIG_PRINTK=n case isn't a miss, though. As above, init falls
back to the panic notifier when kmsg_dump_register() returns -EINVAL,
so that path works. It's just a question of whether it's worth having:
for me it was easy enough to handle it but I wouldn't object to
dropping it if you find it too complex.

Thanks for the careful review!

z

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature