RE: [RFC PATCH] x86, entry: Switch stacks on a paranoid entry from userspace

[Luck, Tony]

> I still wonder whether the timeout code is the real culprit.  My patch
> will slow down entry into do_machine_check by tens of cycles, several
> cachelines, and possibly a couple of TLB misses.  Given that the
> timing seemed marginal to me, it's possible (albeit not that likely)
> that it pushed the time needed for synchronization into the range of
> unreliability.

I think we have very conservative timeouts.  Here are the significant bits from mce

First SPINUNIT ... how many nanoseconds to spin "ndelay(SPINUNIT)"  before pounding on an atomic op to see if we are done:
#define SPINUNIT 100    /* 100ns */

Initialization of our timeout - comment above this claims we are aiming for "one second"

	cfg->monarch_timeout = USEC_PER_SEC;

Inside mce_start() - set up our timeout. Math says this will be 1e9

	u64 timeout = (u64)mca_cfg.monarch_timeout * NSEC_PER_USEC;

Now the actual loop:
        while (atomic_read(&mce_callin) != cpus) {
                if (mce_timed_out(&timeout)) {
                        atomic_set(&global_nwo, 0);
                        return -1;
                }
                ndelay(SPINUNIT);
        }

And the inside of mce_timed_out() ... we decrement our 1e9 by 100 for each call:

        if ((s64)*t < SPINUNIT) {
                if (mca_cfg.tolerant <= 1)
                        mce_panic("Timeout synchronizing machine check over CPUs",
                                  NULL, NULL);
                cpu_missing = 1;
                return 1;
        }
        *t -= SPINUNIT;

Not sure what the worst case is for waiting for other processors to show up. Perhaps it is
when the other cpu is in some deep C-state and we have to wait for it to power the core
back on and resync clock.  Maybe it is when a cpu has just started a "wbinvd" instruction
with a cache entirely full of dirty lines that belong on some other processor so have to
be written back across QPI (may when QPI links are congested with 10gbit network traffic?

But a full second seems like a lot ... you adding a handful of cache and TLB misses shouldn't
push us over the edge.

-Tony