Re: [PATCH 16/17] TSC reset compensation

From: Zachary Amsden
Date: Tue Jun 15 2010 - 16:32:46 EST

On 06/14/2010 10:51 PM, Avi Kivity wrote:
On 06/15/2010 10:34 AM, Zachary Amsden wrote:
Attempt to synchronize TSCs which are reset to the same value. In the
case of a reliable hardware TSC, we can just re-use the same offset, but
on non-reliable hardware, we can get closer by adjusting the offset to
match the elapsed time.

Answers a question from earlier.

I wonder about guests that might try to be clever an compensate for the IPI round trip, so not writing the same value. On the other hand, really clever guests will synchronize though memory, not an IPI.

Really, really clever guests will use an NMI so as to block further NMIs and then synchronize through memory from the NMI handler. Or an SMI, if they have enough understanding of the chipset to make it work.

Signed-off-by: Zachary Amsden<zamsden@xxxxxxxxxx>
arch/x86/kvm/x86.c | 34 ++++++++++++++++++++++++++++++++--
1 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8e836e9..cedb71f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -937,14 +937,44 @@ static inline void kvm_request_guest_time_update(struct kvm_vcpu *v)

+static inline int kvm_tsc_reliable(void)
+ return (boot_cpu_has(X86_FEATURE_CONSTANT_TSC)&&
+ boot_cpu_has(X86_FEATURE_NONSTOP_TSC)&&
+ !check_tsc_unstable());
void guest_write_tsc(struct kvm_vcpu *vcpu, u64 data)
struct kvm *kvm = vcpu->kvm;
- u64 offset;
+ u64 offset, ns, elapsed;

offset = data - native_read_tsc();
- kvm->arch.last_tsc_nsec = get_kernel_ns();
+ ns = get_kernel_ns();
+ elapsed = ns - kvm->arch.last_tsc_nsec;
+ /*
+ * Special case: identical write to TSC within 5 seconds of
+ * another CPU is interpreted as an attempt to synchronize
+ * (the 5 seconds is to accomodate host load / swapping).
+ *
+ * In that case, for a reliable TSC, we can match TSC offsets,
+ * or make a best guest using kernel_ns value.
+ */
+ if (data == kvm->arch.last_tsc_write&& elapsed< 5 * NSEC_PER_SEC) {

5e9 will overflow on i386.

Better make it 4 ;)

+ if (kvm_tsc_reliable()) {
+ offset = kvm->arch.last_tsc_offset;
+ pr_debug("kvm: matched tsc offset for %llu\n", data);
+ } else {
+ u64 tsc_delta = elapsed * __get_cpu_var(cpu_tsc_khz);
+ tsc_delta = tsc_delta / USEC_PER_SEC;
+ offset -= tsc_delta;
+ pr_debug("kvm: adjusted tsc offset by %llu\n", tsc_delta);
+ }
+ ns = kvm->arch.last_tsc_nsec;
+ }
+ kvm->arch.last_tsc_nsec = ns;

Shouldn't we check that the older write was on a different vcpu?

I thought about it; the pattern isn't necessarily the same with every OS (or even every qemu-kvm binary), but at least on AMD hardware, we see 6 writes of the TSC for a 2-VCPU VM. Two are done from ioctls for each VCPU at creation and setup time. The other two are triggered by the INIT and STARTUP IPI signals for each AP from the Linux bootup code.

So if we want to keep the APs in sync with the BSP, they must tolerate four resets in a row of the TSC on a single CPU. We could eliminate the second setup_vcpu reset with refactoring and choose not to reset the TSC for SIPI signals, but for now this seems the simplest and best approach.
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at