[REGRESSION] Needless shutting down of oneshot timer in nohz mode

[Steven Rostedt]

Hi Thomas,

The VMware PhotonOS team is evaluating 4.19-rt compared to CentOS
3.10-rt (franken kernel from Red Hat). They found a regression between
the two kernels that was found to be introduced by:

 d25408756accb ("clockevents: Stop unused clockevent devices")

The issue is running this on a guest, and it causes a noticeable wake
up latency in cyclictest. The 4.19-rt kernel has two extra apic
instructions causing for two extra VMEXITs to occur over the 3.10-rt
kernel. I found out the reason why, and this is true for vanilla 5.9-rc
as well.

When running isocpus with NOHZ_FULL, I see the following.

  tick_nohz_idle_stop_tick() {
	hrtimer_start_range_ns() {
		remove_hrtimer(timer)
			/* no more timers on the base */
			expires = KTIME_MAX;
			tick_program_event() {
				clock_switch_state(ONESHOT_STOPPED);
				/* call to apic to shutdown timer */
			}
		}
		[..]
		hrtimer_reprogram(timer) {
			tick_program_event() {
				clock_switch_state(ONESHOT);
				/* call to apic to enable timer again! */
		}
	}
 }


Thus, we are needlessly shutting down and restarting the apic every
time we call tick_nohz_stop_tick() if there is a timer still on the
queue.

I'm not exactly sure how to fix this. Is there a way we can hold off
disabling the clock here until we know that it isn't going to be
immediately enabled again?

-- Steve