[perf] periodic performance monitoring interrupts causing heavy delays

[ZoltÃn Kiss Z]

Hi,

I'm running some benchmarking on a Dell Poweredge R630 server (Xeon 
E5-2650 v4 Broadwell) with Ubuntu 18.04 (4.15.0-62 kernel). The 
application is busy looping on core's reserved with isolcpus. However 
every 5 seconds there is a ~1 ms glitch where my cores seem to be 
stopped. After checking interrupts it seems these glitches coincide with 
an NMI and a "Performance monitoring interrupt" on all the busy looping 
cores, while the idle ones doesn't receive it. I was not running perf 
during these occasions, but the below post suggests that there is still 
some kernel perf activity in the background which causes these interrupts:

https://serverfault.com/questions/714648/how-to-disable-perf-subsystem-in-linux-kernel

I can also see these "perf interrupt took too long" messages even when I 
never started perf by myself since the last reboot. I've tried to adjust 
the /proc/sys/kernel/perf_* knobs, without any luck.

The above post suggests that these periodic interrupts cannot be 
disabled in any way, and I would like to figure out if this is really 
the case, and if yes, why? A 1 ms glitch in every 5 seconds is quite 
unacceptable not just because of benchmarking noise, but also for 
latency sensitive applications like mine.
Also, I couldn't find any info about these periodic perf background jobs 
generating the interrupts, could anyone point me to at least some 
function names where I could start digging, so I could verify if indeed 
these glitches are due to perf?

Thanks,

Zoltan Kiss