Re: False-positive RCU stall warnings on large systems...

From: Daniel J Blueman
Date: Tue Feb 19 2013 - 22:36:10 EST


On 20/02/2013 02:16, Paul E. McKenney wrote:
On Wed, Feb 20, 2013 at 12:34:12AM +0800, Daniel J Blueman wrote:
Hi Paul,

On some of our larger servers with many hundreds of cores and when
under high duress, we can see scheduler RCU stall warnings [1], so
find we have to increase the hardcoded RCU_STALL_RAT_DELAY up from 2
and RCU_JIFFIES_TILL_FORCE_QS up from 3.

Is there a more sustainable way to account for this to avoid it
being hard-coded, such as making it and dependent timeouts a
fraction of CONFIG_RCU_CPU_STALL_TIMEOUT?

On the other hand, perhaps this is just caused by clock jitter (eg
due to distance from a contended clock source)? So increasing these
a bit may just be adequate in general...

Hmmm... What version of the kernel are you running?

The example below occurs with v3.8, but we see the same with previous kernels eg v3.5.

Of course, when using the local TSC, you'd see no jitter relative to coherent transactions (eg memory writes), but when the HPET is used across a large system, coherent transactions to distant cores are just so much faster, as there's massive congestion to the shared HPET behind various HT and PCIe bridges. This could be where the jitter arises from, if I'm guessing jitter is the problem here.

Thanks,
Daniel

--- [1]

[ 3939.010085] INFO: rcu_sched detected stalls on CPUs/tasks: {}
(detected by 1, t=29662 jiffies, g=3053, c=3052, q=598)
[ 3939.020008] INFO: Stall ended before state dump start
--
Daniel J Blueman
Principal Software Engineer, Numascale Asia
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/