Re: [PATCH v2] tracing/function-graph-tracer: prevent from hrtimerinterrupt infinite loop

[Ingo Molnar]

* Thomas Gleixner <tglx@xxxxxxxxxxxxx> wrote:

> On Thu, 18 Dec 2008, Ingo Molnar wrote:
> > * Frederic Weisbecker <fweisbec@xxxxxxxxx> wrote:
> > 
> > > Impact: fix a system hang on slow systems
> > > 
> > > While testing the function graph tracer on VirtualBox, I had a system hang
> > > immediatly after enabling the tracer.
> > >
> > > If hrtimer is enabled on kernel, a slow system can spend too much time 
> > > during tracing the hrtimer_interrupt which will do eternal loops, 
> > > assuming it always have to retry its process because too much time 
> > > elapsed during its time update. Now we provide a feature which lurks at 
> > > the number of retries on hrtimer_interrupt. After 10 retries, the 
> > > function graph tracer will definetly stop its tracing.
> > 
> > hm, i dont really like this solution - it just works around the problem by 
> > 'speeding up' the system. If we have a _real_ slow system, there's no such 
> > way for us to speed it up.
> > 
> > Thomas, what do you think - would you expect this lockup to happen on 
> > really slow systems? If yes, is there a way we could avoid it from 
> > happening - by driving some sort of 'mandatory interval', that is doubled 
> > in size every time we detect such a bad hrtimer loop?
> 
> In reality I have not seen such a problem yet, even on an old real slow 
> P1 which I tricked to do highres, but of course if we add such time 
> consuming debugs and make it slow enough the system will spend all the 
> time running the tick timer :)
> 
> We should at least warn once about such a loop.
> 
> I'm not sure about the mandatory interval though:
> 
> Try the same test with HZ=1000 periodic mode (HIGHRES/NOHZ=off) and I 
> bet you see the same problem, just not in hrtimer_interrupt().

that would be important to double-check. Frederic, does the system lock up 
with a periodic 1khz HZ tick just as much? I.e. does the processing of a 
single timer interrupt take more than 1 milliseconds?

Granted, if the system is too slow to process the system clock, it's not 
useful.

But that's my point: instead of just randomly disabling functionality 
until the system gets 'fast enough' to process timer IRQs, how about 
dynamically and adaptively extending the required minimal timeout between 
hr-timer IRQs?

That will in essence self-tune the system into some minimally working 
state - instead of locking it up. Note that such a method would work with 
any source of timer IRQ slowness - not just tracing.

( And maybe the lockup is somehow hrtimer IRQ induced. If a 1khz clock
  still works for Frederic then that angle has to be investigated. )

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/