Re: [PATCH] tracing: use raw spinlocks instead of spinlocks

From: Ingo Molnar
Date: Tue Nov 04 2008 - 04:44:00 EST



* Frédéric Weisbecker <fweisbec@xxxxxxxxx> wrote:

> 2008/11/3 Ingo Molnar <mingo@xxxxxxx>:
> > Frederic, do you have trouble finding the source of the deadlock? In
> > theory the NMI watchdog should be able to punch through it. (if not
> > then we need to improve things so that it can)
>
> Ok...actually I still have a deadlock. Or it seems to. I launch the
> return tracer and then...the system can block after several seconds,
> typically after I try to write some characters on the terminal.
>
> It happens more easily on SMP.
>
> I enabled all I could for lock debugging, but I don't have any
> errors printed... Lockdep uses an NMI wachdog?

lockdep works via a completely different principle: it instruments all
the actual lock acquire/release calls and builds a graph of lock
dependencies in the system, as it happens.

It also guarantees that all the observed locking rules are followed
(i.e. it proves that as long as you dont get any messages from
lockdep, all the locking patterns are mathematically safe).

So a lockdep message will most of the time occur much easier than a
real lockup would occur - as lockdep only needs to observe
inconsistent locking patterns to prove that a lockup _could_ occur.

The NMI watchdog just observes the system and complains if it sees
hardirqs not progressing (i.e. a hard lockup). It will detect anything
that causes a hard lockup. (assuming that the NMI watchdog itself is
not locked up)

Regarding your lockup ... it's quite hard. Maybe you can get more
output out of the system by using:

earlyprintk=vga,keep

plus disablig regular tty output. (i.e. not passing any 'console=tty'
line to the kernel bootup.)

this way you wont get any normal printk activities (which might lock
up), you should only get the very simple early-printk output.

it is also probably useful to mark the early-printk code notrace in
arch/x86/kernel/Makefile, via the patch below. (i've just applied to
to tip/x86/debug and pushed it out into tip/master)

... but that's still not a guarantee that you'll be able to debug it
this way :-/ Bootstrapping something this intrusive and wide is one of
the hardest kernel development activities possible.

Ingo

---------------->