Re: Event logging vs enhancing printk

From: Michel Dagenais (michel.dagenais@polymtl.ca)
Date: Tue Apr 09 2002 - 09:21:21 EST


> People want to be able to get better debug information, with more detail
> than is currently possible with printk, hence the Event logging project.

I would like to emphasize that point; logging and tracing is of prime
importance in several areas:

- Telecom and Security. Logging and auditing is a requirement in Telecom
  applications. It is for instance one of the important features of the
  Distributed Security Infrastructure proposed by Ericsson Research and to
  be presented in a couple of months at the Ottawa Linux Symposium.
  The OSDL "Carrier Grade Linux" is certainly no different.

- Real time systems. A low overhead trace is often the only way to debug
  these systems. Lineo, Monte-Vista, and FSM labs all have logging solutions,
  most based on the Linux Trace Toolkit (www.opersys.com/LTT). It would be
  relatively easy to merge the best features of these two systems, present
  on the 2.5 status list, (high volume low overhead of LTT and event
  templates and filtering of EVLOG).

- Kernel/device drivers debugging.

- Detailed performance analysis. Using the Linux Trace Toolkit, it is possible
  to extract very detailed information like the time spent by a process
  executing, executing on behalf of client x, waiting for file y
  (read/page fault), waiting for process z...

The current printk simply does not cut it for these applications. There are
over 80000 printk statements in the kernel, using many different conventions.
A few tens of driver specific tracing facilities (SCSI, ftape, wireless...)
were implemented each with its own mechanism to compile/not the debugging
statements, trigger massive debugging output at runtime...

The EVLOG proposition strikes me as a good balance between solid features,
simplicity, and ease of integration/transition with the current printk.

Here are some of the advantages of more structured logging:

- More consistent activation mechanisms for logging points
  troughout the kernel at configuration/compilation time and at runtime.

- Structured data events for which it is easier to apply filtering, querying,
  analysis and detection tools.

- More compact format, when desired, where data and text descriptions are
  separated. This facilitates high volume applications (lower logging
  overhead, smaller files) and also enables customization/i18n of the static
  text descriptions.

- In its current configuration, klogd rapidly looses events under high volumes.
  The Linux Trace Toolkit with its zero-copy, kernel-daemon shared memory,
  does much better under heavy debugging/tracing output.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/



This archive was generated by hypermail 2b29 : Mon Apr 15 2002 - 22:00:13 EST