Re: [PATCH 00/18] tracing/ring-buffer: fixes for latency tracer[2.6.32]

From: Ingo Molnar
Date: Sun Sep 06 2009 - 00:28:26 EST

* Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:

> Finally (something I should have realized from day 1, not 4 days
> later) I realized I already had a marker. The latency tracers
> always record the timestamp of when the trace began. I can simply
> ignore any trace event that happened before that timestamp. This
> ended up working out very well. The code is quite simple and
> solid.

Yes - that trick came from the original latency tracer and i added
it for that purpose.

The flip side is that if there's anything wrong with timestamps
(and this does happen frequently as well) we suddenly have an empty
trace and no explanation about what happened.

> git://
> tip/tracing/core
> Steven Rostedt (18):
> ring-buffer: do not reset while in a commit
> ring-buffer: do not swap buffers during a commit
> ring-buffer: remove unnecessary cpu_relax
> ring-buffer: fix ring_buffer_read crossing pages
> ring-buffer: remove ring_buffer_event_discard
> ring-buffer: do not count discarded events
> ring-buffer: disable all cpu buffers when one finds a problem
> tracing: print out start and stop in latency traces
> tracing: disable update max tracer while reading trace
> tracing: disable buffers and synchronize_sched before resetting
> tracing: remove users of tracing_reset
> tracing: use timestamp to determine start of latency traces
> tracing: make tracing_reset safe for external use
> tracing: pass around ring buffer instead of tracer
> tracing: add trace_array_printk for internal tracers to use
> tracing: report error in trace if we fail to swap latency buffer
> ring-buffer: check for swapped buffers in start of committing
> ring-buffer: only enable ring_buffer_swap_cpu when needed
> ----
> include/linux/ftrace_event.h | 15 ++-
> include/linux/ring_buffer.h | 23 +--
> include/trace/ftrace.h | 15 ++-
> kernel/trace/Kconfig | 8 +
> kernel/trace/blktrace.c | 12 +-
> kernel/trace/kmemtrace.c | 4 +-
> kernel/trace/ring_buffer.c | 172 +++++++++++++-------
> kernel/trace/trace.c | 297 ++++++++++++++++++++++++----------
> kernel/trace/trace.h | 26 ++--
> kernel/trace/trace_boot.c | 16 +-
> kernel/trace/trace_events.c | 6 +-
> kernel/trace/trace_functions_graph.c | 14 +-
> kernel/trace/trace_irqsoff.c | 3 +-
> kernel/trace/trace_mmiotrace.c | 10 +-
> kernel/trace/trace_power.c | 22 ++-
> kernel/trace/trace_sched_switch.c | 18 ++-
> kernel/trace/trace_sched_wakeup.c | 7 +-
> kernel/trace/trace_syscalls.c | 18 ++-
> 18 files changed, 444 insertions(+), 242 deletions(-)

Pulled, thanks a lot Steve!

We should also think about how to expose the function tracer, and
in particular latency tracing functionality, via perf.

The most natural approach would be to expose it as a 'generic
event' - and simply stream function tracing events as they happen.

The latency tracer itself could be exposed either as a generic
event, or as a software counter. The difference there is that it
needs a per counter 'max latency' state that controls whether a new
trace is sent down the pipe.

This would allow very flexible tooling: for example one could do a
dump of the maximum latency a particular workflow experiences [in a
scheduling invariant manner] - without having to trace the full
system. I.e. individual apps could be validated separately and
latency impact on individual apps could be tracked more accurately
as well.

(beyond the other usecases such as tracking cache-miss rates in
addition to latencies - this would allow the observation of
worst-case cachemisses correlating with large latencies, etc.)

Steve, Frederic, does any of you have interest (and time ;-) to
look into that? It should probably have its own tooling variant,
not be part of 'perf trace' per se (which is the generic tracing

'perf latencytrace' or so - because latency tracing has its own
special needs (for the same reason do we have a separate latency
tracer ftrace plugin).

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at