Re: [RFD] Future tracing/instrumentation directions

From: Ingo Molnar
Date: Thu May 20 2010 - 07:36:54 EST

* Frederic Weisbecker <fweisbec@xxxxxxxxx> wrote:

> On Thu, May 20, 2010 at 11:31:31AM +0200, Ingo Molnar wrote:
> > - [ While it's still a long way off, if this trend continues
> > we eventually might even be able to get rid of the
> > /debug/tracing/ temporary debug API and get rid of
> > the ugly in-kernel pretty-printing bits. This is
> > good: it may make Andrew very happy for a change ;-)
> >
> > The main detail here to be careful of is that lots of
> > people are fond of the simplicity of the
> > /debug/tracing/ debug UI, so when we replace it we
> > want to do it by keeping that simple workflow (or
> > best by making it even simpler). I have a few ideas
> > how to do this.
> How? We can emulate the /debug/tracing result with
> something like perf trace, still that won't replace the
> immediate availability of the result of any trace, which
> makes it valuable for any simplest workflows.

Firstly, one thing is sure: until there's no full
replacement we obviously dont want to phase out
/sys/kernel/debug/tracing. This was more of a 'our future'
email (as i see it), the process that will lead to solve
some of our more strategic problems in tracing land.

Regarding the issues you raised, there are several
solutions that dont need /sys/kernel/debug/tracing but
still support the very useful and usable 'immediate
tracing' workflow that ftrace prototyped. We can have a
combination of several things:

- Have a simple 'ftrace' command aliased to perf trace.

Means less typing, and it also allows a much more
finegrained tracing workflow: per user and per task/job
workflows, instead of the global/exclusive tracing mode
that /sys/kernel/debug/tracing. There would be ready

ftrace --available-tracers
ftrace --current-tracer
ftrace --start
ftrace --stop

... etc ...

- Immediate availability of a trace: persistent events
combined with roundrobin ('flight-recorder') recording
would solve this.

If events are active then if type 'ftrace' you get the
current trace. Simple to scrip and simple to use - no
need to have access to /sys/kernel/debug/tracing, also
can evidently be turned into a per user facility,
supports multiple plugins active at once, etc.

- For lightweight embedded tracing there are two separate
solutions that would work:

- we already have perf 'minimal builds' (when certain
libraries are not available), we could push that
concept some more to create a 'lightweight' command
that embedded systems can run just fine.

- extend our proxy recording and proxy execution/analysis
concepts. We can already run a perf trace recording
through a pipe and netcat, and we have perf archive
and cross-arch analysis code.

If there's interest, then this could be made more
convenient and the functionality around this could
be collected into a handy proxy tool:

ftrace --proxy smallbox --start
ftrace --proxy smallbox --stop
ftrace --proxy smallbox # prints trace

... etc ...

Thus the recording is done on the small box, while
all the analysis (and even all the commands) is
typed/executed on the bigger box.

So there are lots of possibilities - and there are other
options as well.

Does this address your worries?


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at