Re: [PATCH 0/3][RFC] tracing/kprobes: prevent jprobes from crashingfunction graph tracer

From: Masami Hiramatsu
Date: Mon Nov 02 2009 - 10:03:16 EST

Frederic Weisbecker wrote:
On Thu, Oct 29, 2009 at 06:02:20PM -0400, Masami Hiramatsu wrote:
Steven Rostedt wrote:
Lately I've been testing with an allyesconfig. When I ran the function graph
tracer, it immediately crashed the kernel. Thanks to the new frame pointer
test in function graph, it reported directly what the issue was and then
panicked the kernel to prevent any unexpected damage from happening.

It pointed the error to be with jtcp_rcv_established. Which is a jprobe
function added to tcp_rcv_established at bootup when CONFIG_NET_TCPPROBE
is enabled.

Jprobes and the function graph tracer use the same mechanism to trace
the exit of a function. Unfortunately, only one can be done at a time.
The function graph tracer replaces the return address with its own handler,
but so does jprobes. The two are not compatible.

AFAIK, Jprobe doesn't trace the exit of a function. I assume that
jprobe's user handler causes the problem, since the handler never
returns normal way.
Instead of that, it just calls jprobe_return() which causes
int3 to be trapped by kprobe's break handler. And the break handler
fixup regs->ip to back to traced function.

Actually, this will cause a problem with function graph tracer.
The f-g-tracer push the return address into the special stack and replaces
it with fixup function (This is similar (not same) mechanism of kretprobe.)
And then the traced function returns, it returns to the fixup function and
it pops the return address up and back to the real caller.

So, if the f-g-tracer traces jprobe user handler, the pop operation
will be skipped because the the handler never returns.

I'm not sure I've well understood how is performed the call to the jprobe
But if I understand well we have:

func() {
int3() {
jprobe_handler() {
set ip after iret to user_handler()
user_handler() {
jprobe_return() {
int3() {
set ip after iret to func+...()
(execute the rest of func())

If we replace (-) with pause_graph_tracing() and (+) with
unpause_graph_tracing(), this should do the trick...I hope.

I'm not so sure about pause_graph_tracing(), however, it seems that
int3() and jprobe_handler() already pushed on the stack of the
func graph tracer at (-). If it's true, where are those entries
popped up?

Thank you,

Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@xxxxxxxxxx

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at