Re: x86/kprobes: kretprobe fails to triggered if kprobe at function entry is not optimized (trigger by int3 breakpoint)

[Masami Hiramatsu]

On Wed, 26 Aug 2020 18:06:45 +0900
Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote:

> On Wed, 26 Aug 2020 17:22:39 +0900
> Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote:
> 
> > On Wed, 26 Aug 2020 07:07:09 +0000
> > "Eddy_Wu@xxxxxxxxxxxxxx" <Eddy_Wu@xxxxxxxxxxxxxx> wrote:
> > 
> > > 
> > > > -----Original Message-----
> > > > From: peterz@xxxxxxxxxxxxx <peterz@xxxxxxxxxxxxx>
> > > > Sent: Tuesday, August 25, 2020 8:09 PM
> > > > To: Masami Hiramatsu <mhiramat@xxxxxxxxxx>
> > > > Cc: Eddy Wu (RD-TW) <Eddy_Wu@xxxxxxxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx; x86@xxxxxxxxxx; David S. Miller
> > > > <davem@xxxxxxxxxxxxx>
> > > > Subject: Re: x86/kprobes: kretprobe fails to triggered if kprobe at function entry is not optimized (trigger by int3 breakpoint)
> > > >
> > > > Surely we can do a lockless list for this. We have llist_add() and
> > > > llist_del_first() to make a lockless LIFO/stack.
> > > >
> > > 
> > > llist operations require atomic cmpxchg, for some arch doesn't have CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG, in_nmi() check might still needed.
> > > (HAVE_KRETPROBES && !CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG): arc, arm, csky, mips
> > 
> > Good catch. In those cases, we can add in_nmi() check at arch dependent code.
> 
> Oops, in_nmi() check is needed in pre_kretprobe_handler() which has no
> arch dependent code. Hmm, so we still need an weak function to check it...

Oops, again. Sorry I found a big misunderstand. I found the in_nmi() check is
completely unnecessary with Jiri's commit 9b38cc704e84 ("kretprobe: Prevent 
triggering kretprobe from within kprobe_flush_task").

This commit introduced the kprobe_busy_begin/end() to the kretproeb trampoline
handler, which set a dummy kprobe to the per-cpu current kprobe pointer.
This current-kprobe is checked at the kprobe pre handler to prevent kprobes
(including kretprobe) recursion.

This means, if an NMI interrupts a kretprobe operation (both pre-handler and
trampoline-handler) and it hits the same kretprobe, this nested kretprobe
handlers never be called, because there is a current kprobe is already set.
Thus, we are totally safe from double-lock issue in the kretprobe handlers.

So we can just remove the in_nmi() check from pre_kretprobe_handler() if
we introduced a generic trampoline handler, since kprobe_busy_begin/end()
must called from the trampoline handlers. Currently it is used on x86 only.

Of course, this doesn't solve the llist_del_first() contention in the
pre_kretprobe_handler(). So anyway we need a lock for per-probe llist
(if I understand llist.h comment correctly.)

Thank you,

-- 
Masami Hiramatsu <mhiramat@xxxxxxxxxx>