Re: perf test BPF failing on f24: fix

[Masami Hiramatsu]

On Wed, 3 Aug 2016 17:04:15 -0300
Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:

> Em Wed, Aug 03, 2016 at 11:45:57PM +0900, Masami Hiramatsu escreveu:
> > > If we remove vmlinux, perf should use /proc/kallsyms. I think
> 
> I am not removing vmlinux, it is being used, its just that the function
> chosen by the 'perf test BPF' testcase isn't there.
> 
> So lets go again trying to chase this without missing a single step of
> the way:
> 
> We start with:
> 
> [root@jouet ~]# perf test bpf
> 37: Test BPF filter                                          :
> 37.1: Test basic BPF filtering                               : FAILED!
> 37.2: Test BPF prologue generation                           : Skip
> 37.3: Test BPF relocation checker                            : Skip
> [root@jouet ~]# 
> 
> Ok, so we add -v to get more information:
> 
> [root@jouet ~]# perf test -v bpf
> <BIG SNIP>
> bpf: config program 'func=sys_epoll_wait'
> symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> bpf: config 'func=sys_epoll_wait' is ok
> Looking at the vmlinux_path (8 entries long)
> Using /lib/modules/4.7.0+/build/vmlinux for symbols
> Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux
> Try to find probe point from debuginfo.
> Symbol sys_epoll_wait address found : ffffffffbd295b50
> Failed to find debug information for address ffffffffbd295b50
> Probe point 'sys_epoll_wait' not found.
> bpf_probe: failed to convert perf probe eventsFailed to add events
> selected by BPF
> test child finished with -1
> ---- end ----
> Test BPF filter subtest 0: FAILED!
> 
> --------------
> 
> See? It _is_ using /lib/modules/4.7.0+/build/vmlinux, and it should
> because:
> 
> [acme@jouet linux]$ file /lib/modules/4.7.0+/build/vmlinux
> /lib/modules/4.7.0+/build/vmlinux: ELF 64-bit LSB executable, x86-64,
> version 1 (SYSV), statically linked,
> BuildID[sha1]=a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a, not stripped
> [acme@jouet linux]$
> 
> Its the kernel that is in use:
> 
> [acme@jouet linux]$ perf buildid-list --kernel
>               a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a

> [acme@jouet linux]$ perf buildid-list -h --kernel
> 
>  Usage: perf buildid-list [<options>]
> 
>     -k, --kernel          Show current kernel build id
> 
> [acme@jouet linux]$
> 
> And, in this vmlinux file, there is _no_ such function:
> 
> [acme@jouet linux]$ readelf -wi /lib/modules/4.7.0+/build/vmlinux | grep -w sys_epoll_wait
> [acme@jouet linux]$ 
> 
> Exactly like the 'perf probe -v bpf' says:
> 
> Symbol sys_epoll_wait address found : ffffffffbd295b50
> Failed to find debug information for address ffffffffbd295b50
> 
> -----
> 
> It mapped it to an address, sure, it found it in /proc/kallsyms, but
> then it didn't find it in the matching vmlinux file.
> 
> Since the test was working before, when did it stop to be available on
> vmlinux?

Would you remember the previous test was done with this kernel or other one?

Actually, I could put probes on sys_epoll_wait on F24 standard kernel.

-----
[mhiramat@devbox perf]$ sudo ./perf probe -vnf sys_epoll_wait
probe-definition(0): sys_epoll_wait 
symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (8 entries long)
Using /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux for symbols
Open Debuginfo file: /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux
Try to find probe point from debuginfo.
Symbol sys_epoll_wait address found : ffffffff81292d60
Matched function: SyS_epoll_wait
found inline addr: 0xffffffff812930f3
Probe point found: compat_SyS_epoll_pwait+147
found inline addr: 0xffffffff81292ed6
Probe point found: SyS_epoll_pwait+134
found inline addr: 0xffffffff81292d60
Probe point found: SyS_epoll_wait+0
Found 3 probe_trace_events.
Opening /sys/kernel/debug/tracing//kprobe_events write=1
Writing event: p:probe/sys_epoll_wait _text+2699507
Writing event: p:probe/sys_epoll_wait_1 _text+2698966
Writing event: p:probe/sys_epoll_wait_2 _text+2698592
Added new events:
  probe:sys_epoll_wait (on sys_epoll_wait)
  probe:sys_epoll_wait_1 (on sys_epoll_wait)
  probe:sys_epoll_wait_2 (on sys_epoll_wait)

You can now use it in all perf tools, such as:

	perf record -e probe:sys_epoll_wait_2 -aR sleep 1

-----
FYI, if perf-probe failed to find the symbol in debuginfo, it tries to find its address
from symbol table(kallsyms, here), and then tries to convert the address to the symbol
in debuginfo again. It seems that in your case this convert process is failed.

Then, could you grep DEBUG_INFO in .config? I guess your kernel enables some
reduced debuginfo related config enabled...

Thank you,


-- 
Masami Hiramatsu <mhiramat@xxxxxxxxxx>