Re: [PATCH v6 3/4] perf-stat: enable counting events for BPF programs

[Song Liu]

> On Dec 29, 2020, at 7:15 AM, Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:
> 
> Em Mon, Dec 28, 2020 at 11:43:25PM +0000, Song Liu escreveu:
>> 
>> 
>>> On Dec 28, 2020, at 12:11 PM, Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:
>>> 
>>> Em Mon, Dec 28, 2020 at 09:40:53AM -0800, Song Liu escreveu:
>>>> Introduce perf-stat -b option, which counts events for BPF programs, like:
>>>> 
>>>> [root@localhost ~]# ~/perf stat -e ref-cycles,cycles -b 254 -I 1000
>>>>    1.487903822            115,200      ref-cycles
>>>>    1.487903822             86,012      cycles
>>>>    2.489147029             80,560      ref-cycles
>>>>    2.489147029             73,784      cycles
>>>>    3.490341825             60,720      ref-cycles
>>>>    3.490341825             37,797      cycles
>>>>    4.491540887             37,120      ref-cycles
>>>>    4.491540887             31,963      cycles
>>>> 
>>>> The example above counts cycles and ref-cycles of BPF program of id 254.
>>>> This is similar to bpftool-prog-profile command, but more flexible.
>>>> 
>>>> perf-stat -b creates per-cpu perf_event and loads fentry/fexit BPF
>>>> programs (monitor-progs) to the target BPF program (target-prog). The
>>>> monitor-progs read perf_event before and after the target-prog, and
>>>> aggregate the difference in a BPF map. Then the user space reads data
>>>> from these maps.
>>>> 
>>>> A new struct bpf_counter is introduced to provide common interface that
>>>> uses BPF programs/maps to count perf events.
>>> 
>>> Segfaulting here:
>>> 
>>> [root@five ~]# bpftool prog  | grep tracepoint
>>> 110: tracepoint  name syscall_unaugme  tag 57cd311f2e27366b  gpl
>>> 111: tracepoint  name sys_enter_conne  tag 3555418ac9476139  gpl
>>> 112: tracepoint  name sys_enter_sendt  tag bc7fcadbaf7b8145  gpl
>>> 113: tracepoint  name sys_enter_open  tag 0e59c3ac2bea5280  gpl
>>> 114: tracepoint  name sys_enter_opena  tag 0baf443610f59837  gpl
>>> 115: tracepoint  name sys_enter_renam  tag 24664e4aca62d7fa  gpl
>>> 116: tracepoint  name sys_enter_renam  tag 20093e51a8634ebb  gpl
>>> 117: tracepoint  name sys_enter  tag 0bc3fc9d11754ba1  gpl
>>> 118: tracepoint  name sys_exit  tag 29c7ae234d79bd5c  gpl
>>> [root@five ~]#
>>> [root@five ~]# gdb perf
>>> GNU gdb (GDB) Fedora 10.1-2.fc33
>>> Reading symbols from perf...
>>> (gdb) run stat -e instructions,cycles -b 113 -I 1000
>>> Starting program: /root/bin/perf stat -e instructions,cycles -b 113 -I 1000
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>> libbpf: elf: skipping unrecognized data section(9) .eh_frame
>>> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
>>> libbpf: elf: skipping unrecognized data section(9) .eh_frame
>>> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
>>> 
>>> Program received signal SIGSEGV, Segmentation fault.
>>> 0x000000000058d55b in bpf_program_profiler__read (evsel=0xc612c0) at util/bpf_counter.c:217
>>> 217			reading_map_fd = bpf_map__fd(skel->maps.accum_readings);
>>> (gdb) bt
>>> #0  0x000000000058d55b in bpf_program_profiler__read (evsel=0xc612c0) at util/bpf_counter.c:217
>>> #1  0x0000000000000000 in ?? ()
>>> (gdb)
>>> 
>>> [acme@five perf]$ clang -v |& head -2
>>> clang version 11.0.0 (Fedora 11.0.0-2.fc33)
>>> Target: x86_64-unknown-linux-gnu
>>> [acme@five perf]$
>>> 
>>> Do you need any extra info?
>> 
>> Hmm... I am not able to reproduce this. I am trying to setup an environment similar
>> to fc33 (clang 11, etc.). Does this segfault every time, and on all programs? 
> 
> I'll try it with a BPF proggie attached to a kprobes, but here is
> something else I noticed:
> 
> [root@five perf]# export PYTHONPATH=/tmp/build/perf/python
> [root@five perf]# tools/perf/python/twatch.py
> Traceback (most recent call last):
>  File "/home/acme/git/perf/tools/perf/python/twatch.py", line 9, in <module>
>    import perf
> ImportError: /tmp/build/perf/python/perf.cpython-39-x86_64-linux-gnu.so: undefined symbol: bpf_counter__destroy
> [root@five perf]# perf test python
> 19: 'import perf' in python                                         : FAILED!
> [root@five perf]# perf test -v python
> 19: 'import perf' in python                                         :
> --- start ---
> test child forked, pid 3198864
> python usage test: "echo "import sys ; sys.path.append('/tmp/build/perf/python'); import perf" | '/usr/bin/python3' "
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> ImportError: /tmp/build/perf/python/perf.cpython-39-x86_64-linux-gnu.so: undefined symbol: bpf_counter__destroy
> test child finished with -1
> ---- end ----
> 'import perf' in python: FAILED!
> [root@five perf]#
> 
> This should be trivial, I hope, just add the new object file to
> tools/perf/util/python-ext-sources, then do a 'perf test python', if it
> fails, use 'perf test -v python' to see what is preventing the python
> binding from loading.

I fixed the undefined bpf_counter__destroy. But this one looks trickier:

19: 'import perf' in python                                         :
--- start ---
test child forked, pid 2714986
python usage test: "echo "import sys ; sys.path.append('python'); import perf" | '/bin/python2' "
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: XXXXX /tools/perf/python/perf.so: undefined symbol: bpf_map_update_elem

Given I already have:

diff --git i/tools/perf/util/python-ext-sources w/tools/perf/util/python-ext-sources
index a9d9c142eb7c3..2cac55273eca2 100644
--- i/tools/perf/util/python-ext-sources
+++ w/tools/perf/util/python-ext-sources
@@ -35,3 +35,6 @@ util/symbol_fprintf.c
 util/units.c
 util/affinity.c
 util/rwsem.c
+util/bpf_counter.c
+../lib/bpf/bpf.c
+../lib/bpf/libbpf.c


How should I fix this? 

Thanks,
Song

PS: I still cannot reproduce that segfault...

>