Re: [PATCH bpf v2 0/3] bpf: invalidate unused part of bpf_prog_pack

[Song Liu]

> On Apr 27, 2022, at 11:48 PM, Song Liu <songliubraving@xxxxxx> wrote:
> 
> Hi Linus, 
> 
> Thanks for your thorough analysis of the situation, which make a lot of
> sense. 
> 
>> On Apr 27, 2022, at 6:45 PM, Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> wrote:
>> 
>> On Wed, Apr 27, 2022 at 3:24 PM Song Liu <songliubraving@xxxxxx> wrote:
>>> 
>>> Could you please share your suggestions on this set? Shall we ship it
>>> with 5.18?
>> 
>> I'd personally prefer to just not do the prog_pack thing at all, since
>> I don't think it was actually in a "ready to ship" state for this
>> merge window, and the hugepage mapping protection games I'm still
>> leery of.
>> 
>> Yes, the hugepage protection things probably do work from what I saw
>> when I looked through them, but that x86 vmalloc hugepage code was
>> really designed for another use (non-refcounted device pages), so the
>> fact that it all actually seems surprisingly ok certainly wasn't
>> because the code was designed to do that new case.
>> 
>> Does the prog_pack thing work with small pages?
>> 
>> Yes. But that wasn't what it was designed for or its selling point, so
>> it all is a bit suspect to me.
> 
> prog_pack on small pages can also reduce the direct map fragmentation.
> This is because libbpf uses tiny BPF programs to probe kernel features. 
> Before prog_pack, all these BPF programs can fragment the direct map.
> For example, runqslower (tools/bpf/runqslower/) loads total 7 BPF programs 
> (3 actual programs and 4 tiny probe programs). All these programs may 
> cause direct map fragmentation. With prog_pack, OTOH, these BPF programs 
> would fit in a single page (or even share pages with other tools). 

Here are some performance data from our web service production benchmark, 
which is the biggest service in our fleet. We compare 3 kernels:    

  nopack: no bpf_prog_pack; IOW, the same behavior as 5.17
  4kpack: use bpf_prog_pack on 4kB pages (same as 5.18-rc5)
  2mpack: use bpf_prog_pack on 2MB pages

The benchmark measures system throughput under latency constraints. 
4kpack provides 0.5% to 0.7% more throughput than nopack. 
2mpack provides 0.6% to 0.9% more throughput than nopack. 

So the data has confirmed:
1. Direct map fragmentation has non-trivial impact on system performance;
2. While 2MB pages are preferred, bpf_prog_pack on 4kB pages also gives 
   Significant performance improvements.  

Thanks,
Song