Re: [PATCH bpf v2 0/3] bpf: invalidate unused part of bpf_prog_pack

[Song Liu]

> On May 6, 2022, at 11:50 PM, Song Liu <songliubraving@xxxxxx> wrote:
> 
> 
> 
>> On Apr 27, 2022, at 11:48 PM, Song Liu <songliubraving@xxxxxx> wrote:
>> 
>> Hi Linus, 
>> 
>> Thanks for your thorough analysis of the situation, which make a lot of
>> sense. 
>> 
>>> On Apr 27, 2022, at 6:45 PM, Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> wrote:
>>> 
>>> On Wed, Apr 27, 2022 at 3:24 PM Song Liu <songliubraving@xxxxxx> wrote:
>>>> 
>>>> Could you please share your suggestions on this set? Shall we ship it
>>>> with 5.18?
>>> 
>>> I'd personally prefer to just not do the prog_pack thing at all, since
>>> I don't think it was actually in a "ready to ship" state for this
>>> merge window, and the hugepage mapping protection games I'm still
>>> leery of.
>>> 
>>> Yes, the hugepage protection things probably do work from what I saw
>>> when I looked through them, but that x86 vmalloc hugepage code was
>>> really designed for another use (non-refcounted device pages), so the
>>> fact that it all actually seems surprisingly ok certainly wasn't
>>> because the code was designed to do that new case.
>>> 
>>> Does the prog_pack thing work with small pages?
>>> 
>>> Yes. But that wasn't what it was designed for or its selling point, so
>>> it all is a bit suspect to me.
>> 
>> prog_pack on small pages can also reduce the direct map fragmentation.
>> This is because libbpf uses tiny BPF programs to probe kernel features. 
>> Before prog_pack, all these BPF programs can fragment the direct map.
>> For example, runqslower (tools/bpf/runqslower/) loads total 7 BPF programs 
>> (3 actual programs and 4 tiny probe programs). All these programs may 
>> cause direct map fragmentation. With prog_pack, OTOH, these BPF programs 
>> would fit in a single page (or even share pages with other tools). 
> 
> Here are some performance data from our web service production benchmark, 
> which is the biggest service in our fleet. We compare 3 kernels:    
> 
>  nopack: no bpf_prog_pack; IOW, the same behavior as 5.17
>  4kpack: use bpf_prog_pack on 4kB pages (same as 5.18-rc5)
>  2mpack: use bpf_prog_pack on 2MB pages
> 
> The benchmark measures system throughput under latency constraints. 
> 4kpack provides 0.5% to 0.7% more throughput than nopack. 
> 2mpack provides 0.6% to 0.9% more throughput than nopack. 
> 
> So the data has confirmed:
> 1. Direct map fragmentation has non-trivial impact on system performance;
> 2. While 2MB pages are preferred, bpf_prog_pack on 4kB pages also gives 
>   Significant performance improvements.  

Please note that 0.5% is a huge improvement for our fleet. I believe this
is also significant for other companies with many thousand servers. 

Thanks,
Song