Re: [PATCH] net: add big honking pfmemalloc OOM warning

[Eric Dumazet]

On Wed, Apr 10, 2019 at 3:20 AM Juha-Matti Tilli
<juha-matti.tilli@xxxxxxxxxx> wrote:
>
> A system administrator is not notified (except via an obscure SNMP
> counter that most sysadmins don't know to look for) if packets are
> dropped due to out-of-memory condition when SKBs use pfmemalloc
> reserves. This can for example lead to NFS connections hanging on
> high-volume systems.
>
> Implement a ratelimited big honking out of memory warning that directs
> the sysadmin to bump up vm.min_free_kbytes in case this problem happens.
> Our experience shows that with default vm.min_free_kbytes (90112, or
> about 90 megabytes), NFS connections hang approximately once per day,
> whereas with 901120 (default multiplied by 10) NFS connections never
> hang.
>

If NFS sessions hang, then there is a bug to eventually root cause and fix.

Just telling the user : Increase the limit is the same thing than admitting :

Our limit system or TCP or NFS stacks are broken and unable to
recover, so lets disable the limit system and work around a more
serious bug.

Maybe the bug is in a NIC driver, please share more details before
adding yet another noisy signal in syslog

SNMP counters are per netns, and more useful in the modern computing
era,  where a host is shared by many different containers.



>
> Signed-off-by: Juha-Matti Tilli <juha-matti.tilli@xxxxxxxxxx>
> Cc: Eric Dumazet <edumazet@xxxxxxxxxx>
> Cc: Rafael Aquini <aquini@xxxxxxxxxx>
> Cc: Murphy Zhou <xzhou@xxxxxxxxxx>
> Cc: Yongcheng Yang <yoyang@xxxxxxxxxx>
> Cc: Jianhong Yin <jiyin@xxxxxxxxxx>
> ---
>  net/core/filter.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/net/core/filter.c b/net/core/filter.c
> index fc92ebc4e200..7d8ef239b9af 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -99,6 +99,8 @@ int sk_filter_trim_cap(struct sock *sk, struct sk_buff *skb, unsigned int cap)
>          * helping free memory
>          */
>         if (skb_pfmemalloc(skb) && !sock_flag(sk, SOCK_MEMALLOC)) {
> +               printk_ratelimited(KERN_WARNING
> +                                  "dropped packet due to out-of-memory condition, please bump up vm.min_free_kbytes\n");
>                 NET_INC_STATS(sock_net(sk), LINUX_MIB_PFMEMALLOCDROP);
>                 return -ENOMEM;
>         }
> --
> 2.17.1
>