page faults affects the performance

[Feng Li]

Hi experts,


When I use fio to write our distribution storage, I find a lot of page
faults that affect the performance.

My OS is CentOS7,  anyone could give suggestions on how to root cause
this issue? The CPU is running at 100%.

[root@58 15:50:57 ~]$uname -a
Linux 58 3.10.0-693.11.6.el7.smartx.1.x86_64 #1 SMP Tue Jan 16
22:09:10 CST 2018 x86_64 x86_64 x86_64 GNU/Linux

Thanks in advance.

This is the perf report from "perf record -g -p 9881"

+   64.68%     0.01%  zbs-chunkd       [k] system_call_fastpath

       â
+   42.86%     0.01%  zbs-chunkd       [k] inet_recvmsg

       â
+   42.84%     0.00%  zbs-chunkd       [k] sys_recvfrom

       â
+   42.83%     0.13%  zbs-chunkd       [k] tcp_recvmsg

       â
+   42.83%     0.01%  zbs-chunkd       [k] SYSC_recvfrom

       â
+   42.75%     0.02%  zbs-chunkd       [k] sock_recvmsg

       â
+   39.52%     0.09%  zbs-chunkd       [k] skb_copy_datagram_iovec

       â
-   31.97%     7.18%  zbs-chunkd       [k] page_fault

       â
   - 24.79% page_fault

       â
      + 24.78% do_page_fault

       â
   - 7.17% __libc_recv

       â
      - 7.17% system_call_fastpath

       â
           sys_recvfrom

       â
           SYSC_recvfrom

       â
           sock_recvmsg

       â
           inet_recvmsg

       â
           tcp_recvmsg

       â
           skb_copy_datagram_iovec

       â
           page_fault

       â


+   24.79%     0.04%  zbs-chunkd       [k] do_page_fault


This is the output of "perf record -e page-faults -ag":

Samples: 132K of event 'page-faults', Event count (approx.): 9249205
  Children      Self  Command          Shared Object                 Symbol
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] skb_copy_datagram_iovec
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] tcp_recvmsg
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] inet_recvmsg
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] sock_recvmsg
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] SYSC_recvfrom
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] sys_recvfrom
+   78.92%     0.00%  zbs-chunkd       [kernel.kallsyms]
[k] system_call_fastpath
+   78.92%    78.92%  zbs-chunkd       [kernel.kallsyms]
[k] copy_user_enhanced_fast_string
+   78.92%     0.00%  zbs-chunkd       libpthread-2.17.so
[.] __libc_recv




-- 
Thanks and Best Regards,
Feng Li(Alex)