[PATCH 0/8] Reduce writeback from page reclaim context V4

[Mel Gorman]

Sorry for the long delay, I got side-tracked on other bugs.

This is a follow-on series from the series "Avoid overflowing of stack
during page reclaim". It eliminates writeback requiring a filesystem from
direct reclaim and follows on by reducing the amount of IO required from
page reclaim to mitigate any corner cases from the modification.

Changelog since V3
  o Distinguish between file and anon related IO from page reclaim
  o Allow anon writeback from reclaim context
  o Sync old inodes first in background writeback
  o Pre-emptively clean pages when dirty pages are encountered on the LRU
  o Rebase to 2.6.35-rc5

Changelog since V2
  o Add acks and reviewed-bys
  o Do not lock multiple pages at the same time for writeback as it's unsafe
  o Drop the clean_page_list function. It alters timing with very little
    benefit. Without the contiguous writing, it doesn't do much to simplify
    the subsequent patches either
  o Throttle processes that encounter dirty pages in direct reclaim. Instead
    wakeup flusher threads to clean the number of pages encountered that were
    dirty
 
Changelog since V1
  o Merge with series that reduces stack usage in page reclaim in general
  o Allow memcg to writeback pages as they are not expected to overflow stack
  o Drop the contiguous-write patch for the moment

There is a problem in the stack depth usage of page reclaim. Particularly
during direct reclaim, it is possible to overflow the stack if it calls into
the filesystems writepage function. This patch series begins by preventing
writeback from direct reclaim and allowing btrfs and xfs to writeback from
kswapd context. As this is a potentially large change, the remainder of
the series aims to reduce any filesystem writeback from page reclaim and
depend more on background flush.

The first patch in the series is a roll-up of what should currently be
in mmotm. It's provided for convenience of testing.

Patch 2 and 3 note that it is important to distinguish between file and anon
page writeback from page reclaim as they use stack to different depths. It
updates the trace points and scripts appropriately noting which mmotm patch
they should be merged with.

Patch 4 prevents direct reclaim writing out filesystem pages while still
allowing writeback of anon pages which is in less danger of stack overflow
and doesn't have something like background flush to clean the pages.
For filesystem pages, flusher threads are asked to clean the number of
pages encountered, the caller waits on congestion and puts the pages back
on the LRU.  For lumpy reclaim, the caller will wait for a time calling the
flusher multiple times waiting on dirty pages to be written out before trying
to reclaim the dirty pages a second time. This increases the responsibility
of kswapd somewhat because it's now cleaning pages on behalf of direct
reclaimers but unlike background flushers, kswapd knows what zone pages
need to be cleaned from. As it is async IO, it should not cause kswapd to
stall (at least until the queue is congested) but the order that pages are
reclaimed on the LRU is altered. Dirty pages that would have been reclaimed
by direct reclaimers are getting another lap on the LRU. The dirty pages
could have been put on a dedicated list but this increased counter overhead
and the number of lists and it is unclear if it is necessary.

Patches 5 and 6 revert chances on XFS and btrfs that ignore writeback from
reclaim context which is a relatively recent change. extX could be modified
to allow kswapd to writeback but it is a relatively deep change. There may
be some collision with items in the filesystem git trees but it is expected
to be trivial to resolve.

Patch 7 makes background flush behave more like kupdate by syncing old or
expired inodes first as implemented by Wu Fengguang. As filesystem pages are
added onto the inactive queue and only promoted if referenced, it makes sense
to write old pages first to reduce the chances page reclaim is initiating IO.

Patch 8 notes that dirty pages can still be found at the end of the LRU.
If a number of them are encountered, it's reasonable to assume that a similar
number of dirty pages will be discovered in the very near future as that was
the dirtying pattern at the time. The patch pre-emptively kicks background
flusher to clean a number of pages creating feedback from page reclaim to
background flusher that is based on scanning rates. Christoph has described
discussions on this patch as a "band-aid" but Rik liked the idea and the
patch does have interesting results so is worth a closer look.

I ran a number of tests with monitoring on X86, X86-64 and PPC64. Each machine
had 3G of RAM and the CPUs were

X86:	Intel P4 2 core
X86-64:	AMD Phenom 4-core
PPC64:	PPC970MP

Each used a single disk and the onboard IO controller. Dirty ratio was left
at 20. Tests on an earlier series indicated that moving to 40 did not make
much difference. The filesystem used for all tests was XFS.

Four kernels are compared.

traceonly-v4r7		is the first 3 patches of this series
nodirect-v4r7		is the first 6 patches
flusholdest-v4r7	makes background flush behave like kupdated (patch 1-7)
flushforward-v4r7	pre-emptively cleans pages when encountered on the LRU (patch 1-8)

The results on each test is broken up into two parts.  The first part is a
report based on the ftrace postprocessing script in patch 4 and reports on
direct reclaim and kswapd activity. The second part reports what percentage
of time was spent in direct reclaim and kswapd being awake.

To work out the percentage of time spent in direct reclaim, I used
/usr/bin/time to get the User + Sys CPU time. The stalled time was taken
from the post-processing script.  The total time is (User + Sys + Stall)
and obviously the percentage is of stalled over total time.

I am omitting the actual performance results simply because they are not
interesting with very few significant changes.

kernbench
=========

No writeback from reclaim initiated and no performance change of significance.

IOzone
======

No writeback from reclaim initiated and no performance change of significance.


SysBench
========

The results were based on a read/write and as the machine is under-provisioned
for the type of tests, figures are very unstable so not reported.  with
variances up to 15%. Part of the problem is that larger thread counts push
the test into swap as the memory is insufficient and destabilises results
further. I could tune for this, but it was reclaim that was important.

X86
                                 raceonly-v4r7  nodirect-v4r7 flusholdest-v4r7 flushforward-v4r7
Direct reclaims                                 18         25          6        196 
Direct reclaim pages scanned                  1615       1662        605      22233 
Direct reclaim write file async I/O             40          0          0          0 
Direct reclaim write anon async I/O              0          0         13          9 
Direct reclaim write file sync I/O               0          0          0          0 
Direct reclaim write anon sync I/O               0          0          0          0 
Wake kswapd requests                        171039     401450     313156      90960 
Kswapd wakeups                                 685        532        611        262 
Kswapd pages scanned                      14272338   12209663   13799001    5230124 
Kswapd reclaim write file async I/O         581811      23047      23795        759 
Kswapd reclaim write anon async I/O         189590     124947     114948      42906 
Kswapd reclaim write file sync I/O               0          0          0          0 
Kswapd reclaim write anon sync I/O               0          0          0          0 
Time stalled direct reclaim (ms)              0.00       0.91       0.92       1.31 
Time kswapd awake (ms)                     1079.32    1039.42    1194.82    1091.06 

User/Sys Time Running Test (seconds)       1312.24   1241.37   1308.16   1253.15
Percentage Time Spent Direct Reclaim         0.00%     0.00%     0.00%     0.00%
Total Elapsed Time (seconds)               8411.28   7471.15   8292.18   8170.16
Percentage Time kswapd Awake                 3.45%     0.00%     0.00%     0.00%

Dirty file pages from X86 were not much of a problem to begin with and the
patches eliminate them as expected. What is interesting is nodirct-v4r7
made such a large difference to the amount of filesystem pages that had
to be written back. Apparently, background flush must have been doing a
better job getting them cleaned in time and the direct reclaim stalls are
harmful overall. Waking background threads for dirty pages made a very large
difference to the number of pages written back. With all patches applied,
just 759 filesystem pages were written back in comparison to 581811 in the
vanilla kernel and overall the number of pages scanned was reduced.

X86-64
                                 traceonly-v4r7 nodirect-v4r7 flusholdest-v4r7 flushforward-v4r7
Direct reclaims                                795       1662       2131       6459 
Direct reclaim pages scanned                204900     127300     291647     317035 
Direct reclaim write file async I/O          53763          0          0          0 
Direct reclaim write anon async I/O           1256        730       6114         20 
Direct reclaim write file sync I/O              10          0          0          0 
Direct reclaim write anon sync I/O               0          0          0          0 
Wake kswapd requests                        690850    1457411    1713379    1648469 
Kswapd wakeups                                1683       1353       1275       1171 
Kswapd pages scanned                      17976327   15711169   16501926   12634291 
Kswapd reclaim write file async I/O         818222      26560      42081       6311 
Kswapd reclaim write anon async I/O         245442     218708     209703     205254 
Kswapd reclaim write file sync I/O               0          0          0          0 
Kswapd reclaim write anon sync I/O               0          0          0          0 
Time stalled direct reclaim (ms)             13.50      41.19      69.56      51.32 
Time kswapd awake (ms)                     2243.53    2515.34    2767.58    2607.94 

User/Sys Time Running Test (seconds)        687.69    650.83    653.28    640.38
Percentage Time Spent Direct Reclaim         0.01%     0.00%     0.00%     0.00%
Total Elapsed Time (seconds)               6954.05   6472.68   6508.28   6211.11
Percentage Time kswapd Awake                 0.04%     0.00%     0.00%     0.00%

Direct reclaim of filesystem pages is eliminated as expected. Again, the
overall number of pages that need to be written back by page reclaim is
reduced. Flushing just the oldest inode was not much of a help in terms
of how many pages needed to be written back from reclaim but pre-emptively
waking flusher threads helped a lot.

Oddly, more time was spent in direct reclaim with the patches as a greater
number of anon pages needed to be written back. It's possible this was
due to the test making more forward progress as indicated by the shorter
running time.

PPC64
                                 traceonly-v4r7 nodirect-v4r7 flusholdest-v4r7 flushforward-v4r7
Direct reclaims                               1517      34527      32365      51973 
Direct reclaim pages scanned                144496    2041199    1950282    3137493 
Direct reclaim write file async I/O          28147          0          0          0 
Direct reclaim write anon async I/O            463      25258      10894          0 
Direct reclaim write file sync I/O               7          0          0          0 
Direct reclaim write anon sync I/O               0          1          0          0 
Wake kswapd requests                       1126060    6578275    6281512    6649558 
Kswapd wakeups                                 591        262        229        247 
Kswapd pages scanned                      16522849   12277885   11076027    7614475 
Kswapd reclaim write file async I/O        1302640      50301      43308       8658 
Kswapd reclaim write anon async I/O         150876     146600     159229     134919 
Kswapd reclaim write file sync I/O               0          0          0          0 
Kswapd reclaim write anon sync I/O               0          0          0          0 
Time stalled direct reclaim (ms)             32.28     481.52     535.15     342.97 
Time kswapd awake (ms)                     1694.00    4789.76    4426.42    4309.49 

User/Sys Time Running Test (seconds)       1294.96    1264.5   1254.92   1216.92
Percentage Time Spent Direct Reclaim         0.03%     0.00%     0.00%     0.00%
Total Elapsed Time (seconds)               8876.80   8446.49   7644.95   7519.83
Percentage Time kswapd Awake                 0.05%     0.00%     0.00%     0.00%

Direct reclaim filesystem writes are eliminated but the scan rates went way
up. It implies that direct reclaim was spinning quite a bit and finding
clean pages allowing the test to complete 22 minutes faster. S Flushing
oldest inodes helped but pre-emptively waking background flushers helped
more in terms of the number of pages cleaned by page reclaim.

Stress HighAlloc
================

This test builds a large number of kernels simultaneously so that the total
workload is 1.5 times the size of RAM. It then attempts to allocate all of
RAM as huge pages. The metric is the percentage of memory allocated using
load (Pass 1), a second attempt under load (Pass 2) and when the kernel
compiles are finishes and the system is quiet (At Rest). The patches have
little impact on the success rates.

X86
                                 traceonly-v4r7 nodirect-v4r7 flusholdest-v4r7 flushforward-v4r7
Direct reclaims                                623        607        611        491 
Direct reclaim pages scanned                126515     117477     142502      91649 
Direct reclaim write file async I/O            896          0          0          0 
Direct reclaim write anon async I/O          35286      27508      35688      24819 
Direct reclaim write file sync I/O             580          0          0          0 
Direct reclaim write anon sync I/O           13932      12301      15203      11509 
Wake kswapd requests                          1561       1650       1618       1152 
Kswapd wakeups                                 183        209        211         79 
Kswapd pages scanned                       9391908    9144543   11418802    6959545 
Kswapd reclaim write file async I/O          92730       7073       8215        807 
Kswapd reclaim write anon async I/O         946499     831573    1164240     833063 
Kswapd reclaim write file sync I/O               0          0          0          0 
Kswapd reclaim write anon sync I/O               0          0          0          0 
Time stalled direct reclaim (ms)           4653.17    4193.28    5292.97    6954.96 
Time kswapd awake (ms)                     4618.67    3787.74    4856.45   55704.90 

User/Sys Time Running Test (seconds)       2103.48   2161.14      2131   2160.01
Percentage Time Spent Direct Reclaim         0.33%     0.00%     0.00%     0.00%
Total Elapsed Time (seconds)               6996.43   6405.43   7584.74   8904.53
Percentage Time kswapd Awake                 0.80%     0.00%     0.00%     0.00%

Total time running the test was increased unfortunately but this was
the only instance it occurred. Similar story as elsewhere otherwise -
filesystem direct writes are eliminated and overall filesystem writes from
page reclaim are significantly reduced to almost negligible levels (0.01%
of pages scanned by kswapd resulted in a filesystem write for the full
series in comparison to 0.99% in the vanilla kernel).

X86-64
                traceonly-v4r7     nodirect-v4r7  flusholdest-v4r7 flushforward-v4r7
Direct reclaims                               1275       1300       1222       1224 
Direct reclaim pages scanned                156940     152253     148993     148726 
Direct reclaim write file async I/O           2472          0          0          0 
Direct reclaim write anon async I/O          29281      26887      28073      26283 
Direct reclaim write file sync I/O            1943          0          0          0 
Direct reclaim write anon sync I/O           11777       9258      10256       8510 
Wake kswapd requests                          4865      12895       1185       1176 
Kswapd wakeups                                 869        757        789        822 
Kswapd pages scanned                      41664053   30419872   29602438   42603986 
Kswapd reclaim write file async I/O         550544      16092      12775       4414 
Kswapd reclaim write anon async I/O        2409931    1964446    1779486    1667076 
Kswapd reclaim write file sync I/O               0          0          0          0 
Kswapd reclaim write anon sync I/O               0          0          0          0 
Time stalled direct reclaim (ms)           8908.93    7920.53    6192.17    5926.47 
Time kswapd awake (ms)                     6045.11    5486.48    3945.35    3367.01 

User/Sys Time Running Test (seconds)       2813.44   2818.17    2801.8   2803.61
Percentage Time Spent Direct Reclaim         0.21%     0.00%     0.00%     0.00%
Total Elapsed Time (seconds)              11217.45  10286.90   8534.22   8332.84
Percentage Time kswapd Awake                 0.03%     0.00%     0.00%     0.00%

Unlike X86, total time spent on the test was significantly reduced and like
elsewhere, filesystem IO due to reclaim is way down.

PPC64
                traceonly-v4r7     nodirect-v4r7  flusholdest-v4r7 flushforward-v4r7
Direct reclaims                                665        709        652        663 
Direct reclaim pages scanned                145630     125161     116556     124718 
Direct reclaim write file async I/O            946          0          0          0 
Direct reclaim write anon async I/O          26983      23160      28531      23360 
Direct reclaim write file sync I/O             596          0          0          0 
Direct reclaim write anon sync I/O           17517      13635      16114      13121 
Wake kswapd requests                           271        302        299        278 
Kswapd wakeups                                 181        164        158        172 
Kswapd pages scanned                      68789711   68058349   54613548   64905996 
Kswapd reclaim write file async I/O         159196      20569      17538       2475 
Kswapd reclaim write anon async I/O        2311178    1962398    1811115    1829023 
Kswapd reclaim write file sync I/O               0          0          0          0 
Kswapd reclaim write anon sync I/O               0          0          0          0 
Time stalled direct reclaim (ms)          13784.95   12895.39   11132.26   11785.26 
Time kswapd awake (ms)                    13331.51   12603.74   10956.18   11479.22 

User/Sys Time Running Test (seconds)       3567.03   2730.23   2682.86   2668.08
Percentage Time Spent Direct Reclaim         0.33%     0.00%     0.00%     0.00%
Total Elapsed Time (seconds)              15282.74  14347.67  12614.61  13386.85
Percentage Time kswapd Awake                 0.08%     0.00%     0.00%     0.00%

Similar story, the test completed faster and page reclaim IO is down.

Overall, the patches seem to help. Reclaim activity is reduced while test
times are generally improved. A big concern with V3 was that direct reclaim
not being able to write pages could lead to unexpected behaviour. This
series mitigates that risk by reducing the amount of IO initiated by page
reclaim making it a rarer event.

Mel Gorman (7):
  MMOTM MARKER
  vmscan: tracing: Update trace event to track if page reclaim IO is
    for anon or file pages
  vmscan: tracing: Update post-processing script to distinguish between
    anon and file IO from page reclaim
  vmscan: Do not writeback filesystem pages in direct reclaim
  fs,btrfs: Allow kswapd to writeback pages
  fs,xfs: Allow kswapd to writeback pages
  vmscan: Kick flusher threads to clean pages when reclaim is
    encountering dirty pages

Wu Fengguang (1):
  writeback: sync old inodes first in background writeback

 .../trace/postprocess/trace-vmscan-postprocess.pl  |   89 +++++++++-----
 Makefile                                           |    2 +-
 fs/btrfs/disk-io.c                                 |   21 +----
 fs/btrfs/inode.c                                   |    6 -
 fs/fs-writeback.c                                  |   19 +++-
 fs/xfs/linux-2.6/xfs_aops.c                        |   15 ---
 include/trace/events/vmscan.h                      |    8 +-
 mm/vmscan.c                                        |  121 ++++++++++++++++++-
 8 files changed, 195 insertions(+), 86 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/