Re: [PATCH] vmscan: page_check_references() check low order lumpy reclaim properly

[Minchan Kim]

On Fri, Apr 16, 2010 at 12:16 PM, KOSAKI Motohiro
<kosaki.motohiro@xxxxxxxxxxxxxx> wrote:
>> On Thu, Apr 15, 2010 at 12:55:30PM +0800, KOSAKI Motohiro wrote:
>> > > On Thu, Apr 15, 2010 at 12:32:50PM +0800, KOSAKI Motohiro wrote:
>> > > > > On Thu, Apr 15, 2010 at 11:31:52AM +0800, KOSAKI Motohiro wrote:
>> > > > > > > > Many applications (this one and below) are stuck in
>> > > > > > > > wait_on_page_writeback(). I guess this is why "heavy write to
>> > > > > > > > irrelevant partition stalls the whole system". ÂThey are stuck on page
>> > > > > > > > allocation. Your 512MB system memory is a bit tight, so reclaim
>> > > > > > > > pressure is a bit high, which triggers the wait-on-writeback logic.
>> > > > > > >
>> > > > > > > I wonder if this hacking patch may help.
>> > > > > > >
>> > > > > > > When creating 300MB dirty file with dd, it is creating continuous
>> > > > > > > region of hard-to-reclaim pages in the LRU list. priority can easily
>> > > > > > > go low when irrelevant applications' direct reclaim run into these
>> > > > > > > regions..
>> > > > > >
>> > > > > > Sorry I'm confused not. can you please tell us more detail explanation?
>> > > > > > Why did lumpy reclaim cause OOM? lumpy reclaim might cause
>> > > > > > direct reclaim slow down. but IIUC it's not cause OOM because OOM is
>> > > > > > only occur when priority-0 reclaim failure.
>> > > > >
>> > > > > No I'm not talking OOM. Nor lumpy reclaim.
>> > > > >
>> > > > > I mean the direct reclaim can get stuck for long time, when we do
>> > > > > wait_on_page_writeback() on lumpy_reclaim=1.
>> > > > >
>> > > > > > IO get stcking also prevent priority reach to 0.
>> > > > >
>> > > > > Sure. But we can wait for IO a bit later -- after scanning 1/64 LRU
>> > > > > (the below patch) instead of the current 1/1024.
>> > > > >
>> > > > > In Andreas' case, 512MB/1024 = 512KB, this is way too low comparing to
>> > > > > the 22MB writeback pages. There can easily be a continuous range of
>> > > > > 512KB dirty/writeback pages in the LRU, which will trigger the wait
>> > > > > logic.
>> > > >
>> > > > In my feeling from your explanation, we need auto adjustment mechanism
>> > > > instead change default value for special machine. no?
>> > >
>> > > You mean the dumb DEF_PRIORITY/2 may be too large for a 1TB memory box?
>> > >
>> > > However for such boxes, whether it be DEF_PRIORITY-2 or DEF_PRIORITY/2
>> > > shall be irrelevant: it's trivial anyway to reclaim an order-1 or
>> > > order-2 page. In other word, lumpy_reclaim will hardly go 1. ÂDo you
>> > > think so?
>> >
>> > If my remember is correct, Its order-1 lumpy reclaim was introduced
>> > for solving such big box + AIM7 workload made kernel stack (order-1 page)
>> > allocation failure.
>> >
>> > Now, We are living on moore's law. so probably we need to pay attention
>> > scalability always. today's big box is going to become desktop box after
>> > 3-5 years.
>> >
>> > Probably, Lee know such problem than me. cc to him.
>>
>> In Andreas' trace, the processes are blocked in
>> - do_fork: Â Â Â Â Â Â Âconsole-kit-d
>> - __alloc_skb: Â Â Â Â Âx-terminal-em, konqueror
>> - handle_mm_fault: Â Â Âtclsh
>> - filemap_fault: Â Â Â Âls
>>
>> I'm a bit confused by the last one, and wonder what's the typical
>> gfp order of __alloc_skb().
>
> Probably I've found one of reason of low order lumpy reclaim slow down.
> Let's fix obvious bug at first!
>
>
> ============================================================
> From: KOSAKI Motohiro <kosaki.motohiro@xxxxxxxxxxxxxx>
> Subject: [PATCH] vmscan: page_check_references() check low order lumpy reclaim properly
>
> If vmscan is under lumpy reclaim mode, it have to ignore referenced bit
> for making contenious free pages. but current page_check_references()
> doesn't.
>
> Fixes it.
>
> Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@xxxxxxxxxxxxxx>
Reviewed-by: Minchan Kim <minchan.kim@xxxxxxxxx>

I am not sure how the patch affects this problem.
But I think the patch is reasonable.

Nice catch, Kosaiki.
Below is just nitpick. :)

> ---
> Âmm/vmscan.c | Â 32 +++++++++++++++++---------------
> Â1 files changed, 17 insertions(+), 15 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 3ff3311..13d9546 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -77,6 +77,8 @@ struct scan_control {
>
> Â Â Â Âint order;
>
> + Â Â Â int lumpy_reclaim;
> +
> Â Â Â Â/* Which cgroup do we reclaim from */
> Â Â Â Âstruct mem_cgroup *mem_cgroup;
>
> @@ -575,7 +577,7 @@ static enum page_references page_check_references(struct page *page,
> Â Â Â Âreferenced_page = TestClearPageReferenced(page);
>
> Â Â Â Â/* Lumpy reclaim - ignore references */
> - Â Â Â if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
> + Â Â Â if (sc->lumpy_reclaim)
> Â Â Â Â Â Â Â Âreturn PAGEREF_RECLAIM;
>
> Â Â Â Â/*
> @@ -1130,7 +1132,6 @@ static unsigned long shrink_inactive_list(unsigned long max_scan,
> Â Â Â Âunsigned long nr_scanned = 0;
> Â Â Â Âunsigned long nr_reclaimed = 0;
> Â Â Â Âstruct zone_reclaim_stat *reclaim_stat = get_reclaim_stat(zone, sc);
> - Â Â Â int lumpy_reclaim = 0;
>
> Â Â Â Âwhile (unlikely(too_many_isolated(zone, file, sc))) {
> Â Â Â Â Â Â Â Âcongestion_wait(BLK_RW_ASYNC, HZ/10);
> @@ -1140,17 +1141,6 @@ static unsigned long shrink_inactive_list(unsigned long max_scan,
> Â Â Â Â Â Â Â Â Â Â Â Âreturn SWAP_CLUSTER_MAX;
> Â Â Â Â}
>
> - Â Â Â /*
> - Â Â Â Â* If we need a large contiguous chunk of memory, or have
> - Â Â Â Â* trouble getting a small set of contiguous pages, we
> - Â Â Â Â* will reclaim both active and inactive pages.
> - Â Â Â Â*
> - Â Â Â Â* We use the same threshold as pageout congestion_wait below.
> - Â Â Â Â*/
> - Â Â Â if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
> - Â Â Â Â Â Â Â lumpy_reclaim = 1;
> - Â Â Â else if (sc->order && priority < DEF_PRIORITY - 2)
> - Â Â Â Â Â Â Â lumpy_reclaim = 1;
>
> Â Â Â Âpagevec_init(&pvec, 1);
>
> @@ -1163,7 +1153,7 @@ static unsigned long shrink_inactive_list(unsigned long max_scan,
> Â Â Â Â Â Â Â Âunsigned long nr_freed;
> Â Â Â Â Â Â Â Âunsigned long nr_active;
> Â Â Â Â Â Â Â Âunsigned int count[NR_LRU_LISTS] = { 0, };
> - Â Â Â Â Â Â Â int mode = lumpy_reclaim ? ISOLATE_BOTH : ISOLATE_INACTIVE;
> + Â Â Â Â Â Â Â int mode = sc->lumpy_reclaim ? ISOLATE_BOTH : ISOLATE_INACTIVE;
> Â Â Â Â Â Â Â Âunsigned long nr_anon;
> Â Â Â Â Â Â Â Âunsigned long nr_file;
>
> @@ -1216,7 +1206,7 @@ static unsigned long shrink_inactive_list(unsigned long max_scan,
> Â Â Â Â Â Â Â Â * but that should be acceptable to the caller
> Â Â Â Â Â Â Â Â */
> Â Â Â Â Â Â Â Âif (nr_freed < nr_taken && !current_is_kswapd() &&
> - Â Â Â Â Â Â Â Â Â lumpy_reclaim) {
> + Â Â Â Â Â Â Â Â Â sc->lumpy_reclaim) {
> Â Â Â Â Â Â Â Â Â Â Â Âcongestion_wait(BLK_RW_ASYNC, HZ/10);
>
> Â Â Â Â Â Â Â Â Â Â Â Â/*
> @@ -1655,6 +1645,18 @@ static void shrink_zone(int priority, struct zone *zone,
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â&reclaim_stat->nr_saved_scan[l]);
> Â Â Â Â}
>
> + Â Â Â /*
> + Â Â Â Â* If we need a large contiguous chunk of memory, or have
> + Â Â Â Â* trouble getting a small set of contiguous pages, we
> + Â Â Â Â* will reclaim both active and inactive pages.
> + Â Â Â Â*/
> + Â Â Â if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
> + Â Â Â Â Â Â Â sc->lumpy_reclaim = 1;
> + Â Â Â else if (sc->order && priority < DEF_PRIORITY - 2)
> + Â Â Â Â Â Â Â sc->lumpy_reclaim = 1;
> + Â Â Â else
> + Â Â Â Â Â Â Â sc->lumpy_reclaim = 0;

How about making new function for readability instead of nesting else?
int is_lumpy_reclaim(struct scan_control *sc)
{
....
}

If you merge patch reduced stack usage of reclaim path, I think it's
enough alone scan_control argument.
It's just nitpick. :)
If you don't mind, ignore, please.


-- 
Kind regards,
Minchan Kim
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/