Re: [RFC PATCH v2 1/2] mm: tuning hardcoded reserved memory

From: Ric Mason
Date: Fri Mar 01 2013 - 19:29:42 EST

On 03/02/2013 06:41 AM, Andrew Shewmaker wrote:
On Fri, Mar 01, 2013 at 10:40:43AM +0800, Ric Mason wrote:
On 02/28/2013 11:48 AM, Andrew Shewmaker wrote:
On Thu, Feb 28, 2013 at 02:12:00PM -0800, Andrew Morton wrote:
On Wed, 27 Feb 2013 15:56:30 -0500
Andrew Shewmaker <agshew@xxxxxxxxx> wrote:

The following patches are against the mmtom git tree as of February 27th.

The first patch only affects OVERCOMMIT_NEVER mode, entirely removing
the 3% reserve for other user processes.

The second patch affects both OVERCOMMIT_GUESS and OVERCOMMIT_NEVER
modes, replacing the hardcoded 3% reserve for the root user with a
tunable knob.

Gee, it's been years since anyone thought about the overcommit code.

Documentation/vm/overcommit-accounting says that OVERCOMMIT_ALWAYS is
"Appropriate for some scientific applications", but doesn't say why.
You're running a scientific cluster but you're using OVERCOMMIT_NEVER,
I think? Is the documentation wrong?
None of my scientists appeared to use sparse arrays as Alan described.
My users would run jobs that appeared to initialize correctly. However,
they wouldn't write to every page they malloced (and they wouldn't use
calloc), so I saw jobs failing well into a computation once the
simulation tried to access a page and the kernel couldn't give it to them.

I think Roadrunner (http://en.wikipedia.org/wiki/IBM_Roadrunner) was
the first cluster I put into OVERCOMMIT_NEVER mode. Jobs with
infeasible memory requirements fail early and the OOM killer
gets triggered much less often than in guess mode. More often than not
the OOM killer seemed to kill the wrong thing causing a subtle brokenness.
Disabling overcommit worked so well during the stabilization and
early user phases that we did the same with other clusters.
Do you mean OVERCOMMIT_NEVER is more suitable for scientific
application than OVERCOMMIT_GUESS and OVERCOMMIT_ALWAYS? Or should
depend on workload? Since your users would run jobs that wouldn't
write to every page they malloced, so why OVERCOMMIT_GUESS is not
more suitable for you?
It depends on the workload. They eventually wrote to every page,
but not early in the life of the process, so they thought they
were fine until the simulation crashed.

Why overcommit guess is not suitable even they eventually wrote to every page? It takes free pages, file pages, available swap pages, reclaimable slab pages into consideration. In other words, these are all pages available, then why overcommit is not suitable? Actually, I confuse what's the root different of overcommit guess and never?


__vm_enough_memory reserves 3% of free pages with the default
overcommit mode and 6% when overcommit is disabled. These hardcoded
values have become less reasonable as memory sizes have grown.

On scientific clusters, systems are generally dedicated to one user.
Also, overcommit is sometimes disabled in order to prevent a long
running job from suddenly failing days or weeks into a calculation.
In this case, a user wishing to allocate as much memory as possible
to one process may be prevented from using, for example, around 7GB
out of 128GB.

The effect is less, but still significant when a user starts a job
with one process per core. I have repeatedly seen a set of processes
requesting the same amount of memory fail because one of them could
not allocate the amount of memory a user would expect to be able to
allocate.

...

--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -182,11 +182,6 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
allowed -= allowed / 32;
allowed += total_swap_pages;
- /* Don't let a single process grow too big:
- leave 3% of the size of this process for other processes */
- if (mm)
- allowed -= mm->total_vm / 32;
-
if (percpu_counter_read_positive(&vm_committed_as) < allowed)
return 0;
So what might be the downside for this change? root can't log in, I
assume. Have you actually tested for this scenario and observed the
effects?

If there *are* observable risks and/or to preserve back-compatibility,
I guess we could create a fourth overcommit mode which provides the
headroom which you desire.

Also, should we be looking at removing root's 3% from OVERCOMMIT_GUESS
as well?
The downside of the first patch, which removes the "other" reserve
(sorry about the confusing duplicated subject line), is that a user
may not be able to kill their process, even if they have a shell prompt.
When testing, I did sometimes get into spot where I attempted to execute
kill, but got: "bash: fork: Cannot allocate memory". Of course, a
user can get in the same predicament with the current 3% reserve--they
just have to start processes until 3% becomes negligible.

With just the first patch, root still has a 3% reserve, so they can
still log in.

When I resubmit the second patch, adding a tunable rootuser_reserve_pages
variable, I'll test both guess and never overcommit modes to see what
minimum initial values allow root to login and kill a user's memory
hogging process. This will be safer than the current behavior since
root's reserve will never shrink to something useless in the case where
a user has grabbed all available memory with many processes.
The idea of two patches looks reasonable to me.

As an estimate of a useful rootuser_reserve_pages, the rss+share size of
Sorry for my silly, why you mean share size is not consist in rss size?
For some reason I had it in my head that RSS was just the memory
private to the process and that I needed to add memory shared for
libraries. So yeah, it looks like 8MB, or 2000 pages should be
enough of a reserve.

I'm testing new versions now, where the reserve is min(%3, k) as
Alan suggested. k being 2000 pages in this case.

sshd, bash, and top is about 16MB. Overcommit disabled mode would need
closer to 360MB for the same processes. On a 128GB box 3% is 3.8GB, so
the new tunable would still be a win.

I think the tunable would benefit everyone over the current behavior,
but would you prefer it if I only made it tunable in a fourth overcommit
mode in order to preserve back-compatibility?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxxx For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/