Re: Runaway process and oom-killer

From: John Sigler
Date: Thu Jun 14 2007 - 05:43:23 EST

Helge Hafting wrote:

John Sigler wrote:

Andrea Arcangeli wrote:

On Wed, Jun 13, 2007 at 10:49:29AM +0200, John Sigler wrote:

Question 2: how can I tell which process or kernel thread was hogging most of the RAM when the oom-killer kicked in?

Theoretically the one that was killed first but not for sure in
current mainline hence see below.

If I read the logs correctly, oom-killer is "invoked" three times before it effectively kills a process. Then oom-killer kills myapp, syslogd, and boa, in that order. Why didn't oom-killer kill anything the first three times?

My guess:
Something needs memory but finds there is none to be had
oom-killer is invoked and targets myapp.
myapp takes some time to die. Particularly, the memory it uses
isn't freed up instantly. In the meantime something else
needs memory and find none. (Another packet received?)

Possibly. In fact, myapp receives a 40 Mbit/s stream.

The oom-killer is invoked again, this time it targets syslogd.

I went hunting, and found a memory leak in our syslogd. Doh!

And so on. The kernel do many things in parallel, running out
of memory in a multitasking system therefore is a complicated business.
Especially when process killing takes some time.

I didn't mention that there is no swap on this system.

Note that you can turn off memory overcommit, your leaky app
should then get a memory allocation error instead of
triggering the oom-killer.

Are you referring to these /proc/sys/vm entries?

# cat /proc/sys/vm/overcommit_memory
# cat /proc/sys/vm/overcommit_ratio

Are you suggesting I set overcommit_memory to 2?

The manual states:


This file contains the kernel virtual memory accounting mode.
Values are:
0: heuristic overcommit (this is the default)
1: always overcommit, never check
2: always check, never overcommit
In mode 0, calls of mmap(2) with MAP_NORESERVE set are not checked, and the default check is very weak, leading to the risk of getting a process "OOM-killed". Under Linux 2.4 any non-zero value implies mode 1. In mode 2 (available since Linux 2.6), the total virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the size of the swap space, and RAM is the size of the physical memory, and r is the contents of the file /proc/sys/vm/overcommit_ratio.

In my case, SS=0 and RAM=256MB.

If I understand correctly, if I set ratio to 50, then processes can only address 128MB. I'd be, in effect, reserving 128MB for the kernel, right?

Are there other entries in /proc/sys/vm I should be playing with? :-)


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at