Re: Should calculation of vm.overcommit_ratio be changed?
From: Dirk Geschke
Date: Tue Apr 20 2010 - 17:01:46 EST
Hi all,
I am not on the mailing list and a friend pointed me to this
thread...
Probably we had the same problem: We had a linux computer with
16 GB of RAM without swap. There was only one big job running
on it which did a lot of I/O. This program failed to allocate
much memory, we thought that this was due to the high amount
of cached memory use. To avoid problems with overcommit we had
set overcommit_memory to 2.
Now I have seen this thread and now it gets clear: The default
value of overcommit_ratio is 50, therefore one program can not
allocate more than 8GB of memory at all.
After reading this thread I wrote a little programm to allocate
memory in 512MB blocks and fill it with zeros. My test system
has 4GB of RAM and so I started:
qfix:~# free
total used free shared buffers cached
Mem: 4052376 338124 3714252 0 0 17992
-/+ buffers/cache: 320132 3732244
Swap: 0 0 0
geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
malloc failure after 3 * 512 MB
So 1.5 GB are ok, 2 GB of possible 4GB not. I guess some memory of
the 4GB are not useable at all and therefore the limit is slightly
below 2GB with an overcommit_ratio=50.
Next step is to set overcommit_ratio=100:
qfix:~# echo 100 >/proc/sys/vm/overcommit_ratio
and run the porgram again:
geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
got 4 * 512MB
got 5 * 512MB
got 6 * 512MB
malloc failure after 6 * 512 MB
That are more than 3 GB but I would have expected to get at least
3.5GB:
geschke@qfix:~$ free
total used free shared buffers cached
Mem: 4052376 344976 3707400 0 0 22472
-/+ buffers/cache: 322504 3729872
Swap: 0 0 0
Maybe this is due to a reserved percentage for the root user?
However, if I set overcommit_ratio=110 I get more than 3.5 GB:
geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
got 4 * 512MB
got 5 * 512MB
got 6 * 512MB
got 7 * 512MB
malloc failure after 7 * 512 MB
However, now I tested this with a high usage of cached memory, I
die a lot of I/O:
geschke@qfix:~$ free
total used free shared buffers cached
Mem: 4052376 1945512 2106864 0 0 1621200
-/+ buffers/cache: 324312 3728064
Swap: 0 0 0
A new run gives:
geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
got 4 * 512MB
got 5 * 512MB
got 6 * 512MB
got 7 * 512MB
malloc failure after 7 * 512 MB
and:
qfix:~# free
total used free shared buffers cached
Mem: 4052376 346928 3705448 0 0 26008
-/+ buffers/cache: 320920 3731456
Swap: 0 0 0
So the cached memory is not really a problem for malloc.
But since I am testing, I tried what will happens if a lot of memory
is already in use. So I opened a large file with "vi":
geschke@qfix:~$ free
total used free shared buffers cached
Mem: 4052376 1597168 2455208 0 0 391364
-/+ buffers/cache: 1205804 2846572
Swap: 0 0 0
Now I start the program again:
geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
got 4 * 512MB
got 5 * 512MB
malloc failure after 5 * 512 MB
Fine: It seems that there is not really a problem to increase to
overcommit_ratio=100 if there is no swap in the system and one
has set overcommit_memory=2.
So I think, it is not really a problem to run with these settings.
Best regards
Dirk
--
+----------------------------------------------------------------------+
| Dr. Dirk Geschke / Plankensteinweg 61 / 85435 Erding |
| Telefon: 08122-559448 / Mobil: 0176-96906350 / Fax: 08122-9818106 |
| dirk@xxxxxxxxxxxxxxxxx / dirk@xxxxxxxxxxxxx / kontakt@xxxxxxxxxxxxx |
+----------------------------------------------------------------------+
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/