Re: Memory overcommitting (was Re: http://www.redhat.com/redhat/)

Torbjorn Lindgren (tl@funcom.com)
Fri, 21 Feb 1997 09:01:41 +0100 (MET)


On Fri, 21 Feb 1997, Kevin Littlejohn wrote:
> NAME
> fork, vfork - create a child process
[...]
> Under bugs, it mentions that vfork is an alias for fork, but given fork on
> Linux does what vfork on other systems do, I believe you have your vfork :)

False. Linux fork() has a standard fork(), *NOT* a vfork.

All modern *NIX'es uses Copy-on-write fork()'s, but that a very different
animal from a vfork()... Not that many *NIX'es have real vfork()'s...

Cut-n-pasted from a machine with a real vfork and the manual pages to back
it up:

vfork() can be used to create new processes without fully
copying the address space of the old process. It is useful
when the purpose of fork() would have been to create a new
system context for an execve(). vfork() differs from fork()
in that the child borrows the parent's memory and thread of
control until a call to execve() or an exit (either by a
call to _exit() (see exit(2)) or abnormally). The parent
process is suspended while the child is using its resources.

There are at least two possible reasons for using vfork():
1. It doesn't use extra swap-space.
2. The OS doesn't have copy-on-write semantic available.

Neither is relevant for Linux, since it belives in memory over-commitment
and have copy-on-write fork()'s.

> > (2) The claims that statically that this is okay 99.9% is hogwash. The
> > airline industry may sell tickets that way, but they don't want their
> > computers doing it. I've been in line when it has happened to them and
> > its not a pretty site.
>
> Fact remains - the process is going to segfault somewhere if it needs more
> memory than it can get. Given, from the info in this thread, most *nix's
> (especially modern versions) use this sort of 'overcommitting', I can't

FALSE, for non-overcommited systems. Ever heard of *checking* the
return-value of malloc/calloc/realloc/new?

Most modern *NIX's *allows* you to do memory overcommitment (lazy
swapallocation etc), some have it enabled by default, on some it can be
enabled.

One thing that is missing in Linux is the possibility to *disable* the
overcommitment, which is very usefull for some applications...

SGI/Irix has probably the cleanest implementation of this of them all
(including Linux):

You can tell it how much overcommited it is allowed to be by setting up
"virtual swap". Without any "virtual swap" you have your old basic
non-overcommited VM subsystem, and with a really large "virtual swap" you
have a overcommited VM subsystem.

It's cute since you can limit the amount of over-allocation it allows, if
you so wish... Many sites run with essentially unlimited overcommitment
thought.

> see it's such a bad practice - especially when it allows you to run
> programs (netscape springs to mind) that you would not normally be able to
> run.

Which could also be fixed by having sufficent amount of swap...

> I'd be interested in what, if anything, POSIX has to say about memory
> allocation - I'd be willing to bet nothing or very little, in which case
> I'll contend relying on memory being _there_ just because you asked it be
> available is SUICIDE (hey, I like these caps :) if you intend to write
> portable programs.

You don't even have to turn to the POSIX standard. Take the C standard
instead... If you allocate memory it should be available!

> > (3) This is sure a time waster for porting and developing programs to run
> > on Linux. "Hey, you know that program that worked fine on System A could
> > blow up on Linux." This is espcially true for those people who want
> > to move binaries from some other vendor's OS.
>
> hrm. Except, under a scheme that definately allocates that memory, the
> program is going to blow up anyway. Think about it - the program blows up
> because the memory is not there. So, if the memory is not there, is any
> allocation routine going to let the program work? Your scheme: "Hey, you
> know that program that worked fine on System A could not run on Linux."

No, it wouldn't "blow up" on System A, the memory allocations would fall,
but these are *usually* checked, and the program would probably protest.

Note that there are a big difference between:
1. Memory allocations can fail
2. Memory you have successfully allocated might not be there, leading to
seg-fault on access..

It's *MUCH* easier to handle case 1, it happens in a few easily located
spots, and the behavior is well-defined. This isn't the case for case 2.

> Instead, we give the program a chance to allocate all the memory it might
> want (a common, still-taught practice), then not touch and not use any of
> that memory, without impacting on the execution of anything else on the
> system. Seems to me to be a very friendly way of doing things.

Yes, but if it use the memory something may blow up... Really bad practice
for some programs.

> Still, if you'd rather dedicate memory to a program based on the
> programmers original decision of what they might need, you're free to
> build/commission a new memory allocation system. Don't expect me to use
> it, though - none of our three commercial servers would survive that sort
> of scheme, and I doubt I could justify hunting down 128Mb sticks of memory
> for them all. And I suppose that is the acid test - in this production
> system, and in many, many other production systems, this scheme works
> without a hitch :)

Hmm, *extra memory*, sounds like your on the wrong track. A
non-overcommited VM usually doesn't need more MEMORY, it need more *SWAP*.

Frankly, at the current HD prices a non-overcommited VM does make sense,
at least as an option...

-- 
Torbjörn Lindgren
Karenlyst Allè 5, N-0277 Oslo, Norway     Phone: +47 22420102
E-mail: tl@funcom.com
If Santa ever DID deliver presents on Christmas Eve, he's dead now.