Re: Memory intensive processes (fwd)

Richard B. Johnson (root@analogic.com)
Wed, 11 Dec 1996 19:13:54 -0500 (EST)


On Wed, 11 Dec 1996, Jim Nance wrote:

>
> Forwarded message:
> > Date: Wed, 11 Dec 1996 16:14:43 -0500 (EST)
> > From: "Richard B. Johnson" <root@analogic.com>
>
> > idea is that, unless actually written, the zero-filled memory doesn't have
> > to exist at all! This is called "demand-zero" paging and has been used
> > in VAXen forever.
>
[SNIP]
>
> Doesnt linux already do this, or at least something very close? Consider
> the program:

[SNIP DEMO PROGRAM]

> It mallocs 64M of ram, reads from each page, but never dirties any of them.
> You can see from the output that it uses almost no real memory at all:
>
> jlnance-pc> a.out
> total used free shared buffers cached
> Mem: 18808 18144 664 6464 1012 11572
> -/+ buffers: 5560 13248
> Swap: 65388 1444 63944
> total used free shared buffers cached
> Mem: 18808 18212 596 6476 1012 11572
> -/+ buffers: 5628 13180
> Swap: 65388 1444 63944
>
>
> Am I missing something?
>
> Jim

Yes, Linux does something LIKE this. But I beleive that the malloc..ed
memory was allocated (all of it), but marked read-only. When a write
to the memory occurs, the kernel makes a physical allocation and marks
it read/write. However, I think that all of it gets allocated when
a write occurs, at least everything up to the point at which the write
occurred. I think that if you write a single byte at the end of your
20 megabyte buffer, the whole buffer then becomes "REAL" rather than a
single page. You might check this out. Note that the ix86 makes it
fairly easy (perhaps a bad word) to do this. It's just a matter of what
information about a process the kernel needs to keep track of. You
mark something read/only. When a write is attempted, an exception (trap)
occurs. The kernel knows the process that created the trap and the
address that caused it. It can also restart any instruction.

I know that marking pages read/only is done in Linux during fork(). That
allows a child to exist without allocating any real RAM for it. Then
a mechanism called "copy on write" occurs in which real RAM is allocated
as a result of the exception generated when the child attempts to write
to the parent's data. Then, and only then, is the data space allocated and
the parent's data copied over.

Cheers,
Dick Johnson
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Richard B. Johnson
Project Engineer
Analogic Corporation
Voice : (508) 977-3000 ext. 3754
Fax : (508) 532-6097
Modem : (508) 977-6870
Ftp : ftp@boneserver.analogic.com
Email : rjohnson@analogic.com, johnson@analogic.com
Penguin : Linux version 2.1.14 on an i586 machine.
Warning : It's hard to remain at the trailing edge of technology.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-