On Tue, 21 Mar 2000 16:18:13 +0100 (CET), you wrote:
>On Tue, 21 Mar 2000, James Sutherland wrote:
>> On Tue, 21 Mar 2000 13:57:07 +0100 (CET), you wrote:
>> >On Tue, 21 Mar 2000, James Sutherland wrote:
>> >
>> >[...[
>> >> So? If you touch the memory, overcommit doesn't come into play.
>> >
>> >All processes should do that for it to work. Example (you have 128MB of swap):
>> >
>> >you malloc() 1MB of space, and touch every page on it. Does swap gets
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^
>
>> >allocated at this time? Assume it is (I'm not sure: the pages are dirty,
>> >but get written on swap when needed, at later time).
>> >
>> >Another process malloc 128MB of space. It does not touch the pages,
>> >overcommiting allows malloc() to succeed.
>> >
>> >RAM gets used up by some one else. Your pages get swapped out.
>>
>> No - you haven't allocated (touched) the pages yet. They are purely
>> imaginary pages until then.
>
>Please read again. I've already touched all the pages, as you suggested.
>Maybe i'm not been clear:
>
>1) process A malloc()s 1MB. It DOES touch them, right after malloc().
>
>2) process B malloc()s 128MB. It does NOT touch all of them. Only some.
>
>3) process C malloc()s 128MB. It does NOT touch all of them.
>
>4) process C starts accessing its pages sequentially. RAM gets allocated.
>
>5) RAM is short (suppose 64MB). the system starts paging-out A, B, C pages.
> Here are you sure that A swap space is already allocated? I'm not. Anyway,
> let's assume it is. As long as the systems pages out B and C pages,
> it allocates swap space. Sooner or later we're short of swap, because
> both B and C have address spaces larger than available swap (128MB - 1MB
> taken by A).
>
>6) A (who's been sleeping all the time) wakes up, accesses one of its pages.
> No page-frame is available, so the system tries to free one used by C
> or B. But no swap space is available. Page fault for process A fails
> to provide a free page-frame to A. Boom.
That's a fault in the virtual memory system in Linux, if that happens.
Nothing to do with memory allocation, the malloc() implementation or
overcommit - it's a broken VM system.
>> >Later, you access one page of yours. It's not in RAM. The system has to
>> >fetch it from swap. First, it has to locate a free page-frame. There is none.
>> >So, it has to swap-out some other page. But, wait, swap space is full.
>> >What happens? It's your process causing the page fault.
>> >
>> >Even if your process has all swap space allocated, it's not enough.
>> >Overcommiting for other processes still hits you.
>>
>> You seem to misunderstand. Basically, malloc() allocates ADDRESS
>> SPACE, not memory. USING that address space then allocates VM (RAM or
>> swap). Problems only arise if you haven't really allocated your memory
>> yet, you only think you have (because you misunderstand the malloc()
>> implementation.)
>
>That's what *I*'ve been saying in previous messages. I'm not misunderstanding
>malloc(). You are misunderstanding that your 'safe' process may still
>page-fault. Even if its own pages are on swap, you may need to free
>other processes pages, and you can't be sure *they* can be swapped out.
That's nothing to do with memory allocation - that's a faulty VM
subsystem. A badly broken implementation, if that can happen.
>> If YOU touch memory when malloc()ed (i.e. you REALLY allocate memory,
>> not just address space to put memory in) you're fine. Only the other
>> processes can be hit.
>
>No. Only if ALL processes touch ALL pages after malloc() you are
>(partially) safe. They also need NOT to grow their stacks, for example.
OK, disabling overcommit is difficult, and would break all your
existing code. You still haven't shown anything it ever improves,
though.
James.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Thu Mar 23 2000 - 21:00:35 EST