Re: Overcommitable memory??

From: James Sutherland (
Date: Tue Mar 21 2000 - 18:52:08 EST

On Tue, 21 Mar 2000 10:54:04 -0800, you wrote:
>> On Tue, 21 Mar 2000, James Sutherland wrote:
>> > On 21 Mar 2000 13:40:28 +0100, you wrote:
>> >
>> > The process CAN be told there isn't enough memory to do what it wants.
>> > Just signal it (with a signal it can catch.) It can then use that
>> > signal to trigger garbage collection (if it's an ML or Java VM, for
>> > example), shrink the cache (an music player), reduce spare processes
>> > (Apache), whatever.
>> Interesting. The page fault routine should trigger a signal to the process
>> that caused its invocation. And what even makes you hope that that
>> process won't page fault again executing the signal handler (pieces of
>> code and data that have been much likely paged-out months ago)?
>> We are not short of memory. We're short of swap. paging-out is
>> the problem,
>> not paging-in.
> This is why it's important to let the applications know about a potential
>resource problem early enough. If you let them know too late, they may not
>be able to do a thing about it without worsening it.
> Most of my applications can do lots of things if resource consumption is a
>problem. For example, they can tell the load manager to divert connections
>to other machines and let the connections to it close by themselves. Some of
>them can free cached data. They can defer memory-intensive operations for a
>later time. They can refuse 'expensive' commands from clients.
> All of these things will reduce swap consumption eventually, it's just a
>matter of finding out early enough.
>> > IMO, it's much better to get a signal which means "we're getting short
>> > of memory, folks", which can be handled in ONE place, rather than
>> > returning 0 as a pointer - which many apps then try to dereference,
>> > ending up segfaulting themselves anyway.
>> "many apps" do not check malloc() return values? I don't call them apps,
>> I call them first time students exercises. B-)
>> They're are not able to check malloc() return value but they can setup a
>> signal handler that triggers runtime a GC() clever enough not to page
>> fault itself? I don't follow you.
> Well, I can tell you what my apps do. They have two memory allocation
>routines, a hard one and a soft one. The soft one can fail, returning NULL,
>and it's used for large allocations that correlate to starting tasks that
>aren't essential. When the soft allocator is called, the code checks for
>NULL and handles it gracefully.
> If the hard allocator gets a NULL from malloc, it frees a portion of its
>emergency pool and retries. If enough of the emergency pool is freed, it
>begins taking steps to reduce memory consumption as a I discussed above. The
>hard allocator never fails, though it may take it a while to get you the
>memory you asked for. The hard allocator is used for the millions of 'small'
>allocations where it's impractical to deal with an allocation failure.
> With a properly sized emergency pool and enough strategies for reducing
>memory consumption over time, you can avoid the hassle of having to deal
>with memory exhaustion at every memory allocation point (which can be nearly
>impossible in large applications). However, you can't do much about stack
>expansion, can you?
> Sophisticated applications need the tools to allow them to deal with memory
>(or memory+swap) exhaustion. A side issue is that applications can't tell
>when they're hitting swap either, and so can't take steps to reduce memory
>consumption to improve their own (and system) performance.

Giving apps more information about memory usage would certainly help.
If Squid could just be told "use 50% of the free memory as a cache" it
would make life a lot simpler :-)

Ideally, you would get a trappable signal which says "We're getting
short on memory, can you cut down your usage?" Then your well-behaved
app would trim itself a bit, solving the problem without killing


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
Please read the FAQ at

This archive was generated by hypermail 2b29 : Thu Mar 23 2000 - 21:00:35 EST