Re: Overcomittable memory

From: John Ripley (john@empeg.com)
Date: Fri Mar 31 2000 - 08:33:28 EST


James Sutherland wrote:
> On Wed, 29 Mar 2000 17:37:09 +0100, you wrote:
> >James Sutherland wrote:
> >> On 28 Mar 2000 00:26:37 GMT, you wrote:
> >> >On 21 Mar 2000 23:36:59 +0100, Marco Colombo <marco@esi.it> wrote:
> >> >reserving some memory only for kernel and/or root, etc, but it will just
> >> >lower the chances for OOM, not eliminate it.
> >> >OOM will still happen, and we need some handling when it hits us. killing
> >> >random process is at least guaranteed to be fair as far as it goes, but
> >> >fairness is not necessarily the most sane thing to do in all cases...
> >>
> >> Not necessarily "killing" - just "signalling". Properly written, a
> >> program should be able to handle the signal and exit gracefully. If it
> >> doesn't handle a "something's wrong" signal, can we expect it to check
> >> every malloc() call for an equivalent return value?
> >
> >Have you tried implementing a program that handles SIGBUS? It's a bloody
> >nightmare that will drive you insane (especially if you plan on using
> >threads).
>
> I wouldn't have suggested SIGBUS for the job. Something equivalent to
> SIGLOWMEMORY. Remember, this can only ever be called on a memory
> allocation failure (writing to a COW page for the first time, etc.) -
> so you couldn't get a loop, provided you initialise all the structures
> used directly by the signal handler. (Just after every fork, disable
> the handler, touch the structures, reenable it.)
>
> As someone else mentioned, this would be very useful for JVMs,
> interpreters, server daemons etc. to trigger garbage collection or
> cache shrinkage.
>
> With this signal, you could write a pretty simple wrapper to replace
> malloc() etc. with the functionality you want, just by touching all
> the memory before returning, and using the signal to cause an ENOMEM.

I also cannot see how a SIGLOWMEMORY could possibly work. E.g:

        char *p = mmap(...1048576*1024...)
        signal(SIGLOWMEMORY, my_handler);
        for(int i=0; i<1048576*1024; i+=PAGE_SIZE) p[i] = 0;

Example run:

- Memory allocated^H^H^H^H^H^H^H^H^Hmapped
- for loop triggers SIGLOWMEMORY when free memory goes below threshold
- Signal handler can't free up any memory (e.g everything's freed apart
from char *p)
- Returns to for loop
- SIGBUS as p[i] touches a page which can't be handled.

This basically isn't a "guaranteed" solution. It's prone to race
conditions and it's almost as much of a pain to implement as the SIGBUS
method I showed. The only possible way of atomically mapping memory and
paging it in is to do it within mmap (for example, with mlockall).
Everything else cannot guaranteed.

-- 
John Ripley, empeg Ltd.
http://www.empeg.com

- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/



This archive was generated by hypermail 2b29 : Fri Mar 31 2000 - 21:00:29 EST