James Sutherland wrote:
> >But which is nicer:
> >
> >- Program knows if there's enough memory because it gets SIGBUS.
> >
> >- Program knows if there's enough memory because it gets ENOMEM.
>
> It *WILL* get ENOMEM, normally.
>
> The only difference arises when:
> * You are low on memory
> * You "allocate" some memory via malloc(), but do NOT use it.
> * You leave it a while, and more memory is used.
> * THEN you use the memory you had malloc()ed earlier.
>
> If you allocate and then use memory, you're fine. Either it succeeds
> and you have memory, or it fails with ENOMEM.
There is a race condition here, which is extremely likely to occur:
1) 'A' process allocates memory.
2) 'B' process allocates memory.
3) 'A' touches all pages, fine.
4) 'B' touches all pages - SIGBUS.
You need to allocate and touch the pages atomically. The only way of
doing this currently is to use mlockall(MCL_FUTURE), but that has some
ugly semantic issues and side-effects.
> >Why do I feel like we're going in circles. Perhaps the easiest option
> >would be to add a MAP_RESERVED flag to mmap? That seems like the only
> >atomic way of allocating and reserving memory.
>
> Is there really any need? Unless you introduce a significant delay
> between allocating and using the memory, you never see any difference.
> Even then, you only experience problems if there is already a problem
> on the system (being very low on memory). Even that only shows up if
> your process is a "low priority" one.
A race condition is a race condition. It'll happen. A solution with
problems is not a solution.
> IMO, people are just trying to invent problems that don't really exist
> now. Overcommit works perfectly well. Yes, you do have problems if the
> hardware is overloaded and all your swap space is used - but then, you
> ALREADY have a problem!
Perhaps it's best that I describe in detail what I'm trying to achieve.
This is a real, not invented problem:
- 8MB of ram. No swap.
- Need to allocate pretty much all of it.
- Debugging of OOM conditions is made practically impossible because
malloc more often SIGBUS's than returns ENOMEM.
- There are 11 or so threads running, so non-atomically performing an
mmap and then page faulting is ridiculous.
Strictly speaking, none of these problems are caused by overcommit.
They're caused by just allocating too much memory. But it's a bloody
pain finding out if you've actually overcommitted because you might not
hit an out-of-memory condition for hours/days of runtime. You just end
up leaving about 1MB free because you can't be sure. Checking for free
memory is also not possible because doing so and then allocating memory
is non-atomic.
mlockall is not a solution because the make_pages_present call in
mmap_fixup causes a SIGBUS. I have some ideas which I'll be testing,
such as plain non-overcommit. Please try to bear in mind that I and
others do have real world problems that allocation reserve would fix.
-- John Ripley, empeg Ltd. http://www.empeg.com- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Thu Mar 23 2000 - 21:00:33 EST