On Thu, 16 Mar 2000, James Sutherland wrote:
>On Thu, 16 Mar 2000, Paul Jakma wrote:
>
>> On Thu, 16 Mar 2000, James Sutherland wrote:
>>
>> > No. *ANY* memory allocation system can run out of memory. Avoiding
>> > "overcommitting" would make the OOM situation arise SOONER (and more
>> > frequently), as well as killing performance.
>> >
>>
>> well, it's a more a question of whether you make promises that you might
>> not be able to keep. If you do (ie overcommit) then it's your
>> (kernel) problem. If you don't, it's not.
>
>Not really; either way, we are being asked for memory we can't currently
>provide.
>
>> Without overcommit the /system/ can run out memory, of course - it's
>> finite - but it's no longer a kernel problem.
>
>It was never really a "kernel problem" as such anyway. It's a problem the
>kernel is trying to fix.
And where it couldn't detect the beginning of the problem.
>
>> > Right... Now we'll try this on the university's central Unix system, shall
>> > we? Let's see... 6000 users, 2Gb RAM+swap. They get about 300K each.
>> > That's ALMOST enough to log in with!
>>
>> well then get more ram/swap. But at least it has become a hw issue.
>
>It was all along - you don't have enough RAM+swap for the workload the
>system is under, so processes are dying.
The problem is detecting when it is running out of memory.
>> > The alternative just isn't practical in any real-world situation, except
>> > more-or-less single user boxes.
>>
>> the alternative is very practical. boxes where oracle runs as one user /
>> apache as another for example. You might want to lock down something
>> that you know can cause problems.. etc..
>
>That would help, yes, in this particular system. It's not a general
>solution, however.
>
>> > Even then, that single user will still run
>> > out of memory - it's just his quota, rather than a physical limit.
>>
>> obviously.
>>
>> > So his processes STILL end up dying randomly, but they do it sooner rather
>> > than later. Hrm. Wow.
>>
>> but then it's the users problem - not the OS..
>
>But this does nothing at all to prevent runaway root processes blowing the
>system away, for example.
Thats why per user and per process resource limits need to be implemented.
>> > On any realistically specced box, it is almost zero already.
>> >
>> no it's not. a plain linux box today has a terrible tendency to die if
>> an app misbehaves wrt to memory. It's that bad. (unless you think 1GB of
>> RAM + 4GB's+ of swap is a reasonable spec for a desktop)
>
>Yes - I OOMed this machine last night, and almost everything died as a
>result (thanks to me *NOT* having Rik's patch in yet.) These are not
>normal circumstances, though...
>
>> per process limits go a long long way to solving this (easy with
>> pam_limits) but not far enough. But per user limits would be a huge
>> improvement...
>
>Yes, I agree. However, we can still run out of RAM+swap on multiuser
>systems quite easily.
Only if they are mismanaged - allowed to have too many concurrent users,
with too many concurrent processes. I am part of a center that runs
UNIX systems that don't have this problem. The per user resource limits
are taken into account for the number of concurrent interactive users,
and the number of batch queues (with concurrent jobs). The total amount
of required resources is then allocated. There IS no OOM on this system.
Can't happen.
>> > No, it's a risk with *EVERY* OS.
>>
>> no it's not. A non overcommiting OS doesn't run out of VM for
>> processes. it cleanly grants or denies memory requests. What the app
>> does after that is not a kernel problem.
>
>The problem is, denying memory requests leads to processes dying. This is
>what we want to avoid.
No it isn't. What you want to avoid is system crashes/reboots. - If a user
process chooses to exceed its' resource limits then that process is aborted. The
system doesn't crash. The process to abort is clearly known.
>
>In fact, last night I more or less killed this machine. I had overcommit
>turned off, and nothing major loaded (X, WWW server, xmms) and fired up
>"make -j" on a biggish source tree. After quite some time, just about
>everything died from lack of memory.
too bad - user resource limits would have signaled the out of resources to
the make process (it should have been given the OOM signal when it forked/execed
the compiler/linkger). This could then be used by make to determine that
resource limits had been reached, and not to spawn more. Yes this does
require make to be modified. Otherwise the make would have aborted; BUT
the system would not fail.
>This was WITHOUT the OOM killer patch. With it, I think the newly spawned
>make and cc processes would have been killed off much sooner, helping save
>the other processes.
>
>
>How would you define "running out" of memory? I would define it as not
>being able to fulfil new requests - which is the situation we are trying
>to handle here. If you start denying malloc() requests, processes start
>dying (largely at random). If, instead, you kill selected processes, then
>obviously processes still die as a result. However, they are more likely
>to be the "right" ones to kill.
running out of memory: insufficient resources available to a specific
user to continue running processes.
Denying malloc() requests should not "largly at random" cause processes
to die; A very specific process will die - the one making the sbrk system
call. A different user would not see a problem since they should have a
different set of resources. The general system daemons would not see a
problem either - they also would have a different set of reserved resources.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@cats-chateau.net
Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Thu Mar 23 2000 - 21:00:22 EST