Re: running of out memory => kernel crash

[Bryan Donlan]

On Thu, Aug 18, 2011 at 10:26, Pavel Ivanov <paivanof@xxxxxxxxx> wrote:
> On Thu, Aug 18, 2011 at 8:44 AM, Denys Vlasenko
> <vda.linux@xxxxxxxxxxxxxx> wrote:
>>> I have a little concern about this explanation of yours. Suppose we
>>> have some amount of more or less actively executing processes in the
>>> system. Suppose they started to use lots of resident memory. Amount of
>>> memory they use is less than total available physical memory but when
>>> we add total size of code for those processes it would be several
>>> pages more than total size of physical memory. As I understood from
>>> your explanation in such situation one process will execute its time
>>> slice, kernel will switch to other one, find that its code was pushed
>>> out of RAM, read it from disk, execute its time slice, switch to next
>>> process, read its code from disk, execute and so on. So system will be
>>> virtually unusable because of constantly reading from disk just to
>>> execute next small piece of code. But oom will never be firing in such
>>> situation. Is my understanding correct?
>>
>> Yes.
>>
>>> Shouldn't it be considered as an unwanted behavior?
>>
>> Yes. But all alternatives (such as killing some process) seem to be worse.
>
> Could you elaborate on this? We have a completely unusable server
> which can be revived only by hard power cycling (administrators won't
> be able to log in because sshd and shell will fall victims of the same
> unending disk reading). And as an alternative we can kill some process
> and at least allow administrator to log in and check if something else
> can be done to make server feel better. Why is it worse?
>
> I understand that it could be very hard to detect such situation but
> at least it's worth trying I think.

Deciding when to call the server unusable is a policy decision that
the kernel can't make very easily on its own; the point when the
system is considered unusable may be different depending on workload.
You could create a userspace daemon, however, that mlockall()s, then
monitors memory usage, load average, etc and kills processes when
things start to go south. You could also use the memory resource
cgroup controller to set hard limits on memory usage.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/