[2.0.36] Strange behaviour when large BSS causes OOM

Keith Whitwell (keithw@cableinet.co.uk)
Mon, 05 Apr 1999 12:30:47 +0100


I'm debugging Mesa against the Quake3 IHV. The executable I have has a
very large BSS segment, 78MB, and only after quite a bit of effort did I
come to understand that the following wierd behaviour was in fact an
out-of-memory condition triggered when the bss exceeded available core &
swap.

If I have lots of free memory:

# ldd ./quake3
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x40004000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x4009b000)
libdl.so.2 => /lib/libdl.so.2 (0x400a6000)
libm.so.6 => /lib/libm.so.6 (0x400aa000)
libc.so.6 => /lib/libc.so.6 (0x400c3000)

# ./quake3
... runs normally

If I start up netscape and emacs, free core + swap falls below 78MB, and
I get:

# ldd ./quake3
not a dynamic executable

# ./quake3
... segfaults on the first assignment to a static variable.

I don't know if this is a bug or just 'undefined behaviour', but it
seems strange that exec() should succeed even though it must have failed
to allocate the bss segment - there is no way that the call to padzero()
in load_elf_interp() could have succeeded. (Note that I'm looking at an
old 2.1.1xx source, but running a pristine redhat 5.2 with the default
2.0.36 kernel).

I think exec should detect this condition, and fail with ENOMEM. This
would be equivalent to what would happen if you tried to call brk(2)
with a similar value directly from userland.

If you don't have quake3 handy, here is a smaller program with similar
behaviour:

--------
static int m[100000000];

int main()
{
m[1000] = 44;
return m[0];
}
---------

Please CC responses to me as I don't subscribe to this list.

Keith

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/