dynamic allocation problem in linux => DoS for all

Victor STANESCU (bruno@Heineken.lmn.pub.ro)
Sat, 11 Sep 1999 21:41:18 +0300 (EEST)


This small sample of code, as can be found in any c++ book, should
allocate memory until it cannot, and then just exit, with a message:

#include <new>
#include <iostream>

void leave_program()
{
cout<<"Cannot allocate anymore.."<<endl;
exit(1);
}

int main()
{
set_new_handler(leave_program);
while(1)
new int[1000];
}

But, surprise:
Instead of allocating some memory, then exiting with the message "Cannot
allocate anymore..", it allocates all the memory. All other applications
are swapped. When it cannot find any free space (all the RAM and the swap
are full), it dies with something like bus error, or out of memory, BUT
NOT EXECUTING THE NEW HANDLER.

Well, this is not so much a security problem, but the collateral effects
that it generates are sure very ugly, and they are:

If another user is working in X, and when this mallicious program is
running, he tries to move the mouse or launch an application, all the X
server dies because it cannot find any free memory in ram or swap for that
new request. Just run this small code and try to use X in the same time.
This thing allows an user to attack the user logged in locally.

AND MORE SEVERE: if any other application (like syslog, klog, sendmail,
named, httpd) needs to run in that precise moment when this small program
is executed, it dies. This way i have succesfully killed the programs
mentioned above, as a normal user. So anyone with local acces to the
machine can kill the mail, http, dns servers you are running. This is sure
an ugly thing and a very severe denial of service atack from local users.
I do not like to know that all my services can be put down in 5 minutes by
anyone of my users.

Some correction can be made with ulimit -v in bash2, but a program can be
launched in many other ways (from the window manager for example), so it
is not enough, and there are users which do not use bash2.

The problem affects a Redhat 6.0 (glibc 2.1) and a Slackware 3.6
(libc.so.5) , tested both with 2.0.36 and 2.2.5 kernels.

Victor STANESCU-network administrator
The Numerical Methods Laboratory,
Politehnica University of Bucharest

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/