Deadlock in serial driver 2.6.x

[Martin Kögler]

I noticed with different kernel versions (a 2.6.5 FC2 Kernel, a 2.6.7 Knoppix Kernel
and 2.6.10 FC2 and FC3 Kernels (which have no patches for the serial driver)), that it 
is possible for a normal user, which has rw access to /dev/ttySx, to hang a computer.
To exploit it, there must be a device on the other end on the serial line, which sends 
some data.

I tested it on a i686 machine.

At http://www.auto.tuwien.ac.at/~mkoegler/linux/serial_oops.c , I have an example programm,
which exploits the problem (/dev/ttyS0 is hardcoded as serial device).

To trigger the problem, connect two computers with a null modem cable and send some
characters to the programm (The baud rate at the other computer seems to be not important).

With SMP-Kernels, the computer stops responding.
Kernels without SMP print a panic message before they stop working, eg:
Kernel panic - not syncing: drivers/serial/serial_core.c:103: spin_lock(drivers/serial/serial_core.c:c04055e0) already locked  by drivers/serial/8250.c/1170

Photos of a panic messages of a FC3 2.6.10-1.741_FC3 Kernel are available at 
http://www.auto.tuwien.ac.at/~mkoegler/linux .

What the programm does:
It sets the low latency mode, then waiting, until a certain state of the handshake 
lines is reached, then it sends a bytes and waits for a byte. Then it changes the 
handshake lines again and the process starts again.

Martin Kögler
e9925248@xxxxxxxxxxxxxxxxxx
PS: Please CC me on replies.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/