PROBLEM: /proc (procfs) task exit race condition causes a kernelcrash
From: Tony Griffiths
Date: Thu May 25 2006 - 20:42:48 EST
Summary:
A condition exists that crashes the kernel when one or more tasks are
exiting while at the same time another task is reading their /proc
entries. The crash is caused by either a bad VA (NULL, LIST_POISON1, or
LIST_POISON2) in prune_dcache() or a BUG_ON() sanity check in
include/linux/list.h!
Detailed Description:
If there is a great deal of modification activity in /proc caused by
task creation [fork()] and task exiting, and at the same time other
task(s) are reading /proc/<pid>/... files, the dentry_unused list
becomes corrupted and the kernel crashes, usually in function
prune_dcache() in module fs/dcache.c! A simple program that forks
itself run in a continuous loop combined with a 'find /proc ... cat {}
\;' to read the /proc task entries is all that is needed to induce the
condition. A couple of sample crash outputs look like-
(a) BUG_ON() --
------------[ cut here ]------------
kernel BUG at include/linux/list.h:167!
invalid opcode: 0000 [#1]
SMP
last sysfs file: /class/vc/vcs1/dev
Modules linked in: parport_pc lp parport autofs4 i2c_dev i2c_core
microcode binfmt_misc video thermal sony_acpi processor fan button
battery ac ehci_hcd usbcore ide_cd cdrom sg ext3 jbd dm_mod mptspi
scsi_transport_spi mptscsih mptbase sd_mod scsi_mod
CPU: 1
EIP: 0060:[<c017ec60>] Not tainted VLI
EFLAGS: 00010203 (2.6.16-mm2 #1)
EIP is at prune_dcache+0x3c6/0x3d3
eax: 00000010 ebx: f7326b08 ecx: f7326b10 edx: c017e280
esi: f7326ae0 edi: f7e81e5c ebp: 00000001 esp: f7e81e4c
ds: 007b es: 007b ss: 0068
Process init (pid: 1, threadinfo=f7e80000 task=c352eaa0)
Stack: <0>c0401c00 f7e81e5c f7e81ead c017ef28 f7e81e5c f7e81e5c f7326df8
f620c000
f7326df8 f7e81ea8 c017efe6 00000006 f7ec0e00 f620c000 c019c71a
f7326df8
f7e81e98 c036a63a 000077a4 089c21d9 00000005 f7e81ea8 c0117643
32363033
Call Trace:
<c017ef28> select_parent+0x17/0xbc <c017efe6>
shrink_dcache_parent+0x19/0x2c
<c019c71a> proc_flush_task+0x5f/0x1f5 <c0117643> sched_exit+0xb1/0xc8
<c0120552> release_task+0x84/0x101 <c01028c3> handle_signal+0x108/0x143
<c0122094> wait_task_zombie+0x2de/0x3cf <c0102984> do_signal+0x86/0x11c
<c012291d> do_wait+0x36f/0x40f <c0119153> default_wake_function+0x0/0x12
<c01f08ec> copy_to_user+0x3c/0x50 <c0119153>
default_wake_function+0x0/0x12
<c0122a8c> sys_wait4+0x3f/0x43 <c0122ab7> sys_waitpid+0x27/0x2b
<c0102b5f> syscall_call+0x7/0xb
Code: 31 ff ff ff 0f 0b a7 00 9f 69 35 c0 e9 8c fd ff ff 0f 0b a8 00 9f
69 35 c0 e9 8b fd ff ff 0f 0b a8 00 9f 69 35 c0 e9 9e fe ff ff <0f> 0b
a7 00 9f 69 35 c0 e9 85 fe ff ff 55 b8 00 1c 40 c0 57 56
(b) LIST_POISON1/LIST_POISON2 --
# Unable to handle kernel paging request at virtual address 00100104
printing eip:
c0179d12
*pde = 3780b001
Oops: 0002 [#1]
SMP
Modules linked in: parport_pc lp parport autofs4 i2c_dev i2c_core
microcode binfmt_misc video thermal processor fan button battery ac
ehci_hcd usbcore ide_cd cdrom sg ext3 jbd dm_mod mptspi mptscsih mptbase
sd_mod scsi_mod
CPU: 7
EIP: 0060:[<c0179d12>] Not tainted VLI
EFLAGS: 00010202 (2.6.16.18 #1)
EIP is at prune_dcache+0x231/0x327
eax: 00100100 ebx: f553d55c ecx: f553d564 edx: 00200200
esi: f553d534 edi: f7e81e94 ebp: 00000002 esp: f7e81e84
ds: 007b es: 007b ss: 0068
Process init (pid: 1, threadinfo=f7e80000 task=c352ea90)
Stack: <0>c03f2c00 f7e81e94 f65fcac0 c017a03d f7e81e94 f7e81e94 f576a954
f59c2a90
f576a954 00000000 c017a0fa 0000000a f7ecf000 f576a954 c0196c8a
f576a954
f59c2a90 c01212b2 f576a954 c0102993 f59c2a90 00000000 00003b88
00000000
Call Trace:
[<c017a03d>] select_parent+0x17/0xbb
[<c017a0fa>] shrink_dcache_parent+0x19/0x2c
[<c0196c8a>] proc_pid_flush+0x14/0x26
[<c01212b2>] release_task+0xa3/0x12e
[<c0102993>] handle_signal+0x108/0x143
[<c0122dac>] wait_task_zombie+0x2de/0x3c9
[<c0102a5e>] do_signal+0x90/0x127
[<c01235c9>] do_wait+0x34d/0x3de
[<c011a913>] default_wake_function+0x0/0x12
[<c01e9b5e>] copy_to_user+0x3c/0x50
[<c011a913>] default_wake_function+0x0/0x12
[<c0123722>] sys_wait4+0x3f/0x43
[<c012374d>] sys_waitpid+0x27/0x2b
[<c0102c2d>] syscall_call+0x7/0xb
Code: fe ff ff 8b 4e 50 e9 bd fe ff ff 8d 7c 24 10 89 7c 24 10 89 7c 24
14 8b 46 04 a8 10 0f 84 a8 00 00 00 8d 4e 30 8b 46 30 8b 51 04 <89> 50
04 89 02 c7 41 04 00 02 20 00 c7 46 30 00 01 10 00 83 2d
<0>Kernel panic - not syncing: Attempted to kill init!
All kernels from 2.6.15 -> 2.6.17 with any of the applicable patch-sets
(-git or -mm) are affected!!! Also RedHat FC<n> kernels.
Environment:
The environment is any SMP hardware with the kernel build with or
without PREEMPT enabled. Any P4 hyperthreaded chip, or Xeon
multi-processor system [DELL 1425 & 1850 dual-Xeon and also dual-core
dual-Xeon in my case] will exhibit the crash.
The attached forkalot.c program combined with the simple shell scripts
do the job. Running the forkalot shell script while at the same time
running any of the proc-*.sh in a 'while true; do ... ; done' loop
crashes by systems within a couple of minutes.
// Program: forkalot.c
//
// Compile: cc forkalot.c -o forkalot
// Run: ./forkalot [100] [1]
//
// Args: arg1 = # of copies of program to run simultaneously [100]
// arg2 = Sleep time before exiting [1]
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <unistd.h>
#include <fcntl.h>
#define CHILDREN 100
#define SLEEP_FOR 1
int main(int argc, char *argv[])
{
int count, this_long;
int pid;
if (argc > 1)
count = atoi(argv[1]);
else
count = CHILDREN;
if (argc > 2)
this_long = atoi(argv[2]);
else
this_long = SLEEP_FOR;
/* fork count-1 children */
while (count-- > 1) {
pid = fork();
if (pid == 0) {
/* child */
break;
} else if (pid < 0) {
perror("fork");
exit(1);
}
}
/* Sleepy... sleepy... */
sleep(this_long);
/* All done... return success! */
return 0;
}
Attachment:
forkalot-test.sh
Description: application/shellscript
Attachment:
proc-cmdline.sh
Description: application/shellscript
Attachment:
proc-status.sh
Description: application/shellscript
Attachment:
proc-torture.sh
Description: application/shellscript