brk() should check randomize_va_space ratherthen CONFIG_COMPAT_BRK

[Frank Heckenbach]

The brk() syscall checks CONFIG_COMPAT_BRK to decide whether the brk
area starts just after the end of the code+bss.

Since CONFIG_COMPAT_BRK can be overridden by
/proc/sys/kernel/randomize_va_space, it should rather check this
variable (which is initialized in mm/memory.c from
CONFIG_COMPAT_BRK), see the patch below.

I've tested it with an old libc5 binary: Without the patch, if the
kernel was configured without CONFIG_COMPAT_BRK, the first malloc()
fails with any setting of randomize_va_space. With the patch
applied, the program runs correctly with randomize_va_space < 2, but
not with randomize_va_space == 2, as it should be.

--- linux-2.6.37.2/mm/mmap.c.orig	2011-02-25 00:09:00.000000000 +0100
+++ linux-2.6.37.2/mm/mmap.c	2011-02-26 01:13:53.000000000 +0100
@@ -252,11 +252,11 @@
 
 	down_write(&mm->mmap_sem);
 
-#ifdef CONFIG_COMPAT_BRK
-	min_brk = mm->end_code;
-#else
-	min_brk = mm->start_brk;
-#endif
+	if (randomize_va_space < 2)
+		min_brk = mm->end_code;
+	else
+		min_brk = mm->start_brk;
+
 	if (brk < min_brk)
 		goto out;
 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/