[PATCH 2.6] fix mprotect() with len=(size_t)(-1) to return -ENOMEM

[Gordon Jin]

This patch fixes a corner case in sys_mprotect(): 

Case: len is so large that will overflow to 0 after page alignment.
E.g. len=(size_t)(-1), i.e. 0xff...ff.
Expected result: POSIX spec says it should return -ENOMEM.
Current result: len is aligned to 0, then treated the same as len=0 and
return success.

--- linux-2.6.11.3/mm/mprotect.c.orig	2005-03-14 13:40:28.000000000
-0800
+++ linux-2.6.11.3/mm/mprotect.c	2005-03-14 13:42:41.000000000 -0800
@@ -232,14 +232,14 @@ sys_mprotect(unsigned long start, size_t
 
 	if (start & ~PAGE_MASK)
 		return -EINVAL;
+	if (!len)
+		return 0;
 	len = PAGE_ALIGN(len);
 	end = start + len;
-	if (end < start)
+	if (end <= start)
 		return -ENOMEM;
 	if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM))
 		return -EINVAL;
-	if (end == start)
-		return 0;
 	/*
 	 * Does the application expect PROT_READ to imply PROT_EXEC:
 	 */


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/