[RFC PATCH] mmap, aslr: do not enforce legacy mmap on unlimited stacks

[Michal Hocko]

From: Michal Hocko <mhocko@xxxxxxxx>

Since cc503c1b43e0 ("x86: PIE executable randomization") we treat
applications with RLIMIT_STACK configured to unlimited as legacy
and so we a) set the mmap_base to 1/3 of address space + randomization
and b) mmap from bottom to top. This makes some sense as it allows the
stack to grow really large. On the other hand it reduces the address
space usable for default mmaps (wihout address hint) quite a lot. We
have received a bug report that SAP HANA workload has hit into this
limitation.

We could argue that the user just got what he asked for when setting
up the unlimited stack but to be realistic growing stack up to 1/6
TASK_SIZE (allowed by mmap_base) is pretty much unimited in the real
life. This would give mmap 20TB of additional address space which is
quite nice. Especially when it is much more likely to use that address
space than the reserved stack.

Digging into the history the original implementation of the
randomization 8817210d4d96 ("[PATCH] x86_64: Flexmap for 32bit and
randomized mappings for 64bit") didn't have this restriction.

Signed-off-by: Michal Hocko <mhocko@xxxxxxxx>
---

Hi,
I am sending this as a RFC because I am not really sure how to deal with
this. We might as well ignore the reported issue and claim "do not use
unlimited stacks" and be done with it. I just stroke me as an unexpected
behavior.

 arch/x86/mm/mmap.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
index 19ad095b41df..797295e792b2 100644
--- a/arch/x86/mm/mmap.c
+++ b/arch/x86/mm/mmap.c
@@ -74,9 +74,6 @@ static int mmap_is_legacy(void)
 	if (current->personality & ADDR_COMPAT_LAYOUT)
 		return 1;
 
-	if (rlimit(RLIMIT_STACK) == RLIM_INFINITY)
-		return 1;
-
 	return sysctl_legacy_va_layout;
 }
 
-- 
2.11.0