brk() system call, RLIMIT_AS, and RLIMIT_DATA in 32-bit compatibility mode

[Ramon de C Valle]

Hi,

Today I came across an interesting behaviour and would like some enlightenment of what I'm probably missing.

Considering the following program:

.globl _start

_start:
    xorl    %ebx,%ebx
    pushl   $0xc0000000
    popl    %ebx
continue:
    addl    $0x01000000,%ebx
    xorl    %eax,%eax
    movb    $0x2d,%al
    int     $0x80
    jmp     continue


This is the strace output of the above program in a 32-bit machine:

[...]

brk(0xb9000000)                         = 0xb9000000
brk(0xba000000)                         = 0xba000000
brk(0xbb000000)                         = 0xbb000000
brk(0xbc000000)                         = 0xbc000000
brk(0xbd000000)                         = 0xbd000000
brk(0xbe000000)                         = 0xbe000000
brk(0xbf000000)                         = 0xbf000000
brk(0xc0000000)                         = 0xbf000000
brk(0xc1000000)                         = 0xbf000000
brk(0xc2000000)                         = 0xbf000000
brk(0xc3000000)                         = 0xbf000000
brk(0xc4000000)                         = 0xbf000000
brk(0xc5000000)                         = 0xbf000000

[...]

brk(0xf9000000)                         = 0xbf000000
brk(0xfa000000)                         = 0xbf000000
brk(0xfb000000)                         = 0xbf000000
brk(0xfc000000)                         = 0xbf000000
brk(0xfd000000)                         = 0xbf000000
brk(0xfe000000)                         = 0xbf000000
brk(0xff000000)                         = 0xbf000000
brk(0)                                  = 0xbf000000
brk(0x1000000)                          = 0xbf000000
brk(0x2000000)                          = 0xbf000000
brk(0x3000000)                          = 0xbf000000
brk(0x4000000)                          = 0xbf000000
brk(0x5000000)                          = 0xbf000000
brk(0x6000000)                          = 0xbf000000
brk(0x7000000)                          = 0xbf000000
brk(0x8000000)                          = 0xbf000000
brk(0x9000000)                          = 0xbf000000
brk(0xa000000)                          = 0xa000000

[...]

Notice that from 0xc0000000 upwards it fails and returns the current break as expected. However, the following is the strace output of the same program in a 64-bit machine running in 32-bit compatibility mode:

[...]

brk(0xb9000000)                         = 0xffffffffb9000000
brk(0xba000000)                         = 0xffffffffba000000
brk(0xbb000000)                         = 0xffffffffbb000000
brk(0xbc000000)                         = 0xffffffffbc000000
brk(0xbd000000)                         = 0xffffffffbd000000
brk(0xbe000000)                         = 0xffffffffbe000000
brk(0xbf000000)                         = 0xffffffffbf000000
brk(0xc0000000)                         = 0xffffffffc0000000
brk(0xc1000000)                         = 0xffffffffc1000000
brk(0xc2000000)                         = 0xffffffffc2000000
brk(0xc3000000)                         = 0xffffffffc3000000
brk(0xc4000000)                         = 0xffffffffc4000000
brk(0xc5000000)                         = 0xffffffffc5000000

[...]

brk(0xf9000000)                         = 0xfffffffff9000000
brk(0xfa000000)                         = 0xfffffffffa000000
brk(0xfb000000)                         = 0xfffffffffb000000
brk(0xfc000000)                         = 0xfffffffffc000000
brk(0xfd000000)                         = 0xfffffffffd000000
brk(0xfe000000)                         = 0xfffffffffe000000
brk(0xff000000)                         = 0xffffffffff000000
brk(0)                                  = 0xffffffffff000000
brk(0x1000000)                          = 0xffffffffff000000
brk(0x2000000)                          = 0xffffffffff000000
brk(0x3000000)                          = 0xffffffffff000000
brk(0x4000000)                          = 0xffffffffff000000
brk(0x5000000)                          = 0xffffffffff000000
brk(0x6000000)                          = 0xffffffffff000000
brk(0x7000000)                          = 0xffffffffff000000
brk(0x8000000)                          = 0xffffffffff000000
brk(0x9000000)                          = 0x9000000
brk(0xa000000)                          = 0xa000000

[...]

Notice that from 0xc0000000 upwards it continues expanding the program break above 0xc0000000 up to 0xff000000 where it also starts failing and returning the current break. Is this an expected behaviour when running an application in 64-bit machine in 32-bit compatibility mode? Is RLIMIT_AS and RLIMIT_DATA being honoured?

Thanks,

-- 
Ramon de C Valle / Red Hat Security Response Team
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/