> We've been trying to run web stats on especially large log files (some
> totalling 8gig in size), and have been noticing that the stats program
> seg faults at roughly 900meg in resident size. After performing some
> tests, I found was able to allocate only 882 meg before a seg fault
> occured (1gig physical ram, and 4gig swap.. the swap was never used in
> the test). Running the test across two processes i was able to
> allocate 500meg each with the swap then being used (the test was just
> to see if the system would go to swap, so I never tried it higher).
> On another system, with 2gig of memory, I was only able to allocate
> roughly 540meg before segfaulting (more loaded system, but approx
> 1.3gig was free before running the test)...
stupid question: are you running 2.3 kernels and have you enabled High
Memory Support? I'm able to run 1.0 GB RSS processes:
20 0 1000M 1.0G 232 R 1.0G 99.9 26.7 1:08 allocmem
1.5 GB ones:
20 0 1500M 1.5G 232 R 1.5G 99.9 40.0 1:14 allocmem1.5g
2.0 GB ones:
20 0 2000M 2.0G 232 R 2.0G 97.5 53.4 1:05 allocmem2.0g
.. just fine. With the current default virtual memory split of user-space
(3GB total virtual memory for user space - 1GB of that is shared
libraries, 2GB heap and stack) 2GB is the limit per process - this can be
upped to nearly 3GB by lowering processor.h's TASK_UNMAPPED_BASE to eg.
128 MB. This will give 2.9GB freely allocatable (mallocable) memory to a
single process.
> I'm assuming that this is a known issue .. or feature .. or setting ..
> or ... ?? Is there anyway that a single process can allocate all
> available memory?
it should work ... one way to debug such problems is to run the process
through gdb (or attach gdb to it while it's running), then when the
SIGSEGV arrives take a look at /proc/<pid>/maps, and you'll see which area
has overflown.
> (with the new 'high-memory' feature of 2.3 and >2g file sizes, we'd
> like to move our web stats generation over from an irix box to our
> linux machines .. log files totalling 8gig in size and stats utilizing
> at least a gig of memory when generating)... Now that we got the 8gig
> file over .... we need to process it :)
it should be ok for up to 2GB RSS in later 2.3 kernels.
-- mingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/