Re: [PATCH] fs: limit maximum concurrent coredumps

[KOSAKI Motohiro]

> A moderately large number of large core files being dumped simultaneously
> can impose severe latency penalties on other processes due to IO load.
> 
> For example, a common configuration for PHP web-servers includes apache's
> prefork MPM, mod_php and a PHP opcode cache utilizing shared memory. In
> certain failure modes, all requests serviced by PHP result in a segfault.
> Enabling coredumps might lead to 10-20 coredumps per second, all attempting
> to write a 150-200MB core file. This leads to the whole system becoming
> entirely unresponsive for many minutes.
> 
> The ability to limit concurrent coredumps allows dumping core to be safely
> enabled in these situations without affecting responsiveness of the system
> as a whole.
> 
> I have several servers running with this patch applied (actually backported
> to v2.6.26) and it has allowed me to deal successfully with the situation
> described above.
> 
> In do_coredump I have pulled dump_count back out to the top-level scope
> and core_dump_count is now incremented in the normal path.
> 
> Added sysctl parameter for tuning core_max_concurrency.
> 
> checkpatch complains about "extern ... core_max_concurrency" but this seems
> to be an acceptable exception based on the preponderance of other extern
> variables in kernel/sysctl.c

To be honest, My customers have very similar problem for long time. If 
HPC MPI job or apache prefork MPM processces die, The system load to 
make large concurrent core dump I/O and consume all I/O band width. 
Eventually, all other service stall long time and HA cluster software
shutdown such node forcely.

My case used two technique, 1) limit concurrent coredump (but not skip
as sugessted Roland) 2) reduce I/O priority automatically in do_coredump().
I don't know my way was correct way. but I believe we need something case.



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/