Re: huge mem mmap eats all CPU when multiple processes

From: KAMEZAWA Hiroyuki
Date: Mon Jun 08 2009 - 20:43:00 EST


On Mon, 8 Jun 2009 10:27:49 -0400
Matthew Von Maszewski <matthew@xxxxxxxxxxx> wrote:

> [note: not on kernel mailing list, please cc author]
>
> Symptom: 9 processes mmap same 2 Gig memory section for a shared C
> heap (lots of random access). All process begin extreme CPU load in
> top.
>
> - Same code works well when only single process access huge mem.
Does this "huge mem" means HugeTLB(2M/4Mbytes) pages ?

> - Code works well with standard vm based mmap file and 9 processes.
>

What is sys/user ratio in top ? Almost all cpus are used by "sys" ?

> Environment:
>
> - Intel x86_64: Dual core Xeon with hyperthreading (4 logical
> processors)
> - 6 Gig ram, 2.5G allocated to huge mem
by boot option ?

> - tried with kernels 2.6.29.4 and 2.6.30-rc8
> - following mmap() call has base address as NULL on first process,
> then returned address passed to subsequent processes (not threads,
> processes)
>
> m_MemSize=((m_MemSize/(2048*1024))+1)*2048*1024;
> m_BaseAddr=mmap(m_File->GetFixedBase(), m_MemSize,
> (PROT_READ | PROT_WRITE),
> MAP_SHARED, m_File->GetFileId(), m_Offset);
>
>
> I am not a kernel hacker so I have not attempted to debug. Will be
> able to spend time on a sample program for sharing later today or
> tomorrow. Sending this note now in case this is already known.
>

IIUC, all page faults to hugetlb are serialized by system's mutex. Then, touching
in parallel doesn't do fast job..
Then, I wonder touching all necessary maps by one thread is good, in general.



> Don't suppose this is as simple as a Copy-On-Write flag being set wrong?
>
I don't think, so.

> Please send notes as to things I need to capture to better describe
> this bug. Happy to do the work.
>
Add cc to linux-mm.

Thanks,
-Kame


> Thanks,
> Matthew
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/