On 30 Jul 2000, Linus Torvalds wrote:
> >> Talking with an internal engineer here -- NUMA uses MESI
> >> cache control similarly to the i386 cache coherency model,
> >> so a write of one CPU to a an area of memory will be seen
> >> on other processors as soon as they ask for that memory.
> >
> >That's specific to MIPS. A general Linux NUMA model probably won't assume
> >coherence.
>
> I disagree.
>
> If it's not ccNUMA, then the different nodes should run _different_
> copies of the OS, and you'd basically have a multi-OS system rather than
> single image. Non-cache-coherent hardware simply is not worth handling
> as a single image, I suspect.
I actually agree with the non-cc -> cluster idea. The question is whether
we should treat large cc machines largely as clusters as well. Cache
synchronization is expensive and partitioning might just be a performance
boost.
What does it mean to treat a NUMA machine as a cluster for the kernel?
Depending on the memory/disk bandwidth ratio it may continue to make sense
to have a shared page cache of some sort. Most Beowulfs already run with
NFS, which is effectively a shared cache.. Trade ethernet for transparent
memory bus, divide latency by 10..
> Most current/modern NUMA designs are ccNUMA anyway, as far as I can
> tell. I don't know of anybody who seriously makes a non-cc machine any
> more. People know how to make efficient cache controllers, and it just
> makes the machines _soo_ much more usable that anybody who isn't cache
> coherent really deserves to be ridiculed.
I think you'll find that the really big machines (ASCI, etc) are
non-coherent. Coherence is expensive hardware-wise and the programming
model it's designed to support just doesn't scale well. People don't
ridicule Beowulfs and it's only a matter of time before commodity systems
have competitive memory bandwidth with the SGIs and DECs of the world.
-- "Love the dolphins," she advised him. "Write by W.A.S.T.E.."- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Mon Jul 31 2000 - 21:00:35 EST