Hi,
On Mon, 06 Mar 2000 11:36:06 -0800, Aman Singla
<aman@cthulhu.engr.sgi.com> said:
> Both add that additional level of locality that a performance
> conscious application needs to exploit; agreed that KSR was the only
> one to have caching at the remote memory to local memory level (with a
> page size granularity).
Yes, and the granularity was a _big_ problem.
> A DSM system could look very much like the current NUMA h/w with an
> added remote-local memory caching happening at application specific
> granularity. The main issue there is getting access control at the
> desired granularity for the s/w DSM layer
Not just that. On SMP, and even more on NUMA, you only get scalable
performance by going out of your way to avoid unnecessary sharing of
data. It's just not transparent. And when you _do_ want to pass data
from X to Y, you only want to do it once, which requires extra
synchronisation to make sure that the receiver of the data doesn't start
peeking at the data structures until the data is known to arrive.
With message passing, that synchronisation is implicit. With shared
memory, about all you can do is spin waiting for the data to arrive,
unless you add a separate synchronisation mechanism somewhere to do
locking outside the shared memory channel.
It's not just granularity. The synchronisation hit is bad, on _all_
forms of DSM up to and including SMP. (The latest Linux development
kernel includes "big reader" locks which allow a read-mostly data
structure to be locked on each CPU separately with no inter-CPU memory
traffic at all, simply because even on such tightly coupled systems the
cost of passing even one cache line of shared memory traffic between
CPUs is too high.)
--Stephen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Tue Mar 07 2000 - 21:00:22 EST