Albert D. Cahalan writes:
> Richard Gooch writes:
> > Albert D. Cahalan writes:
> >> Richard Gooch writes:
>
> >>> Ach! Not another DSM project :-( Don't do it. There's already a DSM
> >>> implementation for Linux,
> >>
> >> Obviously this is a feature people want.
> >
> > Too many people think they want it because they don't really
> > understand the alternatives and the implications.
>
> No way. Their situation is just different from yours.
No. My point is that while *some* people will be aware that their
threaded programme run on DSM will have to be re-coded to avoid lots
of lock movement, *most* authors of threaded programmes will be
unaware of this. DSM discourages people from thinking.
> Some people have existing software. They want to port it with
> minimal effort, since CPU time is cheaper than BRAIN time.
Why buy a 200 node cluster when you can get the same performance with
a 20 CPU SMP box?
> Some people want to prototype on normal clusters, then run the
> code on fancy hardware. If message passing is slower on the
> fancy hardware, then prototyping with messages is stupid.
Message passing can't be slower. Even with your automatic DMA
registers (from the private email you sent), the message passing call
can simply do a memcpy to the DMA memory area.
> >>> and besides, you're better off with a message-passing interface.
> >>> That way application coders can see how costly operations are.
> >>> Using DSM hides that, resulting in inefficient code.
> >>
> >> Message passing can be more costly! On the hardware I develop for, a
> >> "message" involves setting up some DMA control data. Distributed
> >> shared memory has a one-time setup cost, so it is faster for
> >> frequent access to small bits of data.
> >
> > Message passing must be more efficient than DSM. We are talking about
> > clusters, not a single SMP machine. The reason is that DSM *must* sit
> > on top of a transport layer (aka message passing).
>
> The world is not only clusters and SMP.
>
> Even if it was, you could prototype on a cheap cluster before you
> buy the 8-way SMP Alpha that you need for regular use.
>
> >> You could really mess up performance by using a message-passing API
> >> for repeated random access to 8-byte values. Actually, I think the
> >> break-even point is near 2 kB.
> >
> > If you mean within a single, (hardware) shared memory computer, then
> > yes. Otherwise, no. And DSM is all about pretending you have shared
> > memory across a network of computers.
>
> It all depends on how you define "shared" and "network" I suppose,
> but I'm certainly not using SMP hardware or TCP/IP. I can define
> physical address spaces that, when accessed, cause automatic data
> movement with hardware routing at hundreds of megabytes/second.
> (note BYTES not BITS) I think DSM would be perfect here.
Very nice. What's the latency, though? Can you ship a lock in << 100
ns? Is there a penalty for doing two back-to-back? If you write, is
the data broadcast to all CPUs/local memory pools? Is the MESI cache
protocol supported?
> Try to remember that not everyone has your hardware and needs.
> I've looked at the Linux DSM code, so I know you won't be hurt
> by it being in the kernel.
I'm not saying that DSM cannot ever be a useful thing. I'm sure that
there will be some applications that will scale fairly well, so you
could say that the reduction in brain power is worth the small
performance loss.
What I am saying is that the vast majority of people who jump up and
down saying that DSM will be great for their threaded application are
wrong. If they think their performance won't suck, it's more likely
than not that they haven't thought hard enough about it. The problem
is not that DSM will kill performance in the rest of the kernel (I'm
assuming it won't, otherwise it would never get past Linus, and would
be howled down by me, Larry and plenty of others, I'm sure). The
problem is that DSM is too seductive and easy to use, *when it's
almost certain it shouldn't be used*.
DSM should have a huge red health warning label telling people that
they really don't want it. If it goes into the kernel, it should ask
you 5 times if you really want to do this, and it should be
interactive every time you rebuild (none of this set once in a .config
file and forget about it). And it should randomly not compile 50% of
the time, requiring you to delete your kernel tree and unpack it
again.
Regards,
Richard....
Permanent: rgooch@atnf.csiro.au
Current: rgooch@ras.ucalgary.ca
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Tue Mar 07 2000 - 21:00:17 EST