I thought I would throw my own $.02 into this ring. A few comments....
Process migration, as in the case of MOSIX which is the basis for my
experience, is inherently difficult to maintain when it is supported by a
party other than the system vendor. When it is supported by the system
vendor however, I contend that the cost of support pales in comparison to
the resources required to support the OS as a whole. As a proof I offer
anyone to check the staff hours expended on process scheduling or VMM, as a
kernel subsystem that is on par with migration support, for any major OS as
a percentage of the hours expended on the OS as a whole.
This cost rises almost exponentially, the moment the support for migration
is provided by a third party. The reasons for this are similar to those
for defect correction - once released, it requires a different scale of
effort to integrate an additional module in a seamless manner than to
design it correctly in.
>I don't know that migration impacts performance. As far as reliability goes,
>you can *increase* reliability if you can arrange to migrate off of machines
>that will be taken down for some reason, so there's at least one advantage
to
>go along with whatever disadvantages you might find.
There are living proofs that migration does not significantly impact
performance for a single process, and improves the system throughput and
performance as a whole.
>> So here's a question: if it worked and was useful, why has that facility
>> not made it into any sucessful commercial or free operating systems? Was
>> it ahead of its time or is there some inherent reason?
>
>I have long expected someone to implement it in a system like
>Linux. Takers? In any case, Ken Shirriff's paper in USENIX'97 on
Solaris-MC
>said they provided remote execution at first but expected to support
migration
>for the above reason (offloading nodes before shutdown). I'm copying him
here
>and perhaps he can give us an update :-).
I think one of the major reasons has been the business case. One of the
most illiuminating moments I have ever had in that respect, was hearing IBM
management discuss what their priorities were for the releases one year
out. To those folks AIX was the grease that moved their iron. Every AIX
development was translated into devices and system sales. Items which were
technically extremely desireable but could not show a clear market share
case just did not make the cut.
Process migration does not yet sell systems. In the past, these systems
were either servers, which everyone did not want to be bogged down by
outside processes, or personal workstations, which has a user culture
issue. Note the success or lack thereof of pmake.
>> Past experience has shown that process migration is costly in terms
>> of code required to make it work, time required to do it, and cache
>> utilization (both processor and file system).
I agree that the cost is non trivial. I just don't think that this negates
any possible benefits. Have you looked at any of the results we have
achieved for MOSIX?
>> Since I despise nay-sayers that don't bother to offer a better answer,
>> here's my better answer: cluster small SMP machines. Allow full blown
>> "migration" from one CPU to another CPU on an SMP, but not across machine
>> boundries. This gives some degree of dynamic load balancing, which is
>> the second order throughput term. Use static load balancing at exec()
>> time to get the first order term.
Commonly known as Static placement, it has been shown that dynamic
preemptive migration can provide superior performance and system throughput.
>> I happen to believe that such a system would keep up with, and in some
>> cases outperform, large SMP systems. I have a fair amount of real world
>> experience from Sun and SGI SMP systems that suggests that this approach
>> is better.
Well, thats one take on your experience but my experience has shown me that
SMP scalability is limited and costly to achieve, MPP systems are plain
costly. If you can support migration, then I can benefit from the SMP
support and balance between nodes.
>> Let me say that again. A company with SGI's resources, may full time
>> experienced engineers that would stack up with the best the research
>> community has to offer, couldn't get page migration to work. By "work"
>> I mean come up with a self tuning policy that results in better throughput
>> than just allocating the pages and leaving them where they were allocated
>> and leaving the process near them. /All/ attempts to improve performance
>> by using migration resulted in lower system throughput. The cost of moving
>> the process context and the pages outweighed any performance benefit.
>> I challenge the
>> minds out there to come up with a migration policy that actually
>> improves performance under any realistic workload, not some toy
>> benchmark. Show me a process migration system that makes TPC-C
>> run better. Or fortran jobs. Or make. Or web. Or NFS. Anything
>> that customers will pay money to get.
Have you actually looked into any existing results for such systems?
On MOSIX, runnin a collection of "real apps" that performed diverse
functionalities such as image processing, fluid dynamics analysis, chemical
molecular modeling, etc.. we achieved near linear speedups for a large set
of these. The results were published in a technical report.
>> The point is that yes, you can do it. But that is an academic point,
>> not anything that is actually useful. Not in my experience. I'm happy
>> to be proven wrong, but I'm unhappy with letting people go down a
>> proven rathole.
The chemistry department became avid users of MOSIX once they were shown
the ease with which their applications could be ported and the processing
power of the system harnessed.
-- Shai
============================================================================
Shai Guday |
Director of Commercial Systems Technologies | Voice: (617) 621-8911
The Open Group | Fax : (617) 621-8696
Eleven Cambridge Center | Email: s.guday@opengroup.org
Cambridge, MA 02142 | www.opengroup.org/~shaig/
============================================================================