lm@bitmover.com (Larry McVoy) wrote:
>: The items listed under "BAD" are more powerful that the "GOOD" ones:
>
>Agreed. it does not follow from that, however, that they should be
>implemented. One could implement "tar" in the kernel and that would
>be a more powerful thing than open/close/read/write. It would also
>be a bad idea.
Your example is not suitable for the situation under discussion. However, if
applications used an open-read-write-close sequence _most_ of the time, then
I'd say we could consider adding a single system call to do the work. This
will reduce the number of system calls performed, and also reduce the
possiblity of a bug due to an error in coding the above sequence.
>: *) remote fork() handles the run-time state of a process, while remote
>: exec() does not bother with that. Please keep in mind that the application
>: programmer will have a hard time duplicating the run-time state of a process
>: without support from the operating system.
>
>Before you go about buggering up the kernel with a bunch of things that
>sound good on marketing slides, perhaps you should ask yourself where
>the applications are that need this service [...]
I can't see which part of my technical comments had to do anything with
"marketing"
> [...] Let's try that again: what
>applications need rfork() and can't make do with rexec()? [...]
Most of those programs that use an ordinary fork(), especially if the forked
processes will exist for a long time.
> [...] You might want
>consider that you are taliking to the guy that started 100% of Sun's
>clustering work, and spent 2 years pretty much 7 days a week working on
>these problems, looking at customer applications, sorting through the
>marketing hype versus real issues. I'm very pro clustering. I am not
>in favor of adding complexity where there is no point.
I am sure you are a very experienced and knowledgable person.
>: *) process migration is more flexible and more transparent than an explicit
>: checkpoint / restart mechanism. dynamic load balancing and process migration
>: should come together. The combination of these two eases the optimum use
>: of the available computing resources, without bothering the programmer.
>
>Yeah, and Java is going to save the world, threads are the greatest thing
>since sliced bread, and OSI is coming back. Please step back and consider
>that the things that have survived the test of time are small, simple
>concepts. Just because it is possible to do something does not mean it
>should be done.
Simple for who?? The application programmer or the kernel programmer? I say
let us make things simple for the application programmer. Transparency will
ensure that the size of a feature used by application programmer will be
zero (because they don't see it).
>: After all, the operating system is the only entity that can be aware of the
>: current resource-usage situation in different computers. An application
>: program(mer) can not know this entire state. More important, it can not
>: in general predict the future, so in a static load balancing environment
>: the reaction to changes in resource-usage will not be satisfactory.
>
>Prove it. Show me a real parallel application that will not run
>perfectly when statically load balanced over a bunch of machines (2 or
>4 way SMP boxes) and dynamically load balanced within those machines.
>When you start looking, you start learning about gang scheduling.
>Once you know about that, this whole idea os process migration becomes
>not just unnecessary, it becomes negative.
Just consider a group of people with heavy scientific applications. They
can start their long-running programs anytime and on any machine. A lot of
these people will need hundreds or even thousands of processors to run their
applications in parallel. It is good to have SMP machines in a cluster, but
they are not the answer.
>: What I want to say is that as an all-knowking authority, the operating
>: system is in a good position to use mechanisms like remote fork() and dynamic
>: load balancing for the good of _all_ the running application programs.
>
>See my comment about gang scheduling. Without it, you get nowhere. With it,
>process migration is a horrible idea. Checkpoint/restart is fine and one
>could use it as a coarse level process migration.
Yes. But I believe this should be done by the OS.
>: This will also need minimal recoding: The prgrammer will always call a
>: fork(), and the OS will decide if the system call will be executed
>: locally or remotely.
>
>You really need to think this through. What about /tmp? Are you going to
>have one global /tmp and /usr/tmp? That's a huge performance lose. If not,
>when I fork() and land on another machine and expect my tmp files, where are
>they?
Yes, a distributed file system. Don't worry too much about the network
speeds. They currently are faster that the CPUs in most computers, and
their speed is increasing at a fatser rate that the CPUs. We better make
use of this.
>: All the "GOOD" mechanisms,
>: on the other hand, will ultimately require some work on the part of the
>: application programmers.
>
>Not true at all. login, make, xinit all need to be taught that they are
>in a cluster. Other applications don't know and don't need to know.
Does "Not true at all" mean that you claim the programmer will have no need
to learn _any_ new programming method after modifying login, make and xinit?
then I invite you to please implement your ideas ASAP. I'd really be happy
to use them.
>: Yes, implementing this won't be easy, but it will be done only once, and
>: then a large number of application programmers can use them.
>
>Where exactly are those programmers and what exactly are the applications?
As I said before, many of the current applications.
>: The chicken-egg problem again :-) Many people will not start using clusters
>: while programming them is difficult! What you suggest might ensure that this
>: will remain so. I am sure you agree that we should better "change" the
>: situation rather than "cope" with it.
>
>No, I do not agree. Bad ideas should never be put in the kernel. That's
>why Linux has clone() and not kernel threads.
That's why one sees many more threading applications under Windows NT than
under Linux.
>: > [...] People who
>: >haven't learned this lesson have repeatedly built overly complex and
>: >expensive clusters. Let's not do that to Linux.
>:
>: Linux uses a monolithic kernel. I myself don't think this is a good thing,
>: but we should make our efforts compatible with this design. Adding to the
>: kernel is not a very bad thing here.
>
>Disagree.
Please take a look at DIPC's sources to see how one can add a few kilobytes
to the kernel and bring a lot of transparent and modern functionality to
Linux. DIPC offers its services _via_ the kernel. This is probably different
from what you have in mind about "adding to the kernel"
DIPC's web pages are at http://wallybox.cei.net/dipc
>: In short: It might be better to add a few thousand lines of code to the
>: kernel and be spared the troubles of adding some few hundred lined of code
>: to _many_ application programs. I hope the net effect to be less lines of
>: code and much less bugs.
>
>Look, doing a clustering OS the simple way is going to be more than 100K
>lines of code. Doing it the way you are discribing is an order of
>magnitude worse.
Please refer to the design of DIPC. People who do not need the services of
DIPC at a certain time can stop a "user-space" program called dipcd. After
that, all the penalty of having DIPC is the addition of a few KBs in the
kernel. I consider that a very good price.
>: I hope Mr. Linus Torvalds will have a positive attitude to such changes.
>
>He will, he does, his attitude, however, is much more in line with my
>thinking - we've discussed this issue repeatedly over the years.
Maybe DIPC's approach can be more compatible with yours??
-Kamran Karimi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu