Your technical comments aren't very technical. In my experience, it is well
intentioned people just like you that cause creeping featurism. I'd much
rather see you use those good intentions to keep stuff out of the kernel
rather than shovel stuff into the kernel.
Getting back to your comments. The reason I think they aren't that technical
is that you are arguing for a bunch of features which /sound/ great. They
are not necessary or useful in practice and that is something that only
experience will prove. As someone who has that experience (and has payed
close attention to the experience of others working on these problems), I
am trying to let you know that what you are asking for is not a good idea.
It's not a marginal thing like it might be a good idea, it's an absolute
thing: it's not a good idea.
: > [...] Let's try that again: what
: >applications need rfork() and can't make do with rexec()? [...]
:
: Most of those programs that use an ordinary fork(), especially if the forked
: processes will exist for a long time.
Again, please be more specific. What you are doing is called "armchair
architecture". You /think/ that these applications exist and would use
this facility. I used to think just like you until I went out and waded
through the real applications that can and do make use of clusters. They
are nothing like what you are imagining.
: > Please step back and consider
: >that the things that have survived the test of time are small, simple
: >concepts. Just because it is possible to do something does not mean it
: >should be done.
:
: Simple for who?? The application programmer or the kernel programmer? I say
: let us make things simple for the application programmer.
Wrong answer. Complexity in the base of the system is a bad idea. There are
good reasons that Unix has a "simple" design.
Furthermore, the "application programmers" that you are tlaking about are
more sophisticated than most kernel programmers. They neither need nor
want the features you are asking for. Go spend some time at the national
labs (any national labs, I don't care what country) and listen and learn
what they do. They would be completely uninterested in what you are
proposing, they already know it doesn't work.
: >When you start looking, you start learning about gang scheduling.
: >Once you know about that, this whole idea of process migration becomes
: >not just unnecessary, it becomes negative.
:
: Just consider a group of people with heavy scientific applications. They
: can start their long-running programs anytime and on any machine. A lot of
: these people will need hundreds or even thousands of processors to run their
: applications in parallel. It is good to have SMP machines in a cluster, but
: they are not the answer.
Funny you should mention these applications. Have you ever talked to one
of the people that work on systems like this? Have you ever spent time
understanding the basic issues? How do you propose to have 2 applications
share a cluster at the same time without thrashing? There's only one
answer and it is gang scheduling. Think about gang scheduling for a
while and then try and explain to me how process migration is going to
not completely screw up the scheduling.
Maybe it's just me, but I have a real problem with you trying to tell the
kernel list what ought to be done when it is so obvious that you are not
speaking from experience.
: >You really need to think this through. What about /tmp? Are you going to
: >have one global /tmp and /usr/tmp? That's a huge performance lose. If not,
: >when I fork() and land on another machine and expect my tmp files, where are
: >they?
:
: Yes, a distributed file system. Don't worry too much about the network
: speeds. They currently are faster that the CPUs in most computers, and
: their speed is increasing at a fatser rate that the CPUs. We better make
: use of this.
Another great example of no experience. Go try this and get back to me on how
it works. What you are saying doesn't work at all in practice. It's all
theory and hand waving.
: >Not true at all. login, make, xinit all need to be taught that they are
: >in a cluster. Other applications don't know and don't need to know.
:
: Does "Not true at all" mean that you claim the programmer will have no need
: to learn _any_ new programming method after modifying login, make and xinit?
: then I invite you to please implement your ideas ASAP. I'd really be happy
: to use them.
All of the existing applications do not need to be taught anything.
They are statically load balanced by the startup prgrams I mentioned.
That's one class of application - they should know nothing about the
cluster and things should just work. Note that we need to do none of
the things you promote to make those apps work.
The other class of apps, a /much/ smaller class, are the scientific apps.
Those apps are always hand tuned to the machine. They have to be in
order to make it work efficiently. This always involves gang scheduling,
for example.
: >: Yes, implementing this won't be easy, but it will be done only once, and
: >: then a large number of application programmers can use them.
: >
: >Where exactly are those programmers and what exactly are the applications?
:
: As I said before, many of the current applications.
And as I have said, you don't need to do squat to those in order to have
them work on a cluster.
: >No, I do not agree. Bad ideas should never be put in the kernel. That's
: >why Linux has clone() and not kernel threads.
:
: That's why one sees many more threading applications under Windows NT than
: under Linux.
You don't think it could have anything to do with the thousands of paid
programmers that work on NT apps vs the unpaid programmers that work
o Linux?
: Please refer to the design of DIPC. People who do not need the services of
: DIPC at a certain time can stop a "user-space" program called dipcd. After
: that, all the penalty of having DIPC is the addition of a few KBs in the
: kernel. I consider that a very good price.
DIPC is tiny by comparision to what you are proposing. Let's just
take process migration. Suppose you spend an hour walking through
the kernel and come back with a list of places that you will need to
change to make process migration work. Please consider at least the
following: open sockets, open files, CWD, signals, process relationships,
process namespace, shared memory. I really think it would be very
useful for you to do this /before/ you tell the kernel community what
they should be doing. Better yet, go implement waht you are proposing.
Maybe I'm dead wrong and you can indeed do it in a very small, nice way.
That would be great.
--- Larry McVoy lm@bitmover.com http://www.bitmover.com/lm- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu