Re: Inter-Kernel Communications (Multi Kernel Clusters)
Mon, 24 Feb 97 17:17:39 +1000


>** network memory space **

>1. Upon startup of the first kernel on the network it allocates network
>address space for its physical and swap space, I.e. a machine with 64MB
>ram and 36MB swap would (by udp broadcast etc.) decide to map it's 100mb
>of physical (existent) memory space to the first 100MB of virtual
>addresses, the machine that starts second etc. etc. {using some kind of
>token protocol} would get the second 100MB of virtual addresses. So our
>first effect is we end up with what I will call NETWORK MEMORY SPACE, If
>there are three machines with 100MB each of memory, then each machine will
>be able to address 300MB of memory (100MB of it local the rest remote),
>but this would all operate synonymous to local memory from a user
>standpoint. This is the first step in mounting kernels together in a
>CLUSTER. Of course some type of memory locking would be necessary to make
>this work.

The problem with this is that you can have 2 machines start at the same time when the network is down. They both use the same address space and stuff everything up.

>** network device space **

>2. Each kernel on this three kernel network would deal directly only with
>its physical memory and devices and processes (which is really all I can
>this of that the kernel primarily does). So then, very much like the
>NETWORK MEMORY SPACE we would also need to create a NETWORK DEVICE SPACE,
>so that every machine on the network could see every other machines
>devices. What this really means also is one big network file space. Three
>machines with 3 drives all on one directory tree --- perhaps even with
>some kind of mirroring for redundancy. This really however is not the
>result of some half-ass solution like nfs (I am talking about nfs here not
>the people who created it, no offense).

The mirroring is already written. It should be released in a few weeks. I won't mention the name of the person who's about to release it as he's very busy and doesn't need any more Email.

>** network process space **

>3. Very much like the above there would also be a NETWORK PROCESS SPACE,
>where the heavily loaded machines could offload processes onto idle or
>lightly loaded machines.

Why not have compute servers, disk servers, etc like Amoeba (it's something to consider at least)?

To do this properly you would need to have /dev/* refer to devices on the machine the process started on, and have /localdev/* have the files that are usually in /dev, and have /remotedev/machinename/* refer to remote devices. /dev/* could be some procfs like file system to support this.

>I think it would be really daring for some of us kernel hackers to try
>building a facility like this --- certainly we would be leaving every
>other modern operating system with the taste of dust on their tongues and
>it could be done in such a way as to appear no different from the user
>side --- the three machines become like one machine.


>** the shortcomings **

>This solution would make muliple networked machines like one with more
>memory, devices, and processors --- it still falls short of bonding
>together three processors to look like one really fast one, however you
>wouldn't want to do that over a network anyway unless it was a very fast
>dedicated fiber optic one. In my scenario, cpu hungry processes would
>automatically migrate to the most powerful system on the network. This
>solution also assumes an environment where the like between the computers
>is not a security problem..

If all machines have the same root password and use SSL encryption to transfer data then this shouldn't be a problem.

>Feedback would be much appreciated, I would especially like to hear from
>anyone out there who is interested in such a thing. On major issue would
>be device name space as major and minor numbers are assumed to be
>relatively static.

/remotedev/machine/* would just be named pipes or something (not real devices).

/localdev/* would be normal device files
/dev/* would be sym-links.

I don't see the problem with this.

If you're serious about this then the thing to do is write a web page, arrange a mailing list, and make an entry in the LPM. If you have problems with any of these then let me know and I'll help.

Russell Coker