Re: Overlay writable filesystem over r/o partition [long]

Trevor Johnson (trevor@jpj.net)
Tue, 6 Jan 1998 11:47:56 -0800 (PST)


Bryn Paul Arnold Jones asks:

> Hmm, wasn't ifs abandonded as too hard to do in kernelspace?

Here are some messages from earlier times it was discussed on linux-
kernel.
___
Trevor Johnson

Date: Mon, 11 Mar 96 12:09:01
From: Ray_Van_Tassle-CRV004@email.mot.com
To: jhenders@bogon.com
Cc: linux-kernel@vger.rutgers.edu
Subject: Union fs

>
> >
> > "union fs"??? What's a union fs?
> > One that costs too much and stops working at the drop of a hat, perhaps?
> >
> A union file system can be mounted over another read 0only file system.
> The files underneath show through until you modify one, at which time
> the midifyed file is written to the union fs. This allows you to, for
> instance, compile a kernel on a mounted cdrom drive, where the object
> files would be written onto the union fs over layed on it.

Ah-----This is how clearcase works (it's the source control utility we use
at work on a Sun).

Date: Thu, 14 Mar 96 15:33:27
From: kjahds@kjahds.com
To: linus.torvalds@cs.helsinki.fi
Cc: bpaj@gytha.demon.co.uk,jhenders@bogon.com, linux-kernel@vger.rutgers.edu
Subject: Re: ext2 in a dos file/partition

On Sat, 9 Mar 1996, Linus Torvalds wrote:

> While at USENIX, I saw the _correct_ way to do a union FS. It was done
> as a pre-loaded shared library, and because of that it was a lot more
> flexible than any kernel implementation would ever be (it was not
> limited to the unix mount global state: it was used more like Plan-9's
> dynamic per-process mounting).

You mean by (effectively) wrapping the libc file I/O functions? This seems
feasible, but I wonder about coherency and race conditions with each process
doing it's own "inheritance", and how the information on inheriting mounts
would be disseminated. (I suppose a deamon using IPC could resolve a lot
of this.)

> After having seen that, I don't think I necessarily would even want a
> kernel implementation. It simply was so much better done in user
> space..

If my guess is right, it sounds like it has it's own problems, but even so it
must be less painful then doing it in kernel space.

> The stuff had been done by Dave Korn (or Korn shell fame (and don't
> flame me if his first name isn't "Dave", my mind is going)) at AT&T.
> Damn, but I can't remember his email address..
>
> Linus

--
Kenneth Albanowski (kjahds@kjahds.com, CIS: 70705,126)

Date: Wed, 27 Nov 1996 23:15:28 +0000 (GMT) From: Alan Cox <alan@lxorguk.ukuu.org.uk> To: "Steven N. Hirsch" <shirsch@ibm.net> Cc: mjr@iki.fi, djc@microwave.com, kaos@edison.dialix.com.au, linux-kernel@vger.rutgers.edu Subject: Re: Union file system

> For any interested folks, there is a comprehensive discussion of Union > file systems in the book "Unix Internals" by Uresh Vahalia > (Prentice-Hall, 1996. ISBN 0-13-101908-2). This book is well worth > owning as a reference for some of the deeper technical topics that pop

A union FS btw is _horrible_ to do well. Consider a process with an mmap of a union FS object that is replaced by a file elsewhere and keeping the mmap coherency correct. Now truncate the file at the same instant and enjoy

Alan

Date: 27 Nov 1996 18:00:08 -0600 From: "Eric W. Biederman" <ebiederm@cse.unl.edu> To: Mikko Rauhala <mjr@iki.fi> Cc: "D. Chiodo" <djc@microwave.com>, Keith Owens <kaos@edison.dialix.com.au>, linux-kernel@vger.rutgers.edu Subject: Re: Union file system

>>>>> "Mikko" == Mikko J Rauhala <mjrauhal@cc.helsinki.fi> writes:

Mikko> On Wed, 27 Nov 1996, D. Chiodo wrote:

>> Symlinks? Mount all the cd, network, disk, etc, then make symlinks from >> whereyou want you everything to appear to wherever the appropriate files >> are...

Mikko> Won't cut it. We're talking about a system that would allow you to eg. Mikko> mount a CD-ROM drive and then mount another filesystem "over" that, so Mikko> that if you write to the file system, the CD-ROM files are hidden behind Mikko> the new files on the writable media. Would be rather handy in some cases...

The cyrus people messing with guile have a hacked nfs server that (unless the reporting is all wrong) will allow this capability. You might have to configure it some yourself though. There was a posting on linux-announce a while ago about this. The claim was that it could do everything the hurd could do only slower.

Eric Date: Wed, 27 Nov 1996 18:33:19 -0500 (EST) From: Greg Alexander <galexand@sietch.bloomington.in.us> To: linux-kernel@vger.rutgers.edu Subject: Re: Union file system

I guess no one else really knows about this: Inherited File System. Pitch in and help fix the bugs, I guess.. :) ftp://tsx-11.mit.edu/pub/linux/ALPHA/ifs/

Greg Alexander http://www.cia-g.com/~sietch/

Date: Wed, 27 Nov 1996 14:25:00 -0800 (PST) From: Michael Neuffer <neuffer@nomis.i-connect.net> To: Alan Cox <alan@lxorguk.ukuu.org.uk> Cc: mjr@iki.fi, djc@microwave.com, kaos@edison.dialix.com.au, linux-kernel@vger.rutgers.edu Subject: Re: Union file system

On Wed, 27 Nov 1996, Alan Cox wrote:

> > > Won't cut it. We're talking about a system that would allow you to eg. > > mount a CD-ROM drive and then mount another filesystem "over" that, so > > that if you write to the file system, the CD-ROM files are hidden behind > > the new files on the writable media. Would be rather handy in some cases... > > Sounds like IFS. Im not sure what happened to IFS however.

The original author stopped maintaining it way back after it didn't make it into the kernel.

Mike

Michael Neuffer i-Connect.Net, a Division of iConnect Corp. mike@i-Connect.Net 14355 SW Allen Blvd., Suite 140 503.641.8774 Beaverton, OR 97005

Date: 28 Nov 1996 00:31:05 +0100 From: Matthias Urlichs <smurf@noris.de> To: linux-kernel@vger.rutgers.edu Subject: Re: Union file system

In linux.dev.kernel, article <m0vSnkq-0004omC@pcpaul.wau.mis.ah.nl>, paul@wau.mis.ah.nl (Paul Slootman) writes: > > When I tried Yggdrasil's stuff a couple of years back, that distribution > has something called IFS ("Werner's Inherited File System", IIRC). This > > I recall that it worked quite well. Anyone know why it didn't get into > the mainstream kernel? > Because it was a total hack (just look at the source code), because it had unresolved bugs (like failing to free inodes under some unfathomable circumstances, which meant you were unable to unmount the file systems you used as source for your IFS mount at last startup), and because Linus thinks, quite rightly IMHO, that anything you can do with such a beast in the kernel, you can do even better with a preloaded shared library.

--
Competitive fury is not always anger.  It is the true missionary's
courage and zeal in facing the possibility that one's best may not be enough.
               --Gene Scott
--
Matthias Urlichs         \  noris network GmbH  /  Xlink-POP Nürnberg
Schleiermacherstraße 12   \   Linux+Internet   /   EMail: urlichs@noris.de
90491 Nürnberg (Germany)   \    Consulting+Programming+Networking+etc'ing
   PGP: 1024/4F578875   1B 89 E2 1C 43 EA 80 44  15 D2 29 CF C6 C7 E0 DE
       Click <A HREF="http://info.noris.de/~smurf/finger">here</A>.    42

Date: Fri, 29 Nov 1996 15:12:56 +0100 (MET) From: Werner Almesberger <almesber@lrc.epfl.ch> To: Alan Cox <alan@lxorguk.ukuu.org.uk> Cc: linux-kernel@vger.rutgers.edu Subject: Re: Union file system

Alan Cox wrote: > Sounds like IFS. Im not sure what happened to IFS however.

Well, I completed it to the point where it was a nice proof of concept, but still with problems (leaks inodes, probably has a few races left, was also a bit too liberal with locking, etc.).

Then I looked back at what I did and was disgusted by its complexity. So I decided that, before I might even consider proposing inclusion into the mainstream kernel, I'd have to see how much poorer (performance-wise) a user-space implementation would be. I did some initial hacking on NFS until I convinced myself that userfs might be the better approach. Unfortunately, I never found the time to work on that.

I also noticed that amd should have some functionality that provides at least a subset of what IFS could do (you can quite easily apply the concept of IFS to caching, CD-ROM "writing", storage management, etc.), although I never actually tried that part.

By the way, IFS (Inheriting File System) was indeed inspired by Sun's TFS (Translucent File System). The main difference between the two is that IFS allows the stack of file systems to be arbitrarily large, while TFS can only stack two file systems.

If I had time to work on it today, I'd probably go for a mixed kernel/user space approach: do all the bookkeeping and special cases (device files, named pipes, etc.) in user space, but keep data transfers in the kernel. I'd also investigate into the direction of COW semantics for disk blocks in order to reduce the cost of changing a few bytes of or appending to a "read-only" file.

Sounds like a very nice master's thesis topic for some good Linux hacker ;-)

- Werner

--
  _________________________________________________________________________
 / Werner Almesberger, DI-LRC,EPFL,CH   werner.almesberger@lrc.di.epfl.ch /
/_IN_R_133__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/

Date: Thu, 28 Nov 1996 13:07:58 +1100 (EST) From: Kevin Lentin <kevinl@cs.monash.edu.au> Reply-To: Kevin Lentin <K.Lentin@cs.monash.edu.au> To: Eric Hoeltzel <eric@dogbert.sitewerks.com> Cc: djc@microwave.com, kaos@edison.dialix.com.au, linux-kernel@vger.rutgers.edu Subject: Re: Union file system

Eric Hoeltzel Wrote ... > > I heard something about a "translucent" file system once. Like you > could mount a cdrom and a hard disk partition at the same mount > point. Then when you untar a file on the cdrom it is written to > the disk partition. I suppose there would be other uses too. Was > this fantasy or is this actually being done somewhere?

The automounter program 'amd' has the ability to do this using NFS in cute ways. I tried once to make it work but for the life of me could not understand the config file.

'amd' allows you to automount multiple directories at the same place. I think it writes to the last one listed and reads in reverse order. Or you can configure it. It will also allow you to mount a filesystem over part of another one.

--
[======================================================================]
[     Kevin Lentin               Email: K.Lentin@cs.monash.edu.au      ]
[   finger kevinl@fangorn.cs.monash.edu.au for PGP public key block.   ]
[  KeyId: 06808EED    FingerPrint: 6024308DE1F84314  811B511DBA6FD596  ]
[======================================================================]

Date: Tue, 3 Dec 1996 14:11:49 -0500 (EST) From: "Derrick J. Brashear" <shadow@dementia.org> To: linux-kernel@vger.rutgers.edu Subject: Re: Union file system

> > On Thu, 28 Nov 1996, Matthias Urlichs wrote: > > A loopback mount is far easier than a union file system, and a big > > necessity if you need to set up multiple chrooted environments. > > > > And don't tell me to use a NFS back-mount. NFS is not a solution, NFS is a > > problem in and of itself. (Did anybody check the speed of nfs vs smbfs?) > > Hey, > > I don't really understand why loopback mounts don't exist in Linux then. > It really doesn't sound too hard to implement for someone who knows the fs > code of Linux fairly well. > Any volunteers? Loopback filesystems are great.......

I agree. So does the person who wrote one: ftp://dot.superaje.com/pub/linux has one

-D