Re: 2GIG-file

From: Donald Becker (
Date: Wed Aug 02 2000 - 02:14:12 EST

>Alan Cox wrote:
>> > Dear Al, Linus, and Alan -- 2GB liit is probably something to look at
>> > expanding in the future to support these huge SQL server database files.
>> Indeed. Thats why there are 2.2 patches and 2.4test already zapped it. I think
>> the limit is now about 1Tb for a file and hit due to block layer limits

To elaborate, for most LFS implementations:
  Block offsets are 32 bits.
  The block size is 2^10 or 2^12.
  Thus total file size is limited to 2^42 or 2^44.

>But not for everything. The 2.2.X VFS is limited to 2GB (since the fs
>drivers were coded this way). I treat the off_t field as an unsigned
>long so I'm at 4GB.

The LFS kernel patches for 2.2 change the file offset to 64 bits, and add
new syscalls to the kernels with 64 bit offsets.

The LFS patches I've seen (including Scyld's) don't change the block index,
which is still a 32 bit value. Changing the block index to 64 bits would be
a much bigger change. While many people will soon need 5GB files, very few
people need a over 1TB file.

The kernel was a small fraction of the LFS work. More effort went into
library changes, and the bulk of the time was spent testing (and
occasionally fixing) utilities and applications. I know that people on this
mailing list focus on kernel issues, just keep in mind that adding LFS is a
whole-system issue.

> 2.4.X does support big files, but this guy was
>asking about "commercial" linux distributions he could get his hands on,
>and most folks are still on 2.2.X.

We provide a distribution with LFS and commercial support. It includes a
kernel with LFS and 'bproc' (Beowulf cluster process management) support,
libraries with LFS support, and utilities modified and tested with both
sparse and dense large files on ext2.
It's not a low-cost, mass-market distribution that can be bought at the
corner computer store, but it is available.

>global across all the fs's. He has to get the data to the system
>somehow, and it's a good bet he will be using a network to do it. Which
>client can support copying of a 3.4GB file on Linux in the question--
>SMBFS since it's an NT server -- I don't think so? Since the VFS is
>limited to 2GB, I guess he can install large FS support, then stare at
>the system and wonder how to get the file

Part of the LFS enhancement is FTP client, FTP server, and 'scp' modication.
As you stated, NFS version 2 and Samba will not work with e.g. 5GB file, and
we don't yet support NFS version 3.

Donald Becker
Scyld Computing Corporation
410 Severn Ave. Suite 210 Beowulf Clusters / Linux Installations
Annapolis MD 21403

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
Please read the FAQ at

This archive was generated by hypermail 2b29 : Mon Aug 07 2000 - 21:00:07 EST