I have heard that the Linux NFS client doesn't have the best reputation.
However, I really couldn't believe the following results I got.
The following program writes 8,192 bytes, 1 byte at a time.
#include <fcntl.h>
void main(int ac, char *av[])
{
int fd = open(av[1], O_WRONLY|O_CREAT, 0777);
int i;
for (i = 0; i < 8192; i++) {
write(fd, "0", 1);
}
close(fd);
}
>From two machines with an identical LAN network connection to a SunOS 5.6
server, the program finishes instantaneously on a FreeBSD 3.0 machine:
time ./tw.f3 A1
0.008u 0.091s 0:00.10 90.0% 5+176k 0+1io 1pf+0w
On a Linux 2.2.3/RH 5.2 machine, however, it takes
time ./tw.linux B1
0.000u 0.840s 0:51.16 1.6% 0+0k 0+0io 64pf+0w
51 seconds!
Please tell me that I'm doing something wrong here.
Here's an excerpt from the tcpdump:
23:50:34.178183 peerless.cs.utah.edu.3079405568 > envy.cs.utah.edu.nfs: 172 writ
e [|nfs]
23:50:34.183063 envy.cs.utah.edu.nfs > peerless.cs.utah.edu.3079405568: reply ok
96 write [|nfs] (DF)
23:50:34.183136 peerless.cs.utah.edu.3096182784 > envy.cs.utah.edu.nfs: 152 geta
ttr [|nfs]
23:50:34.183647 envy.cs.utah.edu.nfs > peerless.cs.utah.edu.3096182784: reply ok
96 getattr [|nfs] (DF)
23:50:34.183719 peerless.cs.utah.edu.3112960000 > envy.cs.utah.edu.nfs: 172 writ
e [|nfs]
Linux issues 8,192 writes.
Here's a back of the envelope calculation for the above tcpdump excerpt:
4880 us between write req & write reply
73 us between write reply & getattr req
511 us between getattr req & getattr reply
72 us between getattr reply & next write req
Makes roughly 5536 us per request, which about adds up to 40-50 seconds
for 8192 requests.
So, I guess I have two questions:
+ Why does Linux do the getattr following the write?
+ Assuming the write takes so long (almost 5 ms) because the server does
synchronous writes to disk, then the issue is that Linux doesn't aggregate
write requests?
I assume this is what the README file for nfsserver2.2beta40
means when it says:
* NFS operation between two Linux boxes can be quite
slow. There are a number of reasons for this, only one
of which is that unfsd runs in user space. Another
(and much worse) problem is that the Linux NFS client
code currently does no proper caching, read-ahead and
write-behind of NFS data. This problem can be helped
by increasing the RPC transfer size on the client by
adding the `rsize=8192,wsize=8192' mount options. This
will at least improve throughput when reading or
writing large files. You are still in a lose-lose
situation when applications write data line by line or
with no output buffering at all.
Is anybody working on fixing this situation? I had heard rumors that
this had finally been fixed in 2.2.x, but apparently it hasn't.
This seems like a showstopper for Linux to me. This would be sad
since I rather use Linux than FreeBSD, but I may not be able to
afford the wait when I'm dealing with applications that are unaware
of this problem and don't do their own buffering.
- Godmar
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/