Hello,
i have a really severe autofs4 problem. Kernel is 2.4.7,
Redhat-7.0, Hardware is a Fujitsu/Siemens HPC Line dual
Processor, so kernel is built with SMP turned on. automounter
Daemon is 4.0.0pre10 . gcc is 2.96:
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs
gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-85)
RAM is 2 GB, CPU: Pentium III (Coppermine), 256 kb Cache,
Boot disk is an IDE disk, a SCSI Symbios 53c1010 hostadapter and
a connected additional disk is installed, but not used, the
SCSI-adapter driver isn't loaded. IDE controller is
ServerWorks OSB4.
I'm desperately trying to find the reason for my quite
reproducible problem (it's not the locking thing in the kernel
that has been fixed since quite a while)
Assume an map like that:
kaefer -rw /tmp2 kaefer:/tmp2
gustav -rw /tmp2 gustav:/tmp2
distel -rw /tmp2 distel:/tmp2
averna -rw /tmp2 averna:/tmp2
solp1 -rw /tmp2 solp1:/tmp2
...
host-1 -rw /tmp2 host-1:/tmp2
host-2 -rw /tmp2 host-2:/tmp2
...
i.e. all of those hosts have a /tmp2 directory, that is exported
properly. Automounter Daemon runs with the following args:
automount -t 1 /mnt/my-subdir yp map-name grpid,hard,nodevs,nosymlink
(or with file instead yp, makes no difference)
Basically we don't want a timeout, which is that short. But it
helps to reproduce the problem within minutes. Otherwise it
shows up after some hours, what is not acceptable for users
with long-running compute jobs. The software accesses the
/tmp2 directories and home-direcotries (also autofs-Mounted)
from time to time and once within a few hours the users see
either a "Permission denied" or "Directory does not exist".
And this happens with that stress test, too. Assume the following
script:
#!/bin/sh
# build a list of files to touch within one command
I=1
while [ $I -le 32 ] ; do
FILES="$FILES /mnt/my-subdir/host-$I/tmp2/affile"
I=`expr $I + 1`
done
while true ; do
echo 'trying to touch and ls'
touch $FILES
if [ $? -ne 0 ] ; then
exit 1
fi
ls -ld $FILES
date
/bin/rm -f /tmp/bla
head -4c /dev/random > /tmp/bla
RANDNUM=`sum /tmp/bla | awk '{print $1}'`
WAITTIME=`expr '(' $RANDNUM ')' / 2 + 980000`
# random sleep time between 980 ms and ~ 1020 with script
# interpretatin ... delay - neglible
ls -ld $FILES>/dev/null
usleep $WAITTIME
done
What i try to achieve with that random delay is, that
during expire of the 32 mounts another access to them
occurs and this in fact happens quite frequently, some-
times leading to the mentioned error message during
touch, followed by the exit 1, sometimes not. I wouldn't
have a problem, if the problem showed up only under this
stress test, but it's happening also under `normal'
circumstances. This test runs between few seconds and
several (say, 20) minutes until failing. It is not
depending on the kind of NFS-Server. This can be either
a Linux or Sun machine. No notable difference. I tcpdumped
the network traffic. No clue. Or better: i can see, that
no new mount request is leaving the machine directed to
the server, that supplies one of the directories. Sometimes
the touch complains about olny one missing directory,
sometimes about 8 or so, in successive order (e.g.:
/mnt/my-subdir/host-7/tmp2 through /mnt/my-subdir/host-12/tmp2 )
I suspect, that the autofs4 returns from the revalidate thinking,
sth has been or is mounted, but has not in reality or has been
expired in the meantime. I experimented a lot. I modified
the daemon to serialize the mounts, not just fork and
report to the kernel, when done. Didn't help. Have a lot
of strace output about that. Looks all ok. So i suspected,
it's the kernel autofs4. Found several things, that looked
weird to me and modified them, but no success. What i tried
(please don't laugh, maybe some of the stuff is ridiculous):
* protected the queues linked list using a spinlock during
traversal and modification against parallel accesses
* modified the autofs4_wait function in several more ways
to have the queues list distinguish between the different
notification requests. (If thought it might be a problem,
that a mount request is being added, but finding an expire
request, thinking 'oh, we have that already, can jump on the
request for mount', though in fact waiting for expire, but i
was wrong here, one more time)
* added an additional flag, cause i thought, it might happen
during execution of the first lines of try_to_fill_dentry,
where the dcache is not yet locked, that the test for
AUTOFS_INF_EXPIRING fails and one thread/CPU walks on
through the function while another CPU works in parallel
on an ioctl-issued autofs4_expire_multi and before setting
the flag, the other thread has passed the if. I added another
flag, that is set in try_to_fill_dentry and checked in
the autofs4_expire_multi to avoid, that the mountpoint in
question will be expired, when not expected to expire.
But this wasn't the problem either.
I have tons of syslog and strace -f output of the daemon, but
no more clues here.
This problem is quite a show-stopper here, otherwise we must
set the expire time to infinite, what is not really desirable
or we must use the userspace amd :-(
Any hint how to narrow the problem is appreciated.
Thanks for reading and with kindest regards,
Albert
-- Albert Flügel Tel.: +49-89-636-27690 science+computing AG Fax.: +49-89-636-21950 bei: Infineon AG D1: +49-171-3698673 E-Mail: Albert.Fluegel.gp@mchr2.siemens.de - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Sun Dec 23 2001 - 21:00:13 EST