Re: autofs mount/umount hangs with recent kernel?

[Daire Byrne]

On Fri, 3 Nov 2023 at 14:10, Bagas Sanjaya <bagasdotme@xxxxxxxxx> wrote:
>
> On Fri, Nov 03, 2023 at 09:40:44AM +0000, Daire Byrne wrote:
> > Hi,
> >
> > We have large compute clusters that, amongst other things, spend their
> > day mounting & unounting lots of Linux NFS servers via autofs. This
> > has worked fine for many years and client kernel versions and was
> > working without incident even with our current v6.3.x production
> > kernels.
> >
> > During the v6.6-rc cycle while testing that kernel, I noticed that
> > every now and then, the umount/mount would hang randomly and the
> > compute host would get stuck and not complete it's work until a
> > reboot. I thought I'd wait until v6.6 was released and check again -
> > the issue persists.
> >
> > I have not had the opportunity to test the v6.4 & v6.5 kernels in
> > between yet. The stack traces look something like this:
>
> Please do bisection to find the exact commit that introduces your
> regression. See Documentation/admin-guide/bug-bisect.rst in the kernel
> sources for more information.

I tested the v6.5.10 kernel and I can't reproduce with that, so I
guess I'll start the process of bisecting between v6.5.10 & v6.6.

It takes around 24 hours and 72 client hosts until we see it happen on
at least one, so it'll be slow going.

Weirdly, I also have the v6.6 kernel running on a different set of
hosts and they have not shown the problem at all. The only difference
being AMD clients with higher process concurrency = bad, and Intel
clients with less process concurrency = okay.

> >
> > [202752.264187] INFO: task umount.nfs:58118 blocked for more than 245 seconds.
> > [202752.264237]       Tainted: G            E      6.6.0-1.dneg.x86_64 #1
>
> Can you reproduce on untainted (vanilla) kernel?

Yea, I'm not sure what is going on there. Some compile issue as it's
the "deflate" module complaining about a missing key.

> > And like I said, the mount/umount against the server hangs
> > indefinitely on the client. It is somewhat interesting that autofs
> > still tries to trigger a subsequent mount even though the umount has
> > not completed.
> >
> > The NFS servers are running RHEL8.5 and we are using NFSv3. I also
> > reproduced it with a fairly recent nfs-utils-2.6.2 on the client
> > compute hosts.
>
> What distro on the client?

It's EL79 on the client but I have updated the nfs-utils to something
more modern. I have not updated the autofs package though.

I'll report back in a week or so...

Cheers,

Daire