Re: [PATCH v3 3/3] kselftest: Add new test for detecting unprobed Devicetree devices

[Nícolas F. R. A. Prado]

On Mon, Nov 06, 2023 at 11:09:44AM -0600, Rob Herring wrote:
> On Thu, Nov 2, 2023 at 12:36 PM Mark Brown <broonie@xxxxxxxxxx> wrote:
> >
> > On Thu, Nov 02, 2023 at 07:15:58PM +0530, Naresh Kamboju wrote:
> > > On Thu, 2 Nov 2023 at 17:41, Aishwarya TCV <aishwarya.tcv@xxxxxxx> wrote:
> >
> > > > https://storage.kernelci.org/mainline/master/v6.6-9152-gdeefd5024f07/arm64/defconfig%2Bkselftest/gcc-10/logs/kselftest.log
> >
> > ...
> >
> > > May be due to, A loop of symlinks that are pointing to self / same files ?
> >
> > Right, it does look like something bad is going on with symlinks:
> >
> > > > '/tmp/kci/linux/tools/testing/selftests/../../../build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/build/source/tools/testing/selftests/powerpc/vphn/vphn.c'
> >
> > > Please build by using tuxmake and validate builds are working.
> >
> > Note that tuxmake does an in tree build of kselftest:
> >
> >   make --silent --keep-going --jobs=8 O=/home/tuxbuild/.cache/tuxmake/builds/1/build INSTALL_PATH=/home/tuxbuild/.cache/tuxmake/builds/1/build/kselftest_install ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- CROSS_COMPILE_COMPAT=arm-linux-gnueabihf- 'CC=sccache aarch64-linux-gnu-gcc' 'HOSTCC=sccache gcc' kselftest-install
> >
> > and does it's own tarball build too, whereas kernelci does an out of
> > tree build and uses kselftest-gen_tar:
> >
> >   make KBUILD_BUILD_USER=KernelCI FORMAT=.xz ARCH=arm64 HOSTCC=gcc CROSS_COMPILE=aarch64-linux-gnu- CROSS_COMPILE_COMPAT=arm-linux-gnueabihf- CC="ccache aarch64-linux-gnu-gcc" O=/tmp/kci/linux/build -C/tmp/kci/linux -j10 kselftest-gen_tar
> >
> > and that the error is in the dt-extract-compatibles program which is
> > part of the kernel (well, imported into the kernel from dtc upstream):
> >
> >   File "/tmp/kci/linux/tools/testing/selftests/../../../scripts/dtc/dt-extract-compatibles", line 107, in <module>
> >     compat_ignore_list.extend(parse_compatibles_to_ignore(f))
> >
> > This all suggests that something to do with how the build is set up is
> > resulting in the source symlink that gets created for out of tree builds
> > blowing up, I guess it's not specifically the DT stuff that's blowing it
> > up but rather that it's tripping over an existing bug.  Really does look
> > like a legitimate bug though, the source link is set up by the in tree
> > kernel build infrastructure.
> >
> > I did poke a bit at reproducing outside of the KernelCI scripts but
> > didn't manage to yet.
> 
> I can repro with "make dt_compatible_check". The problem is with an
> 'out of tree' build within the tree. That's my normal setup, but the
> difference is I have ".build" directories. If I use "build" instead,
> then I can repro. The issue is the iglob will recurse into "build" but
> not hidden directories (by default). There's no option to not follow
> symlinks which would solve this (there is an open python issue since
> 2017 to add it). I don't see a simple solution in python other than
> getting a full list with glob(), convert to absolute paths, and remove
> duplicates. I imagine that will be somewhat slow.

Hi, sorry for the delay, I was on vacation last week.

I was able to reproduce the issue the way you described. And I also suspected
an alternative approach would be slower, but after trying it out it ran just as
fast as the current one, even on cold cache, so I sent it out:

https://lore.kernel.org/all/20231107225624.9811-1-nfraprado@xxxxxxxxxxxxx

Let me know your thoughts there.

Thanks,
Nícolas