Re: [PATCH v2 8/8] selftests/mm: Run all tests from run_vmtests.sh

[Peter Xu]

On Mon, Jul 17, 2023 at 07:27:13PM +0200, David Hildenbrand wrote:
> On 17.07.23 12:31, Ryan Roberts wrote:
> > It is very unclear to me how one is supposed to run all the mm selftests
> > consistently and get clear results.
> > 
> > Most of the test programs are launched by both run_vmtests.sh and
> > run_kselftest.sh:
> > 
> >    hugepage-mmap
> >    hugepage-shm
> >    map_hugetlb
> >    hugepage-mremap
> >    hugepage-vmemmap
> >    hugetlb-madvise
> >    map_fixed_noreplace
> >    gup_test
> >    gup_longterm
> >    uffd-unit-tests
> >    uffd-stress
> >    compaction_test
> >    on-fault-limit
> >    map_populate
> >    mlock-random-test
> >    mlock2-tests
> >    mrelease_test
> >    mremap_test
> >    thuge-gen
> >    virtual_address_range
> >    va_high_addr_switch
> >    mremap_dontunmap
> >    hmm-tests
> >    madv_populate
> >    memfd_secret
> >    ksm_tests
> >    ksm_functional_tests
> >    soft-dirty
> >    cow
> > 
> > However, of this set, when launched by run_vmtests.sh, some of the
> > programs are invoked multiple times with different arguments. When
> > invoked by run_kselftest.sh, they are invoked without arguments (and as
> > a consequence, some fail immediately).
> > 
> > Some test programs are only launched by run_vmtests.sh:
> > 
> >    test_vmalloc.sh
> > 
> > And some test programs and only launched by run_kselftest.sh:
> > 
> >    khugepaged
> >    migration
> >    mkdirty
> >    transhuge-stress
> >    split_huge_page_test
> >    mdwe_test
> >    write_to_hugetlbfs
> > 
> > Furthermore, run_vmtests.sh is invoked by run_kselftest.sh, so in this
> > case all the test programs invoked by both scripts are run twice!
> > 
> > Needless to say, this is a bit of a mess. In the absence of fully
> > understanding the history here, it looks to me like the best solution is
> > to launch ALL test programs from run_vmtests.sh, and ONLY invoke
> > run_vmtests.sh from run_kselftest.sh. This way, we get full control over
> > the parameters, each program is only invoked the intended number of
> > times, and regardless of which script is used, the same tests get run in
> > the same way.
> > 
> > The only drawback is that if using run_kselftest.sh, it's top-level tap
> > result reporting reports only a single test and it fails if any of the
> > contained tests fail. I don't see this as a big deal though since we
> > still see all the nested reporting from multiple layers. The other issue
> > with this is that all of run_vmtests.sh must execute within a single
> > kselftest timeout period, so let's increase that to something more
> > suitable.
> > 
> > In the Makefile, TEST_GEN_PROGS will compile and install the tests and
> > will add them to the list of tests that run_kselftest.sh will run.
> > TEST_GEN_FILES will compile and install the tests but will not add them
> > to the test list. So let's move all the programs from TEST_GEN_PROGS to
> > TEST_GEN_FILES so that they are built but not executed by
> > run_kselftest.sh. Note that run_vmtests.sh is added to TEST_PROGS, which
> > means it ends up in the test list. (the lack of "_GEN" means it won't be
> > compiled, but simply copied).
> > 
> > Signed-off-by: Ryan Roberts <ryan.roberts@xxxxxxx>
> > ---
> 
> Acked-by: David Hildenbrand <david@xxxxxxxxxx>

Thanks for letting me know, David.  Sorry for the late response, still
catching up things.

I used to justify that from mm/ itself that everything should be PROG, but
I see that from higher level where TEST_GEN_FILE|PROG is really used this
makes sense.  As long as vm_utils.o will be properly linked I'll be happy
enough..

Acked-by: Peter Xu <peterx@xxxxxxxxxx>

Thanks,

-- 
Peter Xu