Re: [PATCH] kselftest: runner: fix TAP output for skipped tests

[Shuah Khan]

On 6/10/20 11:43 AM, Bird, Tim wrote:
> > -----Original Message-----
> > From: linux-kselftest-owner@xxxxxxxxxxxxxxx <linux-kselftest-owner@xxxxxxxxxxxxxxx> On Behalf Of Shuah Khan
> >
> > On 6/10/20 9:44 AM, Paolo Bonzini wrote:
> > > According to the TAP specification, a skipped test must be marked as "ok"
> > > and annotated with the SKIP directive, for example
> > >
> > >      ok 23 # skip Insufficient flogiston pressure.
> > >      (https://testanything.org/tap-specification.html)
> > >
> > > Fix the runner script to match this.
> > >
> > > Signed-off-by: Paolo Bonzini <pbonzini@xxxxxxxxxx>
> > > ---
> > >    tools/testing/selftests/kselftest/runner.sh | 2 +-
> > >    1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
> > > index 676b3a8b114d..f4815cbcd60f 100644
> > > --- a/tools/testing/selftests/kselftest/runner.sh
> > > +++ b/tools/testing/selftests/kselftest/runner.sh
> > > @@ -77,7 +77,7 @@ run_one()
> > >    		echo "ok $test_num $TEST_HDR_MSG") ||
> > >    		(rc=$?;	\
> > >    		if [ $rc -eq $skip_rc ]; then	\
> > > -			echo "not ok $test_num $TEST_HDR_MSG # SKIP"
> > > +			echo "ok $test_num $TEST_HDR_MSG # SKIP"
>
> This is a pretty big change, and might break upstream CIs that have come to
> rely on kselftest's existing behavior.  I know it's going to break Fuego's parsing
> of results.

Thanks for chiming in. We don't want to break CI workflow.

> kselftest has a few conventions that are different from the TAP spec,
> and a few items it does that are extensions to the TAP spec.
> IMHO, the TAP spec got this one wrong, but I could be convinced
> otherwise.  But I think we should discuss this among CI users of
> kselftest before making the change.
>
> I started work quite a while ago on an effort to document the
> conventions used by kselftest (particularly where it deviates
> from the TAP spec),  but never submitted it.
>
> I'm going to submit what I've got as an RFC now, for discussion,
> even though it's not finished.  I'll do that in a separate thread.
>
>
> > >    		elif [ $rc -eq $timeout_rc ]; then \
> > >    			echo "#"
> > >    			echo "not ok $test_num $TEST_HDR_MSG # TIMEOUT"
> >
> > Thanks. I will pull this in for Linux 5.8-rc2
> Shuah - can you hold off on this until we discuss it?

Of course. Thanks for getting my attention before I pulled it in.

thanks,
-- Shuah