Re: RFC - kernel selftest result documentation (KTAP)

[Brendan Higgins]

On Tue, Jun 16, 2020 at 4:52 PM Kees Cook <keescook@xxxxxxxxxxxx> wrote:
>
> On Mon, Jun 15, 2020 at 07:07:34PM +0000, Bird, Tim wrote:
> > From: Kees Cook <keescook@xxxxxxxxxxxx>
> > > Note: making the plan line required differs from TAP13 and TAP14. I
> > > think it's the right choice, but we should be clear.
> >
> > [...]
> > With regards to making it optional or not, I don't have a strong
> > preference.  The extra info seems helpful in some circumstances.
> > I don't know if it's too onerous to make it a requirement or not.
> > I'd prefer if it was always there (either at the beginning or the end),
> > but if there is some situation where it's quite difficult to calculate,
> > then it would be best not to mandate it. I can't think of any impossible
> > situations at the moment.
>
> I think we should require one of:
>
> - starting plan line
> - ending plan line
> - ending with something that indicates "I'm done, but I have no idea how
>   many tests actually ran" (Maybe "1..?")
>
> To me, the point of the plan line is to be able to say "this test did,
> in fact, finish". So even if some test can't even count how many tests
> it _ran_, it can at least say "I am now finished".

So the counting is actually not the hard part for us, it's figuring
out when we have finished. Again, the change that I am working on (I
REALLY need to get that out) should fix that, but until we get that
upstream, KUnit doesn't actually know when it is done running tests.

> > > TAP13/14 makes description optional, are we making it required (I think
> > > we should). There seems to be a TAP13/14 "convention" of starting
> > > <description> with "- ", which I'm on the fence about it. It does make
> > > parsing maybe a little easier.
> >
> > I would like the description to be required.
> > I don't have a strong opinion on the dash.  I'm OK with either one (dash
> > or no dash), but we should make kselftest and KUnit consistent.
>
> I find the dash to be distracting -- it doesn't help me scan it, and it
> doesn't help a parser (which only needs to find "#").

Yeah, I also prefer spaces and/or "#". I am okay if spaces are
optional only to aid human readability. And honestly I don't care
about this point too much. Just offering my 2 cents.

> > > > Differences between kernel test result format and TAP13:
> > > >  - in KTAP the "# SKIP" directive is placed after the description on
> > > >    the test result line
>
> I sent a bunch of clean-ups for kselftest.h recently[1], but it looks
> like we'll need more for adding "description" to skip (right now it only
> prints the SKIP reason).
>
> [1] https://lore.kernel.org/lkml/20200611224028.3275174-1-keescook@xxxxxxxxxxxx/
>
> > > Yes Documentation/*.rst Not sure on name yet, but where do kselftest
> > > docs live? :)
> > Documentation/dev-tools/kselftest.rst
> >
> > I'll put this at: Documentation/dev-tools/test-results-format.rst
>
> Sounds good!
>
> --
> Kees Cook