Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

[Sorry this turned out a long email, I didn't have the time to write a
short one.]

On Wed, 10 Feb 2016, Daniel Vetter <daniel.vetter@xxxxxxxx> wrote:
> On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet <corbet@xxxxxxx> wrote:
>> On Tue, 26 Jan 2016 14:08:45 +0200
>> Jani Nikula <jani.nikula@xxxxxxxxx> wrote:
>>
>>> I'm afraid we've done some overlapping work in the mean time, but I'm
>>> happy we've both looked at the tool chain, and can have a more
>>> meaningful conversation now.
>>
>> [Adding Keith since you said you wanted to be a part of this - let us know
>> when you've had enough!]
>>
>> So I've spent a bit of time looking at this, and quite a bit more time
>> talking with various folks at LCA.  There is pretty much universal
>> agreement that this is interesting work and the direction we'd like to
>> go.  My current hope is that we can merge some version of it for 4.6 and
>> see where it goes from there.
>>
>> So naturally I have some thoughts on the whole thing...
>>
>>  - I would like to format directly to HTML if at all possible.  It seems
>>    it should be possible to get a table of contents into the files, and
>>    the feedback I got was that a TOC would be enough for navigation - it
>>    would not be necessary to split the files at that point.  We might
>>    still want to try to figure that out too, though.  In any case, this
>>    isn't a show stopper, in that we can change it anytime if a better way
>>    shows up.  But I'd like to have it in mind.
>
> I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
> toolchain, since that's less disruptive. And with that we can also
> fully concentrating on the frontend, and how it'll look and behave.

I'd like to clarify the end goal a bit more before deciding what to do
next. In particular, is the aim to have asciidoc->HTML only or dual
asciidoc->HTML and asciidoc->XML->whatever? Or independent
asciidoc->HTML first, with the existing DocBook on the side until
everything's converted? Something else?

Direct asciidoc->HTML has the problem I mentioned that there is no
chunked output. If the source is big (as-is or via asciidoc includes)
the output is big. The current gpu.tmpl turned way too big. We could
alleviate that by splitting the source documents into smaller pieces (in
gpu.tmpl case it's desirable no matter what), and tying them together
via cross-references and TOC rather than asciidoc includes.

The problem with this, in turn, is that I don't really know how
automatic cross-referencing between kernel-doc comments would turn out
then (e.g. i915 kernel-doc references a symbol in drm core kernel-doc
after gpu.tmpl split) as asciidoc would process the files
independently. A kernel-doc comment writer shouldn't have to know which
document the referenced symbol is in... We could do post-processing I
guess, but I'd really like to get rid of the homebrew aspects here.

Is it acceptable to have dead links when referencing symbols outside of
the document in question, for the time being, until someone figures out
a nice way to do this?

> Once that's solid we can look into the icing on the cake for later
> kernels I think.
>
>>  - Asciidoc templates and processing should happen in a new directory
>>    (perhaps imaginatively called "asciidoc"); having them in a directory
>>    called "DocBook" seems a little weird.  More importantly, though, I'd
>>    like to separate them out as a fresh start, and not mess with the
>>    existing DocBook templates until we decide we don't need them anymore.
>>    If we could end up with a cleaner, simpler makefile in the process,
>>    that would be a bonus.
>
> For the long term dream plan of including other .txt files from the
> existing pile of unstructured docs, do we really want a separate
> asciidoc directory? Or just .asciidoc as a special extension?

Also in my dream world you could have asciidoc files anywhere in the
Documentation tree, with a Makefile per directory identifying which ones
should be processed as asciidoc. I might even name them all .txt, and
you wouldn't have to rename existing "almost markup" plain text files to
have them processed, just fix the markup and update the Makefile. (FWIW
asciidoc suggests .txt extension, though asciidoctor suggests .adoc or
.asciidoc.) I think this would better promote a gradual transition to
lightweight markup, with easier to review patches. Also you mentioned
there's no structure under Documentation. Allowing asciidoc files
anywhere would, I think, help gradual restructuring.

The output could be a subdirectory (one per output format?) under
Documentation.

>>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>>    little files and tracking them for dependencies and such doesn't seem
>>    like a simplification or a path toward better performance.  I would
>>    like to at least consider keeping the direct-from-source inclusion.
>
> The motivation behind the new inclusion mechanism isn't the speed-up
> due to parallelization, but being able to use native asciidoc
> includes. With those you can pass options to e.g. shift the hierarchy.
> With that you can do subheadings in DOC: sections and then seamlessly
> include them. Or similar stuff.
>
> The speed-up due to parallelization is just a small bonus.
>
> Also generating thousands of files is totally not unheard of in the kernel:
>
> $ find include/config | wc -l
> 2623
>
> None of those are in git.

Yes, my main motivation here was to get rid of the preprocessing step
(currently tmpl->xml). I wanted to have the source documents in pure
markup which could be directly processed by asciidoc. I wanted to have
the editor markup helpers and syntax highlighting just work, with no
extra non-markup cruft to confuse it. (For example, emacs tells me the
current tmpl files are invalid XML because of the docproc directives.)
This ties back to the dream above; just have .txt files with no
preprocessing step, IMO it's less confusing for actually writing the
docs.

I didn't think there'd be anything weird about having thousands of
intermediate files generated from source files, with dependencies set
and working, just like we have .o files.

Sure, the mechanism is a proof-of-concept, rough around the edges, and
needs to stow away the intermediate files better, but I still think it's
a conceptually better approach than adding a layer of homebrew when we
have a chance to break away from that. And there's the bonus of getting
parallelization, which I think just backs the concept.

I did try to make asciidoc filters and plugins work for including
kernel-doc, which might have been a better match to what Jon wants, but
without docproc in between. I didn't quite manage to make that work, and
there's the problem they're both incompatible with asciidoctor.

>>  - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
>>    it's going to work for now; that could maybe change after Al's work
>>    goes in, which could be fairly soon.
>
> Hm, assuming Al gets his stuff into 4.6 could we just assume this? It
> holds true for gpu docs already I think, and most other subsystems.
> The trouble iirc is all around asm and similar stuff, and we can't
> kerneldoc asm afaik.

I'd turn this around. IMO the problem isn't insisting EXPORT_SYMBOL is
in the same file as the definition of the symbol. The problem is
insisting that the kernel-doc comment is in the same file as the
EXPORT_SYMBOL and the definition. Particularly include/media has plenty
of kernel-doc in headers with the declarations.

If we can't insist on that, we could teach kernel-doc to scan a list of
other files for the EXPORT_SYMBOLs, instead of having that logic
externally in docproc. This should be trivial, especially if you know
perl. (Unfortunately this might get a little tricky with the include
syntax.)

This was mostly driven by the desire to get rid of the docproc
preprocessing step.

>> Please let me know your thoughts on the above.  Do you think you can find
>> some time over the next month for this?  I'll try to shake loose some time
>> too, but, well, $EXCUSES...

If we can come up with a plan where I can be reasonably sure the
polished effort isn't going down the drain... ;)

> One more thing we discussed: Did you ping kbuild folks already? Or
> want to get some agreement on the overall build process first?

I think CONFIG_BUILD_DOCSRC vs. having documentation targets directly in
Documentation/Makefile (instead of top level make issuing recursive make
in Documentation/DocBook/Makefile) should be reconciliated
somehow. Frankly, I find it odd that the hostprog targets under
Documentation seem to be better class citizens than documentation
targets. Not saying they can't both be there, but they should coexist.

BR,
Jani.



-- 
Jani Nikula, Intel Open Source Technology Center