Re: [RFC] A first shot at asciidoc-based formatted docs

[Daniel Vetter]

On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula <jani.nikula@xxxxxxxxx> wrote:
> On Tue, 26 Jan 2016, Jonathan Corbet <corbet@xxxxxxx> wrote:
>> So here is a proof-of-concept series showing how a fully asciidoc-based
>> toolchain might work.  Lots of hackery here, this isn't meant to be applied
>> to anything at this point, but it's a good start.  What this series has is:
>>
>>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>>    doing this!  It was the kickstart that was needed to get this process
>>    going.
>
> Hooray! :)
>
>>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>>    ".adt", it's an asciidoc template; it's processed pretty much the same
>>    way, except that kernel-doc gets the -asciidoc argument.
>>
>>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>>    Naturally this was where most of the time got spent.  *Only* HTML output
>>    works at the moment.
>>
>>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>>    and the output is much pleasing, IMO.
>>
>> I'm sure there's a thousand details to deal with, and there is the issue of
>> the other output formats.  asciidoctor claims to be able to create man
>> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
>> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
>> straightforward, so it should be possible to use xmlto as is done now.
>>
>> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
>> anybody wants to mess with it.
>>
>> Comments?
>
> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.
>
> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

This is a big bummer since with the parralized kernel-doc processing
using Makefiles and using asciidoctor even building something big like
the gpu docs is down to 2-3 seconds. From a clean tree, so not even
counting incremental speed-ups. Unfortunately asciidoc doesn't have an
built-in tooling (there's some experimental extensions) to split
things up.
-Daniel

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)
>
> One significant difference between our approaches is that I ditched
> docproc out of the equation. Instead of having the docproc ! directives
> in the asciidoc, I opted for using asciidoc's native include macro, with
> specially crafted filename suffixes to specify what parts of the source
> to include. Those files are then generated as intermediate asciidoc
> files using kernel-doc, with dependencies set right. There's a bunch of
> scripting involved, but it's pretty straight forward.
>
> So you'd have, for example,
>
> include::drivers/gpu/drm/drm_ioctl.c,export[]
>
> to include all the exported functions, or
>
> include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]
>
> to include the DOC: section. I think I'd like some version of this,
> regardless of whether asciidoc generates HTML or DocBook. It lets make
> parallelize all of kernel-doc processing for free, which is a big win.
>
> Patches follow, also available in kernel-asciidoc branch of
> git://people.freedesktop.org/~jani/drm (web interface
> http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)
>
> BR,
> Jani.
>
>
> Jani Nikula (10):
>   kernel-doc: rewrite usage description, remove duplicated comments
>   kernel-doc: add support for asciidoc output
>   kernel-doc: support printing exported and non-exported symbols
>   kernel-doc: add support for printing DOC: comments with escaped names
>   scripts: add asciidoc-includes to extract includes from asciidoc
>   scripts: add a kernel-doc helper for special invocation
>   scripts: add tool for generating asciidoc dependencies and rules
>   scripts: add a crude converter from DocBook tmpl to asciidoc
>   Documentation: convert gpu.tmpl to gpu.txt
>   Documentation: build asciidoc documentation
>
>  Documentation/DocBook/Makefile |   37 +-
>  Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
>  Documentation/DocBook/gpu.txt  | 1183 ++++++++++++++
>  scripts/asciidoc-includes      |    6 +
>  scripts/kernel-doc             |  365 ++++-
>  scripts/kernel-doc-deps        |   66 +
>  scripts/kernel-doc-helper      |   70 +
>  scripts/tmpl2asciidoc          |   39 +
>  8 files changed, 1709 insertions(+), 3556 deletions(-)
>  delete mode 100644 Documentation/DocBook/gpu.tmpl
>  create mode 100644 Documentation/DocBook/gpu.txt
>  create mode 100755 scripts/asciidoc-includes
>  create mode 100755 scripts/kernel-doc-deps
>  create mode 100755 scripts/kernel-doc-helper
>  create mode 100755 scripts/tmpl2asciidoc
>
> --
> 2.1.4
>



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch