Merge tag 'docs-6.16' of git://git.lwn.net/linux

Pull documentation updates from Jonathan Corbet:
"A moderately busy cycle for documentation this time around:

- The most significant change is t

Merge tag 'docs-6.16' of git://git.lwn.net/linux

Pull documentation updates from Jonathan Corbet:
"A moderately busy cycle for documentation this time around:

- The most significant change is the replacement of the old
kernel-doc script (a monstrous collection of Perl regexes that
predates the Git era) with a Python reimplementation. That, too, is
a horrifying collection of regexes, but in a much cleaner and more
maintainable structure that integrates far better with the Sphinx
build system.

This change has been in linux-next for the full 6.15 cycle; the
small number of problems that turned up have been addressed,
seemingly to everybody's satisfaction. The Perl kernel-doc script
remains in tree (as scripts/kernel-doc.pl) and can be used with a
command-line option if need be. Unless some reason to keep it
around materializes, it will probably go away in 6.17.

Credit goes to Mauro Carvalho Chehab for doing all this work.

- Some RTLA documentation updates

- A handful of Chinese translations

- The usual collection of typo fixes, general updates, etc"

* tag 'docs-6.16' of git://git.lwn.net/linux: (85 commits)
Docs: doc-guide: update sphinx.rst Sphinx version number
docs: doc-guide: clarify latest theme usage
Documentation/scheduler: Fix typo in sched-stats domain field description
scripts: kernel-doc: prevent a KeyError when checking output
docs: kerneldoc.py: simplify exception handling logic
MAINTAINERS: update linux-doc entry to cover new Python scripts
docs: align with scripts/syscall.tbl migration
Documentation: NTB: Fix typo
Documentation: ioctl-number: Update table intro
docs: conf.py: drop backward support for old Sphinx versions
Docs: driver-api/basics: add kobject_event interfaces
Docs: relay: editing cleanups
docs: fix "incase" typo in coresight/panic.rst
Fix spelling error for 'parallel'
docs: admin-guide: fix typos in reporting-issues.rst
docs: dmaengine: add explanation for DMA_ASYNC_TX capability
Documentation: leds: improve readibility of multicolor doc
docs: fix typo in firmware-related section
docs: Makefile: Inherit PYTHONPYCACHEPREFIX setting as env variable
Documentation: ioctl-number: Update outdated submission info
...

show more ...

Merge branch 'mauro' into docs-mw

Mauro says:

This changeset contains the kernel-doc.py script to replace the verable
kernel-doc originally written in Perl. It replaces the first version and the
se

Merge branch 'mauro' into docs-mw

Mauro says:

This changeset contains the kernel-doc.py script to replace the verable
kernel-doc originally written in Perl. It replaces the first version and the
second series I sent on the top of it.

I tried to stay as close as possible of the original Perl implementation
on the first patch introducing kernel-doc.py, as it helps to double check
if each function was properly translated to Python. This have been
helpful debugging troubles that happened during the conversion.

I worked hard to make it bug-compatible with the original one. Still, its
output has a couple of differences from the original one:

- The tab expansion works better with the Python script. With that, some
outputs that contain tabs at kernel-doc markups are now different;

- The new script works better stripping blank lines. So, there are a couple
of empty new lines that are now stripped with this version;

- There is a buggy logic at kernel-doc to strip empty description and
return sections. I was not able to replicate the exact behavior. So, I ended
adding an extra logic to strip empty sections with a different algorithm.

Yet, on my tests, the results are compatible with the venerable script
output for all .. kernel-doc tags found in Documentation/. I double-checked
this by adding support to output the kernel-doc commands when V=1, and
then I ran a diff between kernel-doc.pl and kernel-doc.py for the same
command lines.

The only patch that doesn't belong to this series is a patch dropping
kernel-doc.pl. I opted to keep it for now, as it can help to better
test the new tools.

With such changes, if one wants to build docs with the old script,
all it is needed is to use KERNELDOC parameter, e.g.:

$ make KERNELDOC=scripts/kernel-doc.pl htmldocs

show more ...

scripts/kernel_doc.py: better handle exported symbols

Change the logic which detects internal/external symbols in a way
that we can re-use it when calling via Sphinx extension.

While here, remove a

scripts/kernel_doc.py: better handle exported symbols

Change the logic which detects internal/external symbols in a way
that we can re-use it when calling via Sphinx extension.

While here, remove an unused self.config var and let it clearer
that self.config variables are read-only. This helps to allow
handling multiple times in parallel if ever needed.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/6a69ba8d2b7ee6a6427abb53e60d09bd4d3565ee.1744106242.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: Properly handle Werror and exit codes

The original kernel-doc script has a logic to return warnings
as errors, and to report the number of warnings found, if in
verbose mode.

scripts/kernel-doc.py: Properly handle Werror and exit codes

The original kernel-doc script has a logic to return warnings
as errors, and to report the number of warnings found, if in
verbose mode.

Implement it to be fully compatible with the original script.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/de33b0cebd9fdf82d8b221bcfe41db7269286222.1744106242.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: move modulename to man class

Only man output requires a modulename. Move its definition
to the man class.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Sig

scripts/kernel-doc.py: move modulename to man class

Only man output requires a modulename. Move its definition
to the man class.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/583085e3885b0075d16ef9961b4f2ad870f30a55.1744106242.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: adjust some coding style issues

Make pylint happier by adding some missing documentation and
addressing a couple of pylint warnings.

Signed-off-by: Mauro Carvalho Chehab <mch

scripts/kernel-doc.py: adjust some coding style issues

Make pylint happier by adding some missing documentation and
addressing a couple of pylint warnings.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/0f9d5473105e4c09c6c41e3db72cc63f1d4d55f9.1744106242.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: implement support for -no-doc-sections

The venerable kernel-doc Perl script has a number of options that
aren't properly documented. Among them, there is -no-doc-sections,
whi

scripts/kernel-doc.py: implement support for -no-doc-sections

The venerable kernel-doc Perl script has a number of options that
aren't properly documented. Among them, there is -no-doc-sections,
which is used by the Sphinx extension.

Implement support for it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/06b18a32142b44d5ba8b41ac64a76c02b03b4969.1744106242.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: move file lists to the parser function

Instead of setting file lists at __init__ time, move it to
the actual parsing function. This allows adding more files
to be parsed in re

scripts/kernel-doc.py: move file lists to the parser function

Instead of setting file lists at __init__ time, move it to
the actual parsing function. This allows adding more files
to be parsed in real time, by calling parse function multiple
times.

With the new way, the export_files logic was rewritten to
avoid parsing twice EXPORT_SYMBOL for partial matches.

Please notice that, with this logic, it can still read the
same file twice when export_file is used.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/ab10bc94050406ce6536d4944b5d718ecd70812f.1744106242.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: convert message output to an interactor

Instead of directly printing output messages, change kdoc classes
to return an interactor with the output message, letting the
actual d

scripts/kernel-doc.py: convert message output to an interactor

Instead of directly printing output messages, change kdoc classes
to return an interactor with the output message, letting the
actual display to happen at the command-line command.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/557304c8458f1fb4aa2e833f4bdaff953094ddcb.1744106242.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: move output classes to a separate file

In preparation for letting kerneldoc Sphinx extension to import
Python libraries, move kernel-doc output logic to a separate file.

Sign

scripts/kernel-doc.py: move output classes to a separate file

In preparation for letting kerneldoc Sphinx extension to import
Python libraries, move kernel-doc output logic to a separate file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/81087eff25d11c265019a8631f7fc8d3904795d0.1744106242.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: move KernelFiles class to a separate file

The KernelFiles class is the main dispatcher which parses each
source file.

In preparation for letting kerneldoc Sphinx extension to

scripts/kernel-doc.py: move KernelFiles class to a separate file

The KernelFiles class is the main dispatcher which parses each
source file.

In preparation for letting kerneldoc Sphinx extension to import
Python libraries, move regex ancillary classes to a separate
file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/80bc855e128a9ff0a11df5afe9ba71775dfc9a0f.1744106241.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: move KernelDoc class to a separate file

In preparation for letting kerneldoc Sphinx extension to import
Python libraries, move regex ancillary classes to a separate
file.

Sig

scripts/kernel-doc.py: move KernelDoc class to a separate file

In preparation for letting kerneldoc Sphinx extension to import
Python libraries, move regex ancillary classes to a separate
file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/c76df228504e711c6b4bcd23d5a0ea1fda678cda.1744106241.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: move regex methods to a separate file

In preparation for letting kerneldoc Sphinx extension to import
Python libraries, move regex ancillary classes to a separate
file.

Signe

scripts/kernel-doc.py: move regex methods to a separate file

In preparation for letting kerneldoc Sphinx extension to import
Python libraries, move regex ancillary classes to a separate
file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/64f96b6744435b51894bb4ab7612851d9d054190.1744106241.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: properly handle struct_group macros

Handing nested parenthesis with regular expressions is not an
easy task. It is even harder with Python's re module, as it
has a limited sub

scripts/kernel-doc.py: properly handle struct_group macros

Handing nested parenthesis with regular expressions is not an
easy task. It is even harder with Python's re module, as it
has a limited subset of regular expressions, missing more
advanced features.

We might use instead Python regex module, but still the
regular expressions are very hard to understand. So, instead,
add a logic to properly match delimiters.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/74dee485f70b7ce85e90496bfdd360283a677a58.1744106241.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: better handle empty sections

While doing the conversion, we opted to skip empty sections
(description, return), but this makes harder to see the differences
between kernel-doc

scripts/kernel-doc.py: better handle empty sections

While doing the conversion, we opted to skip empty sections
(description, return), but this makes harder to see the differences
between kernel-doc (Perl) and kernel-doc.py.

Also, the logic doesn't always work properly. So, change the
way this is done by adding an extra step to remove such
sections, doing it only for Return and Description.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/1b057092a48ba61d92a411f4f6d505b802913785.1744106241.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: output warnings the same way as kerneldoc

Add a formatter to logging to produce outputs in a similar way
to kernel-doc. This should help making it more compatible with
existin

scripts/kernel-doc.py: output warnings the same way as kerneldoc

Add a formatter to logging to produce outputs in a similar way
to kernel-doc. This should help making it more compatible with
existing scripts.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/559f0ad9e6fecfcbb3cc38b6097463bd38d58629.1744106241.git.mchehab+huawei@kernel.org

show more ...

scripts/kernel-doc.py: add a Python parser

Maintaining kernel-doc has been a challenge, as there aren't many
perl developers among maintainers. Also, the logic there is too
complex. Having lots of g

scripts/kernel-doc.py: add a Python parser

Maintaining kernel-doc has been a challenge, as there aren't many
perl developers among maintainers. Also, the logic there is too
complex. Having lots of global variables and using pure functions
doesn't help.

Rewrite the script in Python, placing most global variables
inside classes. This should help maintaining the script in long
term.

It also allows a better integration with kernel-doc Sphinx
extension in the future.

I opted to keep this version as close as possible to what we
have already in Perl. There are some differences though:

1. There is one regular expression that required a rewrite:

/\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;/

As this one uses two features that aren't available by the native
Python regular expression module (re):

- recursive patterns: ?1
- atomic grouping (?>...)

Rewrite it to use a much simpler regular expression:

/\bSTRUCT_GROUP\(([^\)]+)\)[^;]*;/

Extra care should be taken when validating this script, as such
replacement might cause some regressions.

2. The filters are now applied only during output generation.
In particular, "nosymbol" argument is only handled there.

It means that, if the same file is processed twice for
different symbols, the warnings will be duplicated.

I opted to use this behavior as it allows the Sphinx extension
to read the file(s) only once, and apply the filtering only
when producing the ReST output. This hopefully will help
to speed up doc generation

3. This version can handle multiple files and multiple directories.

So, if one just wants to produce a big output with everything
inside a file, this could be done with

$ time ./scripts/kernel-doc.py -man . 2>/dev/null >new
real 0m54.592s
user 0m53.345s
sys 0m0.997s

4. I tried to replicate as much as possible the same arguments
from kernel-doc, with about the same behavior, for the
command line parameters starting with a single dash (-parameter).

I also added one letter aliases for each parameter, and a
--parameter (sometimes with a better name).

5. There are some sutile nuances between how Perl handles
certain regular expressions. In special, the qr operatior,
which compiles a regular expression also works as a
non-capturing group. It means that some regexes like
this one:

my $type1 = qr{[\w\s]+};

needs to be mapped as:

type1 = r'(?:[\w\s]+)?'

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/2fa671a9fb08d03a376a42d46cc0b1d3aab4ae3f.1744106241.git.mchehab+huawei@kernel.org

show more ...