doc tools: better handle KBUILD_VERBOSE

As reported by Jacob, there are troubles when KBUILD_VERBOSE is
set at the environment.

Fix it on both kernel-doc and sphinx-build-wrapper.

Reported-by: Jac

doc tools: better handle KBUILD_VERBOSE

As reported by Jacob, there are troubles when KBUILD_VERBOSE is
set at the environment.

Fix it on both kernel-doc and sphinx-build-wrapper.

Reported-by: Jacob Keller <jacob.e.keller@intel.com>
Closes: https://lore.kernel.org/linux-doc/9367d899-53af-4d9c-9320-22fc4dbadca5@intel.com/
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Tested-by: Jacob Keller <jacob.e.keller@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <7a99788db75630fb14828d612c0fd77c45ec1891.1774591065.git.mchehab+huawei@kernel.org>

show more ...

docs: c_lex.py: store logger on its data

By having the logger stored there, any code using CTokenizer can
log messages there.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed

docs: c_lex.py: store logger on its data

By having the logger stored there, any code using CTokenizer can
log messages there.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <467979dc18149e4b2a7113c178e0cb07919632f2.1774256269.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_output: raise an error if full_proto not available for var

This is mandatory, but if it is missing, we need to know what
symbol had problems.

Signed-off-by: Mauro Carvalho Chehab <mcheha

docs: kdoc_output: raise an error if full_proto not available for var

This is mandatory, but if it is missing, we need to know what
symbol had problems.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <0c3d9dd25889784b999efdb354ade48264c0e03c.1774256269.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_yaml_file: use a better name for the tests

Instead of always using a name with a number on it, use
the name of the object directly whenever possible.

When the name is already used, appen

docs: kdoc_yaml_file: use a better name for the tests

Instead of always using a name with a number on it, use
the name of the object directly whenever possible.

When the name is already used, append a number prefix at
the end.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <d1c4cd94547d843af0debf9e317e006d55d705f1.1774256269.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc: better handle source when producing YAML output

The current logic was storing symbols source code on a list,
not linked to the actual KdocItem. While this works fine when
kernel-doc mark

docs: kdoc: better handle source when producing YAML output

The current logic was storing symbols source code on a list,
not linked to the actual KdocItem. While this works fine when
kernel-doc markups are OK, on places where there is a "/**"
without a valid kernel-doc markup, it ends that the 1:1 match
between source code and KdocItem doesn't happen, causing
problems to generate the YAML output.

Fix it by storing the source code directly into the KdocItem
structure.

This shouldn't affect performance or memory footprint, except
when --yaml option is used.

While here, add a __repr__() function for KdocItem, as it
helps debugging it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <77902dafabb5c3250486aa2dc1568d5fafa95c5b.1774256269.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_output: fix handling of simple tables

Fix check for simple table delimiters.

ReST simple tables use "=" instead of "-". I ended testing it with
a table modified from a complex one, using

docs: kdoc_output: fix handling of simple tables

Fix check for simple table delimiters.

ReST simple tables use "=" instead of "-". I ended testing it with
a table modified from a complex one, using "--- --- ---", instead
of searching for a real Kernel example.

Only noticed when adding an unit test and seek for an actual
example from kernel-doc markups.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <dea95337c05040f95e5a95ae41d69ddef0aaa8d6.1774256269.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_yaml_file: add a representer to make strings look nicer

The strings representation is not ok, currently. Add a helper
function to improve it, and drop blank lines at beginning and
at the

docs: kdoc_yaml_file: add a representer to make strings look nicer

The strings representation is not ok, currently. Add a helper
function to improve it, and drop blank lines at beginning and
at the end of the dumps

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <422041a8b49b2609de5749092fe074b7948c32a6.1774256269.git.mchehab+huawei@kernel.org>

show more ...

docs: kernel-doc: add support to store output on a YAML file

Add a command line parameter and library support to optionally
store:
- KdocItem intermediate format after parsing;
- man pages output;
-

docs: kernel-doc: add support to store output on a YAML file

Add a command line parameter and library support to optionally
store:
- KdocItem intermediate format after parsing;
- man pages output;
- rst output.

inside a YAML file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <ba54277b3c909867153b9547dfa33c1831ca35d9.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: add a new file to write kernel-doc output to a YAML file

Storing kernel-doc output is helpful to allow debugging problems
on it and to preparate unit tests.

Add a class to store such contents

docs: add a new file to write kernel-doc output to a YAML file

Storing kernel-doc output is helpful to allow debugging problems
on it and to preparate unit tests.

Add a class to store such contents at the same format as defined
at kdoc-test-schema.yaml.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <5d084ca1a91f6a620534a1135d1b8183d934319a.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_item: fix a typo on sections_start_lines

Currently, there are 15 occurrences of section?_start_lines,
with 10 using the plural way.

This is an issue, as, while kdoc_output works with Kdo

docs: kdoc_item: fix a typo on sections_start_lines

Currently, there are 15 occurrences of section?_start_lines,
with 10 using the plural way.

This is an issue, as, while kdoc_output works with KdocItem,
the term doesn't match its init value.

The variable sections_start_lines stores multiple sections,
so placing it in plural is its correct way.

So, ensure that, on all parts of kdoc, this will be referred
as sections_start_lines.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <d1e0f1d3f80df41c11a1bbde6a12fd9468bc3813.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_item: add support to generate a KdocItem from a dict

When reading the contents on a KdocItem using YAML, the data
will be imported into a dict.

Add a method to create a new KdocItem from

docs: kdoc_item: add support to generate a KdocItem from a dict

When reading the contents on a KdocItem using YAML, the data
will be imported into a dict.

Add a method to create a new KdocItem from a dict to allow
converting such input into a real KdocItem.

While here, address an issue that, if the class is initialized
with an internal parameter outside the 4 initial arguments,
it would end being added inside other_stuff, which breaks
initializing it from a dict.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <fafeac23d1577927e1a3c32cddfbec1e0209ac73.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_item: fix initial value for parameterdesc_start_lines

Ensure that parameterdesc_start_lines is a dict at init time,
as this is how it will be set later on at the parser.

Signed-off-by: M

docs: kdoc_item: fix initial value for parameterdesc_start_lines

Ensure that parameterdesc_start_lines is a dict at init time,
as this is how it will be set later on at the parser.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <1b4ea24dd4cd82e6711e9be80168684427d74c30.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_files: move output symbols logic to kdoc_output

When writing unittests for kdoc_output, it became clear that
the logic with handles a series of KdocItem symbols from
a single file belons

docs: kdoc_files: move output symbols logic to kdoc_output

When writing unittests for kdoc_output, it became clear that
the logic with handles a series of KdocItem symbols from
a single file belons to kdoc_output, and not to kdoc_files.

Move the code to it.

While here, also ensure that self.config will be placed
together with set.out_style.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <4ebc26e37a0b544c50d50b8077760f147fa6a535.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_files: use a class to group config parameters

Instead of abusing argparse.Namespace, define a class to store
configuration parameters and logger.

Signed-off-by: Mauro Carvalho Chehab <mc

docs: kdoc_files: use a class to group config parameters

Instead of abusing argparse.Namespace, define a class to store
configuration parameters and logger.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <a66ec9872c72a3ba1a5ac567881d67dc8ee581c6.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc: add c_lex to generated documentation

Do some fixes at groups() description for it to be parsed by
Sphinx and add it to the documentation.

Signed-off-by: Mauro Carvalho Chehab <mchehab+h

docs: kdoc: add c_lex to generated documentation

Do some fixes at groups() description for it to be parsed by
Sphinx and add it to the documentation.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <799178cf30dd4022fdb1d029ba998a458e037b52.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_re: better represent long regular expressions

The Sphinx output from autodoc doesn't automatically break long
lines, except on spaces.

Change KernRe __repr__() to break the pattern on mu

docs: kdoc_re: better represent long regular expressions

The Sphinx output from autodoc doesn't automatically break long
lines, except on spaces.

Change KernRe __repr__() to break the pattern on multiple strings,
each one with a maximum limit of 60 characters.

With that, documentation output for KernRe should now be displayable,
even on long strings.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <60c264a9d277fed655b1a62df2195562c8596090.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: xforms_lists: use CMatch for all identifiers

CMatch is lexically correct and replaces only identifiers,
which is exactly where macro transformations happen.

Use it to make the output safer an

docs: xforms_lists: use CMatch for all identifiers

CMatch is lexically correct and replaces only identifiers,
which is exactly where macro transformations happen.

Use it to make the output safer and ensure that all arguments
will be parsed the right way, even on complex cases.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <86d4a07ff0e054207747fabf38d6bb261b52b5fa.1773770483.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_parser: avoid tokenizing structs everytime

Most of the rules inside CTransforms are of the type CMatch.

Don't re-parse the source code every time.

Doing this doesn't change the output,

docs: kdoc_parser: avoid tokenizing structs everytime

Most of the rules inside CTransforms are of the type CMatch.

Don't re-parse the source code every time.

Doing this doesn't change the output, but makes kdoc almost
as fast as before the tokenizer patches:

# Before tokenizer patches
$ time ./scripts/kernel-doc . -man >original 2>&1

real 0m42.933s
user 0m36.523s
sys 0m1.145s

# After tokenizer patches
$ time ./scripts/kernel-doc . -man >before 2>&1

real 1m29.853s
user 1m23.974s
sys 0m1.237s

# After this patch
$ time ./scripts/kernel-doc . -man >after 2>&1

real 0m48.579s
user 0m45.938s
sys 0m0.988s

$ diff -s before after
Files before and after are identical

Manually checked the differences between original and after
with:

$ diff -U0 -prBw original after|grep -v Warning|grep -v "@@"|less

They're due:
- whitespace fixes;
- struct_group are now better handled;
- several badly-generated man pages from broken inline kernel-doc
markups are now fixed.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <1cc2a4286ebf7d4b2d03fcaf42a1ba9fa09004b9.1773770483.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc: ensure that comments are dropped before calling split_struct_proto()

Changeset 2b957decdb6c ("docs: kdoc: don't add broken comments inside prototypes")
revealed a hidden bug at split_str

docs: kdoc: ensure that comments are dropped before calling split_struct_proto()

Changeset 2b957decdb6c ("docs: kdoc: don't add broken comments inside prototypes")
revealed a hidden bug at split_struct_proto(): some comments there may break
its capability of properly identifying a struct.

Fixing it is as simple as stripping comments before calling it.

Fixes: 2b957decdb6c ("docs: kdoc: don't add broken comments inside prototypes")
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <dcff37b6da5329aea415de31f543b6a1c2cbbbce.1773770483.git.mchehab+huawei@kernel.org>

show more ...

docs: c_lex: setup a logger to report tokenizer issues

Report file that has issues detected via CMatch and CTokenizer.

This is done by setting up a logger that will be overriden by
kdoc_parser, whe

docs: c_lex: setup a logger to report tokenizer issues

Report file that has issues detected via CMatch and CTokenizer.

This is done by setting up a logger that will be overriden by
kdoc_parser, when used on it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <903ad83ae176196a50444e66177a4f5bcdef5199.1773770483.git.mchehab+huawei@kernel.org>

show more ...

docs: xforms_lists: better evaluate struct_group macros

The previous approach were to unwind nested structs/unions.

Now that we have a logic that can handle it well, use it to
ensure that struct_gr

docs: xforms_lists: better evaluate struct_group macros

The previous approach were to unwind nested structs/unions.

Now that we have a logic that can handle it well, use it to
ensure that struct_group macros will properly reflect the
actual struct.

Note that the replacemend logic still simplifies the code
a little bit, as the basic build block for struct group is:

union { \
struct { MEMBERS } ATTRS; \
struct __struct_group_tag(TAG) { MEMBERS } ATTRS NAME; \
} ATTRS

There:

- ATTRS is meant to add extra macro attributes like __packed
which we already discard, as they aren't relevant to
document struct members;

- TAG is used only when built with __cplusplus.

So, instead, convert them into just:

struct { MEMBERS };

Please notice that here, we're using the greedy version of the
backrefs, as MEMBERS is actually MEMBERS... on all such macros.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <24bf2c036b08814d9b4aabc27542fd3b2ff54424.1773770483.git.mchehab+huawei@kernel.org>

show more ...

docs: xforms_lists: handle struct_group directly

The previous logic was handling struct_group on two steps.
Remove the previous approach, as CMatch can do it the right
way on a single step.

Signed-

docs: xforms_lists: handle struct_group directly

The previous logic was handling struct_group on two steps.
Remove the previous approach, as CMatch can do it the right
way on a single step.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <da7f879d90e3ffbc1f47771522f212a60df1fab6.1773770483.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc_re: get rid of NestedMatch class

Now that everything was converted to CMatch, we can get rid of
the previous NestedMatch implementation.

Signed-off-by: Mauro Carvalho Chehab <mchehab+hua

docs: kdoc_re: get rid of NestedMatch class

Now that everything was converted to CMatch, we can get rid of
the previous NestedMatch implementation.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <c82dd0d2c0ab330fc04925965091c448ccabb8fd.1773770483.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc: replace NestedMatch with CMatch

Our previous approach to solve nested structs were to use
NestedMatch. It works well, but adding support to parse delimiters
is very complex.

Instead, us

docs: kdoc: replace NestedMatch with CMatch

Our previous approach to solve nested structs were to use
NestedMatch. It works well, but adding support to parse delimiters
is very complex.

Instead, use CMatch, which uses a C tokenizer, making the code more
reliable and simpler.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <900bff66f8093402999f9fe055fbfa3fa33a8d8b.1773770483.git.mchehab+huawei@kernel.org>

show more ...

docs: c_lex: properly implement a sub() method for CMatch

Implement a sub() method to do what it is expected, parsing
backref arguments like \0, \1, \2, ...

Signed-off-by: Mauro Carvalho Chehab <mc

docs: c_lex: properly implement a sub() method for CMatch

Implement a sub() method to do what it is expected, parsing
backref arguments like \0, \1, \2, ...

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <dbc45b86db18783289d94cfdbba4b72792c47929.1773770483.git.mchehab+huawei@kernel.org>

show more ...