docs: kdoc: better handle source when producing YAML output

The current logic was storing symbols source code on a list,
not linked to the actual KdocItem. While this works fine when
kernel-doc mark

docs: kdoc: better handle source when producing YAML output

The current logic was storing symbols source code on a list,
not linked to the actual KdocItem. While this works fine when
kernel-doc markups are OK, on places where there is a "/**"
without a valid kernel-doc markup, it ends that the 1:1 match
between source code and KdocItem doesn't happen, causing
problems to generate the YAML output.

Fix it by storing the source code directly into the KdocItem
structure.

This shouldn't affect performance or memory footprint, except
when --yaml option is used.

While here, add a __repr__() function for KdocItem, as it
helps debugging it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <77902dafabb5c3250486aa2dc1568d5fafa95c5b.1774256269.git.mchehab+huawei@kernel.org>

show more ...

docs: kdoc-test.yaml: add more tests

Add extra tests to check if the new "var" type is properly
handled and to cover mutex context annotations.

Co-developed-by: Randy Dunlap <rdunlap@infradead.org>

docs: kdoc-test.yaml: add more tests

Add extra tests to check if the new "var" type is properly
handled and to cover mutex context annotations.

Co-developed-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <75af93a45151b630c94b7f77637d173e6119fd41.1774256269.git.mchehab+huawei@kernel.org>

show more ...

unittests: test_kdoc_parser: add command line arg to read a YAML file

The test_kdoc_parser.py already supports loading dynamic tests
when running unit tests.

Add support to read from a different fi

unittests: test_kdoc_parser: add command line arg to read a YAML file

The test_kdoc_parser.py already supports loading dynamic tests
when running unit tests.

Add support to read from a different file. This is useful for:
- regression tests before/afer some changes;
- preparing new unit tests;
- test a different yaml before adding its contents at
tools/unittests/kdoc-test.yaml.

It should be noticed that passing an argument to a unit test
is not too trivial, as unittest core will load itself the
runner with a separate environment. The best (only?) way to
do it is by setting the system environment. This way, when
the class is called by the unit test loader, it can pick
the var from the environment without relying on a global
variable.

The unittest_helper has already provision for it, so let's
use its support.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <1d1a56de012c43756f9ca87aa9bf6c285674f113.1774256269.git.mchehab+huawei@kernel.org>

show more ...

docs: test_kdoc_parser: add support for dynamic test creation

Use the content of kdoc-test.yaml to generate unittests to
verify that kernel-doc internal methods are parsing C code
and generating out

docs: test_kdoc_parser: add support for dynamic test creation

Use the content of kdoc-test.yaml to generate unittests to
verify that kernel-doc internal methods are parsing C code
and generating output the expected way.

Depending on what is written at the parser file at
kdoc-test.yaml, up to 5 tests can be generated from a single
test entry inside the YAML file:

1. from source to kdoc_item: test KernelDoc class;
2. from kdoc_item to man: test ManOutput class;
3. from kdoc_item to rst: test RestOutput class;
4. from source to man without checking expected KdocItem;
5. from source to rst without checking expected KdocItem.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <7ec2593c5b19de3e3b1d8de92675f6b751d3fa21.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: add a simple kdoc-test.yaml together with a validation tool

Create a simple kdoc-test.yaml to be used to create unit tests for
kernel-doc parser and output classes.

For now, all we want is a

docs: add a simple kdoc-test.yaml together with a validation tool

Create a simple kdoc-test.yaml to be used to create unit tests for
kernel-doc parser and output classes.

For now, all we want is a simple function mapped on a yaml test
using the defined schema.

To be sure that the schema is followed, add an unittest for
the file, which will also validate that the schema is properly
parsed.

It should be noticed that the .TH definition for the man format
contains a timestamp. We'll need to handle that when dealing with
the actual implementation for the ManOutput class unit tests.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <74883976348c964f00161696d525c33ddd8c7641.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: add a schema to help creating unittests for kernel-doc

Instead of hardcoding lots of tests inside a file, let's place
them inside a yaml file.

Add first a schema to handle it.

Signed-off-by:

docs: add a schema to help creating unittests for kernel-doc

Instead of hardcoding lots of tests inside a file, let's place
them inside a yaml file.

Add first a schema to handle it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <f42fd816ee0d257c736b30977b10f04f6bea27c2.1773823995.git.mchehab+huawei@kernel.org>

show more ...

docs: unittests: add a parser to test kernel-doc parser logic

Validating that kernel-doc is parsing data properly is tricky.

Add an unittest skeleton that alllows passing a source code
and check if

docs: unittests: add a parser to test kernel-doc parser logic

Validating that kernel-doc is parsing data properly is tricky.

Add an unittest skeleton that alllows passing a source code
and check if the corresponding values of export_table and
entries returned by the parser are properly filled.

It works by mocking a file input with the contents of a source
string, an comparing if:

- exports set matches;
- expected KernelItem entries match.

Create a new TestSelfValidate meant to check if the logic
inside KdocParser.run_test() does its job of checking for
differences inside KdocItem.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <8d91bfabd69de7aa44a0f5080ccb01aa41957e6d.1773823995.git.mchehab+huawei@kernel.org>

show more ...

unittests: test_tokenizer: better handle mismatch error

The current logic is too sensitive to how c_lex name is
placed. Also, it doesn't really check the log.

Change it to check if the expected mes

unittests: test_tokenizer: better handle mismatch error

The current logic is too sensitive to how c_lex name is
placed. Also, it doesn't really check the log.

Change it to check if the expected message will be reported
after a call to C tokenizer with an invalid source.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <6e19578bc1ffa96e536dc31997ff658017f60173.1773823995.git.mchehab+huawei@kernel.org>

show more ...

unittests: test_cmatch: add tests for sub()

Now that we have code for sub(), test it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net

unittests: test_cmatch: add tests for sub()

Now that we have code for sub(), test it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <b2621b0d378317f0db5eeb2408c794429dc9f70a.1773770483.git.mchehab+huawei@kernel.org>

show more ...

tools: unittests: add tests for CMatch

The CMatch logic is complex enough to justify tests to ensure
that it is doing its job.

Add unittests to check the functionality provided by CMatch
by replica

tools: unittests: add tests for CMatch

The CMatch logic is complex enough to justify tests to ensure
that it is doing its job.

Add unittests to check the functionality provided by CMatch
by replicating expected patterns.

The CMatch class handles with complex macros. Add an unittest
to check if its doing the right thing and detect eventual regressions
as we improve its code.

The initial version was generated using gpt-oss:latest LLM
on my local GPU, as LLMs aren't bad transforming patterns
into unittests.

Yet, the curent version contains only the skeleton of what
LLM produced, as I ended higly changing its content to be
more representative and to have real case scenarios.

The kdoc_xforms test suite contains 3 test groups. Two of
them tests the basic functionality of CMatch to
replace patterns.

The last one (TestRealUsecases) contains real code snippets
from the Kernel with some cleanups to better fit in 80 columns
and uses the same transforms as kernel-doc, thus allowing
to test the logic used inside kdoc_parser to transform
functions, structs and variable patterns.

Its output is like this:

$ tools/unittests/kdoc_xforms.py
Ran 25 tests in 0.003s

OK
test_cmatch:
TestSearch:
test_search_acquires_multiple: OK
test_search_acquires_nested_paren: OK
test_search_acquires_simple: OK
test_search_must_hold: OK
test_search_must_hold_shared: OK
test_search_no_false_positive: OK
test_search_no_function: OK
test_search_no_macro_remains: OK

Ran 8 tests

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <119712b5bc53b4c6dda6a81b4a783dcbfd1d970d.1773770483.git.mchehab+huawei@kernel.org>

show more ...

unittests: add a runner to execute all unittests

We'll soon have multiple unit tests, add a runner that will
discover all of them and execute all tests.

It was opted to discover only files that sta

unittests: add a runner to execute all unittests

We'll soon have multiple unit tests, add a runner that will
discover all of them and execute all tests.

It was opted to discover only files that starts with "test",
as this way unittest discover won't try adding libraries or
other stuff that might not contain unittest classes.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <2d9dd14f03d3d6394346fdaceeb3167d54d1dd0c.1773770483.git.mchehab+huawei@kernel.org>

show more ...

unittests: test_tokenizer: check if the tokenizer works

Add extra tests to check if the tokenizer is working properly.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by

unittests: test_tokenizer: check if the tokenizer works

Add extra tests to check if the tokenizer is working properly.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <e4b6e880565f40287a69667652adb393879e039b.1773770483.git.mchehab+huawei@kernel.org>

show more ...

unittests: test_private: modify it to use CTokenizer directly

Change the logic to use the tokenizer directly. This allows
adding more unit tests to check the validty of the tokenizer
itself.

Signed

unittests: test_private: modify it to use CTokenizer directly

Change the logic to use the tokenizer directly. This allows
adding more unit tests to check the validty of the tokenizer
itself.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <2672257233ff73a9464c09b50924be51e25d4f59.1773074166.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <66e6320a4d5ad9730c1c0ceea79b5021e90c66c6.1773770483.git.mchehab+huawei@kernel.org>

show more ...

unittests: add a testbench to check public/private kdoc comments

Add unit tests to check if the public/private and comments strip
is working properly.

Running it shows that, on several cases, publi

unittests: add a testbench to check public/private kdoc comments

Add unit tests to check if the public/private and comments strip
is working properly.

Running it shows that, on several cases, public/private is not
doing what it is expected:

test_private:
TestPublicPrivate:
test balanced_inner_private: OK
test balanced_non_greddy_private: OK
test balanced_private: OK
test no private: OK
test unbalanced_inner_private: FAIL
test unbalanced_private: FAIL
test unbalanced_struct_group_tagged_with_private: FAIL
test unbalanced_two_struct_group_tagged_first_with_private: FAIL
test unbalanced_without_end_of_line: FAIL

Ran 9 tests

FAILED (failures=5)

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <144f4952e0cb74fe9c9adc117e9a21ec8aa1cc10.1773074166.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <c1574b5b67b0442de3fab570fe6de9bc28b17a75.1773770483.git.mchehab+huawei@kernel.org>

show more ...