1.. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2.. Copyright 2010 Julia Lawall <julia@diku.dk> 3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5.. highlight:: none 6 7.. _devtools_coccinelle: 8 9Coccinelle 10========== 11 12Coccinelle is a tool for pattern matching and text transformation that has 13many uses in kernel development, including the application of complex, 14tree-wide patches and detection of problematic programming patterns. 15 16Getting Coccinelle 17------------------ 18 19The semantic patches included in the kernel use features and options 20which are provided by Coccinelle version 1.0.0-rc11 and above. 21Using earlier versions will fail as the option names used by 22the Coccinelle files and coccicheck have been updated. 23 24Coccinelle is available through the package manager 25of many distributions, e.g. : 26 27 - Debian 28 - Fedora 29 - Ubuntu 30 - OpenSUSE 31 - Arch Linux 32 - Gentoo 33 - NetBSD 34 - FreeBSD 35 36Some distribution packages are obsolete and it is recommended 37to use the latest version released from the Coccinelle homepage at 38https://coccinelle.gitlabpages.inria.fr/website 39 40Or from Github at: 41 42https://github.com/coccinelle/coccinelle 43 44Once you have it, run the following commands:: 45 46 ./autogen 47 ./configure 48 make 49 50as a regular user, and install it with:: 51 52 sudo make install 53 54More detailed installation instructions to build from source can be 55found at: 56 57https://github.com/coccinelle/coccinelle/blob/master/install.txt 58 59Supplemental documentation 60-------------------------- 61 62For supplemental documentation refer to the wiki: 63 64https://bottest.wiki.kernel.org/coccicheck.html 65 66The wiki documentation always refers to the linux-next version of the script. 67 68For Semantic Patch Language(SmPL) grammar documentation refer to: 69 70https://coccinelle.gitlabpages.inria.fr/website/docs/main_grammar.html 71 72Using Coccinelle on the Linux kernel 73------------------------------------ 74 75A Coccinelle-specific target is defined in the top level 76Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 77front-end in the ``scripts`` directory. 78 79Four basic modes are defined: ``patch``, ``report``, ``context``, and 80``org``. The mode to use is specified by setting the MODE variable with 81``MODE=<mode>``. 82 83- ``patch`` proposes a fix, when possible. 84 85- ``report`` generates a list in the following format: 86 file:line:column-column: message 87 88- ``context`` highlights lines of interest and their context in a 89 diff-like style. Lines of interest are indicated with ``-``. 90 91- ``org`` generates a report in the Org mode format of Emacs. 92 93Note that not all semantic patches implement all modes. For easy use 94of Coccinelle, the default mode is "report". 95 96Two other modes provide some common combinations of these modes. 97 98- ``chain`` tries the previous modes in the order above until one succeeds. 99 100- ``rep+ctxt`` runs successively the report mode and the context mode. 101 It should be used with the C option (described later) 102 which checks the code on a file basis. 103 104Examples 105~~~~~~~~ 106 107To make a report for every semantic patch, run the following command:: 108 109 make coccicheck MODE=report 110 111To produce patches, run:: 112 113 make coccicheck MODE=patch 114 115 116The coccicheck target applies every semantic patch available in the 117sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 118 119For each semantic patch, a commit message is proposed. It gives a 120description of the problem being checked by the semantic patch, and 121includes a reference to Coccinelle. 122 123As with any static code analyzer, Coccinelle produces false 124positives. Thus, reports must be carefully checked, and patches 125reviewed. 126 127To enable verbose messages set the V= variable, for example:: 128 129 make coccicheck MODE=report V=1 130 131By default, coccicheck will print debug logs to stdout and redirect stderr to 132/dev/null. This can make coccicheck output difficult to read and understand. 133Debug and error messages can instead be written to a debug file instead by 134setting the ``DEBUG_FILE`` variable:: 135 136 make coccicheck MODE=report DEBUG_FILE="cocci.log" 137 138Coccinelle cannot overwrite a debug file. Instead of repeatedly deleting a log 139file, you could include the datetime in the debug file name:: 140 141 make coccicheck MODE=report DEBUG_FILE="cocci-$(date -Iseconds).log" 142 143Coccinelle parallelization 144-------------------------- 145 146By default, coccicheck tries to run as parallel as possible. To change 147the parallelism, set the J= variable. For example, to run across 4 CPUs:: 148 149 make coccicheck MODE=report J=4 150 151As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization; 152if support for this is detected you will benefit from parmap parallelization. 153 154When parmap is enabled coccicheck will enable dynamic load balancing by using 155``--chunksize 1`` argument. This ensures we keep feeding threads with work 156one by one, so that we avoid the situation where most work gets done by only 157a few threads. With dynamic load balancing, if a thread finishes early we keep 158feeding it more work. 159 160When parmap is enabled, if an error occurs in Coccinelle, this error 161value is propagated back, and the return value of the ``make coccicheck`` 162command captures this return value. 163 164Using Coccinelle with a single semantic patch 165--------------------------------------------- 166 167The optional make variable COCCI can be used to check a single 168semantic patch. In that case, the variable must be initialized with 169the name of the semantic patch to apply. 170 171For instance:: 172 173 make coccicheck COCCI=<my_SP.cocci> MODE=patch 174 175or:: 176 177 make coccicheck COCCI=<my_SP.cocci> MODE=report 178 179 180Controlling Which Files are Processed by Coccinelle 181--------------------------------------------------- 182 183By default the entire kernel source tree is checked. 184 185To apply Coccinelle to a specific directory, ``M=`` can be used. 186For example, to check drivers/net/wireless/ one may write:: 187 188 make coccicheck M=drivers/net/wireless/ 189 190To apply Coccinelle on a file basis, instead of a directory basis, the 191C variable is used by the makefile to select which files to work with. 192This variable can be used to run scripts for the entire kernel, a 193specific directory, or for a single file. 194 195For example, to check drivers/bluetooth/bfusb.c, the value 1 is 196passed to the C variable to check files that make considers 197need to be compiled.:: 198 199 make C=1 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o 200 201The value 2 is passed to the C variable to check files regardless of 202whether they need to be compiled or not.:: 203 204 make C=2 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o 205 206In these modes, which work on a file basis, there is no information 207about semantic patches displayed, and no commit message proposed. 208 209This runs every semantic patch in scripts/coccinelle by default. The 210COCCI variable may additionally be used to only apply a single 211semantic patch as shown in the previous section. 212 213The "report" mode is the default. You can select another one with the 214MODE variable explained above. 215 216Debugging Coccinelle SmPL patches 217--------------------------------- 218 219Using coccicheck is best as it provides in the spatch command line 220include options matching the options used when we compile the kernel. 221You can learn what these options are by using V=1; you could then 222manually run Coccinelle with debug options added. 223 224An easier approach to debug running Coccinelle against SmPL patches is to ask 225coccicheck to redirect stderr to a debug file. As mentioned in the examples, by 226default stderr is redirected to /dev/null; if you'd like to capture stderr you 227can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For instance:: 228 229 rm -f cocci.err 230 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 231 cat cocci.err 232 233You can use SPFLAGS to add debugging flags; for instance you may want to 234add both ``--profile --show-trying`` to SPFLAGS when debugging. For example 235you may want to use:: 236 237 rm -f err.log 238 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 239 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd 240 241err.log will now have the profiling information, while stdout will 242provide some progress information as Coccinelle moves forward with 243work. 244 245NOTE: 246 247DEBUG_FILE support is only supported when using coccinelle >= 1.0.2. 248 249Currently, DEBUG_FILE support is only available to check folders, and 250not single files. This is because checking a single file requires spatch 251to be called twice leading to DEBUG_FILE being set both times to the same value, 252giving rise to an error. 253 254.cocciconfig support 255-------------------- 256 257Coccinelle supports reading .cocciconfig for default Coccinelle options that 258should be used every time spatch is spawned. The order of precedence for 259variables for .cocciconfig is as follows: 260 261- Your current user's home directory is processed first 262- Your directory from which spatch is called is processed next 263- The directory provided with the ``--dir`` option is processed last, if used 264 265``make coccicheck`` also supports using M= targets. If you do not supply 266any M= target, it is assumed you want to target the entire kernel. 267The kernel coccicheck script has:: 268 269 OPTIONS="--dir $srcroot $COCCIINCLUDE" 270 271Here, $srcroot refers to the source directory of the target: it points to the 272external module's source directory when M= used, and otherwise, to the kernel 273source directory. The third rule ensures the spatch reads the .cocciconfig from 274the target directory, allowing external modules to have their own .cocciconfig 275file. 276 277If not using the kernel's coccicheck target, keep the above precedence 278order logic of .cocciconfig reading. If using the kernel's coccicheck target, 279override any of the kernel's .coccicheck's settings using SPFLAGS. 280 281We help Coccinelle when used against Linux with a set of sensible default 282options for Linux with our own Linux .cocciconfig. This hints to coccinelle 283that git can be used for ``git grep`` queries over coccigrep. A timeout of 200 284seconds should suffice for now. 285 286The options picked up by coccinelle when reading a .cocciconfig do not appear 287as arguments to spatch processes running on your system. To confirm what 288options will be used by Coccinelle run:: 289 290 spatch --print-options-only 291 292You can override with your own preferred index option by using SPFLAGS. Take 293note that when there are conflicting options Coccinelle takes precedence for 294the last options passed. Using .cocciconfig is possible to use idutils, however 295given the order of precedence followed by Coccinelle, since the kernel now 296carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 297desired. See below section "Additional flags" for more details on how to use 298idutils. 299 300Additional flags 301---------------- 302 303Additional flags can be passed to spatch through the SPFLAGS 304variable. This works as Coccinelle respects the last flags 305given to it when options are in conflict. :: 306 307 make SPFLAGS=--use-glimpse coccicheck 308 309Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 310When no ID file is specified coccinelle assumes your ID database file 311is in the file .id-utils.index on the top level of the kernel. Coccinelle 312carries a script scripts/idutils_index.sh which creates the database with:: 313 314 mkid -i C --output .id-utils.index 315 316If you have another database filename you can also just symlink with this 317name. :: 318 319 make SPFLAGS=--use-idutils coccicheck 320 321Alternatively you can specify the database filename explicitly, for 322instance:: 323 324 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 325 326See ``spatch --help`` to learn more about spatch options. 327 328Note that the ``--use-glimpse`` and ``--use-idutils`` options 329require external tools for indexing the code. None of them is 330thus active by default. However, by indexing the code with 331one of these tools, and according to the cocci file used, 332spatch could proceed the entire code base more quickly. 333 334SmPL patch specific options 335--------------------------- 336 337SmPL patches can have their own requirements for options passed 338to Coccinelle. SmPL patch-specific options can be provided by 339providing them at the top of the SmPL patch, for instance:: 340 341 // Options: --no-includes --include-headers 342 343SmPL patch Coccinelle requirements 344---------------------------------- 345 346As Coccinelle features get added some more advanced SmPL patches 347may require newer versions of Coccinelle. If an SmPL patch requires 348a minimum version of Coccinelle, this can be specified as follows, 349as an example if requiring at least Coccinelle >= 1.0.5:: 350 351 // Requires: 1.0.5 352 353Proposing new semantic patches 354------------------------------ 355 356New semantic patches can be proposed and submitted by kernel 357developers. For sake of clarity, they should be organized in the 358sub-directories of ``scripts/coccinelle/``. 359 360 361Detailed description of the ``report`` mode 362------------------------------------------- 363 364``report`` generates a list in the following format:: 365 366 file:line:column-column: message 367 368Example 369~~~~~~~ 370 371Running:: 372 373 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 374 375will execute the following part of the SmPL script:: 376 377 <smpl> 378 @r depends on !context && !patch && (org || report)@ 379 expression x; 380 position p; 381 @@ 382 383 ERR_PTR@p(PTR_ERR(x)) 384 385 @script:python depends on report@ 386 p << r.p; 387 x << r.x; 388 @@ 389 390 msg="ERR_CAST can be used with %s" % (x) 391 coccilib.report.print_report(p[0], msg) 392 </smpl> 393 394This SmPL excerpt generates entries on the standard output, as 395illustrated below:: 396 397 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 398 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 399 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 400 401 402Detailed description of the ``patch`` mode 403------------------------------------------ 404 405When the ``patch`` mode is available, it proposes a fix for each problem 406identified. 407 408Example 409~~~~~~~ 410 411Running:: 412 413 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 414 415will execute the following part of the SmPL script:: 416 417 <smpl> 418 @ depends on !context && patch && !org && !report @ 419 expression x; 420 @@ 421 422 - ERR_PTR(PTR_ERR(x)) 423 + ERR_CAST(x) 424 </smpl> 425 426This SmPL excerpt generates patch hunks on the standard output, as 427illustrated below:: 428 429 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 430 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 431 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 432 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 433 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 434 CRYPTO_ALG_TYPE_MASK); 435 if (IS_ERR(alg)) 436 - return ERR_PTR(PTR_ERR(alg)); 437 + return ERR_CAST(alg); 438 439 /* Block size must be >= 4 bytes. */ 440 err = -EINVAL; 441 442Detailed description of the ``context`` mode 443-------------------------------------------- 444 445``context`` highlights lines of interest and their context 446in a diff-like style. 447 448 **NOTE**: The diff-like output generated is NOT an applicable patch. The 449 intent of the ``context`` mode is to highlight the important lines 450 (annotated with minus, ``-``) and gives some surrounding context 451 lines around. This output can be used with the diff mode of 452 Emacs to review the code. 453 454Example 455~~~~~~~ 456 457Running:: 458 459 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 460 461will execute the following part of the SmPL script:: 462 463 <smpl> 464 @ depends on context && !patch && !org && !report@ 465 expression x; 466 @@ 467 468 * ERR_PTR(PTR_ERR(x)) 469 </smpl> 470 471This SmPL excerpt generates diff hunks on the standard output, as 472illustrated below:: 473 474 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 475 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 476 +++ /tmp/nothing 477 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 478 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 479 CRYPTO_ALG_TYPE_MASK); 480 if (IS_ERR(alg)) 481 - return ERR_PTR(PTR_ERR(alg)); 482 483 /* Block size must be >= 4 bytes. */ 484 err = -EINVAL; 485 486Detailed description of the ``org`` mode 487---------------------------------------- 488 489``org`` generates a report in the Org mode format of Emacs. 490 491Example 492~~~~~~~ 493 494Running:: 495 496 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 497 498will execute the following part of the SmPL script:: 499 500 <smpl> 501 @r depends on !context && !patch && (org || report)@ 502 expression x; 503 position p; 504 @@ 505 506 ERR_PTR@p(PTR_ERR(x)) 507 508 @script:python depends on org@ 509 p << r.p; 510 x << r.x; 511 @@ 512 513 msg="ERR_CAST can be used with %s" % (x) 514 msg_safe=msg.replace("[","@(").replace("]",")") 515 coccilib.org.print_todo(p[0], msg_safe) 516 </smpl> 517 518This SmPL excerpt generates Org entries on the standard output, as 519illustrated below:: 520 521 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 522 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 523 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 524