1.. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2.. Copyright 2010 Julia Lawall <julia@diku.dk> 3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5.. highlight:: none 6 7.. _devtools_coccinelle: 8 9Coccinelle 10========== 11 12Coccinelle is a tool for pattern matching and text transformation that has 13many uses in kernel development, including the application of complex, 14tree-wide patches and detection of problematic programming patterns. 15 16Getting Coccinelle 17------------------ 18 19The semantic patches included in the kernel use features and options 20which are provided by Coccinelle version 1.0.0-rc11 and above. 21Using earlier versions will fail as the option names used by 22the Coccinelle files and coccicheck have been updated. 23 24Coccinelle is available through the package manager 25of many distributions, e.g. : 26 27 - Debian 28 - Fedora 29 - Ubuntu 30 - OpenSUSE 31 - Arch Linux 32 - NetBSD 33 - FreeBSD 34 35Some distribution packages are obsolete and it is recommended 36to use the latest version released from the Coccinelle homepage at 37http://coccinelle.lip6.fr/ 38 39Or from Github at: 40 41https://github.com/coccinelle/coccinelle 42 43Once you have it, run the following commands:: 44 45 ./autogen 46 ./configure 47 make 48 49as a regular user, and install it with:: 50 51 sudo make install 52 53More detailed installation instructions to build from source can be 54found at: 55 56https://github.com/coccinelle/coccinelle/blob/master/install.txt 57 58Supplemental documentation 59-------------------------- 60 61For supplemental documentation refer to the wiki: 62 63https://bottest.wiki.kernel.org/coccicheck 64 65The wiki documentation always refers to the linux-next version of the script. 66 67For Semantic Patch Language(SmPL) grammar documentation refer to: 68 69https://coccinelle.gitlabpages.inria.fr/website/docs/main_grammar.html 70 71Using Coccinelle on the Linux kernel 72------------------------------------ 73 74A Coccinelle-specific target is defined in the top level 75Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 76front-end in the ``scripts`` directory. 77 78Four basic modes are defined: ``patch``, ``report``, ``context``, and 79``org``. The mode to use is specified by setting the MODE variable with 80``MODE=<mode>``. 81 82- ``patch`` proposes a fix, when possible. 83 84- ``report`` generates a list in the following format: 85 file:line:column-column: message 86 87- ``context`` highlights lines of interest and their context in a 88 diff-like style. Lines of interest are indicated with ``-``. 89 90- ``org`` generates a report in the Org mode format of Emacs. 91 92Note that not all semantic patches implement all modes. For easy use 93of Coccinelle, the default mode is "report". 94 95Two other modes provide some common combinations of these modes. 96 97- ``chain`` tries the previous modes in the order above until one succeeds. 98 99- ``rep+ctxt`` runs successively the report mode and the context mode. 100 It should be used with the C option (described later) 101 which checks the code on a file basis. 102 103Examples 104~~~~~~~~ 105 106To make a report for every semantic patch, run the following command:: 107 108 make coccicheck MODE=report 109 110To produce patches, run:: 111 112 make coccicheck MODE=patch 113 114 115The coccicheck target applies every semantic patch available in the 116sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 117 118For each semantic patch, a commit message is proposed. It gives a 119description of the problem being checked by the semantic patch, and 120includes a reference to Coccinelle. 121 122As with any static code analyzer, Coccinelle produces false 123positives. Thus, reports must be carefully checked, and patches 124reviewed. 125 126To enable verbose messages set the V= variable, for example:: 127 128 make coccicheck MODE=report V=1 129 130Coccinelle parallelization 131-------------------------- 132 133By default, coccicheck tries to run as parallel as possible. To change 134the parallelism, set the J= variable. For example, to run across 4 CPUs:: 135 136 make coccicheck MODE=report J=4 137 138As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization; 139if support for this is detected you will benefit from parmap parallelization. 140 141When parmap is enabled coccicheck will enable dynamic load balancing by using 142``--chunksize 1`` argument. This ensures we keep feeding threads with work 143one by one, so that we avoid the situation where most work gets done by only 144a few threads. With dynamic load balancing, if a thread finishes early we keep 145feeding it more work. 146 147When parmap is enabled, if an error occurs in Coccinelle, this error 148value is propagated back, and the return value of the ``make coccicheck`` 149command captures this return value. 150 151Using Coccinelle with a single semantic patch 152--------------------------------------------- 153 154The optional make variable COCCI can be used to check a single 155semantic patch. In that case, the variable must be initialized with 156the name of the semantic patch to apply. 157 158For instance:: 159 160 make coccicheck COCCI=<my_SP.cocci> MODE=patch 161 162or:: 163 164 make coccicheck COCCI=<my_SP.cocci> MODE=report 165 166 167Controlling Which Files are Processed by Coccinelle 168--------------------------------------------------- 169 170By default the entire kernel source tree is checked. 171 172To apply Coccinelle to a specific directory, ``M=`` can be used. 173For example, to check drivers/net/wireless/ one may write:: 174 175 make coccicheck M=drivers/net/wireless/ 176 177To apply Coccinelle on a file basis, instead of a directory basis, the 178C variable is used by the makefile to select which files to work with. 179This variable can be used to run scripts for the entire kernel, a 180specific directory, or for a single file. 181 182For example, to check drivers/bluetooth/bfusb.c, the value 1 is 183passed to the C variable to check files that make considers 184need to be compiled.:: 185 186 make C=1 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o 187 188The value 2 is passed to the C variable to check files regardless of 189whether they need to be compiled or not.:: 190 191 make C=2 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o 192 193In these modes, which work on a file basis, there is no information 194about semantic patches displayed, and no commit message proposed. 195 196This runs every semantic patch in scripts/coccinelle by default. The 197COCCI variable may additionally be used to only apply a single 198semantic patch as shown in the previous section. 199 200The "report" mode is the default. You can select another one with the 201MODE variable explained above. 202 203Debugging Coccinelle SmPL patches 204--------------------------------- 205 206Using coccicheck is best as it provides in the spatch command line 207include options matching the options used when we compile the kernel. 208You can learn what these options are by using V=1; you could then 209manually run Coccinelle with debug options added. 210 211Alternatively you can debug running Coccinelle against SmPL patches 212by asking for stderr to be redirected to stderr. By default stderr 213is redirected to /dev/null; if you'd like to capture stderr you 214can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For 215instance:: 216 217 rm -f cocci.err 218 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 219 cat cocci.err 220 221You can use SPFLAGS to add debugging flags; for instance you may want to 222add both ``--profile --show-trying`` to SPFLAGS when debugging. For example 223you may want to use:: 224 225 rm -f err.log 226 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 227 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd 228 229err.log will now have the profiling information, while stdout will 230provide some progress information as Coccinelle moves forward with 231work. 232 233NOTE: 234 235DEBUG_FILE support is only supported when using coccinelle >= 1.0.2. 236 237Currently, DEBUG_FILE support is only available to check folders, and 238not single files. This is because checking a single file requires spatch 239to be called twice leading to DEBUG_FILE being set both times to the same value, 240giving rise to an error. 241 242.cocciconfig support 243-------------------- 244 245Coccinelle supports reading .cocciconfig for default Coccinelle options that 246should be used every time spatch is spawned. The order of precedence for 247variables for .cocciconfig is as follows: 248 249- Your current user's home directory is processed first 250- Your directory from which spatch is called is processed next 251- The directory provided with the ``--dir`` option is processed last, if used 252 253``make coccicheck`` also supports using M= targets. If you do not supply 254any M= target, it is assumed you want to target the entire kernel. 255The kernel coccicheck script has:: 256 257 OPTIONS="--dir $srcroot $COCCIINCLUDE" 258 259Here, $srcroot refers to the source directory of the target: it points to the 260external module's source directory when M= used, and otherwise, to the kernel 261source directory. The third rule ensures the spatch reads the .cocciconfig from 262the target directory, allowing external modules to have their own .cocciconfig 263file. 264 265If not using the kernel's coccicheck target, keep the above precedence 266order logic of .cocciconfig reading. If using the kernel's coccicheck target, 267override any of the kernel's .coccicheck's settings using SPFLAGS. 268 269We help Coccinelle when used against Linux with a set of sensible default 270options for Linux with our own Linux .cocciconfig. This hints to coccinelle 271that git can be used for ``git grep`` queries over coccigrep. A timeout of 200 272seconds should suffice for now. 273 274The options picked up by coccinelle when reading a .cocciconfig do not appear 275as arguments to spatch processes running on your system. To confirm what 276options will be used by Coccinelle run:: 277 278 spatch --print-options-only 279 280You can override with your own preferred index option by using SPFLAGS. Take 281note that when there are conflicting options Coccinelle takes precedence for 282the last options passed. Using .cocciconfig is possible to use idutils, however 283given the order of precedence followed by Coccinelle, since the kernel now 284carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 285desired. See below section "Additional flags" for more details on how to use 286idutils. 287 288Additional flags 289---------------- 290 291Additional flags can be passed to spatch through the SPFLAGS 292variable. This works as Coccinelle respects the last flags 293given to it when options are in conflict. :: 294 295 make SPFLAGS=--use-glimpse coccicheck 296 297Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 298When no ID file is specified coccinelle assumes your ID database file 299is in the file .id-utils.index on the top level of the kernel. Coccinelle 300carries a script scripts/idutils_index.sh which creates the database with:: 301 302 mkid -i C --output .id-utils.index 303 304If you have another database filename you can also just symlink with this 305name. :: 306 307 make SPFLAGS=--use-idutils coccicheck 308 309Alternatively you can specify the database filename explicitly, for 310instance:: 311 312 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 313 314See ``spatch --help`` to learn more about spatch options. 315 316Note that the ``--use-glimpse`` and ``--use-idutils`` options 317require external tools for indexing the code. None of them is 318thus active by default. However, by indexing the code with 319one of these tools, and according to the cocci file used, 320spatch could proceed the entire code base more quickly. 321 322SmPL patch specific options 323--------------------------- 324 325SmPL patches can have their own requirements for options passed 326to Coccinelle. SmPL patch-specific options can be provided by 327providing them at the top of the SmPL patch, for instance:: 328 329 // Options: --no-includes --include-headers 330 331SmPL patch Coccinelle requirements 332---------------------------------- 333 334As Coccinelle features get added some more advanced SmPL patches 335may require newer versions of Coccinelle. If an SmPL patch requires 336a minimum version of Coccinelle, this can be specified as follows, 337as an example if requiring at least Coccinelle >= 1.0.5:: 338 339 // Requires: 1.0.5 340 341Proposing new semantic patches 342------------------------------ 343 344New semantic patches can be proposed and submitted by kernel 345developers. For sake of clarity, they should be organized in the 346sub-directories of ``scripts/coccinelle/``. 347 348 349Detailed description of the ``report`` mode 350------------------------------------------- 351 352``report`` generates a list in the following format:: 353 354 file:line:column-column: message 355 356Example 357~~~~~~~ 358 359Running:: 360 361 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 362 363will execute the following part of the SmPL script:: 364 365 <smpl> 366 @r depends on !context && !patch && (org || report)@ 367 expression x; 368 position p; 369 @@ 370 371 ERR_PTR@p(PTR_ERR(x)) 372 373 @script:python depends on report@ 374 p << r.p; 375 x << r.x; 376 @@ 377 378 msg="ERR_CAST can be used with %s" % (x) 379 coccilib.report.print_report(p[0], msg) 380 </smpl> 381 382This SmPL excerpt generates entries on the standard output, as 383illustrated below:: 384 385 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 386 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 387 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 388 389 390Detailed description of the ``patch`` mode 391------------------------------------------ 392 393When the ``patch`` mode is available, it proposes a fix for each problem 394identified. 395 396Example 397~~~~~~~ 398 399Running:: 400 401 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 402 403will execute the following part of the SmPL script:: 404 405 <smpl> 406 @ depends on !context && patch && !org && !report @ 407 expression x; 408 @@ 409 410 - ERR_PTR(PTR_ERR(x)) 411 + ERR_CAST(x) 412 </smpl> 413 414This SmPL excerpt generates patch hunks on the standard output, as 415illustrated below:: 416 417 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 418 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 419 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 420 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 421 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 422 CRYPTO_ALG_TYPE_MASK); 423 if (IS_ERR(alg)) 424 - return ERR_PTR(PTR_ERR(alg)); 425 + return ERR_CAST(alg); 426 427 /* Block size must be >= 4 bytes. */ 428 err = -EINVAL; 429 430Detailed description of the ``context`` mode 431-------------------------------------------- 432 433``context`` highlights lines of interest and their context 434in a diff-like style. 435 436 **NOTE**: The diff-like output generated is NOT an applicable patch. The 437 intent of the ``context`` mode is to highlight the important lines 438 (annotated with minus, ``-``) and gives some surrounding context 439 lines around. This output can be used with the diff mode of 440 Emacs to review the code. 441 442Example 443~~~~~~~ 444 445Running:: 446 447 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 448 449will execute the following part of the SmPL script:: 450 451 <smpl> 452 @ depends on context && !patch && !org && !report@ 453 expression x; 454 @@ 455 456 * ERR_PTR(PTR_ERR(x)) 457 </smpl> 458 459This SmPL excerpt generates diff hunks on the standard output, as 460illustrated below:: 461 462 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 463 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 464 +++ /tmp/nothing 465 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 466 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 467 CRYPTO_ALG_TYPE_MASK); 468 if (IS_ERR(alg)) 469 - return ERR_PTR(PTR_ERR(alg)); 470 471 /* Block size must be >= 4 bytes. */ 472 err = -EINVAL; 473 474Detailed description of the ``org`` mode 475---------------------------------------- 476 477``org`` generates a report in the Org mode format of Emacs. 478 479Example 480~~~~~~~ 481 482Running:: 483 484 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 485 486will execute the following part of the SmPL script:: 487 488 <smpl> 489 @r depends on !context && !patch && (org || report)@ 490 expression x; 491 position p; 492 @@ 493 494 ERR_PTR@p(PTR_ERR(x)) 495 496 @script:python depends on org@ 497 p << r.p; 498 x << r.x; 499 @@ 500 501 msg="ERR_CAST can be used with %s" % (x) 502 msg_safe=msg.replace("[","@(").replace("]",")") 503 coccilib.org.print_todo(p[0], msg_safe) 504 </smpl> 505 506This SmPL excerpt generates Org entries on the standard output, as 507illustrated below:: 508 509 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 510 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 511 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 512