1.. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2.. Copyright 2010 Julia Lawall <julia@diku.dk> 3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5.. highlight:: none 6 7.. _devtools_coccinelle: 8 9Coccinelle 10========== 11 12Coccinelle is a tool for pattern matching and text transformation that has 13many uses in kernel development, including the application of complex, 14tree-wide patches and detection of problematic programming patterns. 15 16Getting Coccinelle 17------------------ 18 19The semantic patches included in the kernel use features and options 20which are provided by Coccinelle version 1.0.0-rc11 and above. 21Using earlier versions will fail as the option names used by 22the Coccinelle files and coccicheck have been updated. 23 24Coccinelle is available through the package manager 25of many distributions, e.g. : 26 27 - Debian 28 - Fedora 29 - Ubuntu 30 - OpenSUSE 31 - Arch Linux 32 - NetBSD 33 - FreeBSD 34 35Some distribution packages are obsolete and it is recommended 36to use the latest version released from the Coccinelle homepage at 37http://coccinelle.lip6.fr/ 38 39Or from Github at: 40 41https://github.com/coccinelle/coccinelle 42 43Once you have it, run the following commands:: 44 45 ./autogen 46 ./configure 47 make 48 49as a regular user, and install it with:: 50 51 sudo make install 52 53More detailed installation instructions to build from source can be 54found at: 55 56https://github.com/coccinelle/coccinelle/blob/master/install.txt 57 58Supplemental documentation 59-------------------------- 60 61For supplemental documentation refer to the wiki: 62 63https://bottest.wiki.kernel.org/coccicheck 64 65The wiki documentation always refers to the linux-next version of the script. 66 67For Semantic Patch Language(SmPL) grammar documentation refer to: 68 69https://coccinelle.gitlabpages.inria.fr/website/docs/main_grammar.html 70 71Using Coccinelle on the Linux kernel 72------------------------------------ 73 74A Coccinelle-specific target is defined in the top level 75Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 76front-end in the ``scripts`` directory. 77 78Four basic modes are defined: ``patch``, ``report``, ``context``, and 79``org``. The mode to use is specified by setting the MODE variable with 80``MODE=<mode>``. 81 82- ``patch`` proposes a fix, when possible. 83 84- ``report`` generates a list in the following format: 85 file:line:column-column: message 86 87- ``context`` highlights lines of interest and their context in a 88 diff-like style. Lines of interest are indicated with ``-``. 89 90- ``org`` generates a report in the Org mode format of Emacs. 91 92Note that not all semantic patches implement all modes. For easy use 93of Coccinelle, the default mode is "report". 94 95Two other modes provide some common combinations of these modes. 96 97- ``chain`` tries the previous modes in the order above until one succeeds. 98 99- ``rep+ctxt`` runs successively the report mode and the context mode. 100 It should be used with the C option (described later) 101 which checks the code on a file basis. 102 103Examples 104~~~~~~~~ 105 106To make a report for every semantic patch, run the following command:: 107 108 make coccicheck MODE=report 109 110To produce patches, run:: 111 112 make coccicheck MODE=patch 113 114 115The coccicheck target applies every semantic patch available in the 116sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 117 118For each semantic patch, a commit message is proposed. It gives a 119description of the problem being checked by the semantic patch, and 120includes a reference to Coccinelle. 121 122As with any static code analyzer, Coccinelle produces false 123positives. Thus, reports must be carefully checked, and patches 124reviewed. 125 126To enable verbose messages set the V= variable, for example:: 127 128 make coccicheck MODE=report V=1 129 130Coccinelle parallelization 131-------------------------- 132 133By default, coccicheck tries to run as parallel as possible. To change 134the parallelism, set the J= variable. For example, to run across 4 CPUs:: 135 136 make coccicheck MODE=report J=4 137 138As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization; 139if support for this is detected you will benefit from parmap parallelization. 140 141When parmap is enabled coccicheck will enable dynamic load balancing by using 142``--chunksize 1`` argument. This ensures we keep feeding threads with work 143one by one, so that we avoid the situation where most work gets done by only 144a few threads. With dynamic load balancing, if a thread finishes early we keep 145feeding it more work. 146 147When parmap is enabled, if an error occurs in Coccinelle, this error 148value is propagated back, and the return value of the ``make coccicheck`` 149command captures this return value. 150 151Using Coccinelle with a single semantic patch 152--------------------------------------------- 153 154The optional make variable COCCI can be used to check a single 155semantic patch. In that case, the variable must be initialized with 156the name of the semantic patch to apply. 157 158For instance:: 159 160 make coccicheck COCCI=<my_SP.cocci> MODE=patch 161 162or:: 163 164 make coccicheck COCCI=<my_SP.cocci> MODE=report 165 166 167Controlling Which Files are Processed by Coccinelle 168--------------------------------------------------- 169 170By default the entire kernel source tree is checked. 171 172To apply Coccinelle to a specific directory, ``M=`` can be used. 173For example, to check drivers/net/wireless/ one may write:: 174 175 make coccicheck M=drivers/net/wireless/ 176 177To apply Coccinelle on a file basis, instead of a directory basis, the 178C variable is used by the makefile to select which files to work with. 179This variable can be used to run scripts for the entire kernel, a 180specific directory, or for a single file. 181 182For example, to check drivers/bluetooth/bfusb.c, the value 1 is 183passed to the C variable to check files that make considers 184need to be compiled.:: 185 186 make C=1 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o 187 188The value 2 is passed to the C variable to check files regardless of 189whether they need to be compiled or not.:: 190 191 make C=2 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o 192 193In these modes, which work on a file basis, there is no information 194about semantic patches displayed, and no commit message proposed. 195 196This runs every semantic patch in scripts/coccinelle by default. The 197COCCI variable may additionally be used to only apply a single 198semantic patch as shown in the previous section. 199 200The "report" mode is the default. You can select another one with the 201MODE variable explained above. 202 203Debugging Coccinelle SmPL patches 204--------------------------------- 205 206Using coccicheck is best as it provides in the spatch command line 207include options matching the options used when we compile the kernel. 208You can learn what these options are by using V=1; you could then 209manually run Coccinelle with debug options added. 210 211Alternatively you can debug running Coccinelle against SmPL patches 212by asking for stderr to be redirected to stderr. By default stderr 213is redirected to /dev/null; if you'd like to capture stderr you 214can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For 215instance:: 216 217 rm -f cocci.err 218 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 219 cat cocci.err 220 221You can use SPFLAGS to add debugging flags; for instance you may want to 222add both --profile --show-trying to SPFLAGS when debugging. For example 223you may want to use:: 224 225 rm -f err.log 226 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 227 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd 228 229err.log will now have the profiling information, while stdout will 230provide some progress information as Coccinelle moves forward with 231work. 232 233NOTE: 234 235DEBUG_FILE support is only supported when using coccinelle >= 1.0.2. 236 237Currently, DEBUG_FILE support is only available to check folders, and 238not single files. This is because checking a single file requires spatch 239to be called twice leading to DEBUG_FILE being set both times to the same value, 240giving rise to an error. 241 242.cocciconfig support 243-------------------- 244 245Coccinelle supports reading .cocciconfig for default Coccinelle options that 246should be used every time spatch is spawned. The order of precedence for 247variables for .cocciconfig is as follows: 248 249- Your current user's home directory is processed first 250- Your directory from which spatch is called is processed next 251- The directory provided with the --dir option is processed last, if used 252 253Since coccicheck runs through make, it naturally runs from the kernel 254proper dir; as such the second rule above would be implied for picking up a 255.cocciconfig when using ``make coccicheck``. 256 257``make coccicheck`` also supports using M= targets. If you do not supply 258any M= target, it is assumed you want to target the entire kernel. 259The kernel coccicheck script has:: 260 261 if [ "$KBUILD_EXTMOD" = "" ] ; then 262 OPTIONS="--dir $srctree $COCCIINCLUDE" 263 else 264 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" 265 fi 266 267KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases 268the spatch --dir argument is used, as such third rule applies when whether M= 269is used or not, and when M= is used the target directory can have its own 270.cocciconfig file. When M= is not passed as an argument to coccicheck the 271target directory is the same as the directory from where spatch was called. 272 273If not using the kernel's coccicheck target, keep the above precedence 274order logic of .cocciconfig reading. If using the kernel's coccicheck target, 275override any of the kernel's .coccicheck's settings using SPFLAGS. 276 277We help Coccinelle when used against Linux with a set of sensible default 278options for Linux with our own Linux .cocciconfig. This hints to coccinelle 279that git can be used for ``git grep`` queries over coccigrep. A timeout of 200 280seconds should suffice for now. 281 282The options picked up by coccinelle when reading a .cocciconfig do not appear 283as arguments to spatch processes running on your system. To confirm what 284options will be used by Coccinelle run:: 285 286 spatch --print-options-only 287 288You can override with your own preferred index option by using SPFLAGS. Take 289note that when there are conflicting options Coccinelle takes precedence for 290the last options passed. Using .cocciconfig is possible to use idutils, however 291given the order of precedence followed by Coccinelle, since the kernel now 292carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 293desired. See below section "Additional flags" for more details on how to use 294idutils. 295 296Additional flags 297---------------- 298 299Additional flags can be passed to spatch through the SPFLAGS 300variable. This works as Coccinelle respects the last flags 301given to it when options are in conflict. :: 302 303 make SPFLAGS=--use-glimpse coccicheck 304 305Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 306When no ID file is specified coccinelle assumes your ID database file 307is in the file .id-utils.index on the top level of the kernel. Coccinelle 308carries a script scripts/idutils_index.sh which creates the database with:: 309 310 mkid -i C --output .id-utils.index 311 312If you have another database filename you can also just symlink with this 313name. :: 314 315 make SPFLAGS=--use-idutils coccicheck 316 317Alternatively you can specify the database filename explicitly, for 318instance:: 319 320 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 321 322See ``spatch --help`` to learn more about spatch options. 323 324Note that the ``--use-glimpse`` and ``--use-idutils`` options 325require external tools for indexing the code. None of them is 326thus active by default. However, by indexing the code with 327one of these tools, and according to the cocci file used, 328spatch could proceed the entire code base more quickly. 329 330SmPL patch specific options 331--------------------------- 332 333SmPL patches can have their own requirements for options passed 334to Coccinelle. SmPL patch-specific options can be provided by 335providing them at the top of the SmPL patch, for instance:: 336 337 // Options: --no-includes --include-headers 338 339SmPL patch Coccinelle requirements 340---------------------------------- 341 342As Coccinelle features get added some more advanced SmPL patches 343may require newer versions of Coccinelle. If an SmPL patch requires 344a minimum version of Coccinelle, this can be specified as follows, 345as an example if requiring at least Coccinelle >= 1.0.5:: 346 347 // Requires: 1.0.5 348 349Proposing new semantic patches 350------------------------------ 351 352New semantic patches can be proposed and submitted by kernel 353developers. For sake of clarity, they should be organized in the 354sub-directories of ``scripts/coccinelle/``. 355 356 357Detailed description of the ``report`` mode 358------------------------------------------- 359 360``report`` generates a list in the following format:: 361 362 file:line:column-column: message 363 364Example 365~~~~~~~ 366 367Running:: 368 369 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 370 371will execute the following part of the SmPL script:: 372 373 <smpl> 374 @r depends on !context && !patch && (org || report)@ 375 expression x; 376 position p; 377 @@ 378 379 ERR_PTR@p(PTR_ERR(x)) 380 381 @script:python depends on report@ 382 p << r.p; 383 x << r.x; 384 @@ 385 386 msg="ERR_CAST can be used with %s" % (x) 387 coccilib.report.print_report(p[0], msg) 388 </smpl> 389 390This SmPL excerpt generates entries on the standard output, as 391illustrated below:: 392 393 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 394 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 395 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 396 397 398Detailed description of the ``patch`` mode 399------------------------------------------ 400 401When the ``patch`` mode is available, it proposes a fix for each problem 402identified. 403 404Example 405~~~~~~~ 406 407Running:: 408 409 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 410 411will execute the following part of the SmPL script:: 412 413 <smpl> 414 @ depends on !context && patch && !org && !report @ 415 expression x; 416 @@ 417 418 - ERR_PTR(PTR_ERR(x)) 419 + ERR_CAST(x) 420 </smpl> 421 422This SmPL excerpt generates patch hunks on the standard output, as 423illustrated below:: 424 425 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 426 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 427 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 428 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 429 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 430 CRYPTO_ALG_TYPE_MASK); 431 if (IS_ERR(alg)) 432 - return ERR_PTR(PTR_ERR(alg)); 433 + return ERR_CAST(alg); 434 435 /* Block size must be >= 4 bytes. */ 436 err = -EINVAL; 437 438Detailed description of the ``context`` mode 439-------------------------------------------- 440 441``context`` highlights lines of interest and their context 442in a diff-like style. 443 444 **NOTE**: The diff-like output generated is NOT an applicable patch. The 445 intent of the ``context`` mode is to highlight the important lines 446 (annotated with minus, ``-``) and gives some surrounding context 447 lines around. This output can be used with the diff mode of 448 Emacs to review the code. 449 450Example 451~~~~~~~ 452 453Running:: 454 455 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 456 457will execute the following part of the SmPL script:: 458 459 <smpl> 460 @ depends on context && !patch && !org && !report@ 461 expression x; 462 @@ 463 464 * ERR_PTR(PTR_ERR(x)) 465 </smpl> 466 467This SmPL excerpt generates diff hunks on the standard output, as 468illustrated below:: 469 470 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 471 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 472 +++ /tmp/nothing 473 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 474 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 475 CRYPTO_ALG_TYPE_MASK); 476 if (IS_ERR(alg)) 477 - return ERR_PTR(PTR_ERR(alg)); 478 479 /* Block size must be >= 4 bytes. */ 480 err = -EINVAL; 481 482Detailed description of the ``org`` mode 483---------------------------------------- 484 485``org`` generates a report in the Org mode format of Emacs. 486 487Example 488~~~~~~~ 489 490Running:: 491 492 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 493 494will execute the following part of the SmPL script:: 495 496 <smpl> 497 @r depends on !context && !patch && (org || report)@ 498 expression x; 499 position p; 500 @@ 501 502 ERR_PTR@p(PTR_ERR(x)) 503 504 @script:python depends on org@ 505 p << r.p; 506 x << r.x; 507 @@ 508 509 msg="ERR_CAST can be used with %s" % (x) 510 msg_safe=msg.replace("[","@(").replace("]",")") 511 coccilib.org.print_todo(p[0], msg_safe) 512 </smpl> 513 514This SmPL excerpt generates Org entries on the standard output, as 515illustrated below:: 516 517 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 518 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 519 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 520