1.. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2.. Copyright 2010 Julia Lawall <julia@diku.dk> 3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5.. highlight:: none 6 7.. _devtools_coccinelle: 8 9Coccinelle 10========== 11 12Coccinelle is a tool for pattern matching and text transformation that has 13many uses in kernel development, including the application of complex, 14tree-wide patches and detection of problematic programming patterns. 15 16Getting Coccinelle 17------------------ 18 19The semantic patches included in the kernel use features and options 20which are provided by Coccinelle version 1.0.0-rc11 and above. 21Using earlier versions will fail as the option names used by 22the Coccinelle files and coccicheck have been updated. 23 24Coccinelle is available through the package manager 25of many distributions, e.g. : 26 27 - Debian 28 - Fedora 29 - Ubuntu 30 - OpenSUSE 31 - Arch Linux 32 - NetBSD 33 - FreeBSD 34 35Some distribution packages are obsolete and it is recommended 36to use the latest version released from the Coccinelle homepage at 37http://coccinelle.lip6.fr/ 38 39Or from Github at: 40 41https://github.com/coccinelle/coccinelle 42 43Once you have it, run the following commands:: 44 45 ./autogen 46 ./configure 47 make 48 49as a regular user, and install it with:: 50 51 sudo make install 52 53More detailed installation instructions to build from source can be 54found at: 55 56https://github.com/coccinelle/coccinelle/blob/master/install.txt 57 58Supplemental documentation 59-------------------------- 60 61For supplemental documentation refer to the wiki: 62 63https://bottest.wiki.kernel.org/coccicheck 64 65The wiki documentation always refers to the linux-next version of the script. 66 67For Semantic Patch Language(SmPL) grammar documentation refer to: 68 69https://coccinelle.gitlabpages.inria.fr/website/docs/main_grammar.html 70 71Using Coccinelle on the Linux kernel 72------------------------------------ 73 74A Coccinelle-specific target is defined in the top level 75Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 76front-end in the ``scripts`` directory. 77 78Four basic modes are defined: ``patch``, ``report``, ``context``, and 79``org``. The mode to use is specified by setting the MODE variable with 80``MODE=<mode>``. 81 82- ``patch`` proposes a fix, when possible. 83 84- ``report`` generates a list in the following format: 85 file:line:column-column: message 86 87- ``context`` highlights lines of interest and their context in a 88 diff-like style. Lines of interest are indicated with ``-``. 89 90- ``org`` generates a report in the Org mode format of Emacs. 91 92Note that not all semantic patches implement all modes. For easy use 93of Coccinelle, the default mode is "report". 94 95Two other modes provide some common combinations of these modes. 96 97- ``chain`` tries the previous modes in the order above until one succeeds. 98 99- ``rep+ctxt`` runs successively the report mode and the context mode. 100 It should be used with the C option (described later) 101 which checks the code on a file basis. 102 103Examples 104~~~~~~~~ 105 106To make a report for every semantic patch, run the following command:: 107 108 make coccicheck MODE=report 109 110To produce patches, run:: 111 112 make coccicheck MODE=patch 113 114 115The coccicheck target applies every semantic patch available in the 116sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 117 118For each semantic patch, a commit message is proposed. It gives a 119description of the problem being checked by the semantic patch, and 120includes a reference to Coccinelle. 121 122As with any static code analyzer, Coccinelle produces false 123positives. Thus, reports must be carefully checked, and patches 124reviewed. 125 126To enable verbose messages set the V= variable, for example:: 127 128 make coccicheck MODE=report V=1 129 130By default, coccicheck will print debug logs to stdout and redirect stderr to 131/dev/null. This can make coccicheck output difficult to read and understand. 132Debug and error messages can instead be written to a debug file instead by 133setting the ``DEBUG_FILE`` variable:: 134 135 make coccicheck MODE=report DEBUG_FILE="cocci.log" 136 137Coccinelle cannot overwrite a debug file. Instead of repeatedly deleting a log 138file, you could include the datetime in the debug file name:: 139 140 make coccicheck MODE=report DEBUG_FILE="cocci-$(date -Iseconds).log" 141 142Coccinelle parallelization 143-------------------------- 144 145By default, coccicheck tries to run as parallel as possible. To change 146the parallelism, set the J= variable. For example, to run across 4 CPUs:: 147 148 make coccicheck MODE=report J=4 149 150As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization; 151if support for this is detected you will benefit from parmap parallelization. 152 153When parmap is enabled coccicheck will enable dynamic load balancing by using 154``--chunksize 1`` argument. This ensures we keep feeding threads with work 155one by one, so that we avoid the situation where most work gets done by only 156a few threads. With dynamic load balancing, if a thread finishes early we keep 157feeding it more work. 158 159When parmap is enabled, if an error occurs in Coccinelle, this error 160value is propagated back, and the return value of the ``make coccicheck`` 161command captures this return value. 162 163Using Coccinelle with a single semantic patch 164--------------------------------------------- 165 166The optional make variable COCCI can be used to check a single 167semantic patch. In that case, the variable must be initialized with 168the name of the semantic patch to apply. 169 170For instance:: 171 172 make coccicheck COCCI=<my_SP.cocci> MODE=patch 173 174or:: 175 176 make coccicheck COCCI=<my_SP.cocci> MODE=report 177 178 179Controlling Which Files are Processed by Coccinelle 180--------------------------------------------------- 181 182By default the entire kernel source tree is checked. 183 184To apply Coccinelle to a specific directory, ``M=`` can be used. 185For example, to check drivers/net/wireless/ one may write:: 186 187 make coccicheck M=drivers/net/wireless/ 188 189To apply Coccinelle on a file basis, instead of a directory basis, the 190C variable is used by the makefile to select which files to work with. 191This variable can be used to run scripts for the entire kernel, a 192specific directory, or for a single file. 193 194For example, to check drivers/bluetooth/bfusb.c, the value 1 is 195passed to the C variable to check files that make considers 196need to be compiled.:: 197 198 make C=1 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o 199 200The value 2 is passed to the C variable to check files regardless of 201whether they need to be compiled or not.:: 202 203 make C=2 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o 204 205In these modes, which work on a file basis, there is no information 206about semantic patches displayed, and no commit message proposed. 207 208This runs every semantic patch in scripts/coccinelle by default. The 209COCCI variable may additionally be used to only apply a single 210semantic patch as shown in the previous section. 211 212The "report" mode is the default. You can select another one with the 213MODE variable explained above. 214 215Debugging Coccinelle SmPL patches 216--------------------------------- 217 218Using coccicheck is best as it provides in the spatch command line 219include options matching the options used when we compile the kernel. 220You can learn what these options are by using V=1; you could then 221manually run Coccinelle with debug options added. 222 223An easier approach to debug running Coccinelle against SmPL patches is to ask 224coccicheck to redirect stderr to a debug file. As mentioned in the examples, by 225default stderr is redirected to /dev/null; if you'd like to capture stderr you 226can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For instance:: 227 228 rm -f cocci.err 229 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 230 cat cocci.err 231 232You can use SPFLAGS to add debugging flags; for instance you may want to 233add both ``--profile --show-trying`` to SPFLAGS when debugging. For example 234you may want to use:: 235 236 rm -f err.log 237 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 238 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd 239 240err.log will now have the profiling information, while stdout will 241provide some progress information as Coccinelle moves forward with 242work. 243 244NOTE: 245 246DEBUG_FILE support is only supported when using coccinelle >= 1.0.2. 247 248Currently, DEBUG_FILE support is only available to check folders, and 249not single files. This is because checking a single file requires spatch 250to be called twice leading to DEBUG_FILE being set both times to the same value, 251giving rise to an error. 252 253.cocciconfig support 254-------------------- 255 256Coccinelle supports reading .cocciconfig for default Coccinelle options that 257should be used every time spatch is spawned. The order of precedence for 258variables for .cocciconfig is as follows: 259 260- Your current user's home directory is processed first 261- Your directory from which spatch is called is processed next 262- The directory provided with the ``--dir`` option is processed last, if used 263 264``make coccicheck`` also supports using M= targets. If you do not supply 265any M= target, it is assumed you want to target the entire kernel. 266The kernel coccicheck script has:: 267 268 OPTIONS="--dir $srcroot $COCCIINCLUDE" 269 270Here, $srcroot refers to the source directory of the target: it points to the 271external module's source directory when M= used, and otherwise, to the kernel 272source directory. The third rule ensures the spatch reads the .cocciconfig from 273the target directory, allowing external modules to have their own .cocciconfig 274file. 275 276If not using the kernel's coccicheck target, keep the above precedence 277order logic of .cocciconfig reading. If using the kernel's coccicheck target, 278override any of the kernel's .coccicheck's settings using SPFLAGS. 279 280We help Coccinelle when used against Linux with a set of sensible default 281options for Linux with our own Linux .cocciconfig. This hints to coccinelle 282that git can be used for ``git grep`` queries over coccigrep. A timeout of 200 283seconds should suffice for now. 284 285The options picked up by coccinelle when reading a .cocciconfig do not appear 286as arguments to spatch processes running on your system. To confirm what 287options will be used by Coccinelle run:: 288 289 spatch --print-options-only 290 291You can override with your own preferred index option by using SPFLAGS. Take 292note that when there are conflicting options Coccinelle takes precedence for 293the last options passed. Using .cocciconfig is possible to use idutils, however 294given the order of precedence followed by Coccinelle, since the kernel now 295carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 296desired. See below section "Additional flags" for more details on how to use 297idutils. 298 299Additional flags 300---------------- 301 302Additional flags can be passed to spatch through the SPFLAGS 303variable. This works as Coccinelle respects the last flags 304given to it when options are in conflict. :: 305 306 make SPFLAGS=--use-glimpse coccicheck 307 308Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 309When no ID file is specified coccinelle assumes your ID database file 310is in the file .id-utils.index on the top level of the kernel. Coccinelle 311carries a script scripts/idutils_index.sh which creates the database with:: 312 313 mkid -i C --output .id-utils.index 314 315If you have another database filename you can also just symlink with this 316name. :: 317 318 make SPFLAGS=--use-idutils coccicheck 319 320Alternatively you can specify the database filename explicitly, for 321instance:: 322 323 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 324 325See ``spatch --help`` to learn more about spatch options. 326 327Note that the ``--use-glimpse`` and ``--use-idutils`` options 328require external tools for indexing the code. None of them is 329thus active by default. However, by indexing the code with 330one of these tools, and according to the cocci file used, 331spatch could proceed the entire code base more quickly. 332 333SmPL patch specific options 334--------------------------- 335 336SmPL patches can have their own requirements for options passed 337to Coccinelle. SmPL patch-specific options can be provided by 338providing them at the top of the SmPL patch, for instance:: 339 340 // Options: --no-includes --include-headers 341 342SmPL patch Coccinelle requirements 343---------------------------------- 344 345As Coccinelle features get added some more advanced SmPL patches 346may require newer versions of Coccinelle. If an SmPL patch requires 347a minimum version of Coccinelle, this can be specified as follows, 348as an example if requiring at least Coccinelle >= 1.0.5:: 349 350 // Requires: 1.0.5 351 352Proposing new semantic patches 353------------------------------ 354 355New semantic patches can be proposed and submitted by kernel 356developers. For sake of clarity, they should be organized in the 357sub-directories of ``scripts/coccinelle/``. 358 359 360Detailed description of the ``report`` mode 361------------------------------------------- 362 363``report`` generates a list in the following format:: 364 365 file:line:column-column: message 366 367Example 368~~~~~~~ 369 370Running:: 371 372 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 373 374will execute the following part of the SmPL script:: 375 376 <smpl> 377 @r depends on !context && !patch && (org || report)@ 378 expression x; 379 position p; 380 @@ 381 382 ERR_PTR@p(PTR_ERR(x)) 383 384 @script:python depends on report@ 385 p << r.p; 386 x << r.x; 387 @@ 388 389 msg="ERR_CAST can be used with %s" % (x) 390 coccilib.report.print_report(p[0], msg) 391 </smpl> 392 393This SmPL excerpt generates entries on the standard output, as 394illustrated below:: 395 396 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 397 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 398 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 399 400 401Detailed description of the ``patch`` mode 402------------------------------------------ 403 404When the ``patch`` mode is available, it proposes a fix for each problem 405identified. 406 407Example 408~~~~~~~ 409 410Running:: 411 412 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 413 414will execute the following part of the SmPL script:: 415 416 <smpl> 417 @ depends on !context && patch && !org && !report @ 418 expression x; 419 @@ 420 421 - ERR_PTR(PTR_ERR(x)) 422 + ERR_CAST(x) 423 </smpl> 424 425This SmPL excerpt generates patch hunks on the standard output, as 426illustrated below:: 427 428 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 429 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 430 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 431 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 432 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 433 CRYPTO_ALG_TYPE_MASK); 434 if (IS_ERR(alg)) 435 - return ERR_PTR(PTR_ERR(alg)); 436 + return ERR_CAST(alg); 437 438 /* Block size must be >= 4 bytes. */ 439 err = -EINVAL; 440 441Detailed description of the ``context`` mode 442-------------------------------------------- 443 444``context`` highlights lines of interest and their context 445in a diff-like style. 446 447 **NOTE**: The diff-like output generated is NOT an applicable patch. The 448 intent of the ``context`` mode is to highlight the important lines 449 (annotated with minus, ``-``) and gives some surrounding context 450 lines around. This output can be used with the diff mode of 451 Emacs to review the code. 452 453Example 454~~~~~~~ 455 456Running:: 457 458 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 459 460will execute the following part of the SmPL script:: 461 462 <smpl> 463 @ depends on context && !patch && !org && !report@ 464 expression x; 465 @@ 466 467 * ERR_PTR(PTR_ERR(x)) 468 </smpl> 469 470This SmPL excerpt generates diff hunks on the standard output, as 471illustrated below:: 472 473 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 474 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 475 +++ /tmp/nothing 476 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 477 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 478 CRYPTO_ALG_TYPE_MASK); 479 if (IS_ERR(alg)) 480 - return ERR_PTR(PTR_ERR(alg)); 481 482 /* Block size must be >= 4 bytes. */ 483 err = -EINVAL; 484 485Detailed description of the ``org`` mode 486---------------------------------------- 487 488``org`` generates a report in the Org mode format of Emacs. 489 490Example 491~~~~~~~ 492 493Running:: 494 495 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 496 497will execute the following part of the SmPL script:: 498 499 <smpl> 500 @r depends on !context && !patch && (org || report)@ 501 expression x; 502 position p; 503 @@ 504 505 ERR_PTR@p(PTR_ERR(x)) 506 507 @script:python depends on org@ 508 p << r.p; 509 x << r.x; 510 @@ 511 512 msg="ERR_CAST can be used with %s" % (x) 513 msg_safe=msg.replace("[","@(").replace("]",")") 514 coccilib.org.print_todo(p[0], msg_safe) 515 </smpl> 516 517This SmPL excerpt generates Org entries on the standard output, as 518illustrated below:: 519 520 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 521 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 522 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 523