1.. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2.. Copyright 2010 Julia Lawall <julia@diku.dk> 3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5.. highlight:: none 6 7.. _devtools_coccinelle: 8 9Coccinelle 10========== 11 12Coccinelle is a tool for pattern matching and text transformation that has 13many uses in kernel development, including the application of complex, 14tree-wide patches and detection of problematic programming patterns. 15 16Getting Coccinelle 17------------------- 18 19The semantic patches included in the kernel use features and options 20which are provided by Coccinelle version 1.0.0-rc11 and above. 21Using earlier versions will fail as the option names used by 22the Coccinelle files and coccicheck have been updated. 23 24Coccinelle is available through the package manager 25of many distributions, e.g. : 26 27 - Debian 28 - Fedora 29 - Ubuntu 30 - OpenSUSE 31 - Arch Linux 32 - NetBSD 33 - FreeBSD 34 35Some distribution packages are obsolete and it is recommended 36to use the latest version released from the Coccinelle homepage at 37http://coccinelle.lip6.fr/ 38 39Or from Github at: 40 41https://github.com/coccinelle/coccinelle 42 43Once you have it, run the following commands:: 44 45 ./autogen 46 ./configure 47 make 48 49as a regular user, and install it with:: 50 51 sudo make install 52 53More detailed installation instructions to build from source can be 54found at: 55 56https://github.com/coccinelle/coccinelle/blob/master/install.txt 57 58Supplemental documentation 59--------------------------- 60 61For supplemental documentation refer to the wiki: 62 63https://bottest.wiki.kernel.org/coccicheck 64 65The wiki documentation always refers to the linux-next version of the script. 66 67For Semantic Patch Language(SmPL) grammar documentation refer to: 68 69http://coccinelle.lip6.fr/documentation.php 70 71Using Coccinelle on the Linux kernel 72------------------------------------ 73 74A Coccinelle-specific target is defined in the top level 75Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 76front-end in the ``scripts`` directory. 77 78Four basic modes are defined: ``patch``, ``report``, ``context``, and 79``org``. The mode to use is specified by setting the MODE variable with 80``MODE=<mode>``. 81 82- ``patch`` proposes a fix, when possible. 83 84- ``report`` generates a list in the following format: 85 file:line:column-column: message 86 87- ``context`` highlights lines of interest and their context in a 88 diff-like style.Lines of interest are indicated with ``-``. 89 90- ``org`` generates a report in the Org mode format of Emacs. 91 92Note that not all semantic patches implement all modes. For easy use 93of Coccinelle, the default mode is "report". 94 95Two other modes provide some common combinations of these modes. 96 97- ``chain`` tries the previous modes in the order above until one succeeds. 98 99- ``rep+ctxt`` runs successively the report mode and the context mode. 100 It should be used with the C option (described later) 101 which checks the code on a file basis. 102 103Examples 104~~~~~~~~ 105 106To make a report for every semantic patch, run the following command:: 107 108 make coccicheck MODE=report 109 110To produce patches, run:: 111 112 make coccicheck MODE=patch 113 114 115The coccicheck target applies every semantic patch available in the 116sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 117 118For each semantic patch, a commit message is proposed. It gives a 119description of the problem being checked by the semantic patch, and 120includes a reference to Coccinelle. 121 122As any static code analyzer, Coccinelle produces false 123positives. Thus, reports must be carefully checked, and patches 124reviewed. 125 126To enable verbose messages set the V= variable, for example:: 127 128 make coccicheck MODE=report V=1 129 130Coccinelle parallelization 131--------------------------- 132 133By default, coccicheck tries to run as parallel as possible. To change 134the parallelism, set the J= variable. For example, to run across 4 CPUs:: 135 136 make coccicheck MODE=report J=4 137 138As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization, 139if support for this is detected you will benefit from parmap parallelization. 140 141When parmap is enabled coccicheck will enable dynamic load balancing by using 142``--chunksize 1`` argument, this ensures we keep feeding threads with work 143one by one, so that we avoid the situation where most work gets done by only 144a few threads. With dynamic load balancing, if a thread finishes early we keep 145feeding it more work. 146 147When parmap is enabled, if an error occurs in Coccinelle, this error 148value is propagated back, the return value of the ``make coccicheck`` 149captures this return value. 150 151Using Coccinelle with a single semantic patch 152--------------------------------------------- 153 154The optional make variable COCCI can be used to check a single 155semantic patch. In that case, the variable must be initialized with 156the name of the semantic patch to apply. 157 158For instance:: 159 160 make coccicheck COCCI=<my_SP.cocci> MODE=patch 161 162or:: 163 164 make coccicheck COCCI=<my_SP.cocci> MODE=report 165 166 167Controlling Which Files are Processed by Coccinelle 168--------------------------------------------------- 169 170By default the entire kernel source tree is checked. 171 172To apply Coccinelle to a specific directory, ``M=`` can be used. 173For example, to check drivers/net/wireless/ one may write:: 174 175 make coccicheck M=drivers/net/wireless/ 176 177To apply Coccinelle on a file basis, instead of a directory basis, the 178following command may be used:: 179 180 make C=1 CHECK="scripts/coccicheck" 181 182To check only newly edited code, use the value 2 for the C flag, i.e.:: 183 184 make C=2 CHECK="scripts/coccicheck" 185 186In these modes, which works on a file basis, there is no information 187about semantic patches displayed, and no commit message proposed. 188 189This runs every semantic patch in scripts/coccinelle by default. The 190COCCI variable may additionally be used to only apply a single 191semantic patch as shown in the previous section. 192 193The "report" mode is the default. You can select another one with the 194MODE variable explained above. 195 196Debugging Coccinelle SmPL patches 197--------------------------------- 198 199Using coccicheck is best as it provides in the spatch command line 200include options matching the options used when we compile the kernel. 201You can learn what these options are by using V=1, you could then 202manually run Coccinelle with debug options added. 203 204Alternatively you can debug running Coccinelle against SmPL patches 205by asking for stderr to be redirected to stderr, by default stderr 206is redirected to /dev/null, if you'd like to capture stderr you 207can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For 208instance:: 209 210 rm -f cocci.err 211 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 212 cat cocci.err 213 214You can use SPFLAGS to add debugging flags, for instance you may want to 215add both --profile --show-trying to SPFLAGS when debugging. For instance 216you may want to use:: 217 218 rm -f err.log 219 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 220 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c 221 222err.log will now have the profiling information, while stdout will 223provide some progress information as Coccinelle moves forward with 224work. 225 226DEBUG_FILE support is only supported when using coccinelle >= 1.0.2. 227 228.cocciconfig support 229-------------------- 230 231Coccinelle supports reading .cocciconfig for default Coccinelle options that 232should be used every time spatch is spawned, the order of precedence for 233variables for .cocciconfig is as follows: 234 235- Your current user's home directory is processed first 236- Your directory from which spatch is called is processed next 237- The directory provided with the --dir option is processed last, if used 238 239Since coccicheck runs through make, it naturally runs from the kernel 240proper dir, as such the second rule above would be implied for picking up a 241.cocciconfig when using ``make coccicheck``. 242 243``make coccicheck`` also supports using M= targets. If you do not supply 244any M= target, it is assumed you want to target the entire kernel. 245The kernel coccicheck script has:: 246 247 if [ "$KBUILD_EXTMOD" = "" ] ; then 248 OPTIONS="--dir $srctree $COCCIINCLUDE" 249 else 250 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" 251 fi 252 253KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases 254the spatch --dir argument is used, as such third rule applies when whether M= 255is used or not, and when M= is used the target directory can have its own 256.cocciconfig file. When M= is not passed as an argument to coccicheck the 257target directory is the same as the directory from where spatch was called. 258 259If not using the kernel's coccicheck target, keep the above precedence 260order logic of .cocciconfig reading. If using the kernel's coccicheck target, 261override any of the kernel's .coccicheck's settings using SPFLAGS. 262 263We help Coccinelle when used against Linux with a set of sensible defaults 264options for Linux with our own Linux .cocciconfig. This hints to coccinelle 265git can be used for ``git grep`` queries over coccigrep. A timeout of 200 266seconds should suffice for now. 267 268The options picked up by coccinelle when reading a .cocciconfig do not appear 269as arguments to spatch processes running on your system, to confirm what 270options will be used by Coccinelle run:: 271 272 spatch --print-options-only 273 274You can override with your own preferred index option by using SPFLAGS. Take 275note that when there are conflicting options Coccinelle takes precedence for 276the last options passed. Using .cocciconfig is possible to use idutils, however 277given the order of precedence followed by Coccinelle, since the kernel now 278carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 279desired. See below section "Additional flags" for more details on how to use 280idutils. 281 282Additional flags 283---------------- 284 285Additional flags can be passed to spatch through the SPFLAGS 286variable. This works as Coccinelle respects the last flags 287given to it when options are in conflict. :: 288 289 make SPFLAGS=--use-glimpse coccicheck 290 291Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 292When no ID file is specified coccinelle assumes your ID database file 293is in the file .id-utils.index on the top level of the kernel, coccinelle 294carries a script scripts/idutils_index.sh which creates the database with:: 295 296 mkid -i C --output .id-utils.index 297 298If you have another database filename you can also just symlink with this 299name. :: 300 301 make SPFLAGS=--use-idutils coccicheck 302 303Alternatively you can specify the database filename explicitly, for 304instance:: 305 306 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 307 308See ``spatch --help`` to learn more about spatch options. 309 310Note that the ``--use-glimpse`` and ``--use-idutils`` options 311require external tools for indexing the code. None of them is 312thus active by default. However, by indexing the code with 313one of these tools, and according to the cocci file used, 314spatch could proceed the entire code base more quickly. 315 316SmPL patch specific options 317--------------------------- 318 319SmPL patches can have their own requirements for options passed 320to Coccinelle. SmPL patch specific options can be provided by 321providing them at the top of the SmPL patch, for instance:: 322 323 // Options: --no-includes --include-headers 324 325SmPL patch Coccinelle requirements 326---------------------------------- 327 328As Coccinelle features get added some more advanced SmPL patches 329may require newer versions of Coccinelle. If an SmPL patch requires 330at least a version of Coccinelle, this can be specified as follows, 331as an example if requiring at least Coccinelle >= 1.0.5:: 332 333 // Requires: 1.0.5 334 335Proposing new semantic patches 336------------------------------- 337 338New semantic patches can be proposed and submitted by kernel 339developers. For sake of clarity, they should be organized in the 340sub-directories of ``scripts/coccinelle/``. 341 342 343Detailed description of the ``report`` mode 344------------------------------------------- 345 346``report`` generates a list in the following format:: 347 348 file:line:column-column: message 349 350Example 351~~~~~~~ 352 353Running:: 354 355 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 356 357will execute the following part of the SmPL script:: 358 359 <smpl> 360 @r depends on !context && !patch && (org || report)@ 361 expression x; 362 position p; 363 @@ 364 365 ERR_PTR@p(PTR_ERR(x)) 366 367 @script:python depends on report@ 368 p << r.p; 369 x << r.x; 370 @@ 371 372 msg="ERR_CAST can be used with %s" % (x) 373 coccilib.report.print_report(p[0], msg) 374 </smpl> 375 376This SmPL excerpt generates entries on the standard output, as 377illustrated below:: 378 379 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 380 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 381 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 382 383 384Detailed description of the ``patch`` mode 385------------------------------------------ 386 387When the ``patch`` mode is available, it proposes a fix for each problem 388identified. 389 390Example 391~~~~~~~ 392 393Running:: 394 395 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 396 397will execute the following part of the SmPL script:: 398 399 <smpl> 400 @ depends on !context && patch && !org && !report @ 401 expression x; 402 @@ 403 404 - ERR_PTR(PTR_ERR(x)) 405 + ERR_CAST(x) 406 </smpl> 407 408This SmPL excerpt generates patch hunks on the standard output, as 409illustrated below:: 410 411 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 412 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 413 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 414 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 415 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 416 CRYPTO_ALG_TYPE_MASK); 417 if (IS_ERR(alg)) 418 - return ERR_PTR(PTR_ERR(alg)); 419 + return ERR_CAST(alg); 420 421 /* Block size must be >= 4 bytes. */ 422 err = -EINVAL; 423 424Detailed description of the ``context`` mode 425-------------------------------------------- 426 427``context`` highlights lines of interest and their context 428in a diff-like style. 429 430 **NOTE**: The diff-like output generated is NOT an applicable patch. The 431 intent of the ``context`` mode is to highlight the important lines 432 (annotated with minus, ``-``) and gives some surrounding context 433 lines around. This output can be used with the diff mode of 434 Emacs to review the code. 435 436Example 437~~~~~~~ 438 439Running:: 440 441 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 442 443will execute the following part of the SmPL script:: 444 445 <smpl> 446 @ depends on context && !patch && !org && !report@ 447 expression x; 448 @@ 449 450 * ERR_PTR(PTR_ERR(x)) 451 </smpl> 452 453This SmPL excerpt generates diff hunks on the standard output, as 454illustrated below:: 455 456 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 457 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 458 +++ /tmp/nothing 459 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 460 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 461 CRYPTO_ALG_TYPE_MASK); 462 if (IS_ERR(alg)) 463 - return ERR_PTR(PTR_ERR(alg)); 464 465 /* Block size must be >= 4 bytes. */ 466 err = -EINVAL; 467 468Detailed description of the ``org`` mode 469---------------------------------------- 470 471``org`` generates a report in the Org mode format of Emacs. 472 473Example 474~~~~~~~ 475 476Running:: 477 478 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 479 480will execute the following part of the SmPL script:: 481 482 <smpl> 483 @r depends on !context && !patch && (org || report)@ 484 expression x; 485 position p; 486 @@ 487 488 ERR_PTR@p(PTR_ERR(x)) 489 490 @script:python depends on org@ 491 p << r.p; 492 x << r.x; 493 @@ 494 495 msg="ERR_CAST can be used with %s" % (x) 496 msg_safe=msg.replace("[","@(").replace("]",")") 497 coccilib.org.print_todo(p[0], msg_safe) 498 </smpl> 499 500This SmPL excerpt generates Org entries on the standard output, as 501illustrated below:: 502 503 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 504 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 505 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 506