1.. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2.. Copyright 2010 Julia Lawall <julia@diku.dk> 3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5.. highlight:: none 6 7Coccinelle 8========== 9 10Coccinelle is a tool for pattern matching and text transformation that has 11many uses in kernel development, including the application of complex, 12tree-wide patches and detection of problematic programming patterns. 13 14Getting Coccinelle 15------------------- 16 17The semantic patches included in the kernel use features and options 18which are provided by Coccinelle version 1.0.0-rc11 and above. 19Using earlier versions will fail as the option names used by 20the Coccinelle files and coccicheck have been updated. 21 22Coccinelle is available through the package manager 23of many distributions, e.g. : 24 25 - Debian 26 - Fedora 27 - Ubuntu 28 - OpenSUSE 29 - Arch Linux 30 - NetBSD 31 - FreeBSD 32 33You can get the latest version released from the Coccinelle homepage at 34http://coccinelle.lip6.fr/ 35 36Once you have it, run the following command:: 37 38 ./configure 39 make 40 41as a regular user, and install it with:: 42 43 sudo make install 44 45Supplemental documentation 46--------------------------- 47 48For supplemental documentation refer to the wiki: 49 50https://bottest.wiki.kernel.org/coccicheck 51 52The wiki documentation always refers to the linux-next version of the script. 53 54Using Coccinelle on the Linux kernel 55------------------------------------ 56 57A Coccinelle-specific target is defined in the top level 58Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 59front-end in the ``scripts`` directory. 60 61Four basic modes are defined: ``patch``, ``report``, ``context``, and 62``org``. The mode to use is specified by setting the MODE variable with 63``MODE=<mode>``. 64 65- ``patch`` proposes a fix, when possible. 66 67- ``report`` generates a list in the following format: 68 file:line:column-column: message 69 70- ``context`` highlights lines of interest and their context in a 71 diff-like style.Lines of interest are indicated with ``-``. 72 73- ``org`` generates a report in the Org mode format of Emacs. 74 75Note that not all semantic patches implement all modes. For easy use 76of Coccinelle, the default mode is "report". 77 78Two other modes provide some common combinations of these modes. 79 80- ``chain`` tries the previous modes in the order above until one succeeds. 81 82- ``rep+ctxt`` runs successively the report mode and the context mode. 83 It should be used with the C option (described later) 84 which checks the code on a file basis. 85 86Examples 87~~~~~~~~ 88 89To make a report for every semantic patch, run the following command:: 90 91 make coccicheck MODE=report 92 93To produce patches, run:: 94 95 make coccicheck MODE=patch 96 97 98The coccicheck target applies every semantic patch available in the 99sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 100 101For each semantic patch, a commit message is proposed. It gives a 102description of the problem being checked by the semantic patch, and 103includes a reference to Coccinelle. 104 105As any static code analyzer, Coccinelle produces false 106positives. Thus, reports must be carefully checked, and patches 107reviewed. 108 109To enable verbose messages set the V= variable, for example:: 110 111 make coccicheck MODE=report V=1 112 113Coccinelle parallelization 114--------------------------- 115 116By default, coccicheck tries to run as parallel as possible. To change 117the parallelism, set the J= variable. For example, to run across 4 CPUs:: 118 119 make coccicheck MODE=report J=4 120 121As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization, 122if support for this is detected you will benefit from parmap parallelization. 123 124When parmap is enabled coccicheck will enable dynamic load balancing by using 125``--chunksize 1`` argument, this ensures we keep feeding threads with work 126one by one, so that we avoid the situation where most work gets done by only 127a few threads. With dynamic load balancing, if a thread finishes early we keep 128feeding it more work. 129 130When parmap is enabled, if an error occurs in Coccinelle, this error 131value is propagated back, the return value of the ``make coccicheck`` 132captures this return value. 133 134Using Coccinelle with a single semantic patch 135--------------------------------------------- 136 137The optional make variable COCCI can be used to check a single 138semantic patch. In that case, the variable must be initialized with 139the name of the semantic patch to apply. 140 141For instance:: 142 143 make coccicheck COCCI=<my_SP.cocci> MODE=patch 144 145or:: 146 147 make coccicheck COCCI=<my_SP.cocci> MODE=report 148 149 150Controlling Which Files are Processed by Coccinelle 151--------------------------------------------------- 152 153By default the entire kernel source tree is checked. 154 155To apply Coccinelle to a specific directory, ``M=`` can be used. 156For example, to check drivers/net/wireless/ one may write:: 157 158 make coccicheck M=drivers/net/wireless/ 159 160To apply Coccinelle on a file basis, instead of a directory basis, the 161following command may be used:: 162 163 make C=1 CHECK="scripts/coccicheck" 164 165To check only newly edited code, use the value 2 for the C flag, i.e.:: 166 167 make C=2 CHECK="scripts/coccicheck" 168 169In these modes, which works on a file basis, there is no information 170about semantic patches displayed, and no commit message proposed. 171 172This runs every semantic patch in scripts/coccinelle by default. The 173COCCI variable may additionally be used to only apply a single 174semantic patch as shown in the previous section. 175 176The "report" mode is the default. You can select another one with the 177MODE variable explained above. 178 179Debugging Coccinelle SmPL patches 180--------------------------------- 181 182Using coccicheck is best as it provides in the spatch command line 183include options matching the options used when we compile the kernel. 184You can learn what these options are by using V=1, you could then 185manually run Coccinelle with debug options added. 186 187Alternatively you can debug running Coccinelle against SmPL patches 188by asking for stderr to be redirected to stderr, by default stderr 189is redirected to /dev/null, if you'd like to capture stderr you 190can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For 191instance:: 192 193 rm -f cocci.err 194 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 195 cat cocci.err 196 197You can use SPFLAGS to add debugging flags, for instance you may want to 198add both --profile --show-trying to SPFLAGS when debugging. For instance 199you may want to use:: 200 201 rm -f err.log 202 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 203 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c 204 205err.log will now have the profiling information, while stdout will 206provide some progress information as Coccinelle moves forward with 207work. 208 209DEBUG_FILE support is only supported when using coccinelle >= 1.0.2. 210 211.cocciconfig support 212-------------------- 213 214Coccinelle supports reading .cocciconfig for default Coccinelle options that 215should be used every time spatch is spawned, the order of precedence for 216variables for .cocciconfig is as follows: 217 218- Your current user's home directory is processed first 219- Your directory from which spatch is called is processed next 220- The directory provided with the --dir option is processed last, if used 221 222Since coccicheck runs through make, it naturally runs from the kernel 223proper dir, as such the second rule above would be implied for picking up a 224.cocciconfig when using ``make coccicheck``. 225 226``make coccicheck`` also supports using M= targets.If you do not supply 227any M= target, it is assumed you want to target the entire kernel. 228The kernel coccicheck script has:: 229 230 if [ "$KBUILD_EXTMOD" = "" ] ; then 231 OPTIONS="--dir $srctree $COCCIINCLUDE" 232 else 233 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" 234 fi 235 236KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases 237the spatch --dir argument is used, as such third rule applies when whether M= 238is used or not, and when M= is used the target directory can have its own 239.cocciconfig file. When M= is not passed as an argument to coccicheck the 240target directory is the same as the directory from where spatch was called. 241 242If not using the kernel's coccicheck target, keep the above precedence 243order logic of .cocciconfig reading. If using the kernel's coccicheck target, 244override any of the kernel's .coccicheck's settings using SPFLAGS. 245 246We help Coccinelle when used against Linux with a set of sensible defaults 247options for Linux with our own Linux .cocciconfig. This hints to coccinelle 248git can be used for ``git grep`` queries over coccigrep. A timeout of 200 249seconds should suffice for now. 250 251The options picked up by coccinelle when reading a .cocciconfig do not appear 252as arguments to spatch processes running on your system, to confirm what 253options will be used by Coccinelle run:: 254 255 spatch --print-options-only 256 257You can override with your own preferred index option by using SPFLAGS. Take 258note that when there are conflicting options Coccinelle takes precedence for 259the last options passed. Using .cocciconfig is possible to use idutils, however 260given the order of precedence followed by Coccinelle, since the kernel now 261carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 262desired. See below section "Additional flags" for more details on how to use 263idutils. 264 265Additional flags 266---------------- 267 268Additional flags can be passed to spatch through the SPFLAGS 269variable. This works as Coccinelle respects the last flags 270given to it when options are in conflict. :: 271 272 make SPFLAGS=--use-glimpse coccicheck 273 274Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 275When no ID file is specified coccinelle assumes your ID database file 276is in the file .id-utils.index on the top level of the kernel, coccinelle 277carries a script scripts/idutils_index.sh which creates the database with:: 278 279 mkid -i C --output .id-utils.index 280 281If you have another database filename you can also just symlink with this 282name. :: 283 284 make SPFLAGS=--use-idutils coccicheck 285 286Alternatively you can specify the database filename explicitly, for 287instance:: 288 289 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 290 291See ``spatch --help`` to learn more about spatch options. 292 293Note that the ``--use-glimpse`` and ``--use-idutils`` options 294require external tools for indexing the code. None of them is 295thus active by default. However, by indexing the code with 296one of these tools, and according to the cocci file used, 297spatch could proceed the entire code base more quickly. 298 299SmPL patch specific options 300--------------------------- 301 302SmPL patches can have their own requirements for options passed 303to Coccinelle. SmPL patch specific options can be provided by 304providing them at the top of the SmPL patch, for instance:: 305 306 // Options: --no-includes --include-headers 307 308SmPL patch Coccinelle requirements 309---------------------------------- 310 311As Coccinelle features get added some more advanced SmPL patches 312may require newer versions of Coccinelle. If an SmPL patch requires 313at least a version of Coccinelle, this can be specified as follows, 314as an example if requiring at least Coccinelle >= 1.0.5:: 315 316 // Requires: 1.0.5 317 318Proposing new semantic patches 319------------------------------- 320 321New semantic patches can be proposed and submitted by kernel 322developers. For sake of clarity, they should be organized in the 323sub-directories of ``scripts/coccinelle/``. 324 325 326Detailed description of the ``report`` mode 327------------------------------------------- 328 329``report`` generates a list in the following format:: 330 331 file:line:column-column: message 332 333Example 334~~~~~~~ 335 336Running:: 337 338 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 339 340will execute the following part of the SmPL script:: 341 342 <smpl> 343 @r depends on !context && !patch && (org || report)@ 344 expression x; 345 position p; 346 @@ 347 348 ERR_PTR@p(PTR_ERR(x)) 349 350 @script:python depends on report@ 351 p << r.p; 352 x << r.x; 353 @@ 354 355 msg="ERR_CAST can be used with %s" % (x) 356 coccilib.report.print_report(p[0], msg) 357 </smpl> 358 359This SmPL excerpt generates entries on the standard output, as 360illustrated below:: 361 362 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 363 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 364 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 365 366 367Detailed description of the ``patch`` mode 368------------------------------------------ 369 370When the ``patch`` mode is available, it proposes a fix for each problem 371identified. 372 373Example 374~~~~~~~ 375 376Running:: 377 378 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 379 380will execute the following part of the SmPL script:: 381 382 <smpl> 383 @ depends on !context && patch && !org && !report @ 384 expression x; 385 @@ 386 387 - ERR_PTR(PTR_ERR(x)) 388 + ERR_CAST(x) 389 </smpl> 390 391This SmPL excerpt generates patch hunks on the standard output, as 392illustrated below:: 393 394 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 395 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 396 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 397 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 398 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 399 CRYPTO_ALG_TYPE_MASK); 400 if (IS_ERR(alg)) 401 - return ERR_PTR(PTR_ERR(alg)); 402 + return ERR_CAST(alg); 403 404 /* Block size must be >= 4 bytes. */ 405 err = -EINVAL; 406 407Detailed description of the ``context`` mode 408-------------------------------------------- 409 410``context`` highlights lines of interest and their context 411in a diff-like style. 412 413 **NOTE**: The diff-like output generated is NOT an applicable patch. The 414 intent of the ``context`` mode is to highlight the important lines 415 (annotated with minus, ``-``) and gives some surrounding context 416 lines around. This output can be used with the diff mode of 417 Emacs to review the code. 418 419Example 420~~~~~~~ 421 422Running:: 423 424 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 425 426will execute the following part of the SmPL script:: 427 428 <smpl> 429 @ depends on context && !patch && !org && !report@ 430 expression x; 431 @@ 432 433 * ERR_PTR(PTR_ERR(x)) 434 </smpl> 435 436This SmPL excerpt generates diff hunks on the standard output, as 437illustrated below:: 438 439 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 440 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 441 +++ /tmp/nothing 442 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 443 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 444 CRYPTO_ALG_TYPE_MASK); 445 if (IS_ERR(alg)) 446 - return ERR_PTR(PTR_ERR(alg)); 447 448 /* Block size must be >= 4 bytes. */ 449 err = -EINVAL; 450 451Detailed description of the ``org`` mode 452---------------------------------------- 453 454``org`` generates a report in the Org mode format of Emacs. 455 456Example 457~~~~~~~ 458 459Running:: 460 461 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 462 463will execute the following part of the SmPL script:: 464 465 <smpl> 466 @r depends on !context && !patch && (org || report)@ 467 expression x; 468 position p; 469 @@ 470 471 ERR_PTR@p(PTR_ERR(x)) 472 473 @script:python depends on org@ 474 p << r.p; 475 x << r.x; 476 @@ 477 478 msg="ERR_CAST can be used with %s" % (x) 479 msg_safe=msg.replace("[","@(").replace("]",")") 480 coccilib.org.print_todo(p[0], msg_safe) 481 </smpl> 482 483This SmPL excerpt generates Org entries on the standard output, as 484illustrated below:: 485 486 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 487 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 488 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 489