1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) 2.. [see the bottom of this file for redistribution information] 3 4========================================= 5How to verify bugs and bisect regressions 6========================================= 7 8This document describes how to check if some Linux kernel problem occurs in code 9currently supported by developers -- to then explain how to locate the change 10causing the issue, if it is a regression (e.g. did not happen with earlier 11versions). 12 13The text aims at people running kernels from mainstream Linux distributions on 14commodity hardware who want to report a kernel bug to the upstream Linux 15developers. Despite this intent, the instructions work just as well for users 16who are already familiar with building their own kernels: they help avoid 17mistakes occasionally made even by experienced developers. 18 19.. 20 Note: if you see this note, you are reading the text's source file. You 21 might want to switch to a rendered version: it makes it a lot easier to 22 read and navigate this document -- especially when you want to look something 23 up in the reference section, then jump back to where you left off. 24.. 25 Find the latest rendered version of this text here: 26 https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.rst.html 27 28The essence of the process (aka 'TL;DR') 29======================================== 30 31*[If you are new to building or bisecting Linux, ignore this section and head 32over to the* ":ref:`step-by-step guide<introguide_bissbs>`" *below. It utilizes 33the same commands as this section while describing them in brief fashion. The 34steps are nevertheless easy to follow and together with accompanying entries 35in a reference section mention many alternatives, pitfalls, and additional 36aspects, all of which might be essential in your present case.]* 37 38**In case you want to check if a bug is present in code currently supported by 39developers**, execute just the *preparations* and *segment 1*; while doing so, 40consider the newest Linux kernel you regularly use to be the 'working' kernel. 41In the following example that's assumed to be 6.0.13, which is why the sources 42of v6.0 will be used to prepare the .config file. 43 44**In case you face a regression**, follow the steps at least till the end of 45*segment 2*. Then you can submit a preliminary report -- or continue with 46*segment 3*, which describes how to perform a bisection needed for a 47full-fledged regression report. In the following example 6.0.13 is assumed to be 48the 'working' kernel and 6.1.5 to be the first 'broken', which is why v6.0 49will be considered the 'good' release and used to prepare the .config file. 50 51* **Preparations**: set up everything to build your own kernels:: 52 53 # * Remove any software that depends on externally maintained kernel modules 54 # or builds any automatically during bootup. 55 # * Ensure Secure Boot permits booting self-compiled Linux kernels. 56 # * If you are not already running the 'working' kernel, reboot into it. 57 # * Install compilers and everything else needed for building Linux. 58 # * Ensure to have 15 Gigabyte free space in your home directory. 59 git clone -o mainline --no-checkout \ 60 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/ 61 cd ~/linux/ 62 git remote add -t master stable \ 63 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git 64 git checkout --detach v6.0 65 # * Hint: if you used an existing clone, ensure no stale .config is around. 66 make olddefconfig 67 # * Ensure the former command picked the .config of the 'working' kernel. 68 # * Connect external hardware (USB keys, tokens, ...), start a VM, bring up 69 # VPNs, mount network shares, and briefly try the feature that is broken. 70 yes '' | make localmodconfig 71 ./scripts/config --set-str CONFIG_LOCALVERSION '-local' 72 ./scripts/config -e CONFIG_LOCALVERSION_AUTO 73 # * Note, when short on storage space, check the guide for an alternative: 74 ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \ 75 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS 76 # * Hint: at this point you might want to adjust the build configuration; 77 # you'll have to, if you are running Debian. 78 make olddefconfig 79 cp .config ~/kernel-config-working 80 81* **Segment 1**: build a kernel from the latest mainline codebase. 82 83 This among others checks if the problem was fixed already and which developers 84 later need to be told about the problem; in case of a regression, this rules 85 out a .config change as root of the problem. 86 87 a) Checking out latest mainline code:: 88 89 cd ~/linux/ 90 git checkout --force --detach mainline/master 91 92 b) Build, install, and boot a kernel:: 93 94 cp ~/kernel-config-working .config 95 make olddefconfig 96 make -j $(nproc --all) 97 # * Make sure there is enough disk space to hold another kernel: 98 df -h /boot/ /lib/modules/ 99 # * Note: on Arch Linux, its derivatives and a few other distributions 100 # the following commands will do nothing at all or only part of the 101 # job. See the step-by-step guide for further details. 102 command -v installkernel && sudo make modules_install install 103 # * Check how much space your self-built kernel actually needs, which 104 # enables you to make better estimates later: 105 du -ch /boot/*$(make -s kernelrelease)* | tail -n 1 106 du -sh /lib/modules/$(make -s kernelrelease)/ 107 # * Hint: the output of the following command will help you pick the 108 # right kernel from the boot menu: 109 make -s kernelrelease | tee -a ~/kernels-built 110 reboot 111 # * Once booted, ensure you are running the kernel you just built by 112 # checking if the output of the next two commands matches: 113 tail -n 1 ~/kernels-built 114 uname -r 115 116 c) Check if the problem occurs with this kernel as well. 117 118* **Segment 2**: ensure the 'good' kernel is also a 'working' kernel. 119 120 This among others verifies the trimmed .config file actually works well, as 121 bisecting with it otherwise would be a waste of time: 122 123 a) Start by checking out the sources of the 'good' version:: 124 125 cd ~/linux/ 126 git checkout --force --detach v6.0 127 128 b) Build, install, and boot a kernel as described earlier in *segment 1, 129 section b* -- just feel free to skip the 'du' commands, as you have a rough 130 estimate already. 131 132 c) Ensure the feature that regressed with the 'broken' kernel actually works 133 with this one. 134 135* **Segment 3**: perform and validate the bisection. 136 137 a) In case your 'broken' version is a stable/longterm release, add the Git 138 branch holding it:: 139 140 git remote set-branches --add stable linux-6.1.y 141 git fetch stable 142 143 b) Initialize the bisection:: 144 145 cd ~/linux/ 146 git bisect start 147 git bisect good v6.0 148 git bisect bad v6.1.5 149 150 c) Build, install, and boot a kernel as described earlier in *segment 1, 151 section b*. 152 153 In case building or booting the kernel fails for unrelated reasons, run 154 ``git bisect skip``. In all other outcomes, check if the regressed feature 155 works with the newly built kernel. If it does, tell Git by executing 156 ``git bisect good``; if it does not, run ``git bisect bad`` instead. 157 158 All three commands will make Git checkout another commit; then re-execute 159 this step (e.g. build, install, boot, and test a kernel to then tell Git 160 the outcome). Do so again and again until Git shows which commit broke 161 things. If you run short of disk space during this process, check the 162 "Supplementary tasks" section below. 163 164 d) Once your finished the bisection, put a few things away:: 165 166 cd ~/linux/ 167 git bisect log > ~/bisect-log 168 cp .config ~/bisection-config-culprit 169 git bisect reset 170 171 e) Try to verify the bisection result:: 172 173 git checkout --force --detach mainline/master 174 git revert --no-edit cafec0cacaca0 175 176 This is optional, as some commits are impossible to revert. But if the 177 second command worked flawlessly, build, install, and boot one more kernel 178 kernel, which should not show the regression. 179 180* **Supplementary tasks**: cleanup during and after the process. 181 182 a) To avoid running out of disk space during a bisection, you might need to 183 remove some kernels you built earlier. You most likely want to keep those 184 you built during segment 1 and 2 around for a while, but you will most 185 likely no longer need kernels tested during the actual bisection 186 (Segment 3 c). You can list them in build order using:: 187 188 ls -ltr /lib/modules/*-local* 189 190 To then for example erase a kernel that identifies itself as 191 '6.0-rc1-local-gcafec0cacaca0', use this:: 192 193 sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0 194 sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0 195 # * Note, on some distributions kernel-install is missing 196 # or does only part of the job. 197 198 b) If you performed a bisection and successfully validated the result, feel 199 free to remove all kernels built during the actual bisection (Segment 3 c); 200 the kernels you built earlier and later you might want to keep around for 201 a week or two. 202 203.. _introguide_bissbs: 204 205Step-by-step guide on how to verify bugs and bisect regressions 206=============================================================== 207 208This guide describes how to set up your own Linux kernels for investigating bugs 209or regressions you intent to report. How far you want to follow the instructions 210depends on your issue: 211 212Execute all steps till the end of *segment 1* to **verify if your kernel problem 213is present in code supported by Linux kernel developers**. If it is, you are all 214set to report the bug -- unless it did not happen with earlier kernel versions, 215as then your want to at least continue with *segment 2* to **check if the issue 216qualifies as regression** which receive priority treatment. Depending on the 217outcome you then are ready to report a bug or submit a preliminary regression 218report; instead of the latter your could also head straight on and follow 219*segment 3* to **perform a bisection** for a full-fledged regression report 220developers are obliged to act upon. 221 222 :ref:`Preparations: set up everything to build your own kernels.<introprep_bissbs>` 223 224 :ref:`Segment 1: try to reproduce the problem with the latest codebase.<introlatestcheck_bissbs>` 225 226 :ref:`Segment 2: check if the kernels you build work fine.<introworkingcheck_bissbs>` 227 228 :ref:`Segment 3: perform a bisection and validate the result.<introbisect_bissbs>` 229 230 :ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>` 231 232The steps in each segment illustrate the important aspects of the process, while 233a comprehensive reference section holds additional details. The latter sometimes 234also outlines alternative approaches, pitfalls, as well as problems that might 235occur at the particular step -- and how to get things rolling again. 236 237For further details on how to report Linux kernel issues or regressions check 238out Documentation/admin-guide/reporting-issues.rst, which works in conjunction 239with this document. It among others explains why you need to verify bugs with 240the latest 'mainline' kernel, even if you face a problem with a kernel from a 241'stable/longterm' series; for users facing a regression it also explains that 242sending a preliminary report after finishing segment 2 might be wise, as the 243regression and its culprit might be known already. For further details on 244what actually qualifies as a regression check out 245Documentation/admin-guide/reporting-regressions.rst. 246 247.. _introprep_bissbs: 248 249Preparations: set up everything to build your own kernels 250--------------------------------------------------------- 251 252.. _backup_bissbs: 253 254* Create a fresh backup and put system repair and restore tools at hand, just 255 to be prepared for the unlikely case of something going sideways. 256 257 [:ref:`details<backup_bisref>`] 258 259.. _vanilla_bissbs: 260 261* Remove all software that depends on externally developed kernel drivers or 262 builds them automatically. That includes but is not limited to DKMS, openZFS, 263 VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module). 264 265 [:ref:`details<vanilla_bisref>`] 266 267.. _secureboot_bissbs: 268 269* On platforms with 'Secure Boot' or similar solutions, prepare everything to 270 ensure the system will permit your self-compiled kernel to boot. The 271 quickest and easiest way to achieve this on commodity x86 systems is to 272 disable such techniques in the BIOS setup utility; alternatively, remove 273 their restrictions through a process initiated by 274 ``mokutil --disable-validation``. 275 276 [:ref:`details<secureboot_bisref>`] 277 278.. _rangecheck_bissbs: 279 280* Determine the kernel versions considered 'good' and 'bad' throughout this 281 guide. 282 283 Do you follow this guide to verify if a bug is present in the code developers 284 care for? Then consider the mainline release your 'working' kernel (the newest 285 one you regularly use) is based on to be the 'good' version; if your 'working' 286 kernel for example is '6.0.11', then your 'good' kernel is 'v6.0'. 287 288 In case you face a regression, it depends on the version range where the 289 regression was introduced: 290 291 * Something which used to work in Linux 6.0 broke when switching to Linux 292 6.1-rc1? Then henceforth regard 'v6.0' as the last known 'good' version 293 and 'v6.1-rc1' as the first 'bad' one. 294 295 * Some function stopped working when updating from 6.0.11 to 6.1.4? Then for 296 the time being consider 'v6.0' as the last 'good' version and 'v6.1.4' as 297 the 'bad' one. Note, at this point it is merely assumed that 6.0 is fine; 298 this assumption will be checked in segment 2. 299 300 * A feature you used in 6.0.11 does not work at all or worse in 6.1.13? In 301 that case you want to bisect within a stable/longterm series: consider 302 'v6.0.11' as the last known 'good' version and 'v6.0.13' as the first 'bad' 303 one. Note, in this case you still want to compile and test a mainline kernel 304 as explained in segment 1: the outcome will determine if you need to report 305 your issue to the regular developers or the stable team. 306 307 *Note, do not confuse 'good' version with 'working' kernel; the latter term 308 throughout this guide will refer to the last kernel that has been working 309 fine.* 310 311 [:ref:`details<rangecheck_bisref>`] 312 313.. _bootworking_bissbs: 314 315* Boot into the 'working' kernel and briefly use the apparently broken feature. 316 317 [:ref:`details<bootworking_bisref>`] 318 319.. _diskspace_bissbs: 320 321* Ensure to have enough free space for building Linux. 15 Gigabyte in your home 322 directory should typically suffice. If you have less available, be sure to pay 323 attention to later steps about retrieving the Linux sources and handling of 324 debug symbols: both explain approaches reducing the amount of space, which 325 should allow you to master these tasks with about 4 Gigabytes free space. 326 327 [:ref:`details<diskspace_bisref>`] 328 329.. _buildrequires_bissbs: 330 331* Install all software required to build a Linux kernel. Often you will need: 332 'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl', 333 'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The 334 reference section shows how to quickly install those on various popular Linux 335 distributions. 336 337 [:ref:`details<buildrequires_bisref>`] 338 339.. _sources_bissbs: 340 341* Retrieve the mainline Linux sources; then change into the directory holding 342 them, as all further commands in this guide are meant to be executed from 343 there. 344 345 *Note, the following describe how to retrieve the sources using a full 346 mainline clone, which downloads about 2,75 GByte as of early 2024. The* 347 :ref:`reference section describes two alternatives <sources_bisref>` *: 348 one downloads less than 500 MByte, the other works better with unreliable 349 internet connections.* 350 351 Execute the following command to retrieve a fresh mainline codebase while 352 preparing things to add stable/longterm branches later:: 353 354 git clone -o mainline --no-checkout \ 355 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/ 356 cd ~/linux/ 357 git remote add -t master stable \ 358 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git 359 360 [:ref:`details<sources_bisref>`] 361 362.. _oldconfig_bissbs: 363 364* Start preparing a kernel build configuration (the '.config' file). 365 366 Before doing so, ensure you are still running the 'working' kernel an earlier 367 step told you to boot; if you are unsure, check the current kernel release 368 identifier using ``uname -r``. 369 370 Afterwards check out the source code for the version earlier established as 371 'good' (in this example this is assumed to be 6.0) and create a .config file:: 372 373 git checkout --detach v6.0 374 make olddefconfig 375 376 The second command will try to locate the build configuration file for the 377 running kernel and then adjust it for the needs of the kernel sources you 378 checked out. While doing so, it will print a few lines you need to check. 379 380 Look out for a line starting with '# using defaults found in'. It should be 381 followed by a path to a file in '/boot/' that contains the release identifier 382 of your currently working kernel. If the line instead continues with something 383 like 'arch/x86/configs/x86_64_defconfig', then the build infra failed to find 384 the .config file for your running kernel -- in which case you have to put one 385 there manually, as explained in the reference section. 386 387 In case you can not find such a line, look for one containing '# configuration 388 written to .config'. If that's the case you have a stale build configuration 389 lying around. Unless you intend to use it, delete it; afterwards run 390 'make olddefconfig' again and check if it now picked up the right config file 391 as base. 392 393 [:ref:`details<oldconfig_bisref>`] 394 395.. _localmodconfig_bissbs: 396 397* Disable any kernel modules apparently superfluous for your setup. This is 398 optional, but especially wise for bisections, as it speeds up the build 399 process enormously -- at least unless the .config file picked up in the 400 previous step was already tailored to your and your hardware needs, in which 401 case you should skip this step. 402 403 To prepare the trimming, connect external hardware you occasionally use (USB 404 keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted 405 since you started that guide, ensure that you tried using the feature causing 406 trouble since you started the system. Only then trim your .config:: 407 408 yes '' | make localmodconfig 409 410 There is a catch to this, as the 'apparently' in initial sentence of this step 411 and the preparation instructions already hinted at: 412 413 The 'localmodconfig' target easily disables kernel modules for features only 414 used occasionally -- like modules for external peripherals not yet connected 415 since booting, virtualization software not yet utilized, VPN tunnels, and a 416 few other things. That's because some tasks rely on kernel modules Linux only 417 loads when you execute tasks like the aforementioned ones for the first time. 418 419 This drawback of localmodconfig is nothing you should lose sleep over, but 420 something to keep in mind: if something is misbehaving with the kernels built 421 during this guide, this is most likely the reason. You can reduce or nearly 422 eliminate the risk with tricks outlined in the reference section; but when 423 building a kernel just for quick testing purposes this is usually not worth 424 spending much effort on, as long as it boots and allows to properly test the 425 feature that causes trouble. 426 427 [:ref:`details<localmodconfig_bisref>`] 428 429.. _tagging_bissbs: 430 431* Ensure all the kernels you will build are clearly identifiable using a special 432 tag and a unique version number:: 433 434 ./scripts/config --set-str CONFIG_LOCALVERSION '-local' 435 ./scripts/config -e CONFIG_LOCALVERSION_AUTO 436 437 [:ref:`details<tagging_bisref>`] 438 439.. _debugsymbols_bissbs: 440 441* Decide how to handle debug symbols. 442 443 In the context of this document it is often wise to enable them, as there is a 444 decent chance you will need to decode a stack trace from a 'panic', 'Oops', 445 'warning', or 'BUG':: 446 447 ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \ 448 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS 449 450 But if you are extremely short on storage space, you might want to disable 451 debug symbols instead:: 452 453 ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \ 454 -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE 455 456 [:ref:`details<debugsymbols_bisref>`] 457 458.. _configmods_bissbs: 459 460* Check if you may want or need to adjust some other kernel configuration 461 options: 462 463 * Are you running Debian? Then you want to avoid known problems by performing 464 additional adjustments explained in the reference section. 465 466 [:ref:`details<configmods_distros_bisref>`]. 467 468 * If you want to influence other aspects of the configuration, do so now using 469 your preferred tool. Note, to use make targets like 'menuconfig' or 470 'nconfig', you will need to install the development files of ncurses; for 471 'xconfig' you likewise need the Qt5 or Qt6 headers. 472 473 [:ref:`details<configmods_individual_bisref>`]. 474 475.. _saveconfig_bissbs: 476 477* Reprocess the .config after the latest adjustments and store it in a safe 478 place:: 479 480 make olddefconfig 481 cp .config ~/kernel-config-working 482 483 [:ref:`details<saveconfig_bisref>`] 484 485.. _introlatestcheck_bissbs: 486 487Segment 1: try to reproduce the problem with the latest codebase 488---------------------------------------------------------------- 489 490The following steps verify if the problem occurs with the code currently 491supported by developers. In case you face a regression, it also checks that the 492problem is not caused by some .config change, as reporting the issue then would 493be a waste of time. [:ref:`details<introlatestcheck_bisref>`] 494 495.. _checkoutmaster_bissbs: 496 497* Check out the latest Linux codebase:: 498 499 cd ~/linux/ 500 git checkout --force --detach mainline/master 501 502 [:ref:`details<checkoutmaster_bisref>`] 503 504.. _build_bissbs: 505 506* Build the image and the modules of your first kernel using the config file you 507 prepared:: 508 509 cp ~/kernel-config-working .config 510 make olddefconfig 511 make -j $(nproc --all) 512 513 If you want your kernel packaged up as deb, rpm, or tar file, see the 514 reference section for alternatives, which obviously will require other 515 steps to install as well. 516 517 [:ref:`details<build_bisref>`] 518 519.. _install_bissbs: 520 521* Install your newly built kernel. 522 523 Before doing so, consider checking if there is still enough room for it:: 524 525 df -h /boot/ /lib/modules/ 526 527 150 MByte in /boot/ and 200 in /lib/modules/ usually suffice. Those are rough 528 estimates assuming the worst case. How much your kernels actually require will 529 be determined later. 530 531 Now install the kernel, which will be saved in parallel to the kernels from 532 your Linux distribution:: 533 534 command -v installkernel && sudo make modules_install install 535 536 On many commodity Linux distributions this will take care of everything 537 required to boot your kernel. You might want to ensure that's the case by 538 checking if your boot loader's configuration was updated; furthermore ensure 539 an initramfs (also known as initrd) exists, which on many distributions can be 540 achieved by running ``ls -l /boot/init*$(make -s kernelrelease)*``. Those 541 steps are recommended, as there are quite a few Linux distribution where above 542 command is insufficient: 543 544 * On Arch Linux, its derivatives, many immutable Linux distributions, and a 545 few others the above command does nothing at, as they lack 'installkernel' 546 executable. 547 548 * Some distributions install the kernel, but don't add an entry for your 549 kernel in your boot loader's configuration -- the kernel thus won't show up 550 in the boot menu. 551 552 * Some distributions add a boot loader menu entry, but don't create an 553 initramfs on installation -- in that case your kernel most likely will be 554 unable to mount the root partition during bootup. 555 556 If any of that applies to you, see the reference section for further guidance. 557 Once you figured out what to do, consider writing down the necessary 558 installation steps: if you will build more kernels as described in 559 segment 2 and 3, you will have to execute these commands every time that 560 ``command -v installkernel [...]`` comes up again. 561 562 [:ref:`details<install_bisref>`] 563 564.. _storagespace_bissbs: 565 566* In case you plan to follow this guide further, check how much storage space 567 the kernel, its modules, and other related files like the initramfs consume:: 568 569 du -ch /boot/*$(make -s kernelrelease)* | tail -n 1 570 du -sh /lib/modules/$(make -s kernelrelease)/ 571 572 Write down or remember those two values for later: they enable you to prevent 573 running out of disk space accidentally during a bisection. 574 575 [:ref:`details<storagespace_bisref>`] 576 577.. _kernelrelease_bissbs: 578 579* Show and store the kernelrelease identifier of the kernel you just built:: 580 581 make -s kernelrelease | tee -a ~/kernels-built 582 583 Remember the identifier momentarily, as it will help you pick the right kernel 584 from the boot menu upon restarting. 585 586.. _recheckbroken_bissbs: 587 588* Reboot into the kernel you just built and check if the feature that is 589 expected to be broken really is. 590 591 Start by making sure the kernel you booted is the one you just built. When 592 unsure, check if the output of these commands show the exact same release 593 identifier:: 594 595 tail -n 1 ~/kernels-built 596 uname -r 597 598 Now verify if the feature that causes trouble works with your newly built 599 kernel. If things work while investigating a regression, check the reference 600 section for further details. 601 602 [:ref:`details<recheckbroken_bisref>`] 603 604.. _recheckstablebroken_bissbs: 605 606* Are you facing a problem within a stable/longterm release, but failed to 607 reproduce it with the mainline kernel you just built? Then check if the latest 608 codebase for the particular series might already fix the problem. To do so, 609 add the stable series Git branch for your 'good' kernel (again, this here is 610 assumed to be 6.0) and check out the latest version:: 611 612 cd ~/linux/ 613 git remote set-branches --add stable linux-6.0.y 614 git fetch stable 615 git checkout --force --detach linux-6.0.y 616 617 Now use the checked out code to build and install another kernel using the 618 commands the earlier steps already described in more detail:: 619 620 cp ~/kernel-config-working .config 621 make olddefconfig 622 make -j $(nproc --all) 623 # * Check if the free space suffices holding another kernel: 624 df -h /boot/ /lib/modules/ 625 command -v installkernel && sudo make modules_install install 626 make -s kernelrelease | tee -a ~/kernels-built 627 reboot 628 629 Now verify if you booted the kernel you intended to start, to then check if 630 everything works fine with this kernel:: 631 632 tail -n 1 ~/kernels-built 633 uname -r 634 635 [:ref:`details<recheckstablebroken_bisref>`] 636 637Do you follow this guide to verify if a problem is present in the code 638currently supported by Linux kernel developers? Then you are done at this 639point. If you later want to remove the kernel you just built, check out 640:ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>`. 641 642In case you face a regression, move on and execute at least the next segment 643as well. 644 645.. _introworkingcheck_bissbs: 646 647Segment 2: check if the kernels you build work fine 648--------------------------------------------------- 649 650In case of a regression, you now want to ensure the trimmed configuration file 651you created earlier works as expected; a bisection with the .config file 652otherwise would be a waste of time. [:ref:`details<introworkingcheck_bisref>`] 653 654.. _recheckworking_bissbs: 655 656* Build your own variant of the 'working' kernel and check if the feature that 657 regressed works as expected with it. 658 659 Start by checking out the sources for the version earlier established as 660 'good' (once again assumed to be 6.0 here):: 661 662 cd ~/linux/ 663 git checkout --detach v6.0 664 665 Now use the checked out code to configure, build, and install another kernel 666 using the commands the previous subsection explained in more detail:: 667 668 cp ~/kernel-config-working .config 669 make olddefconfig 670 make -j $(nproc --all) 671 # * Check if the free space suffices holding another kernel: 672 df -h /boot/ /lib/modules/ 673 command -v installkernel && sudo make modules_install install 674 make -s kernelrelease | tee -a ~/kernels-built 675 reboot 676 677 When the system booted, you may want to verify once again that the 678 kernel you started is the one you just built: 679 680 tail -n 1 ~/kernels-built 681 uname -r 682 683 Now check if this kernel works as expected; if not, consult the reference 684 section for further instructions. 685 686 [:ref:`details<recheckworking_bisref>`] 687 688.. _introbisect_bissbs: 689 690Segment 3: perform the bisection and validate the result 691-------------------------------------------------------- 692 693With all the preparations and precaution builds taken care of, you are now ready 694to begin the bisection. This will make you build quite a few kernels -- usually 695about 15 in case you encountered a regression when updating to a newer series 696(say from 6.0.11 to 6.1.3). But do not worry, due to the trimmed build 697configuration created earlier this works a lot faster than many people assume: 698overall on average it will often just take about 10 to 15 minutes to compile 699each kernel on commodity x86 machines. 700 701* In case your 'bad' version is a stable/longterm release (say v6.1.5), add its 702 stable branch, unless you already did so earlier:: 703 704 cd ~/linux/ 705 git remote set-branches --add stable linux-6.1.y 706 git fetch stable 707 708.. _bisectstart_bissbs: 709 710* Start the bisection and tell Git about the versions earlier established as 711 'good' (6.0 in the following example command) and 'bad' (6.1.5):: 712 713 cd ~/linux/ 714 git bisect start 715 git bisect good v6.0 716 git bisect bad v6.1.5 717 718 [:ref:`details<bisectstart_bisref>`] 719 720.. _bisectbuild_bissbs: 721 722* Now use the code Git checked out to build, install, and boot a kernel using 723 the commands introduced earlier:: 724 725 cp ~/kernel-config-working .config 726 make olddefconfig 727 make -j $(nproc --all) 728 # * Check if the free space suffices holding another kernel: 729 df -h /boot/ /lib/modules/ 730 command -v installkernel && sudo make modules_install install 731 make -s kernelrelease | tee -a ~/kernels-built 732 reboot 733 734 If compilation fails for some reason, run ``git bisect skip`` and restart 735 executing the stack of commands from the beginning. 736 737 In case you skipped the "test latest codebase" step in the guide, check its 738 description as for why the 'df [...]' and 'make -s kernelrelease [...]' 739 commands are here. 740 741 Important note: the latter command from this point on will print release 742 identifiers that might look odd or wrong to you -- which they are not, as it's 743 totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0' 744 if you bisect between versions 6.1 and 6.2 for example. 745 746 [:ref:`details<bisectbuild_bisref>`] 747 748.. _bisecttest_bissbs: 749 750* Now check if the feature that regressed works in the kernel you just built. 751 752 You again might want to start by making sure the kernel you booted is the one 753 you just built:: 754 755 cd ~/linux/ 756 tail -n 1 ~/kernels-built 757 uname -r 758 759 Now verify if the feature that regressed works at this kernel bisection point. 760 If it does, run this:: 761 762 git bisect good 763 764 If it does not, run this:: 765 766 git bisect bad 767 768 Be sure about what you tell Git, as getting this wrong just once will send the 769 rest of the bisection totally off course. 770 771 While the bisection is ongoing, Git will use the information you provided to 772 find and check out another bisection point for you to test. While doing so, it 773 will print something like 'Bisecting: 675 revisions left to test after this 774 (roughly 10 steps)' to indicate how many further changes it expects to be 775 tested. Now build and install another kernel using the instructions from the 776 previous step; afterwards follow the instructions in this step again. 777 778 Repeat this again and again until you finish the bisection -- that's the case 779 when Git after tagging a change as 'good' or 'bad' prints something like 780 'cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit'; right 781 afterwards it will show some details about the culprit including the patch 782 description of the change. The latter might fill your terminal screen, so you 783 might need to scroll up to see the message mentioning the culprit; 784 alternatively, run ``git bisect log > ~/bisection-log``. 785 786 [:ref:`details<bisecttest_bisref>`] 787 788.. _bisectlog_bissbs: 789 790* Store Git's bisection log and the current .config file in a safe place before 791 telling Git to reset the sources to the state before the bisection:: 792 793 cd ~/linux/ 794 git bisect log > ~/bisection-log 795 cp .config ~/bisection-config-culprit 796 git bisect reset 797 798 [:ref:`details<bisectlog_bisref>`] 799 800.. _revert_bissbs: 801 802* Try reverting the culprit on top of latest mainline to see if this fixes your 803 regression. 804 805 This is optional, as it might be impossible or hard to realize. The former is 806 the case, if the bisection determined a merge commit as the culprit; the 807 latter happens if other changes depend on the culprit. But if the revert 808 succeeds, it is worth building another kernel, as it validates the result of 809 a bisection, which can easily deroute; it furthermore will let kernel 810 developers know, if they can resolve the regression with a quick revert. 811 812 Begin by checking out the latest codebase depending on the range you bisected: 813 814 * Did you face a regression within a stable/longterm series (say between 815 6.0.11 and 6.0.13) that does not happen in mainline? Then check out the 816 latest codebase for the affected series like this:: 817 818 git fetch stable 819 git checkout --force --detach linux-6.0.y 820 821 * In all other cases check out latest mainline:: 822 823 git fetch mainline 824 git checkout --force --detach mainline/master 825 826 If you bisected a regression within a stable/longterm series that also 827 happens in mainline, there is one more thing to do: look up the mainline 828 commit-id. To do so, use a command like ``git show abcdcafecabcd`` to 829 view the patch description of the culprit. There will be a line near 830 the top which looks like 'commit cafec0cacaca0 upstream.' or 831 'Upstream commit cafec0cacaca0'; use that commit-id in the next command 832 and not the one the bisection blamed. 833 834 Now try reverting the culprit by specifying its commit id:: 835 836 git revert --no-edit cafec0cacaca0 837 838 If that fails, give up trying and move on to the next step. But if it works, 839 build a kernel again using the familiar command sequence:: 840 841 cp ~/kernel-config-working .config 842 make olddefconfig && 843 make -j $(nproc --all) && 844 # * Check if the free space suffices holding another kernel: 845 df -h /boot/ /lib/modules/ 846 command -v installkernel && sudo make modules_install install 847 Make -s kernelrelease | tee -a ~/kernels-built 848 reboot 849 850 Now check one last time if the feature that made you perform a bisection work 851 with that kernel. 852 853 [:ref:`details<revert_bisref>`] 854 855.. _introclosure_bissbs: 856 857Supplementary tasks: cleanup during and after the bisection 858----------------------------------------------------------- 859 860During and after following this guide you might want or need to remove some of 861the kernels you installed: the boot menu otherwise will become confusing or 862space might run out. 863 864.. _makeroom_bissbs: 865 866* To remove one of the kernels you installed, look up its 'kernelrelease' 867 identifier. This guide stores them in '~/kernels-built', but the following 868 command will print them as well:: 869 870 ls -ltr /lib/modules/*-local* 871 872 You in most situations want to remove the oldest kernels built during the 873 actual bisection (e.g. segment 3 of this guide). The two ones you created 874 beforehand (e.g. to test the latest codebase and the version considered 875 'good') might become handy to verify something later -- thus better keep them 876 around, unless you are really short on storage space. 877 878 To remove the modules of a kernel with the kernelrelease identifier 879 '*6.0-rc1-local-gcafec0cacaca0*', start by removing the directory holding its 880 modules:: 881 882 sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0 883 884 Afterwards try the following command:: 885 886 sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0 887 888 On quite a few distributions this will delete all other kernel files installed 889 while also removing the kernel's entry from the boot menu. But on some 890 distributions kernel-install does not exist or leaves boot-loader entries or 891 kernel image and related files behind; in that case remove them as described 892 in the reference section. 893 894 [:ref:`details<makeroom_bisref>`] 895 896.. _finishingtouch_bissbs: 897 898* Once you have finished the bisection, do not immediately remove anything you 899 set up, as you might need a few things again. What is safe to remove depends 900 on the outcome of the bisection: 901 902 * Could you initially reproduce the regression with the latest codebase and 903 after the bisection were able to fix the problem by reverting the culprit on 904 top of the latest codebase? Then you want to keep those two kernels around 905 for a while, but safely remove all others with a '-local' in the release 906 identifier. 907 908 * Did the bisection end on a merge-commit or seems questionable for other 909 reasons? Then you want to keep as many kernels as possible around for a few 910 days: it's pretty likely that you will be asked to recheck something. 911 912 * In other cases it likely is a good idea to keep the following kernels around 913 for some time: the one built from the latest codebase, the one created from 914 the version considered 'good', and the last three or four you compiled 915 during the actual bisection process. 916 917 [:ref:`details<finishingtouch_bisref>`] 918 919.. _submit_improvements: 920 921This concludes the step-by-step guide. 922 923Did you run into trouble following any of the above steps not cleared up by the 924reference section below? Did you spot errors? Or do you have ideas how to 925improve the guide? Then please take a moment and let the maintainer of this 926document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while 927CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is 928vital to improve this document further, which is in everybody's interest, as it 929will enable more people to master the task described here -- and hopefully also 930improve similar guides inspired by this one. 931 932 933Reference section for the step-by-step guide 934============================================ 935 936This section holds additional information for almost all the items in the above 937step-by-step guide. 938 939.. _backup_bisref: 940 941Prepare for emergencies 942----------------------- 943 944 *Create a fresh backup and put system repair and restore tools at hand.* 945 [:ref:`... <backup_bissbs>`] 946 947Remember, you are dealing with computers, which sometimes do unexpected things 948-- especially if you fiddle with crucial parts like the kernel of an operating 949system. That's what you are about to do in this process. Hence, better prepare 950for something going sideways, even if that should not happen. 951 952[:ref:`back to step-by-step guide <backup_bissbs>`] 953 954.. _vanilla_bisref: 955 956Remove anything related to externally maintained kernel modules 957--------------------------------------------------------------- 958 959 *Remove all software that depends on externally developed kernel drivers or 960 builds them automatically.* [:ref:`...<vanilla_bissbs>`] 961 962Externally developed kernel modules can easily cause trouble during a bisection. 963 964But there is a more important reason why this guide contains this step: most 965kernel developers will not care about reports about regressions occurring with 966kernels that utilize such modules. That's because such kernels are not 967considered 'vanilla' anymore, as Documentation/admin-guide/reporting-issues.rst 968explains in more detail. 969 970[:ref:`back to step-by-step guide <vanilla_bissbs>`] 971 972.. _secureboot_bisref: 973 974Deal with techniques like Secure Boot 975------------------------------------- 976 977 *On platforms with 'Secure Boot' or similar techniques, prepare everything to 978 ensure the system will permit your self-compiled kernel to boot later.* 979 [:ref:`... <secureboot_bissbs>`] 980 981Many modern systems allow only certain operating systems to start; that's why 982they reject booting self-compiled kernels by default. 983 984You ideally deal with this by making your platform trust your self-built kernels 985with the help of a certificate. How to do that is not described 986here, as it requires various steps that would take the text too far away from 987its purpose; 'Documentation/admin-guide/module-signing.rst' and various web 988sides already explain everything needed in more detail. 989 990Temporarily disabling solutions like Secure Boot is another way to make your own 991Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup 992utility; the required steps vary a lot between machines and therefore cannot be 993described here. 994 995On mainstream x86 Linux distributions there is a third and universal option: 996disable all Secure Boot restrictions for your Linux environment. You can 997initiate this process by running ``mokutil --disable-validation``; this will 998tell you to create a one-time password, which is safe to write down. Now 999restart; right after your BIOS performed all self-tests the bootloader Shim will 1000show a blue box with a message 'Press any key to perform MOK management'. Hit 1001some key before the countdown exposes, which will open a menu. Choose 'Change 1002Secure Boot state'. Shim's 'MokManager' will now ask you to enter three 1003randomly chosen characters from the one-time password specified earlier. Once 1004you provided them, confirm you really want to disable the validation. 1005Afterwards, permit MokManager to reboot the machine. 1006 1007[:ref:`back to step-by-step guide <secureboot_bissbs>`] 1008 1009.. _bootworking_bisref: 1010 1011Boot the last kernel that was working 1012------------------------------------- 1013 1014 *Boot into the last working kernel and briefly recheck if the feature that 1015 regressed really works.* [:ref:`...<bootworking_bissbs>`] 1016 1017This will make later steps that cover creating and trimming the configuration do 1018the right thing. 1019 1020[:ref:`back to step-by-step guide <bootworking_bissbs>`] 1021 1022.. _diskspace_bisref: 1023 1024Space requirements 1025------------------ 1026 1027 *Ensure to have enough free space for building Linux.* 1028 [:ref:`... <diskspace_bissbs>`] 1029 1030The numbers mentioned are rough estimates with a big extra charge to be on the 1031safe side, so often you will need less. 1032 1033If you have space constraints, be sure to hay attention to the :ref:`step about 1034debug symbols' <debugsymbols_bissbs>` and its :ref:`accompanying reference 1035section' <debugsymbols_bisref>`, as disabling then will reduce the consumed disk 1036space by quite a few gigabytes. 1037 1038[:ref:`back to step-by-step guide <diskspace_bissbs>`] 1039 1040.. _rangecheck_bisref: 1041 1042Bisection range 1043--------------- 1044 1045 *Determine the kernel versions considered 'good' and 'bad' throughout this 1046 guide.* [:ref:`...<rangecheck_bissbs>`] 1047 1048Establishing the range of commits to be checked is mostly straightforward, 1049except when a regression occurred when switching from a release of one stable 1050series to a release of a later series (e.g. from 6.0.11 to 6.1.4). In that case 1051Git will need some hand holding, as there is no straight line of descent. 1052 1053That's because with the release of 6.0 mainline carried on to 6.1 while the 1054stable series 6.0.y branched to the side. It's therefore theoretically possible 1055that the issue you face with 6.1.4 only worked in 6.0.11, as it was fixed by a 1056commit that went into one of the 6.0.y releases, but never hit mainline or the 10576.1.y series. Thankfully that normally should not happen due to the way the 1058stable/longterm maintainers maintain the code. It's thus pretty safe to assume 10596.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel 1060will be built and tested in the segment '2' of this guide; Git would force you 1061to do this as well, if you tried bisecting between 6.0.11 and 6.1.13. 1062 1063[:ref:`back to step-by-step guide <rangecheck_bissbs>`] 1064 1065.. _buildrequires_bisref: 1066 1067Install build requirements 1068-------------------------- 1069 1070 *Install all software required to build a Linux kernel.* 1071 [:ref:`...<buildrequires_bissbs>`] 1072 1073The kernel is pretty stand-alone, but besides tools like the compiler you will 1074sometimes need a few libraries to build one. How to install everything needed 1075depends on your Linux distribution and the configuration of the kernel you are 1076about to build. 1077 1078Here are a few examples what you typically need on some mainstream 1079distributions: 1080 1081* Arch Linux and derivatives:: 1082 1083 sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \ 1084 pahole perl zlib ncurses qt6-base 1085 1086* Debian, Ubuntu, and derivatives:: 1087 1088 sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \ 1089 libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \ 1090 libncurses-dev qt6-base-dev g++ 1091 1092* Fedora and derivatives:: 1093 1094 sudo dnf install binutils \ 1095 /usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \ 1096 /usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction} 1097 1098* openSUSE and derivatives:: 1099 1100 sudo zypper install bc binutils bison dwarves flex gcc git \ 1101 kernel-install-tools libelf-devel make modutils openssl openssl-devel \ 1102 perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel 1103 1104These commands install a few packages that are often, but not always needed. You 1105for example might want to skip installing the development headers for ncurses, 1106which you will only need in case you later might want to adjust the kernel build 1107configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit 1108the headers of Qt6 is you do not plan to adjust the .config using 'xconfig'. 1109 1110You furthermore might need additional libraries and their development headers 1111for tasks not covered in this guide -- for example when building utilities from 1112the kernel's tools/ directory. 1113 1114[:ref:`back to step-by-step guide <buildrequires_bissbs>`] 1115 1116.. _sources_bisref: 1117 1118Download the sources using Git 1119------------------------------ 1120 1121 *Retrieve the Linux mainline sources.* 1122 [:ref:`...<sources_bissbs>`] 1123 1124The step-by-step guide outlines how to download the Linux sources using a full 1125Git clone of Linus' mainline repository. There is nothing more to say about 1126that -- but there are two alternatives ways to retrieve the sources that might 1127work better for you: 1128 1129 * If you have an unreliable internet connection, consider 1130 :ref:`using a 'Git bundle'<sources_bundle_bisref>`. 1131 1132 * If downloading the complete repository would take too long or requires too 1133 much storage space, consider :ref:`using a 'shallow 1134 clone'<sources_shallow_bisref>`. 1135 1136.. _sources_bundle_bisref: 1137 1138Downloading Linux mainline sources using a bundle 1139~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1140 1141Use the following commands to retrieve the Linux mainline sources using a 1142bundle:: 1143 1144 wget -c \ 1145 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle 1146 git clone --no-checkout clone.bundle ~/linux/ 1147 cd ~/linux/ 1148 git remote remove origin 1149 git remote add mainline \ 1150 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 1151 git fetch mainline 1152 git remote add -t master stable \ 1153 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git 1154 1155In case the 'wget' command fails, just re-execute it, it will pick up where 1156it left off. 1157 1158[:ref:`back to step-by-step guide <sources_bissbs>`] 1159[:ref:`back to section intro <sources_bisref>`] 1160 1161.. _sources_shallow_bisref: 1162 1163Downloading Linux mainline sources using a shallow clone 1164~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1165 1166First, execute the following command to retrieve the latest mainline codebase:: 1167 1168 git clone -o mainline --no-checkout --depth 1 -b master \ 1169 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/ 1170 cd ~/linux/ 1171 git remote add -t master stable \ 1172 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git 1173 1174Now deepen your clone's history to the second predecessor of the mainline 1175release of your 'good' version. In case the latter are 6.0 or 6.0.11, 5.19 would 1176be the first predecessor and 5.18 the second -- hence deepen the history up to 1177that version:: 1178 1179 git fetch --shallow-exclude=v5.18 mainline 1180 1181Afterwards add the stable Git repository as remote and all required stable 1182branches as explained in the step-by-step guide. 1183 1184Note, shallow clones have a few peculiar characteristics: 1185 1186 * For bisections the history needs to be deepened a few mainline versions 1187 farther than it seems necessary, as explained above already. That's because 1188 Git otherwise will be unable to revert or describe most of the commits within 1189 a range (say v6.1..v6.2), as they are internally based on earlier kernels 1190 releases (like v6.0-rc2 or 5.19-rc3). 1191 1192 * This document in most places uses ``git fetch`` with ``--shallow-exclude=`` 1193 to specify the earliest version you care about (or to be precise: its git 1194 tag). You alternatively can use the parameter ``--shallow-since=`` to specify 1195 an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to 1196 define the depth of the history you want to download. When using them while 1197 bisecting mainline, ensure to deepen the history to at least 7 months before 1198 the release of the mainline release your 'good' kernel is based on. 1199 1200 * Be warned, when deepening your clone you might encounter an error like 1201 'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'. 1202 In that case run ``git repack -d`` and try again. 1203 1204[:ref:`back to step-by-step guide <sources_bissbs>`] 1205[:ref:`back to section intro <sources_bisref>`] 1206 1207.. _oldconfig_bisref: 1208 1209Start defining the build configuration for your kernel 1210------------------------------------------------------ 1211 1212 *Start preparing a kernel build configuration (the '.config' file).* 1213 [:ref:`... <oldconfig_bissbs>`] 1214 1215*Note, this is the first of multiple steps in this guide that create or modify 1216build artifacts. The commands used in this guide store them right in the source 1217tree to keep things simple. In case you prefer storing the build artifacts 1218separately, create a directory like '~/linux-builddir/' and add the parameter 1219``O=~/linux-builddir/`` to all make calls used throughout this guide. You will 1220have to point other commands there as well -- among them the ``./scripts/config 1221[...]`` commands, which will require ``--file ~/linux-builddir/.config`` to 1222locate the right build configuration.* 1223 1224Two things can easily go wrong when creating a .config file as advised: 1225 1226 * The oldconfig target will use a .config file from your build directory, if 1227 one is already present there (e.g. '~/linux/.config'). That's totally fine if 1228 that's what you intend (see next step), but in all other cases you want to 1229 delete it. This for example is important in case you followed this guide 1230 further, but due to problems come back here to redo the configuration from 1231 scratch. 1232 1233 * Sometimes olddefconfig is unable to locate the .config file for your running 1234 kernel and will use defaults, as briefly outlined in the guide. In that case 1235 check if your distribution ships the configuration somewhere and manually put 1236 it in the right place (e.g. '~/linux/.config') if it does. On distributions 1237 where /proc/config.gz exists this can be achieved using this command:: 1238 1239 zcat /proc/config.gz > .config 1240 1241 Once you put it there, run ``make olddefconfig`` again to adjust it to the 1242 needs of the kernel about to be built. 1243 1244Note, the olddefconfig target will set any undefined build options to their 1245default value. If you prefer to set such configuration options manually, use 1246``make oldconfig`` instead. Then for each undefined configuration option you 1247will be asked how to proceed; in case you are unsure what to answer, simply hit 1248'enter' to apply the default value. Note though that for bisections you normally 1249want to go with the defaults, as you otherwise might enable a new feature that 1250causes a problem looking like regressions (for example due to security 1251restrictions). 1252 1253Occasionally odd things happen when trying to use a config file prepared for one 1254kernel (say 6.1) on an older mainline release -- especially if it is much older 1255(say v5.15). That's one of the reasons why the previous step in the guide told 1256you to boot the kernel where everything works. If you manually add a .config 1257file you thus want to ensure it's from the working kernel and not from a one 1258that shows the regression. 1259 1260In case you want to build kernels for another machine, locate its kernel build 1261configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy 1262that file to the build machine and store it as ~/linux/.config; afterwards run 1263``make olddefconfig`` to adjust it. 1264 1265[:ref:`back to step-by-step guide <oldconfig_bissbs>`] 1266 1267.. _localmodconfig_bisref: 1268 1269Trim the build configuration for your kernel 1270-------------------------------------------- 1271 1272 *Disable any kernel modules apparently superfluous for your setup.* 1273 [:ref:`... <localmodconfig_bissbs>`] 1274 1275As explained briefly in the step-by-step guide already: with localmodconfig it 1276can easily happen that your self-built kernels will lack modules for tasks you 1277did not perform at least once before utilizing this make target. That happens 1278when a task requires kernel modules which are only autoloaded when you execute 1279it for the first time. So when you never performed that task since starting your 1280kernel the modules will not have been loaded -- and from localmodonfig's point 1281of view look superfluous, which thus disables them to reduce the amount of code 1282to be compiled. 1283 1284You can try to avoid this by performing typical tasks that often will autoload 1285additional kernel modules: start a VM, establish VPN connections, loop-mount a 1286CD/DVD ISO, mount network shares (CIFS, NFS, ...), and connect all external 1287devices (2FA keys, headsets, webcams, ...) as well as storage devices with file 1288systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, ...). But it 1289is hard to think of everything that might be needed -- even kernel developers 1290often forget one thing or another at this point. 1291 1292Do not let that risk bother you, especially when compiling a kernel only for 1293testing purposes: everything typically crucial will be there. And if you forget 1294something important you can turn on a missing feature manually later and quickly 1295run the commands again to compile and install a kernel that has everything you 1296need. 1297 1298But if you plan to build and use self-built kernels regularly, you might want to 1299reduce the risk by recording which modules your system loads over the course of 1300a few weeks. You can automate this with `modprobed-db 1301<https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to 1302point localmodconfig to the list of modules modprobed-db noticed being used:: 1303 1304 yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig 1305 1306That parameter also allows you to build trimmed kernels for another machine in 1307case you copied a suitable .config over to use as base (see previous step). Just 1308run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to 1309your build's host home directory. Then run these commands instead of the one the 1310step-by-step guide mentions:: 1311 1312 yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig 1313 1314[:ref:`back to step-by-step guide <localmodconfig_bissbs>`] 1315 1316.. _tagging_bisref: 1317 1318Tag the kernels about to be build 1319--------------------------------- 1320 1321 *Ensure all the kernels you will build are clearly identifiable using a 1322 special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`] 1323 1324This allows you to differentiate your distribution's kernels from those created 1325during this process, as the file or directories for the latter will contain 1326'-local' in the name; it also helps picking the right entry in the boot menu and 1327not lose track of you kernels, as their version numbers will look slightly 1328confusing during the bisection. 1329 1330[:ref:`back to step-by-step guide <tagging_bissbs>`] 1331 1332.. _debugsymbols_bisref: 1333 1334Decide to enable or disable debug symbols 1335----------------------------------------- 1336 1337 *Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`] 1338 1339Having debug symbols available can be important when your kernel throws a 1340'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be 1341able to find the exact place where the problem occurred in the code. But 1342collecting and embedding the needed debug information takes time and consumes 1343quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel 1344trimmed with localmodconfig consumed around 5 Gigabyte of space with debug 1345symbols, but less than 1 when they were disabled. The resulting kernel image and 1346modules are bigger as well, which increases storage requirements for /boot/ and 1347load times. 1348 1349In case you want a small kernel and are unlikely to decode a stack trace later, 1350you thus might want to disable debug symbols to avoid those downsides. If it 1351later turns out that you need them, just enable them as shown and rebuild the 1352kernel. 1353 1354You on the other hand definitely want to enable them for this process, if there 1355is a decent chance that you need to decode a stack trace later. The section 1356'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst 1357explains this process in more detail. 1358 1359[:ref:`back to step-by-step guide <debugsymbols_bissbs>`] 1360 1361.. _configmods_bisref: 1362 1363Adjust build configuration 1364-------------------------- 1365 1366 *Check if you may want or need to adjust some other kernel configuration 1367 options:* 1368 1369Depending on your needs you at this point might want or have to adjust some 1370kernel configuration options. 1371 1372.. _configmods_distros_bisref: 1373 1374Distro specific adjustments 1375~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1376 1377 *Are you running* [:ref:`... <configmods_bissbs>`] 1378 1379The following sections help you to avoid build problems that are known to occur 1380when following this guide on a few commodity distributions. 1381 1382**Debian:** 1383 1384 * Remove a stale reference to a certificate file that would cause your build to 1385 fail:: 1386 1387 ./scripts/config --set-str SYSTEM_TRUSTED_KEYS '' 1388 1389 Alternatively, download the needed certificate and make that configuration 1390 option point to it, as `the Debian handbook explains in more detail 1391 <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_ 1392 -- or generate your own, as explained in 1393 Documentation/admin-guide/module-signing.rst. 1394 1395[:ref:`back to step-by-step guide <configmods_bissbs>`] 1396 1397.. _configmods_individual_bisref: 1398 1399Individual adjustments 1400~~~~~~~~~~~~~~~~~~~~~~ 1401 1402 *If you want to influence the other aspects of the configuration, do so 1403 now.* [:ref:`... <configmods_bissbs>`] 1404 1405You at this point can use a command like ``make menuconfig`` to enable or 1406disable certain features using a text-based user interface; to use a graphical 1407configuration utility, call the make target ``xconfig`` or ``gconfig`` instead. 1408All of them require development libraries from toolkits they are based on 1409(ncurses, Qt5, Gtk2); an error message will tell you if something required is 1410missing. 1411 1412[:ref:`back to step-by-step guide <configmods_bissbs>`] 1413 1414.. _saveconfig_bisref: 1415 1416Put the .config file aside 1417-------------------------- 1418 1419 *Reprocess the .config after the latest changes and store it in a safe place.* 1420 [:ref:`... <saveconfig_bissbs>`] 1421 1422Put the .config you prepared aside, as you want to copy it back to the build 1423directory every time during this guide before you start building another 1424kernel. That's because going back and forth between different versions can alter 1425.config files in odd ways; those occasionally cause side effects that could 1426confuse testing or in some cases render the result of your bisection 1427meaningless. 1428 1429[:ref:`back to step-by-step guide <saveconfig_bissbs>`] 1430 1431.. _introlatestcheck_bisref: 1432 1433Try to reproduce the regression 1434----------------------------------------- 1435 1436 *Verify the regression is not caused by some .config change and check if it 1437 still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`] 1438 1439For some readers it might seem unnecessary to check the latest codebase at this 1440point, especially if you did that already with a kernel prepared by your 1441distributor or face a regression within a stable/longterm series. But it's 1442highly recommended for these reasons: 1443 1444* You will run into any problems caused by your setup before you actually begin 1445 a bisection. That will make it a lot easier to differentiate between 'this 1446 most likely is some problem in my setup' and 'this change needs to be skipped 1447 during the bisection, as the kernel sources at that stage contain an unrelated 1448 problem that causes building or booting to fail'. 1449 1450* These steps will rule out if your problem is caused by some change in the 1451 build configuration between the 'working' and the 'broken' kernel. This for 1452 example can happen when your distributor enabled an additional security 1453 feature in the newer kernel which was disabled or not yet supported by the 1454 older kernel. That security feature might get into the way of something you 1455 do -- in which case your problem from the perspective of the Linux kernel 1456 upstream developers is not a regression, as 1457 Documentation/admin-guide/reporting-regressions.rst explains in more detail. 1458 You thus would waste your time if you'd try to bisect this. 1459 1460* If the cause for your regression was already fixed in the latest mainline 1461 codebase, you'd perform the bisection for nothing. This holds true for a 1462 regression you encountered with a stable/longterm release as well, as they are 1463 often caused by problems in mainline changes that were backported -- in which 1464 case the problem will have to be fixed in mainline first. Maybe it already was 1465 fixed there and the fix is already in the process of being backported. 1466 1467* For regressions within a stable/longterm series it's furthermore crucial to 1468 know if the issue is specific to that series or also happens in the mainline 1469 kernel, as the report needs to be sent to different people: 1470 1471 * Regressions specific to a stable/longterm series are the stable team's 1472 responsibility; mainline Linux developers might or might not care. 1473 1474 * Regressions also happening in mainline are something the regular Linux 1475 developers and maintainers have to handle; the stable team does not care 1476 and does not need to be involved in the report, they just should be told 1477 to backport the fix once it's ready. 1478 1479 Your report might be ignored if you send it to the wrong party -- and even 1480 when you get a reply there is a decent chance that developers tell you to 1481 evaluate which of the two cases it is before they take a closer look. 1482 1483[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`] 1484 1485.. _checkoutmaster_bisref: 1486 1487Checkout the latest Linux codebase 1488---------------------------------- 1489 1490 *Checkout the latest Linux codebase.* 1491 [:ref:`... <introlatestcheck_bissbs>`] 1492 1493In case you later want to recheck if an ever newer codebase might fix the 1494problem, remember to run that ``git fetch --shallow-exclude [...]`` command 1495again mentioned earlier to update your local Git repository. 1496 1497[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`] 1498 1499.. _build_bisref: 1500 1501Build your kernel 1502----------------- 1503 1504 *Build the image and the modules of your first kernel using the config file 1505 you prepared.* [:ref:`... <build_bissbs>`] 1506 1507A lot can go wrong at this stage, but the instructions below will help you help 1508yourself. Another subsection explains how to directly package your kernel up as 1509deb, rpm or tar file. 1510 1511Dealing with build errors 1512~~~~~~~~~~~~~~~~~~~~~~~~~ 1513 1514When a build error occurs, it might be caused by some aspect of your machine's 1515setup that often can be fixed quickly; other times though the problem lies in 1516the code and can only be fixed by a developer. A close examination of the 1517failure messages coupled with some research on the internet will often tell you 1518which of the two it is. To perform such a investigation, restart the build 1519process like this:: 1520 1521 make V=1 1522 1523The ``V=1`` activates verbose output, which might be needed to see the actual 1524error. To make it easier to spot, this command also omits the ``-j $(nproc 1525--all)`` used earlier to utilize every CPU core in the system for the job -- but 1526this parallelism also results in some clutter when failures occur. 1527 1528After a few seconds the build process should run into the error again. Now try 1529to find the most crucial line describing the problem. Then search the internet 1530for the most important and non-generic section of that line (say 4 to 8 words); 1531avoid or remove anything that looks remotely system-specific, like your username 1532or local path names like ``/home/username/linux/``. First try your regular 1533internet search engine with that string, afterwards search Linux kernel mailing 1534lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_. 1535 1536This most of the time will find something that will explain what is wrong; quite 1537often one of the hits will provide a solution for your problem, too. If you 1538do not find anything that matches your problem, try again from a different angle 1539by modifying your search terms or using another line from the error messages. 1540 1541In the end, most trouble you are to run into has likely been encountered and 1542reported by others already. That includes issues where the cause is not your 1543system, but lies the code. If you run into one of those, you might thus find a 1544solution (e.g. a patch) or workaround for your problem, too. 1545 1546Package your kernel up 1547~~~~~~~~~~~~~~~~~~~~~~ 1548 1549The step-by-step guide uses the default make targets (e.g. 'bzImage' and 1550'modules' on x86) to build the image and the modules of your kernel, which later 1551steps of the guide then install. You instead can also directly build everything 1552and directly package it up by using one of the following targets: 1553 1554 * ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package 1555 1556 * ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package 1557 1558 * ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball 1559 1560This is just a selection of available make targets for this purpose, see 1561``make help`` for others. You can also use these targets after running 1562``make -j $(nproc --all)``, as they will pick up everything already built. 1563 1564If you employ the targets to generate deb or rpm packages, ignore the 1565step-by-step guide's instructions on installing and removing your kernel; 1566instead install and remove the packages using the package utility for the format 1567(e.g. dpkg and rpm) or a package management utility build on top of them (apt, 1568aptitude, dnf/yum, zypper, ...). Be aware that the packages generated using 1569these two make targets are designed to work on various distributions utilizing 1570those formats, they thus will sometimes behave differently than your 1571distribution's kernel packages. 1572 1573[:ref:`back to step-by-step guide <build_bissbs>`] 1574 1575.. _install_bisref: 1576 1577Put the kernel in place 1578----------------------- 1579 1580 *Install the kernel you just built.* [:ref:`... <install_bissbs>`] 1581 1582What you need to do after executing the command in the step-by-step guide 1583depends on the existence and the implementation of an ``installkernel`` 1584executable. Many commodity Linux distributions ship such a kernel installer in 1585'/sbin/' that does everything needed, hence there is nothing left for you 1586except rebooting. But some distributions contain an installkernel that does 1587only part of the job -- and a few lack it completely and leave all the work to 1588you. 1589 1590If ``installkernel`` is found, the kernel's build system will delegate the 1591actual installation of your kernel's image and related files to this executable. 1592On almost all Linux distributions it will store the image as '/boot/vmlinuz- 1593<kernelrelease identifier>' and put a 'System.map-<kernelrelease 1594identifier>' alongside it. Your kernel will thus be installed in parallel to any 1595existing ones, unless you already have one with exactly the same release name. 1596 1597Installkernel on many distributions will afterwards generate an 'initramfs' 1598(often also called 'initrd'), which commodity distributions rely on for booting; 1599hence be sure to keep the order of the two make targets used in the step-by-step 1600guide, as things will go sideways if you install your kernel's image before its 1601modules. Often installkernel will then add your kernel to the bootloader 1602configuration, too. You have to take care of one or both of these tasks 1603yourself, if your distributions installkernel doesn't handle them. 1604 1605A few distributions like Arch Linux and its derivatives totally lack an 1606installkernel executable. On those just install the modules using the kernel's 1607build system and then install the image and the System.map file manually:: 1608 1609 sudo make modules_install 1610 sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease) 1611 sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease) 1612 1613If your distribution boots with the help of an initramfs, now generate one for 1614your kernel using the tools your distribution provides for this process. 1615Afterwards add your kernel to your bootloader configuration and reboot. 1616 1617[:ref:`back to step-by-step guide <install_bissbs>`] 1618 1619.. _storagespace_bisref: 1620 1621Storage requirements per kernel 1622------------------------------- 1623 1624 *Check how much storage space the kernel, its modules, and other related files 1625 like the initramfs consume.* [:ref:`... <storagespace_bissbs>`] 1626 1627The kernels built during a bisection consume quite a bit of space in /boot/ and 1628/lib/modules/, especially if you enabled debug symbols. That makes it easy to 1629fill up volumes during a bisection -- and due to that even kernels which used to 1630work earlier might fail to boot. To prevent that you will need to know how much 1631space each installed kernel typically requires. 1632 1633Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in 1634the guide will match all files needed to boot your kernel -- but neither the 1635path nor the naming scheme are mandatory. On some distributions you thus will 1636need to look in different places. 1637 1638[:ref:`back to step-by-step guide <storagespace_bissbs>`] 1639 1640.. _recheckbroken_bisref: 1641 1642Check the kernel built from the latest codebase 1643----------------------------------------------- 1644 1645 *Reboot into the kernel you just built and check if the feature that regressed 1646 is really broken there.* [:ref:`... <recheckbroken_bissbs>`] 1647 1648There are a couple of reasons why the regression you face might not show up with 1649your own kernel built from the latest codebase. These are the most frequent: 1650 1651* The cause for the regression was fixed meanwhile. 1652 1653* The regression with the broken kernel was caused by a change in the build 1654 configuration the provider of your kernel carried out. 1655 1656* Your problem might be a race condition that does not show up with your kernel; 1657 the trimmed build configuration, a different setting for debug symbols, the 1658 compiler used, and various other things can cause this. 1659 1660* In case you encountered the regression with a stable/longterm kernel it might 1661 be a problem that is specific to that series; the next step in this guide will 1662 check this. 1663 1664[:ref:`back to step-by-step guide <recheckbroken_bissbs>`] 1665 1666.. _recheckstablebroken_bisref: 1667 1668Check the kernel built from the latest stable/longterm codebase 1669--------------------------------------------------------------- 1670 1671 *Are you facing a regression within a stable/longterm release, but failed to 1672 reproduce it with the kernel you just built using the latest mainline sources? 1673 Then check if the latest codebase for the particular series might already fix 1674 the problem.* [:ref:`... <recheckstablebroken_bissbs>`] 1675 1676If this kernel does not show the regression either, there most likely is no need 1677for a bisection. 1678 1679[:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`] 1680 1681.. _introworkingcheck_bisref: 1682 1683Ensure the 'good' version is really working well 1684------------------------------------------------ 1685 1686 *Check if the kernels you build work fine.* 1687 [:ref:`... <introworkingcheck_bissbs>`] 1688 1689This section will reestablish a known working base. Skipping it might be 1690appealing, but is usually a bad idea, as it does something important: 1691 1692It will ensure the .config file you prepared earlier actually works as expected. 1693That is in your own interest, as trimming the configuration is not foolproof -- 1694and you might be building and testing ten or more kernels for nothing before 1695starting to suspect something might be wrong with the build configuration. 1696 1697That alone is reason enough to spend the time on this, but not the only reason. 1698 1699Many readers of this guide normally run kernels that are patched, use add-on 1700modules, or both. Those kernels thus are not considered 'vanilla' -- therefore 1701it's possible that the thing that regressed might never have worked in vanilla 1702builds of the 'good' version in the first place. 1703 1704There is a third reason for those that noticed a regression between 1705stable/longterm kernels of different series (e.g. v6.0.13..v6.1.5): it will 1706ensure the kernel version you assumed to be 'good' earlier in the process (e.g. 1707v6.0) actually is working. 1708 1709[:ref:`back to step-by-step guide <introworkingcheck_bissbs>`] 1710 1711.. _recheckworking_bisref: 1712 1713Build your own version of the 'good' kernel 1714------------------------------------------- 1715 1716 *Build your own variant of the working kernel and check if the feature that 1717 regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`] 1718 1719In case the feature that broke with newer kernels does not work with your first 1720self-built kernel, find and resolve the cause before moving on. There are a 1721multitude of reasons why this might happen. Some ideas where to look: 1722 1723* Maybe localmodconfig did something odd and disabled the module required to 1724 test the feature? Then you might want to recreate a .config file based on the 1725 one from the last working kernel and skip trimming it down; manually disabling 1726 some features in the .config might work as well to reduce the build time. 1727 1728* Maybe it's not a kernel regression and something that is caused by some fluke, 1729 a broken initramfs (also known as initrd), new firmware files, or an updated 1730 userland software? 1731 1732* Maybe it was a feature added to your distributor's kernel which vanilla Linux 1733 at that point never supported? 1734 1735Note, if you found and fixed problems with the .config file, you want to use it 1736to build another kernel from the latest codebase, as your earlier tests with 1737mainline and the latest version from an affected stable/longterm series most 1738likely has been flawed. 1739 1740[:ref:`back to step-by-step guide <recheckworking_bissbs>`] 1741 1742.. _bisectstart_bisref: 1743 1744Start the bisection 1745------------------- 1746 1747 *Start the bisection and tell Git about the versions earlier established as 1748 'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`] 1749 1750This will start the bisection process; the last of the commands will make Git 1751checkout a commit round about half-way between the 'good' and the 'bad' changes 1752for your to test. 1753 1754[:ref:`back to step-by-step guide <bisectstart_bissbs>`] 1755 1756.. _bisectbuild_bisref: 1757 1758Build a kernel from the bisection point 1759--------------------------------------- 1760 1761 *Build, install, and boot a kernel from the code Git checked out using the 1762 same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`] 1763 1764There are two things worth of note here: 1765 1766* Occasionally building the kernel will fail or it might not boot due some 1767 problem in the code at the bisection point. In that case run this command:: 1768 1769 git bisect skip 1770 1771 Git will then check out another commit nearby which with a bit of luck should 1772 work better. Afterwards restart executing this step. 1773 1774* Those slightly odd looking version identifiers can happen during bisections, 1775 because the Linux kernel subsystems prepare their changes for a new mainline 1776 release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus 1777 base them on a somewhat earlier point like v6.1-rc1 or even v6.0 -- and then 1778 get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This 1779 leads to those slightly odd looking version identifiers coming up during 1780 bisections. 1781 1782[:ref:`back to step-by-step guide <bisectbuild_bissbs>`] 1783 1784.. _bisecttest_bisref: 1785 1786Bisection checkpoint 1787-------------------- 1788 1789 *Check if the feature that regressed works in the kernel you just built.* 1790 [:ref:`... <bisecttest_bissbs>`] 1791 1792Ensure what you tell Git is accurate: getting it wrong just one time will bring 1793the rest of the bisection totally of course, hence all testing after that point 1794will be for nothing. 1795 1796[:ref:`back to step-by-step guide <bisecttest_bissbs>`] 1797 1798.. _bisectlog_bisref: 1799 1800Put the bisection log away 1801-------------------------- 1802 1803 *Store Git's bisection log and the current .config file in a safe place.* 1804 [:ref:`... <bisectlog_bissbs>`] 1805 1806As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will 1807render the end result of a bisection useless. In that case you'd normally have 1808to restart the bisection from scratch. The log can prevent that, as it might 1809allow someone to point out where a bisection likely went sideways -- and then 1810instead of testing ten or more kernels you might only have to build a few to 1811resolve things. 1812 1813The .config file is put aside, as there is a decent chance that developers might 1814ask for it after you reported the regression. 1815 1816[:ref:`back to step-by-step guide <bisectlog_bissbs>`] 1817 1818.. _revert_bisref: 1819 1820Try reverting the culprit 1821------------------------- 1822 1823 *Try reverting the culprit on top of the latest codebase to see if this fixes 1824 your regression.* [:ref:`... <revert_bissbs>`] 1825 1826This is an optional step, but whenever possible one you should try: there is a 1827decent chance that developers will ask you to perform this step when you bring 1828the bisection result up. So give it a try, you are in the flow already, building 1829one more kernel shouldn't be a big deal at this point. 1830 1831The step-by-step guide covers everything relevant already except one slightly 1832rare thing: did you bisected a regression that also happened with mainline using 1833a stable/longterm series, but Git failed to revert the commit in mainline? Then 1834try to revert the culprit in the affected stable/longterm series -- and if that 1835succeeds, test that kernel version instead. 1836 1837[:ref:`back to step-by-step guide <revert_bissbs>`] 1838 1839 1840Supplementary tasks: cleanup during and after the bisection 1841----------------------------------------------------------- 1842 1843.. _makeroom_bisref: 1844 1845Cleaning up during the bisection 1846-------------------------------- 1847 1848 *To remove one of the kernels you installed, look up its 'kernelrelease' 1849 identifier.* [:ref:`... <makeroom_bissbs>`] 1850 1851The kernels you install during this process are easy to remove later, as its 1852parts are only stored in two places and clearly identifiable. You thus do not 1853need to worry to mess up your machine when you install a kernel manually (and 1854thus bypass your distribution's packaging system): all parts of your kernels are 1855relatively easy to remove later. 1856 1857One of the two places is a directory in /lib/modules/, which holds the modules 1858for each installed kernel. This directory is named after the kernel's release 1859identifier; hence, to remove all modules for one of the kernels you built, 1860simply remove its modules directory in /lib/modules/. 1861 1862The other place is /boot/, where typically two up to five files will be placed 1863during installation of a kernel. All of them usually contain the release name in 1864their file name, but how many files and their exact name depends somewhat on 1865your distribution's installkernel executable and its initramfs generator. On 1866some distributions the ``kernel-install remove...`` command mentioned in the 1867step-by-step guide will delete all of these files for you while also removing 1868the menu entry for the kernel from your bootloader configuration. On others you 1869have to take care of these two tasks yourself. The following command should 1870interactively remove the three main files of a kernel with the release name 1871'6.0-rc1-local-gcafec0cacaca0':: 1872 1873 rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0 1874 1875Afterwards check for other files in /boot/ that have 1876'6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well. 1877Now remove the boot entry for the kernel from your bootloader's configuration; 1878the steps to do that vary quite a bit between Linux distributions. 1879 1880Note, be careful with wildcards like '*' when deleting files or directories 1881for kernels manually: you might accidentally remove files of a 6.0.11 kernel 1882when all you want is to remove 6.0 or 6.0.1. 1883 1884[:ref:`back to step-by-step guide <makeroom_bissbs>`] 1885 1886Cleaning up after the bisection 1887------------------------------- 1888 1889.. _finishingtouch_bisref: 1890 1891 *Once you have finished the bisection, do not immediately remove anything 1892 you set up, as you might need a few things again.* 1893 [:ref:`... <finishingtouch_bissbs>`] 1894 1895When you are really short of storage space removing the kernels as described in 1896the step-by-step guide might not free as much space as you would like. In that 1897case consider running ``rm -rf ~/linux/*`` as well now. This will remove the 1898build artifacts and the Linux sources, but will leave the Git repository 1899(~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the 1900sources back. 1901 1902Removing the repository as well would likely be unwise at this point: there is a 1903decent chance developers will ask you to build another kernel to perform 1904additional tests. This is often required to debug an issue or check proposed 1905fixes. Before doing so you want to run the ``git fetch mainline`` command again 1906followed by ``git checkout mainline/master`` to bring your clone up to date and 1907checkout the latest codebase. Then apply the patch using ``git apply 1908<filename>`` or ``git am <filename>`` and build yet another kernel using the 1909familiar commands. 1910 1911Additional tests are also the reason why you want to keep the 1912~/kernel-config-working file around for a few weeks. 1913 1914[:ref:`back to step-by-step guide <finishingtouch_bissbs>`] 1915 1916 1917Additional reading material 1918=========================== 1919 1920Further sources 1921--------------- 1922 1923* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and 1924 `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_ 1925 in the Git documentation. 1926* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_ 1927 from kernel developer Nathan Chancellor. 1928* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_. 1929* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_. 1930 1931.. 1932 end-of-content 1933.. 1934 This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If 1935 you spot a typo or small mistake, feel free to let him know directly and 1936 he'll fix it. You are free to do the same in a mostly informal way if you 1937 want to contribute changes to the text -- but for copyright reasons please CC 1938 linux-doc@vger.kernel.org and 'sign-off' your contribution as 1939 Documentation/process/submitting-patches.rst explains in the section 'Sign 1940 your work - the Developer's Certificate of Origin'. 1941.. 1942 This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top 1943 of the file. If you want to distribute this text under CC-BY-4.0 only, 1944 please use 'The Linux kernel development community' for author attribution 1945 and link this as source: 1946 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst 1947 1948.. 1949 Note: Only the content of this RST file as found in the Linux kernel sources 1950 is available under CC-BY-4.0, as versions of this text that were processed 1951 (for example by the kernel's build system) might contain content taken from 1952 files which use a more restrictive license. 1953