xref: /linux/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst (revision f6e0a4984c2e7244689ea87b62b433bed9d07e94)
1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
2.. [see the bottom of this file for redistribution information]
3
4=========================================
5How to verify bugs and bisect regressions
6=========================================
7
8This document describes how to check if some Linux kernel problem occurs in code
9currently supported by developers -- to then explain how to locate the change
10causing the issue, if it is a regression (e.g. did not happen with earlier
11versions).
12
13The text aims at people running kernels from mainstream Linux distributions on
14commodity hardware who want to report a kernel bug to the upstream Linux
15developers. Despite this intent, the instructions work just as well for users
16who are already familiar with building their own kernels: they help avoid
17mistakes occasionally made even by experienced developers.
18
19..
20   Note: if you see this note, you are reading the text's source file. You
21   might want to switch to a rendered version: it makes it a lot easier to
22   read and navigate this document -- especially when you want to look something
23   up in the reference section, then jump back to where you left off.
24..
25   Find the latest rendered version of this text here:
26   https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.rst.html
27
28The essence of the process (aka 'TL;DR')
29========================================
30
31*[If you are new to building or bisecting Linux, ignore this section and head
32over to the* ":ref:`step-by-step guide<introguide_bissbs>`" *below. It utilizes
33the same commands as this section while describing them in brief fashion. The
34steps are nevertheless easy to follow and together with accompanying entries
35in a reference section mention many alternatives, pitfalls, and additional
36aspects, all of which might be essential in your present case.]*
37
38**In case you want to check if a bug is present in code currently supported by
39developers**, execute just the *preparations* and *segment 1*; while doing so,
40consider the newest Linux kernel you regularly use to be the 'working' kernel.
41In the following example that's assumed to be 6.0.13, which is why the sources
42of v6.0 will be used to prepare the .config file.
43
44**In case you face a regression**, follow the steps at least till the end of
45*segment 2*. Then you can submit a preliminary report -- or continue with
46*segment 3*, which describes how to perform a bisection needed for a
47full-fledged regression report. In the following example 6.0.13 is assumed to be
48the 'working' kernel and 6.1.5 to be the first 'broken', which is why v6.0
49will be considered the 'good' release and used to prepare the .config file.
50
51* **Preparations**: set up everything to build your own kernels::
52
53    # * Remove any software that depends on externally maintained kernel modules
54    #   or builds any automatically during bootup.
55    # * Ensure Secure Boot permits booting self-compiled Linux kernels.
56    # * If you are not already running the 'working' kernel, reboot into it.
57    # * Install compilers and everything else needed for building Linux.
58    # * Ensure to have 15 Gigabyte free space in your home directory.
59    git clone -o mainline --no-checkout \
60      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
61    cd ~/linux/
62    git remote add -t master stable \
63      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
64    git checkout --detach v6.0
65    # * Hint: if you used an existing clone, ensure no stale .config is around.
66    make olddefconfig
67    # * Ensure the former command picked the .config of the 'working' kernel.
68    # * Connect external hardware (USB keys, tokens, ...), start a VM, bring up
69    #   VPNs, mount network shares, and briefly try the feature that is broken.
70    yes '' | make localmodconfig
71    ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
72    ./scripts/config -e CONFIG_LOCALVERSION_AUTO
73    # * Note, when short on storage space, check the guide for an alternative:
74    ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
75      -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
76    # * Hint: at this point you might want to adjust the build configuration;
77    #   you'll have to, if you are running Debian.
78    make olddefconfig
79    cp .config ~/kernel-config-working
80
81* **Segment 1**: build a kernel from the latest mainline codebase.
82
83  This among others checks if the problem was fixed already and which developers
84  later need to be told about the problem; in case of a regression, this rules
85  out a .config change as root of the problem.
86
87  a) Checking out latest mainline code::
88
89       cd ~/linux/
90       git checkout --force --detach mainline/master
91
92  b) Build, install, and boot a kernel::
93
94       cp ~/kernel-config-working .config
95       make olddefconfig
96       make -j $(nproc --all)
97       # * Make sure there is enough disk space to hold another kernel:
98       df -h /boot/ /lib/modules/
99       # * Note: on Arch Linux, its derivatives and a few other distributions
100       #   the following commands will do nothing at all or only part of the
101       #   job. See the step-by-step guide for further details.
102       command -v installkernel && sudo make modules_install install
103       # * Check how much space your self-built kernel actually needs, which
104       #   enables you to make better estimates later:
105       du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
106       du -sh /lib/modules/$(make -s kernelrelease)/
107       # * Hint: the output of the following command will help you pick the
108       #   right kernel from the boot menu:
109       make -s kernelrelease | tee -a ~/kernels-built
110       reboot
111       # * Once booted, ensure you are running the kernel you just built by
112       #   checking if the output of the next two commands matches:
113       tail -n 1 ~/kernels-built
114       uname -r
115
116  c) Check if the problem occurs with this kernel as well.
117
118* **Segment 2**: ensure the 'good' kernel is also a 'working' kernel.
119
120  This among others verifies the trimmed .config file actually works well, as
121  bisecting with it otherwise would be a waste of time:
122
123  a) Start by checking out the sources of the 'good' version::
124
125       cd ~/linux/
126       git checkout --force --detach v6.0
127
128  b) Build, install, and boot a kernel as described earlier in *segment 1,
129     section b* -- just feel free to skip the 'du' commands, as you have a rough
130     estimate already.
131
132  c) Ensure the feature that regressed with the 'broken' kernel actually works
133     with this one.
134
135* **Segment 3**: perform and validate the bisection.
136
137  a) In case your 'broken' version is a stable/longterm release, add the Git
138     branch holding it::
139
140       git remote set-branches --add stable linux-6.1.y
141       git fetch stable
142
143  b) Initialize the bisection::
144
145       cd ~/linux/
146       git bisect start
147       git bisect good v6.0
148       git bisect bad v6.1.5
149
150  c) Build, install, and boot a kernel as described earlier in *segment 1,
151     section b*.
152
153     In case building or booting the kernel fails for unrelated reasons, run
154     ``git bisect skip``. In all other outcomes, check if the regressed feature
155     works with the newly built kernel. If it does, tell Git by executing
156     ``git bisect good``; if it does not, run ``git bisect bad`` instead.
157
158     All three commands will make Git checkout another commit; then re-execute
159     this step (e.g. build, install, boot, and test a kernel to then tell Git
160     the outcome). Do so again and again until Git shows which commit broke
161     things. If you run short of disk space during this process, check the
162     "Supplementary tasks" section below.
163
164  d) Once your finished the bisection, put a few things away::
165
166       cd ~/linux/
167       git bisect log > ~/bisect-log
168       cp .config ~/bisection-config-culprit
169       git bisect reset
170
171  e) Try to verify the bisection result::
172
173       git checkout --force --detach mainline/master
174       git revert --no-edit cafec0cacaca0
175
176    This is optional, as some commits are impossible to revert. But if the
177    second command worked flawlessly, build, install, and boot one more kernel
178    kernel, which should not show the regression.
179
180* **Supplementary tasks**: cleanup during and after the process.
181
182  a) To avoid running out of disk space during a bisection, you might need to
183     remove some kernels you built earlier. You most likely want to keep those
184     you built during segment 1 and 2 around for a while, but you will most
185     likely no longer need kernels tested during the actual bisection
186     (Segment 3 c). You can list them in build order using::
187
188       ls -ltr /lib/modules/*-local*
189
190    To then for example erase a kernel that identifies itself as
191    '6.0-rc1-local-gcafec0cacaca0', use this::
192
193       sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
194       sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
195       # * Note, on some distributions kernel-install is missing
196       #   or does only part of the job.
197
198  b) If you performed a bisection and successfully validated the result, feel
199     free to remove all kernels built during the actual bisection (Segment 3 c);
200     the kernels you built earlier and later you might want to keep around for
201     a week or two.
202
203.. _introguide_bissbs:
204
205Step-by-step guide on how to verify bugs and bisect regressions
206===============================================================
207
208This guide describes how to set up your own Linux kernels for investigating bugs
209or regressions you intent to report. How far you want to follow the instructions
210depends on your issue:
211
212Execute all steps till the end of *segment 1* to **verify if your kernel problem
213is present in code supported by Linux kernel developers**. If it is, you are all
214set to report the bug -- unless it did not happen with earlier kernel versions,
215as then your want to at least continue with *segment 2* to **check if the issue
216qualifies as regression** which receive priority treatment. Depending on the
217outcome you then are ready to report a bug or submit a preliminary regression
218report; instead of the latter your could also head straight on and follow
219*segment 3* to **perform a bisection** for a full-fledged regression report
220developers are obliged to act upon.
221
222 :ref:`Preparations: set up everything to build your own kernels.<introprep_bissbs>`
223
224 :ref:`Segment 1: try to reproduce the problem with the latest codebase.<introlatestcheck_bissbs>`
225
226 :ref:`Segment 2: check if the kernels you build work fine.<introworkingcheck_bissbs>`
227
228 :ref:`Segment 3: perform a bisection and validate the result.<introbisect_bissbs>`
229
230 :ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>`
231
232The steps in each segment illustrate the important aspects of the process, while
233a comprehensive reference section holds additional details. The latter sometimes
234also outlines alternative approaches, pitfalls, as well as problems that might
235occur at the particular step -- and how to get things rolling again.
236
237For further details on how to report Linux kernel issues or regressions check
238out Documentation/admin-guide/reporting-issues.rst, which works in conjunction
239with this document. It among others explains why you need to verify bugs with
240the latest 'mainline' kernel, even if you face a problem with a kernel from a
241'stable/longterm' series; for users facing a regression it also explains that
242sending a preliminary report after finishing segment 2 might be wise, as the
243regression and its culprit might be known already. For further details on
244what actually qualifies as a regression check out
245Documentation/admin-guide/reporting-regressions.rst.
246
247.. _introprep_bissbs:
248
249Preparations: set up everything to build your own kernels
250---------------------------------------------------------
251
252.. _backup_bissbs:
253
254* Create a fresh backup and put system repair and restore tools at hand, just
255  to be prepared for the unlikely case of something going sideways.
256
257  [:ref:`details<backup_bisref>`]
258
259.. _vanilla_bissbs:
260
261* Remove all software that depends on externally developed kernel drivers or
262  builds them automatically. That includes but is not limited to DKMS, openZFS,
263  VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module).
264
265  [:ref:`details<vanilla_bisref>`]
266
267.. _secureboot_bissbs:
268
269* On platforms with 'Secure Boot' or similar solutions, prepare everything to
270  ensure the system will permit your self-compiled kernel to boot. The
271  quickest and easiest way to achieve this on commodity x86 systems is to
272  disable such techniques in the BIOS setup utility; alternatively, remove
273  their restrictions through a process initiated by
274  ``mokutil --disable-validation``.
275
276  [:ref:`details<secureboot_bisref>`]
277
278.. _rangecheck_bissbs:
279
280* Determine the kernel versions considered 'good' and 'bad' throughout this
281  guide.
282
283  Do you follow this guide to verify if a bug is present in the code developers
284  care for? Then consider the mainline release your 'working' kernel (the newest
285  one you regularly use) is based on to be the 'good' version; if your 'working'
286  kernel for example is '6.0.11', then your 'good' kernel is 'v6.0'.
287
288  In case you face a regression, it depends on the version range where the
289  regression was introduced:
290
291  * Something which used to work in Linux 6.0 broke when switching to Linux
292    6.1-rc1? Then henceforth regard 'v6.0' as the last known 'good' version
293    and 'v6.1-rc1' as the first 'bad' one.
294
295  * Some function stopped working when updating from 6.0.11 to 6.1.4? Then for
296    the time being consider 'v6.0' as the last 'good' version and 'v6.1.4' as
297    the 'bad' one. Note, at this point it is merely assumed that 6.0 is fine;
298    this assumption will be checked in segment 2.
299
300  * A feature you used in 6.0.11 does not work at all or worse in 6.1.13? In
301    that case you want to bisect within a stable/longterm series: consider
302    'v6.0.11' as the last known 'good' version and 'v6.0.13' as the first 'bad'
303    one. Note, in this case you still want to compile and test a mainline kernel
304    as explained in segment 1: the outcome will determine if you need to report
305    your issue to the regular developers or the stable team.
306
307  *Note, do not confuse 'good' version with 'working' kernel; the latter term
308  throughout this guide will refer to the last kernel that has been working
309  fine.*
310
311  [:ref:`details<rangecheck_bisref>`]
312
313.. _bootworking_bissbs:
314
315* Boot into the 'working' kernel and briefly use the apparently broken feature.
316
317  [:ref:`details<bootworking_bisref>`]
318
319.. _diskspace_bissbs:
320
321* Ensure to have enough free space for building Linux. 15 Gigabyte in your home
322  directory should typically suffice. If you have less available, be sure to pay
323  attention to later steps about retrieving the Linux sources and handling of
324  debug symbols: both explain approaches reducing the amount of space, which
325  should allow you to master these tasks with about 4 Gigabytes free space.
326
327  [:ref:`details<diskspace_bisref>`]
328
329.. _buildrequires_bissbs:
330
331* Install all software required to build a Linux kernel. Often you will need:
332  'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl',
333  'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The
334  reference section shows how to quickly install those on various popular Linux
335  distributions.
336
337  [:ref:`details<buildrequires_bisref>`]
338
339.. _sources_bissbs:
340
341* Retrieve the mainline Linux sources; then change into the directory holding
342  them, as all further commands in this guide are meant to be executed from
343  there.
344
345  *Note, the following describe how to retrieve the sources using a full
346  mainline clone, which downloads about 2,75 GByte as of early 2024. The*
347  :ref:`reference section describes two alternatives <sources_bisref>` *:
348  one downloads less than 500 MByte, the other works better with unreliable
349  internet connections.*
350
351  Execute the following command to retrieve a fresh mainline codebase while
352  preparing things to add stable/longterm branches later::
353
354    git clone -o mainline --no-checkout \
355      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
356    cd ~/linux/
357    git remote add -t master stable \
358      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
359
360  [:ref:`details<sources_bisref>`]
361
362.. _oldconfig_bissbs:
363
364* Start preparing a kernel build configuration (the '.config' file).
365
366  Before doing so, ensure you are still running the 'working' kernel an earlier
367  step told you to boot; if you are unsure, check the current kernel release
368  identifier using ``uname -r``.
369
370  Afterwards check out the source code for the version earlier established as
371  'good' (in this example this is assumed to be 6.0) and create a .config file::
372
373    git checkout --detach v6.0
374    make olddefconfig
375
376  The second command will try to locate the build configuration file for the
377  running kernel and then adjust it for the needs of the kernel sources you
378  checked out. While doing so, it will print a few lines you need to check.
379
380  Look out for a line starting with '# using defaults found in'. It should be
381  followed by a path to a file in '/boot/' that contains the release identifier
382  of your currently working kernel. If the line instead continues with something
383  like 'arch/x86/configs/x86_64_defconfig', then the build infra failed to find
384  the .config file for your running kernel -- in which case you have to put one
385  there manually, as explained in the reference section.
386
387  In case you can not find such a line, look for one containing '# configuration
388  written to .config'. If that's the case you have a stale build configuration
389  lying around. Unless you intend to use it, delete it; afterwards run
390  'make olddefconfig' again and check if it now picked up the right config file
391  as base.
392
393  [:ref:`details<oldconfig_bisref>`]
394
395.. _localmodconfig_bissbs:
396
397* Disable any kernel modules apparently superfluous for your setup. This is
398  optional, but especially wise for bisections, as it speeds up the build
399  process enormously -- at least unless the .config file picked up in the
400  previous step was already tailored to your and your hardware needs, in which
401  case you should skip this step.
402
403  To prepare the trimming, connect external hardware you occasionally use (USB
404  keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted
405  since you started that guide, ensure that you tried using the feature causing
406  trouble since you started the system. Only then trim your .config::
407
408     yes '' | make localmodconfig
409
410  There is a catch to this, as the 'apparently' in initial sentence of this step
411  and the preparation instructions already hinted at:
412
413  The 'localmodconfig' target easily disables kernel modules for features only
414  used occasionally -- like modules for external peripherals not yet connected
415  since booting, virtualization software not yet utilized, VPN tunnels, and a
416  few other things. That's because some tasks rely on kernel modules Linux only
417  loads when you execute tasks like the aforementioned ones for the first time.
418
419  This drawback of localmodconfig is nothing you should lose sleep over, but
420  something to keep in mind: if something is misbehaving with the kernels built
421  during this guide, this is most likely the reason. You can reduce or nearly
422  eliminate the risk with tricks outlined in the reference section; but when
423  building a kernel just for quick testing purposes this is usually not worth
424  spending much effort on, as long as it boots and allows to properly test the
425  feature that causes trouble.
426
427  [:ref:`details<localmodconfig_bisref>`]
428
429.. _tagging_bissbs:
430
431* Ensure all the kernels you will build are clearly identifiable using a special
432  tag and a unique version number::
433
434    ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
435    ./scripts/config -e CONFIG_LOCALVERSION_AUTO
436
437  [:ref:`details<tagging_bisref>`]
438
439.. _debugsymbols_bissbs:
440
441* Decide how to handle debug symbols.
442
443  In the context of this document it is often wise to enable them, as there is a
444  decent chance you will need to decode a stack trace from a 'panic', 'Oops',
445  'warning', or 'BUG'::
446
447    ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
448      -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
449
450  But if you are extremely short on storage space, you might want to disable
451  debug symbols instead::
452
453    ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
454      -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE
455
456  [:ref:`details<debugsymbols_bisref>`]
457
458.. _configmods_bissbs:
459
460* Check if you may want or need to adjust some other kernel configuration
461  options:
462
463  * Are you running Debian? Then you want to avoid known problems by performing
464    additional adjustments explained in the reference section.
465
466    [:ref:`details<configmods_distros_bisref>`].
467
468  * If you want to influence other aspects of the configuration, do so now using
469    your preferred tool. Note, to use make targets like 'menuconfig' or
470    'nconfig', you will need to install the development files of ncurses; for
471    'xconfig' you likewise need the Qt5 or Qt6 headers.
472
473    [:ref:`details<configmods_individual_bisref>`].
474
475.. _saveconfig_bissbs:
476
477* Reprocess the .config after the latest adjustments and store it in a safe
478  place::
479
480     make olddefconfig
481     cp .config ~/kernel-config-working
482
483  [:ref:`details<saveconfig_bisref>`]
484
485.. _introlatestcheck_bissbs:
486
487Segment 1: try to reproduce the problem with the latest codebase
488----------------------------------------------------------------
489
490The following steps verify if the problem occurs with the code currently
491supported by developers. In case you face a regression, it also checks that the
492problem is not caused by some .config change, as reporting the issue then would
493be a waste of time. [:ref:`details<introlatestcheck_bisref>`]
494
495.. _checkoutmaster_bissbs:
496
497* Check out the latest Linux codebase::
498
499    cd ~/linux/
500    git checkout --force --detach mainline/master
501
502  [:ref:`details<checkoutmaster_bisref>`]
503
504.. _build_bissbs:
505
506* Build the image and the modules of your first kernel using the config file you
507  prepared::
508
509    cp ~/kernel-config-working .config
510    make olddefconfig
511    make -j $(nproc --all)
512
513  If you want your kernel packaged up as deb, rpm, or tar file, see the
514  reference section for alternatives, which obviously will require other
515  steps to install as well.
516
517  [:ref:`details<build_bisref>`]
518
519.. _install_bissbs:
520
521* Install your newly built kernel.
522
523  Before doing so, consider checking if there is still enough room for it::
524
525    df -h /boot/ /lib/modules/
526
527  150 MByte in /boot/ and 200 in /lib/modules/ usually suffice. Those are rough
528  estimates assuming the worst case. How much your kernels actually require will
529  be determined later.
530
531  Now install the kernel, which will be saved in parallel to the kernels from
532  your Linux distribution::
533
534    command -v installkernel && sudo make modules_install install
535
536  On many commodity Linux distributions this will take care of everything
537  required to boot your kernel. You might want to ensure that's the case by
538  checking if your boot loader's configuration was updated; furthermore ensure
539  an initramfs (also known as initrd) exists, which on many distributions can be
540  achieved by running ``ls -l /boot/init*$(make -s kernelrelease)*``. Those
541  steps are recommended, as there are quite a few Linux distribution where above
542  command is insufficient:
543
544  * On Arch Linux, its derivatives, many immutable Linux distributions, and a
545    few others the above command does nothing at, as they lack 'installkernel'
546    executable.
547
548  * Some distributions install the kernel, but don't add an entry for your
549    kernel in your boot loader's configuration -- the kernel thus won't show up
550    in the boot menu.
551
552  * Some distributions add a boot loader menu entry, but don't create an
553    initramfs on installation -- in that case your kernel most likely will be
554    unable to mount the root partition during bootup.
555
556  If any of that applies to you, see the reference section for further guidance.
557  Once you figured out what to do, consider writing down the necessary
558  installation steps: if you will build more kernels as described in
559  segment 2 and 3, you will have to execute these commands every time that
560  ``command -v installkernel [...]`` comes up again.
561
562  [:ref:`details<install_bisref>`]
563
564.. _storagespace_bissbs:
565
566* In case you plan to follow this guide further, check how much storage space
567  the kernel, its modules, and other related files like the initramfs consume::
568
569    du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
570    du -sh /lib/modules/$(make -s kernelrelease)/
571
572  Write down or remember those two values for later: they enable you to prevent
573  running out of disk space accidentally during a bisection.
574
575  [:ref:`details<storagespace_bisref>`]
576
577.. _kernelrelease_bissbs:
578
579* Show and store the kernelrelease identifier of the kernel you just built::
580
581    make -s kernelrelease | tee -a ~/kernels-built
582
583  Remember the identifier momentarily, as it will help you pick the right kernel
584  from the boot menu upon restarting.
585
586.. _recheckbroken_bissbs:
587
588* Reboot into the kernel you just built and check if the feature that is
589  expected to be broken really is.
590
591  Start by making sure the kernel you booted is the one you just built. When
592  unsure, check if the output of these commands show the exact same release
593  identifier::
594
595    tail -n 1 ~/kernels-built
596    uname -r
597
598 Now verify if the feature that causes trouble works with your newly built
599 kernel. If things work while investigating a regression, check the reference
600 section for further details.
601
602  [:ref:`details<recheckbroken_bisref>`]
603
604.. _recheckstablebroken_bissbs:
605
606* Are you facing a problem within a stable/longterm release, but failed to
607  reproduce it with the mainline kernel you just built? Then check if the latest
608  codebase for the particular series might already fix the problem. To do so,
609  add the stable series Git branch for your 'good' kernel (again, this here is
610  assumed to be 6.0) and check out the latest version::
611
612    cd ~/linux/
613    git remote set-branches --add stable linux-6.0.y
614    git fetch stable
615    git checkout --force --detach linux-6.0.y
616
617  Now use the checked out code to build and install another kernel using the
618  commands the earlier steps already described in more detail::
619
620    cp ~/kernel-config-working .config
621    make olddefconfig
622    make -j $(nproc --all)
623    # * Check if the free space suffices holding another kernel:
624    df -h /boot/ /lib/modules/
625    command -v installkernel && sudo make modules_install install
626    make -s kernelrelease | tee -a ~/kernels-built
627    reboot
628
629  Now verify if you booted the kernel you intended to start, to then check if
630  everything works fine with this kernel::
631
632    tail -n 1 ~/kernels-built
633    uname -r
634
635  [:ref:`details<recheckstablebroken_bisref>`]
636
637Do you follow this guide to verify if a problem is present in the code
638currently supported by Linux kernel developers? Then you are done at this
639point. If you later want to remove the kernel you just built, check out
640:ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>`.
641
642In case you face a regression, move on and execute at least the next segment
643as well.
644
645.. _introworkingcheck_bissbs:
646
647Segment 2: check if the kernels you build work fine
648---------------------------------------------------
649
650In case of a regression, you now want to ensure the trimmed configuration file
651you created earlier works as expected; a bisection with the .config file
652otherwise would be a waste of time. [:ref:`details<introworkingcheck_bisref>`]
653
654.. _recheckworking_bissbs:
655
656* Build your own variant of the 'working' kernel and check if the feature that
657  regressed works as expected with it.
658
659  Start by checking out the sources for the version earlier established as
660  'good' (once again assumed to be 6.0 here)::
661
662    cd ~/linux/
663    git checkout --detach v6.0
664
665  Now use the checked out code to configure, build, and install another kernel
666  using the commands the previous subsection explained in more detail::
667
668    cp ~/kernel-config-working .config
669    make olddefconfig
670    make -j $(nproc --all)
671    # * Check if the free space suffices holding another kernel:
672    df -h /boot/ /lib/modules/
673    command -v installkernel && sudo make modules_install install
674    make -s kernelrelease | tee -a ~/kernels-built
675    reboot
676
677  When the system booted, you may want to verify once again that the
678  kernel you started is the one you just built:
679
680    tail -n 1 ~/kernels-built
681    uname -r
682
683  Now check if this kernel works as expected; if not, consult the reference
684  section for further instructions.
685
686  [:ref:`details<recheckworking_bisref>`]
687
688.. _introbisect_bissbs:
689
690Segment 3: perform the bisection and validate the result
691--------------------------------------------------------
692
693With all the preparations and precaution builds taken care of, you are now ready
694to begin the bisection. This will make you build quite a few kernels -- usually
695about 15 in case you encountered a regression when updating to a newer series
696(say from 6.0.11 to 6.1.3). But do not worry, due to the trimmed build
697configuration created earlier this works a lot faster than many people assume:
698overall on average it will often just take about 10 to 15 minutes to compile
699each kernel on commodity x86 machines.
700
701* In case your 'bad' version is a stable/longterm release (say v6.1.5), add its
702  stable branch, unless you already did so earlier::
703
704    cd ~/linux/
705    git remote set-branches --add stable linux-6.1.y
706    git fetch stable
707
708.. _bisectstart_bissbs:
709
710* Start the bisection and tell Git about the versions earlier established as
711  'good' (6.0 in the following example command) and 'bad' (6.1.5)::
712
713    cd ~/linux/
714    git bisect start
715    git bisect good v6.0
716    git bisect bad v6.1.5
717
718  [:ref:`details<bisectstart_bisref>`]
719
720.. _bisectbuild_bissbs:
721
722* Now use the code Git checked out to build, install, and boot a kernel using
723  the commands introduced earlier::
724
725    cp ~/kernel-config-working .config
726    make olddefconfig
727    make -j $(nproc --all)
728    # * Check if the free space suffices holding another kernel:
729    df -h /boot/ /lib/modules/
730    command -v installkernel && sudo make modules_install install
731    make -s kernelrelease | tee -a ~/kernels-built
732    reboot
733
734  If compilation fails for some reason, run ``git bisect skip`` and restart
735  executing the stack of commands from the beginning.
736
737  In case you skipped the "test latest codebase" step in the guide, check its
738  description as for why the 'df [...]' and 'make -s kernelrelease [...]'
739  commands are here.
740
741  Important note: the latter command from this point on will print release
742  identifiers that might look odd or wrong to you -- which they are not, as it's
743  totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0'
744  if you bisect between versions 6.1 and 6.2 for example.
745
746  [:ref:`details<bisectbuild_bisref>`]
747
748.. _bisecttest_bissbs:
749
750* Now check if the feature that regressed works in the kernel you just built.
751
752  You again might want to start by making sure the kernel you booted is the one
753  you just built::
754
755    cd ~/linux/
756    tail -n 1 ~/kernels-built
757    uname -r
758
759  Now verify if the feature that regressed works at this kernel bisection point.
760  If it does, run this::
761
762    git bisect good
763
764  If it does not, run this::
765
766    git bisect bad
767
768  Be sure about what you tell Git, as getting this wrong just once will send the
769  rest of the bisection totally off course.
770
771  While the bisection is ongoing, Git will use the information you provided to
772  find and check out another bisection point for you to test. While doing so, it
773  will print something like 'Bisecting: 675 revisions left to test after this
774  (roughly 10 steps)' to indicate how many further changes it expects to be
775  tested. Now build and install another kernel using the instructions from the
776  previous step; afterwards follow the instructions in this step again.
777
778  Repeat this again and again until you finish the bisection -- that's the case
779  when Git after tagging a change as 'good' or 'bad' prints something like
780  'cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit'; right
781  afterwards it will show some details about the culprit including the patch
782  description of the change. The latter might fill your terminal screen, so you
783  might need to scroll up to see the message mentioning the culprit;
784  alternatively, run ``git bisect log > ~/bisection-log``.
785
786  [:ref:`details<bisecttest_bisref>`]
787
788.. _bisectlog_bissbs:
789
790* Store Git's bisection log and the current .config file in a safe place before
791  telling Git to reset the sources to the state before the bisection::
792
793    cd ~/linux/
794    git bisect log > ~/bisection-log
795    cp .config ~/bisection-config-culprit
796    git bisect reset
797
798  [:ref:`details<bisectlog_bisref>`]
799
800.. _revert_bissbs:
801
802* Try reverting the culprit on top of latest mainline to see if this fixes your
803  regression.
804
805  This is optional, as it might be impossible or hard to realize. The former is
806  the case, if the bisection determined a merge commit as the culprit; the
807  latter happens if other changes depend on the culprit. But if the revert
808  succeeds, it is worth building another kernel, as it validates the result of
809  a bisection, which can easily deroute; it furthermore will let kernel
810  developers know, if they can resolve the regression with a quick revert.
811
812  Begin by checking out the latest codebase depending on the range you bisected:
813
814  * Did you face a regression within a stable/longterm series (say between
815    6.0.11 and 6.0.13) that does not happen in mainline? Then check out the
816    latest codebase for the affected series like this::
817
818      git fetch stable
819      git checkout --force --detach linux-6.0.y
820
821  * In all other cases check out latest mainline::
822
823      git fetch mainline
824      git checkout --force --detach mainline/master
825
826    If you bisected a regression within a stable/longterm series that also
827    happens in mainline, there is one more thing to do: look up the mainline
828    commit-id. To do so, use a command like ``git show abcdcafecabcd`` to
829    view the patch description of the culprit. There will be a line near
830    the top which looks like 'commit cafec0cacaca0 upstream.' or
831    'Upstream commit cafec0cacaca0'; use that commit-id in the next command
832    and not the one the bisection blamed.
833
834  Now try reverting the culprit by specifying its commit id::
835
836    git revert --no-edit cafec0cacaca0
837
838  If that fails, give up trying and move on to the next step. But if it works,
839  build a kernel again using the familiar command sequence::
840
841    cp ~/kernel-config-working .config
842    make olddefconfig &&
843    make -j $(nproc --all) &&
844    # * Check if the free space suffices holding another kernel:
845    df -h /boot/ /lib/modules/
846    command -v installkernel && sudo make modules_install install
847    Make -s kernelrelease | tee -a ~/kernels-built
848    reboot
849
850  Now check one last time if the feature that made you perform a bisection work
851  with that kernel.
852
853  [:ref:`details<revert_bisref>`]
854
855.. _introclosure_bissbs:
856
857Supplementary tasks: cleanup during and after the bisection
858-----------------------------------------------------------
859
860During and after following this guide you might want or need to remove some of
861the kernels you installed: the boot menu otherwise will become confusing or
862space might run out.
863
864.. _makeroom_bissbs:
865
866* To remove one of the kernels you installed, look up its 'kernelrelease'
867  identifier. This guide stores them in '~/kernels-built', but the following
868  command will print them as well::
869
870    ls -ltr /lib/modules/*-local*
871
872  You in most situations want to remove the oldest kernels built during the
873  actual bisection (e.g. segment 3 of this guide). The two ones you created
874  beforehand (e.g. to test the latest codebase and the version considered
875  'good') might become handy to verify something later -- thus better keep them
876  around, unless you are really short on storage space.
877
878  To remove the modules of a kernel with the kernelrelease identifier
879  '*6.0-rc1-local-gcafec0cacaca0*', start by removing the directory holding its
880  modules::
881
882    sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
883
884  Afterwards try the following command::
885
886    sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
887
888  On quite a few distributions this will delete all other kernel files installed
889  while also removing the kernel's entry from the boot menu. But on some
890  distributions kernel-install does not exist or leaves boot-loader entries or
891  kernel image and related files behind; in that case remove them as described
892  in the reference section.
893
894  [:ref:`details<makeroom_bisref>`]
895
896.. _finishingtouch_bissbs:
897
898* Once you have finished the bisection, do not immediately remove anything you
899  set up, as you might need a few things again. What is safe to remove depends
900  on the outcome of the bisection:
901
902  * Could you initially reproduce the regression with the latest codebase and
903    after the bisection were able to fix the problem by reverting the culprit on
904    top of the latest codebase? Then you want to keep those two kernels around
905    for a while, but safely remove all others with a '-local' in the release
906    identifier.
907
908  * Did the bisection end on a merge-commit or seems questionable for other
909    reasons? Then you want to keep as many kernels as possible around for a few
910    days: it's pretty likely that you will be asked to recheck something.
911
912  * In other cases it likely is a good idea to keep the following kernels around
913    for some time: the one built from the latest codebase, the one created from
914    the version considered 'good', and the last three or four you compiled
915    during the actual bisection process.
916
917  [:ref:`details<finishingtouch_bisref>`]
918
919.. _submit_improvements:
920
921This concludes the step-by-step guide.
922
923Did you run into trouble following any of the above steps not cleared up by the
924reference section below? Did you spot errors? Or do you have ideas how to
925improve the guide? Then please take a moment and let the maintainer of this
926document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while
927CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is
928vital to improve this document further, which is in everybody's interest, as it
929will enable more people to master the task described here -- and hopefully also
930improve similar guides inspired by this one.
931
932
933Reference section for the step-by-step guide
934============================================
935
936This section holds additional information for almost all the items in the above
937step-by-step guide.
938
939.. _backup_bisref:
940
941Prepare for emergencies
942-----------------------
943
944  *Create a fresh backup and put system repair and restore tools at hand.*
945  [:ref:`... <backup_bissbs>`]
946
947Remember, you are dealing with computers, which sometimes do unexpected things
948-- especially if you fiddle with crucial parts like the kernel of an operating
949system. That's what you are about to do in this process. Hence, better prepare
950for something going sideways, even if that should not happen.
951
952[:ref:`back to step-by-step guide <backup_bissbs>`]
953
954.. _vanilla_bisref:
955
956Remove anything related to externally maintained kernel modules
957---------------------------------------------------------------
958
959  *Remove all software that depends on externally developed kernel drivers or
960  builds them automatically.* [:ref:`...<vanilla_bissbs>`]
961
962Externally developed kernel modules can easily cause trouble during a bisection.
963
964But there is a more important reason why this guide contains this step: most
965kernel developers will not care about reports about regressions occurring with
966kernels that utilize such modules. That's because such kernels are not
967considered 'vanilla' anymore, as Documentation/admin-guide/reporting-issues.rst
968explains in more detail.
969
970[:ref:`back to step-by-step guide <vanilla_bissbs>`]
971
972.. _secureboot_bisref:
973
974Deal with techniques like Secure Boot
975-------------------------------------
976
977  *On platforms with 'Secure Boot' or similar techniques, prepare everything to
978  ensure the system will permit your self-compiled kernel to boot later.*
979  [:ref:`... <secureboot_bissbs>`]
980
981Many modern systems allow only certain operating systems to start; that's why
982they reject booting self-compiled kernels by default.
983
984You ideally deal with this by making your platform trust your self-built kernels
985with the help of a certificate. How to do that is not described
986here, as it requires various steps that would take the text too far away from
987its purpose; 'Documentation/admin-guide/module-signing.rst' and various web
988sides already explain everything needed in more detail.
989
990Temporarily disabling solutions like Secure Boot is another way to make your own
991Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup
992utility; the required steps vary a lot between machines and therefore cannot be
993described here.
994
995On mainstream x86 Linux distributions there is a third and universal option:
996disable all Secure Boot restrictions for your Linux environment. You can
997initiate this process by running ``mokutil --disable-validation``; this will
998tell you to create a one-time password, which is safe to write down. Now
999restart; right after your BIOS performed all self-tests the bootloader Shim will
1000show a blue box with a message 'Press any key to perform MOK management'. Hit
1001some key before the countdown exposes, which will open a menu. Choose 'Change
1002Secure Boot state'. Shim's 'MokManager' will now ask you to enter three
1003randomly chosen characters from the one-time password specified earlier. Once
1004you provided them, confirm you really want to disable the validation.
1005Afterwards, permit MokManager to reboot the machine.
1006
1007[:ref:`back to step-by-step guide <secureboot_bissbs>`]
1008
1009.. _bootworking_bisref:
1010
1011Boot the last kernel that was working
1012-------------------------------------
1013
1014  *Boot into the last working kernel and briefly recheck if the feature that
1015  regressed really works.* [:ref:`...<bootworking_bissbs>`]
1016
1017This will make later steps that cover creating and trimming the configuration do
1018the right thing.
1019
1020[:ref:`back to step-by-step guide <bootworking_bissbs>`]
1021
1022.. _diskspace_bisref:
1023
1024Space requirements
1025------------------
1026
1027  *Ensure to have enough free space for building Linux.*
1028  [:ref:`... <diskspace_bissbs>`]
1029
1030The numbers mentioned are rough estimates with a big extra charge to be on the
1031safe side, so often you will need less.
1032
1033If you have space constraints, be sure to hay attention to the :ref:`step about
1034debug symbols' <debugsymbols_bissbs>` and its :ref:`accompanying reference
1035section' <debugsymbols_bisref>`, as disabling then will reduce the consumed disk
1036space by quite a few gigabytes.
1037
1038[:ref:`back to step-by-step guide <diskspace_bissbs>`]
1039
1040.. _rangecheck_bisref:
1041
1042Bisection range
1043---------------
1044
1045  *Determine the kernel versions considered 'good' and 'bad' throughout this
1046  guide.* [:ref:`...<rangecheck_bissbs>`]
1047
1048Establishing the range of commits to be checked is mostly straightforward,
1049except when a regression occurred when switching from a release of one stable
1050series to a release of a later series (e.g. from 6.0.11 to 6.1.4). In that case
1051Git will need some hand holding, as there is no straight line of descent.
1052
1053That's because with the release of 6.0 mainline carried on to 6.1 while the
1054stable series 6.0.y branched to the side. It's therefore theoretically possible
1055that the issue you face with 6.1.4 only worked in 6.0.11, as it was fixed by a
1056commit that went into one of the 6.0.y releases, but never hit mainline or the
10576.1.y series. Thankfully that normally should not happen due to the way the
1058stable/longterm maintainers maintain the code. It's thus pretty safe to assume
10596.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel
1060will be built and tested in the segment '2' of this guide; Git would force you
1061to do this as well, if you tried bisecting between 6.0.11 and 6.1.13.
1062
1063[:ref:`back to step-by-step guide <rangecheck_bissbs>`]
1064
1065.. _buildrequires_bisref:
1066
1067Install build requirements
1068--------------------------
1069
1070  *Install all software required to build a Linux kernel.*
1071  [:ref:`...<buildrequires_bissbs>`]
1072
1073The kernel is pretty stand-alone, but besides tools like the compiler you will
1074sometimes need a few libraries to build one. How to install everything needed
1075depends on your Linux distribution and the configuration of the kernel you are
1076about to build.
1077
1078Here are a few examples what you typically need on some mainstream
1079distributions:
1080
1081* Arch Linux and derivatives::
1082
1083    sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \
1084      pahole perl zlib ncurses qt6-base
1085
1086* Debian, Ubuntu, and derivatives::
1087
1088    sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \
1089      libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \
1090      libncurses-dev qt6-base-dev g++
1091
1092* Fedora and derivatives::
1093
1094    sudo dnf install binutils \
1095      /usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \
1096      /usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction}
1097
1098* openSUSE and derivatives::
1099
1100    sudo zypper install bc binutils bison dwarves flex gcc git \
1101      kernel-install-tools libelf-devel make modutils openssl openssl-devel \
1102      perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel
1103
1104These commands install a few packages that are often, but not always needed. You
1105for example might want to skip installing the development headers for ncurses,
1106which you will only need in case you later might want to adjust the kernel build
1107configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit
1108the headers of Qt6 is you do not plan to adjust the .config using 'xconfig'.
1109
1110You furthermore might need additional libraries and their development headers
1111for tasks not covered in this guide -- for example when building utilities from
1112the kernel's tools/ directory.
1113
1114[:ref:`back to step-by-step guide <buildrequires_bissbs>`]
1115
1116.. _sources_bisref:
1117
1118Download the sources using Git
1119------------------------------
1120
1121  *Retrieve the Linux mainline sources.*
1122  [:ref:`...<sources_bissbs>`]
1123
1124The step-by-step guide outlines how to download the Linux sources using a full
1125Git clone of Linus' mainline repository. There is nothing more to say about
1126that -- but there are two alternatives ways to retrieve the sources that might
1127work better for you:
1128
1129 * If you have an unreliable internet connection, consider
1130   :ref:`using a 'Git bundle'<sources_bundle_bisref>`.
1131
1132 * If downloading the complete repository would take too long or requires too
1133   much storage space, consider :ref:`using a 'shallow
1134   clone'<sources_shallow_bisref>`.
1135
1136.. _sources_bundle_bisref:
1137
1138Downloading Linux mainline sources using a bundle
1139~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1140
1141Use the following commands to retrieve the Linux mainline sources using a
1142bundle::
1143
1144    wget -c \
1145      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
1146    git clone --no-checkout clone.bundle ~/linux/
1147    cd ~/linux/
1148    git remote remove origin
1149    git remote add mainline \
1150      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
1151    git fetch mainline
1152    git remote add -t master stable \
1153      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1154
1155In case the 'wget' command fails, just re-execute it, it will pick up where
1156it left off.
1157
1158[:ref:`back to step-by-step guide <sources_bissbs>`]
1159[:ref:`back to section intro <sources_bisref>`]
1160
1161.. _sources_shallow_bisref:
1162
1163Downloading Linux mainline sources using a shallow clone
1164~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1165
1166First, execute the following command to retrieve the latest mainline codebase::
1167
1168    git clone -o mainline --no-checkout --depth 1 -b master \
1169      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
1170    cd ~/linux/
1171    git remote add -t master stable \
1172      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1173
1174Now deepen your clone's history to the second predecessor of the mainline
1175release of your 'good' version. In case the latter are 6.0 or 6.0.11, 5.19 would
1176be the first predecessor and 5.18 the second -- hence deepen the history up to
1177that version::
1178
1179    git fetch --shallow-exclude=v5.18 mainline
1180
1181Afterwards add the stable Git repository as remote and all required stable
1182branches as explained in the step-by-step guide.
1183
1184Note, shallow clones have a few peculiar characteristics:
1185
1186 * For bisections the history needs to be deepened a few mainline versions
1187   farther than it seems necessary, as explained above already. That's because
1188   Git otherwise will be unable to revert or describe most of the commits within
1189   a range (say v6.1..v6.2), as they are internally based on earlier kernels
1190   releases (like v6.0-rc2 or 5.19-rc3).
1191
1192 * This document in most places uses ``git fetch`` with ``--shallow-exclude=``
1193   to specify the earliest version you care about (or to be precise: its git
1194   tag). You alternatively can use the parameter ``--shallow-since=`` to specify
1195   an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to
1196   define the depth of the history you want to download. When using them while
1197   bisecting mainline, ensure to deepen the history to at least 7 months before
1198   the release of the mainline release your 'good' kernel is based on.
1199
1200 * Be warned, when deepening your clone you might encounter an error like
1201   'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'.
1202   In that case run ``git repack -d`` and try again.
1203
1204[:ref:`back to step-by-step guide <sources_bissbs>`]
1205[:ref:`back to section intro <sources_bisref>`]
1206
1207.. _oldconfig_bisref:
1208
1209Start defining the build configuration for your kernel
1210------------------------------------------------------
1211
1212  *Start preparing a kernel build configuration (the '.config' file).*
1213  [:ref:`... <oldconfig_bissbs>`]
1214
1215*Note, this is the first of multiple steps in this guide that create or modify
1216build artifacts. The commands used in this guide store them right in the source
1217tree to keep things simple. In case you prefer storing the build artifacts
1218separately, create a directory like '~/linux-builddir/' and add the parameter
1219``O=~/linux-builddir/`` to all make calls used throughout this guide. You will
1220have to point other commands there as well -- among them the ``./scripts/config
1221[...]`` commands, which will require ``--file ~/linux-builddir/.config`` to
1222locate the right build configuration.*
1223
1224Two things can easily go wrong when creating a .config file as advised:
1225
1226 * The oldconfig target will use a .config file from your build directory, if
1227   one is already present there (e.g. '~/linux/.config'). That's totally fine if
1228   that's what you intend (see next step), but in all other cases you want to
1229   delete it. This for example is important in case you followed this guide
1230   further, but due to problems come back here to redo the configuration from
1231   scratch.
1232
1233 * Sometimes olddefconfig is unable to locate the .config file for your running
1234   kernel and will use defaults, as briefly outlined in the guide. In that case
1235   check if your distribution ships the configuration somewhere and manually put
1236   it in the right place (e.g. '~/linux/.config') if it does. On distributions
1237   where /proc/config.gz exists this can be achieved using this command::
1238
1239     zcat /proc/config.gz > .config
1240
1241   Once you put it there, run ``make olddefconfig`` again to adjust it to the
1242   needs of the kernel about to be built.
1243
1244Note, the olddefconfig target will set any undefined build options to their
1245default value. If you prefer to set such configuration options manually, use
1246``make oldconfig`` instead. Then for each undefined configuration option you
1247will be asked how to proceed; in case you are unsure what to answer, simply hit
1248'enter' to apply the default value. Note though that for bisections you normally
1249want to go with the defaults, as you otherwise might enable a new feature that
1250causes a problem looking like regressions (for example due to security
1251restrictions).
1252
1253Occasionally odd things happen when trying to use a config file prepared for one
1254kernel (say 6.1) on an older mainline release -- especially if it is much older
1255(say v5.15). That's one of the reasons why the previous step in the guide told
1256you to boot the kernel where everything works. If you manually add a .config
1257file you thus want to ensure it's from the working kernel and not from a one
1258that shows the regression.
1259
1260In case you want to build kernels for another machine, locate its kernel build
1261configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy
1262that file to the build machine and store it as ~/linux/.config; afterwards run
1263``make olddefconfig`` to adjust it.
1264
1265[:ref:`back to step-by-step guide <oldconfig_bissbs>`]
1266
1267.. _localmodconfig_bisref:
1268
1269Trim the build configuration for your kernel
1270--------------------------------------------
1271
1272  *Disable any kernel modules apparently superfluous for your setup.*
1273  [:ref:`... <localmodconfig_bissbs>`]
1274
1275As explained briefly in the step-by-step guide already: with localmodconfig it
1276can easily happen that your self-built kernels will lack modules for tasks you
1277did not perform at least once before utilizing this make target. That happens
1278when a task requires kernel modules which are only autoloaded when you execute
1279it for the first time. So when you never performed that task since starting your
1280kernel the modules will not have been loaded -- and from localmodonfig's point
1281of view look superfluous, which thus disables them to reduce the amount of code
1282to be compiled.
1283
1284You can try to avoid this by performing typical tasks that often will autoload
1285additional kernel modules: start a VM, establish VPN connections, loop-mount a
1286CD/DVD ISO, mount network shares (CIFS, NFS, ...), and connect all external
1287devices (2FA keys, headsets, webcams, ...) as well as storage devices with file
1288systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, ...). But it
1289is hard to think of everything that might be needed -- even kernel developers
1290often forget one thing or another at this point.
1291
1292Do not let that risk bother you, especially when compiling a kernel only for
1293testing purposes: everything typically crucial will be there. And if you forget
1294something important you can turn on a missing feature manually later and quickly
1295run the commands again to compile and install a kernel that has everything you
1296need.
1297
1298But if you plan to build and use self-built kernels regularly, you might want to
1299reduce the risk by recording which modules your system loads over the course of
1300a few weeks. You can automate this with `modprobed-db
1301<https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to
1302point localmodconfig to the list of modules modprobed-db noticed being used::
1303
1304  yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig
1305
1306That parameter also allows you to build trimmed kernels for another machine in
1307case you copied a suitable .config over to use as base (see previous step). Just
1308run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to
1309your build's host home directory. Then run these commands instead of the one the
1310step-by-step guide mentions::
1311
1312  yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig
1313
1314[:ref:`back to step-by-step guide <localmodconfig_bissbs>`]
1315
1316.. _tagging_bisref:
1317
1318Tag the kernels about to be build
1319---------------------------------
1320
1321  *Ensure all the kernels you will build are clearly identifiable using a
1322  special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`]
1323
1324This allows you to differentiate your distribution's kernels from those created
1325during this process, as the file or directories for the latter will contain
1326'-local' in the name; it also helps picking the right entry in the boot menu and
1327not lose track of you kernels, as their version numbers will look slightly
1328confusing during the bisection.
1329
1330[:ref:`back to step-by-step guide <tagging_bissbs>`]
1331
1332.. _debugsymbols_bisref:
1333
1334Decide to enable or disable debug symbols
1335-----------------------------------------
1336
1337  *Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`]
1338
1339Having debug symbols available can be important when your kernel throws a
1340'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be
1341able to find the exact place where the problem occurred in the code. But
1342collecting and embedding the needed debug information takes time and consumes
1343quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel
1344trimmed with localmodconfig consumed around 5 Gigabyte of space with debug
1345symbols, but less than 1 when they were disabled. The resulting kernel image and
1346modules are bigger as well, which increases storage requirements for /boot/ and
1347load times.
1348
1349In case you want a small kernel and are unlikely to decode a stack trace later,
1350you thus might want to disable debug symbols to avoid those downsides. If it
1351later turns out that you need them, just enable them as shown and rebuild the
1352kernel.
1353
1354You on the other hand definitely want to enable them for this process, if there
1355is a decent chance that you need to decode a stack trace later. The section
1356'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst
1357explains this process in more detail.
1358
1359[:ref:`back to step-by-step guide <debugsymbols_bissbs>`]
1360
1361.. _configmods_bisref:
1362
1363Adjust build configuration
1364--------------------------
1365
1366  *Check if you may want or need to adjust some other kernel configuration
1367  options:*
1368
1369Depending on your needs you at this point might want or have to adjust some
1370kernel configuration options.
1371
1372.. _configmods_distros_bisref:
1373
1374Distro specific adjustments
1375~~~~~~~~~~~~~~~~~~~~~~~~~~~
1376
1377  *Are you running* [:ref:`... <configmods_bissbs>`]
1378
1379The following sections help you to avoid build problems that are known to occur
1380when following this guide on a few commodity distributions.
1381
1382**Debian:**
1383
1384 * Remove a stale reference to a certificate file that would cause your build to
1385   fail::
1386
1387    ./scripts/config --set-str SYSTEM_TRUSTED_KEYS ''
1388
1389   Alternatively, download the needed certificate and make that configuration
1390   option point to it, as `the Debian handbook explains in more detail
1391   <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_
1392   -- or generate your own, as explained in
1393   Documentation/admin-guide/module-signing.rst.
1394
1395[:ref:`back to step-by-step guide <configmods_bissbs>`]
1396
1397.. _configmods_individual_bisref:
1398
1399Individual adjustments
1400~~~~~~~~~~~~~~~~~~~~~~
1401
1402  *If you want to influence the other aspects of the configuration, do so
1403  now.* [:ref:`... <configmods_bissbs>`]
1404
1405You at this point can use a command like ``make menuconfig`` to enable or
1406disable certain features using a text-based user interface; to use a graphical
1407configuration utility, call the make target ``xconfig`` or ``gconfig`` instead.
1408All of them require development libraries from toolkits they are based on
1409(ncurses, Qt5, Gtk2); an error message will tell you if something required is
1410missing.
1411
1412[:ref:`back to step-by-step guide <configmods_bissbs>`]
1413
1414.. _saveconfig_bisref:
1415
1416Put the .config file aside
1417--------------------------
1418
1419  *Reprocess the .config after the latest changes and store it in a safe place.*
1420  [:ref:`... <saveconfig_bissbs>`]
1421
1422Put the .config you prepared aside, as you want to copy it back to the build
1423directory every time  during this guide before you start building another
1424kernel. That's because going back and forth between different versions can alter
1425.config files in odd ways; those occasionally cause side effects that could
1426confuse testing or in some cases render the result of your bisection
1427meaningless.
1428
1429[:ref:`back to step-by-step guide <saveconfig_bissbs>`]
1430
1431.. _introlatestcheck_bisref:
1432
1433Try to reproduce the regression
1434-----------------------------------------
1435
1436  *Verify the regression is not caused by some .config change and check if it
1437  still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`]
1438
1439For some readers it might seem unnecessary to check the latest codebase at this
1440point, especially if you did that already with a kernel prepared by your
1441distributor or face a regression within a stable/longterm series. But it's
1442highly recommended for these reasons:
1443
1444* You will run into any problems caused by your setup before you actually begin
1445  a bisection. That will make it a lot easier to differentiate between 'this
1446  most likely is some problem in my setup' and 'this change needs to be skipped
1447  during the bisection, as the kernel sources at that stage contain an unrelated
1448  problem that causes building or booting to fail'.
1449
1450* These steps will rule out if your problem is caused by some change in the
1451  build configuration between the 'working' and the 'broken' kernel. This for
1452  example can happen when your distributor enabled an additional security
1453  feature in the newer kernel which was disabled or not yet supported by the
1454  older kernel. That security feature might get into the way of something you
1455  do -- in which case your problem from the perspective of the Linux kernel
1456  upstream developers is not a regression, as
1457  Documentation/admin-guide/reporting-regressions.rst explains in more detail.
1458  You thus would waste your time if you'd try to bisect this.
1459
1460* If the cause for your regression was already fixed in the latest mainline
1461  codebase, you'd perform the bisection for nothing. This holds true for a
1462  regression you encountered with a stable/longterm release as well, as they are
1463  often caused by problems in mainline changes that were backported -- in which
1464  case the problem will have to be fixed in mainline first. Maybe it already was
1465  fixed there and the fix is already in the process of being backported.
1466
1467* For regressions within a stable/longterm series it's furthermore crucial to
1468  know if the issue is specific to that series or also happens in the mainline
1469  kernel, as the report needs to be sent to different people:
1470
1471  * Regressions specific to a stable/longterm series are the stable team's
1472    responsibility; mainline Linux developers might or might not care.
1473
1474  * Regressions also happening in mainline are something the regular Linux
1475    developers and maintainers have to handle; the stable team does not care
1476    and does not need to be involved in the report, they just should be told
1477    to backport the fix once it's ready.
1478
1479  Your report might be ignored if you send it to the wrong party -- and even
1480  when you get a reply there is a decent chance that developers tell you to
1481  evaluate   which of the two cases it is before they take a closer look.
1482
1483[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]
1484
1485.. _checkoutmaster_bisref:
1486
1487Checkout the latest Linux codebase
1488----------------------------------
1489
1490  *Checkout the latest Linux codebase.*
1491  [:ref:`... <introlatestcheck_bissbs>`]
1492
1493In case you later want to recheck if an ever newer codebase might fix the
1494problem, remember to run that ``git fetch --shallow-exclude [...]`` command
1495again mentioned earlier to update your local Git repository.
1496
1497[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]
1498
1499.. _build_bisref:
1500
1501Build your kernel
1502-----------------
1503
1504  *Build the image and the modules of your first kernel using the config file
1505  you prepared.* [:ref:`... <build_bissbs>`]
1506
1507A lot can go wrong at this stage, but the instructions below will help you help
1508yourself. Another subsection explains how to directly package your kernel up as
1509deb, rpm or tar file.
1510
1511Dealing with build errors
1512~~~~~~~~~~~~~~~~~~~~~~~~~
1513
1514When a build error occurs, it might be caused by some aspect of your machine's
1515setup that often can be fixed quickly; other times though the problem lies in
1516the code and can only be fixed by a developer. A close examination of the
1517failure messages coupled with some research on the internet will often tell you
1518which of the two it is. To perform such a investigation, restart the build
1519process like this::
1520
1521  make V=1
1522
1523The ``V=1`` activates verbose output, which might be needed to see the actual
1524error. To make it easier to spot, this command also omits the ``-j $(nproc
1525--all)`` used earlier to utilize every CPU core in the system for the job -- but
1526this parallelism also results in some clutter when failures occur.
1527
1528After a few seconds the build process should run into the error again. Now try
1529to find the most crucial line describing the problem. Then search the internet
1530for the most important and non-generic section of that line (say 4 to 8 words);
1531avoid or remove anything that looks remotely system-specific, like your username
1532or local path names like ``/home/username/linux/``. First try your regular
1533internet search engine with that string, afterwards search Linux kernel mailing
1534lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_.
1535
1536This most of the time will find something that will explain what is wrong; quite
1537often one of the hits will provide a solution for your problem, too. If you
1538do not find anything that matches your problem, try again from a different angle
1539by modifying your search terms or using another line from the error messages.
1540
1541In the end, most trouble you are to run into has likely been encountered and
1542reported by others already. That includes issues where the cause is not your
1543system, but lies the code. If you run into one of those, you might thus find a
1544solution (e.g. a patch) or workaround for your problem, too.
1545
1546Package your kernel up
1547~~~~~~~~~~~~~~~~~~~~~~
1548
1549The step-by-step guide uses the default make targets (e.g. 'bzImage' and
1550'modules' on x86) to build the image and the modules of your kernel, which later
1551steps of the guide then install. You instead can also directly build everything
1552and directly package it up by using one of the following targets:
1553
1554 * ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package
1555
1556 * ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package
1557
1558 * ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball
1559
1560This is just a selection of available make targets for this purpose, see
1561``make help`` for others. You can also use these targets after running
1562``make -j $(nproc --all)``, as they will pick up everything already built.
1563
1564If you employ the targets to generate deb or rpm packages, ignore the
1565step-by-step guide's instructions on installing and removing your kernel;
1566instead install and remove the packages using the package utility for the format
1567(e.g. dpkg and rpm) or a package management utility build on top of them (apt,
1568aptitude, dnf/yum, zypper, ...). Be aware that the packages generated using
1569these two make targets are designed to work on various distributions utilizing
1570those formats, they thus will sometimes behave differently than your
1571distribution's kernel packages.
1572
1573[:ref:`back to step-by-step guide <build_bissbs>`]
1574
1575.. _install_bisref:
1576
1577Put the kernel in place
1578-----------------------
1579
1580  *Install the kernel you just built.* [:ref:`... <install_bissbs>`]
1581
1582What you need to do after executing the command in the step-by-step guide
1583depends on the existence and the implementation of an ``installkernel``
1584executable. Many commodity Linux distributions ship such a kernel installer in
1585'/sbin/' that does everything needed, hence there is nothing left for you
1586except rebooting. But some distributions contain an installkernel that does
1587only part of the job -- and a few lack it completely and leave all the work to
1588you.
1589
1590If ``installkernel`` is found, the kernel's build system will delegate the
1591actual installation of your kernel's image and related files to this executable.
1592On almost all Linux distributions it will store the image as '/boot/vmlinuz-
1593<kernelrelease identifier>' and put a 'System.map-<kernelrelease
1594identifier>' alongside it. Your kernel will thus be installed in parallel to any
1595existing ones, unless you already have one with exactly the same release name.
1596
1597Installkernel on many distributions will afterwards generate an 'initramfs'
1598(often also called 'initrd'), which commodity distributions rely on for booting;
1599hence be sure to keep the order of the two make targets used in the step-by-step
1600guide, as things will go sideways if you install your kernel's image before its
1601modules. Often installkernel will then add your kernel to the bootloader
1602configuration, too. You have to take care of one or both of these tasks
1603yourself, if your distributions installkernel doesn't handle them.
1604
1605A few distributions like Arch Linux and its derivatives totally lack an
1606installkernel executable. On those just install the modules using the kernel's
1607build system and then install the image and the System.map file manually::
1608
1609   sudo make modules_install
1610   sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)
1611   sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)
1612
1613If your distribution boots with the help of an initramfs, now generate one for
1614your kernel using the tools your distribution provides for this process.
1615Afterwards add your kernel to your bootloader configuration and reboot.
1616
1617[:ref:`back to step-by-step guide <install_bissbs>`]
1618
1619.. _storagespace_bisref:
1620
1621Storage requirements per kernel
1622-------------------------------
1623
1624  *Check how much storage space the kernel, its modules, and other related files
1625  like the initramfs consume.* [:ref:`... <storagespace_bissbs>`]
1626
1627The kernels built during a bisection consume quite a bit of space in /boot/ and
1628/lib/modules/, especially if you enabled debug symbols. That makes it easy to
1629fill up volumes during a bisection -- and due to that even kernels which used to
1630work earlier might fail to boot. To prevent that you will need to know how much
1631space each installed kernel typically requires.
1632
1633Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in
1634the guide will match all files needed to boot your kernel -- but neither the
1635path nor the naming scheme are mandatory. On some distributions you thus will
1636need to look in different places.
1637
1638[:ref:`back to step-by-step guide <storagespace_bissbs>`]
1639
1640.. _recheckbroken_bisref:
1641
1642Check the kernel built from the latest codebase
1643-----------------------------------------------
1644
1645  *Reboot into the kernel you just built and check if the feature that regressed
1646  is really broken there.* [:ref:`... <recheckbroken_bissbs>`]
1647
1648There are a couple of reasons why the regression you face might not show up with
1649your own kernel built from the latest codebase. These are the most frequent:
1650
1651* The cause for the regression was fixed meanwhile.
1652
1653* The regression with the broken kernel was caused by a change in the build
1654  configuration the provider of your kernel carried out.
1655
1656* Your problem might be a race condition that does not show up with your kernel;
1657  the trimmed build configuration, a different setting for debug symbols, the
1658  compiler used, and various other things can cause this.
1659
1660* In case you encountered the regression with a stable/longterm kernel it might
1661  be a problem that is specific to that series; the next step in this guide will
1662  check this.
1663
1664[:ref:`back to step-by-step guide <recheckbroken_bissbs>`]
1665
1666.. _recheckstablebroken_bisref:
1667
1668Check the kernel built from the latest stable/longterm codebase
1669---------------------------------------------------------------
1670
1671  *Are you facing a regression within a stable/longterm release, but failed to
1672  reproduce it with the kernel you just built using the latest mainline sources?
1673  Then check if the latest codebase for the particular series might already fix
1674  the problem.* [:ref:`... <recheckstablebroken_bissbs>`]
1675
1676If this kernel does not show the regression either, there most likely is no need
1677for a bisection.
1678
1679[:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`]
1680
1681.. _introworkingcheck_bisref:
1682
1683Ensure the 'good' version is really working well
1684------------------------------------------------
1685
1686  *Check if the kernels you build work fine.*
1687  [:ref:`... <introworkingcheck_bissbs>`]
1688
1689This section will reestablish a known working base. Skipping it might be
1690appealing, but is usually a bad idea, as it does something important:
1691
1692It will ensure the .config file you prepared earlier actually works as expected.
1693That is in your own interest, as trimming the configuration is not foolproof --
1694and you might be building and testing ten or more kernels for nothing before
1695starting to suspect something might be wrong with the build configuration.
1696
1697That alone is reason enough to spend the time on this, but not the only reason.
1698
1699Many readers of this guide normally run kernels that are patched, use add-on
1700modules, or both. Those kernels thus are not considered 'vanilla' -- therefore
1701it's possible that the thing that regressed might never have worked in vanilla
1702builds of the 'good' version in the first place.
1703
1704There is a third reason for those that noticed a regression between
1705stable/longterm kernels of different series (e.g. v6.0.13..v6.1.5): it will
1706ensure the kernel version you assumed to be 'good' earlier in the process (e.g.
1707v6.0) actually is working.
1708
1709[:ref:`back to step-by-step guide <introworkingcheck_bissbs>`]
1710
1711.. _recheckworking_bisref:
1712
1713Build your own version of the 'good' kernel
1714-------------------------------------------
1715
1716  *Build your own variant of the working kernel and check if the feature that
1717  regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`]
1718
1719In case the feature that broke with newer kernels does not work with your first
1720self-built kernel, find and resolve the cause before moving on. There are a
1721multitude of reasons why this might happen. Some ideas where to look:
1722
1723* Maybe localmodconfig did something odd and disabled the module required to
1724  test the feature? Then you might want to recreate a .config file based on the
1725  one from the last working kernel and skip trimming it down; manually disabling
1726  some features in the .config might work as well to reduce the build time.
1727
1728* Maybe it's not a kernel regression and something that is caused by some fluke,
1729  a broken initramfs (also known as initrd), new firmware files, or an updated
1730  userland software?
1731
1732* Maybe it was a feature added to your distributor's kernel which vanilla Linux
1733  at that point never supported?
1734
1735Note, if you found and fixed problems with the .config file, you want to use it
1736to build another kernel from the latest codebase, as your earlier tests with
1737mainline and the latest version from an affected stable/longterm series most
1738likely has been flawed.
1739
1740[:ref:`back to step-by-step guide <recheckworking_bissbs>`]
1741
1742.. _bisectstart_bisref:
1743
1744Start the bisection
1745-------------------
1746
1747  *Start the bisection and tell Git about the versions earlier established as
1748  'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`]
1749
1750This will start the bisection process; the last of the commands will make Git
1751checkout a commit round about half-way between the 'good' and the 'bad' changes
1752for your to test.
1753
1754[:ref:`back to step-by-step guide <bisectstart_bissbs>`]
1755
1756.. _bisectbuild_bisref:
1757
1758Build a kernel from the bisection point
1759---------------------------------------
1760
1761  *Build, install, and boot a kernel from the code Git checked out using the
1762  same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`]
1763
1764There are two things worth of note here:
1765
1766* Occasionally building the kernel will fail or it might not boot due some
1767  problem in the code at the bisection point. In that case run this command::
1768
1769    git bisect skip
1770
1771  Git will then check out another commit nearby which with a bit of luck should
1772  work better. Afterwards restart executing this step.
1773
1774* Those slightly odd looking version identifiers can happen during bisections,
1775  because the Linux kernel subsystems prepare their changes for a new mainline
1776  release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus
1777  base them on a somewhat earlier point like v6.1-rc1 or even v6.0 -- and then
1778  get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This
1779  leads to those slightly odd looking version identifiers coming up during
1780  bisections.
1781
1782[:ref:`back to step-by-step guide <bisectbuild_bissbs>`]
1783
1784.. _bisecttest_bisref:
1785
1786Bisection checkpoint
1787--------------------
1788
1789  *Check if the feature that regressed works in the kernel you just built.*
1790  [:ref:`... <bisecttest_bissbs>`]
1791
1792Ensure what you tell Git is accurate: getting it wrong just one time will bring
1793the rest of the bisection totally of course, hence all testing after that point
1794will be for nothing.
1795
1796[:ref:`back to step-by-step guide <bisecttest_bissbs>`]
1797
1798.. _bisectlog_bisref:
1799
1800Put the bisection log away
1801--------------------------
1802
1803  *Store Git's bisection log and the current .config file in a safe place.*
1804  [:ref:`... <bisectlog_bissbs>`]
1805
1806As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will
1807render the end result of a bisection useless. In that case you'd normally have
1808to restart the bisection from scratch. The log can prevent that, as it might
1809allow someone to point out where a bisection likely went sideways -- and then
1810instead of testing ten or more kernels you might only have to build a few to
1811resolve things.
1812
1813The .config file is put aside, as there is a decent chance that developers might
1814ask for it after you reported the regression.
1815
1816[:ref:`back to step-by-step guide <bisectlog_bissbs>`]
1817
1818.. _revert_bisref:
1819
1820Try reverting the culprit
1821-------------------------
1822
1823  *Try reverting the culprit on top of the latest codebase to see if this fixes
1824  your regression.* [:ref:`... <revert_bissbs>`]
1825
1826This is an optional step, but whenever possible one you should try: there is a
1827decent chance that developers will ask you to perform this step when you bring
1828the bisection result up. So give it a try, you are in the flow already, building
1829one more kernel shouldn't be a big deal at this point.
1830
1831The step-by-step guide covers everything relevant already except one slightly
1832rare thing: did you bisected a regression that also happened with mainline using
1833a stable/longterm series, but Git failed to revert the commit in mainline? Then
1834try to revert the culprit in the affected stable/longterm series -- and if that
1835succeeds, test that kernel version instead.
1836
1837[:ref:`back to step-by-step guide <revert_bissbs>`]
1838
1839
1840Supplementary tasks: cleanup during and after the bisection
1841-----------------------------------------------------------
1842
1843.. _makeroom_bisref:
1844
1845Cleaning up during the bisection
1846--------------------------------
1847
1848  *To remove one of the kernels you installed, look up its 'kernelrelease'
1849  identifier.* [:ref:`... <makeroom_bissbs>`]
1850
1851The kernels you install during this process are easy to remove later, as its
1852parts are only stored in two places and clearly identifiable. You thus do not
1853need to worry to mess up your machine when you install a kernel manually (and
1854thus bypass your distribution's packaging system): all parts of your kernels are
1855relatively easy to remove later.
1856
1857One of the two places is a directory in /lib/modules/, which holds the modules
1858for each installed kernel. This directory is named after the kernel's release
1859identifier; hence, to remove all modules for one of the kernels you built,
1860simply remove its modules directory in /lib/modules/.
1861
1862The other place is /boot/, where typically two up to five files will be placed
1863during installation of a kernel. All of them usually contain the release name in
1864their file name, but how many files and their exact name depends somewhat on
1865your distribution's installkernel executable and its initramfs generator. On
1866some distributions the ``kernel-install remove...`` command mentioned in the
1867step-by-step guide will delete all of these files for you while also removing
1868the menu entry for the kernel from your bootloader configuration. On others you
1869have to take care of these two tasks yourself. The following command should
1870interactively remove the three main files of a kernel with the release name
1871'6.0-rc1-local-gcafec0cacaca0'::
1872
1873  rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0
1874
1875Afterwards check for other files in /boot/ that have
1876'6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well.
1877Now remove the boot entry for the kernel from your bootloader's configuration;
1878the steps to do that vary quite a bit between Linux distributions.
1879
1880Note, be careful with wildcards like '*' when deleting files or directories
1881for kernels manually: you might accidentally remove files of a 6.0.11 kernel
1882when all you want is to remove 6.0 or 6.0.1.
1883
1884[:ref:`back to step-by-step guide <makeroom_bissbs>`]
1885
1886Cleaning up after the bisection
1887-------------------------------
1888
1889.. _finishingtouch_bisref:
1890
1891  *Once you have finished the bisection, do not immediately remove anything
1892  you set up, as you might need a few things again.*
1893  [:ref:`... <finishingtouch_bissbs>`]
1894
1895When you are really short of storage space removing the kernels as described in
1896the step-by-step guide might not free as much space as you would like. In that
1897case consider running ``rm -rf ~/linux/*`` as well now. This will remove the
1898build artifacts and the Linux sources, but will leave the Git repository
1899(~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the
1900sources back.
1901
1902Removing the repository as well would likely be unwise at this point: there is a
1903decent chance developers will ask you to build another kernel to perform
1904additional tests. This is often required to debug an issue or check proposed
1905fixes. Before doing so you want to run the ``git fetch mainline`` command again
1906followed by ``git checkout mainline/master`` to bring your clone up to date and
1907checkout the latest codebase. Then apply the patch using ``git apply
1908<filename>`` or ``git am <filename>`` and build yet another kernel using the
1909familiar commands.
1910
1911Additional tests are also the reason why you want to keep the
1912~/kernel-config-working file around for a few weeks.
1913
1914[:ref:`back to step-by-step guide <finishingtouch_bissbs>`]
1915
1916
1917Additional reading material
1918===========================
1919
1920Further sources
1921---------------
1922
1923* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
1924  `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
1925  in the Git documentation.
1926* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
1927  from kernel developer Nathan Chancellor.
1928* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
1929* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.
1930
1931..
1932   end-of-content
1933..
1934   This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
1935   you spot a typo or small mistake, feel free to let him know directly and
1936   he'll fix it. You are free to do the same in a mostly informal way if you
1937   want to contribute changes to the text -- but for copyright reasons please CC
1938   linux-doc@vger.kernel.org and 'sign-off' your contribution as
1939   Documentation/process/submitting-patches.rst explains in the section 'Sign
1940   your work - the Developer's Certificate of Origin'.
1941..
1942   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
1943   of the file. If you want to distribute this text under CC-BY-4.0 only,
1944   please use 'The Linux kernel development community' for author attribution
1945   and link this as source:
1946   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
1947
1948..
1949   Note: Only the content of this RST file as found in the Linux kernel sources
1950   is available under CC-BY-4.0, as versions of this text that were processed
1951   (for example by the kernel's build system) might contain content taken from
1952   files which use a more restrictive license.
1953