1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) 2.. See the bottom of this file for additional redistribution information. 3 4Reporting issues 5++++++++++++++++ 6 7 8The short guide (aka TL;DR) 9=========================== 10 11Are you facing a regression with vanilla kernels from the same stable or 12longterm series? One still supported? Then search the `LKML 13<https://lore.kernel.org/lkml/>`_ and the `Linux stable mailing list 14<https://lore.kernel.org/stable/>`_ archives for matching reports to join. If 15you don't find any, install `the latest release from that series 16<https://kernel.org/>`_. If it still shows the issue, report it to the stable 17mailing list (stable@vger.kernel.org) and CC the regressions list 18(regressions@lists.linux.dev); ideally also CC the maintainer and the mailing 19list for the subsystem in question. 20 21In all other cases try your best guess which kernel part might be causing the 22issue. Check the :ref:`MAINTAINERS <maintainers>` file for how its developers 23expect to be told about problems, which most of the time will be by email with a 24mailing list in CC. Check the destination's archives for matching reports; 25search the `LKML <https://lore.kernel.org/lkml/>`_ and the web, too. If you 26don't find any to join, install `the latest mainline kernel 27<https://kernel.org/>`_. If the issue is present there, send a report. 28 29The issue was fixed there, but you would like to see it resolved in a still 30supported stable or longterm series as well? Then install its latest release. 31If it shows the problem, search for the change that fixed it in mainline and 32check if backporting is in the works or was discarded; if it's neither, ask 33those who handled the change for it. 34 35**General remarks**: When installing and testing a kernel as outlined above, 36ensure it's vanilla (IOW: not patched and not using add-on modules). Also make 37sure it's built and running in a healthy environment and not already tainted 38before the issue occurs. 39 40If you are facing multiple issues with the Linux kernel at once, report each 41separately. While writing your report, include all information relevant to the 42issue, like the kernel and the distro used. In case of a regression, CC the 43regressions mailing list (regressions@lists.linux.dev) to your report. Also try 44to pinpoint the culprit with a bisection; if you succeed, include its 45commit-id and CC everyone in the sign-off-by chain. 46 47Once the report is out, answer any questions that come up and help where you 48can. That includes keeping the ball rolling by occasionally retesting with newer 49releases and sending a status update afterwards. 50 51.. 52 Note: If you see this note, you are reading the text's source file. You 53 might want to switch to a rendered version: It makes it a lot easier to 54 read and navigate this document -- especially when you want to look something 55 up in the reference section, then jump back to where you left off. 56.. 57 Find the latest rendered version of this text here: 58 https://docs.kernel.org/admin-guide/reporting-issues.html 59 60 61Step-by-step guide how to report issues to the kernel maintainers 62================================================================= 63 64The above TL;DR outlines roughly how to report issues to the Linux kernel 65developers. It might be all that's needed for people already familiar with 66reporting issues to Free/Libre & Open Source Software (FLOSS) projects. For 67everyone else there is this section. It is more detailed and uses a 68step-by-step approach. It still tries to be brief for readability and leaves 69out a lot of details; those are described below the step-by-step guide in a 70reference section, which explains each of the steps in more detail. 71 72Note: this section covers a few more aspects than the TL;DR and does things in 73a slightly different order. That's in your interest, to make sure you notice 74early if an issue that looks like a Linux kernel problem is actually caused by 75something else. These steps thus help to ensure the time you invest in this 76process won't feel wasted in the end: 77 78 * Are you facing an issue with a Linux kernel a hardware or software vendor 79 provided? Then in almost all cases you are better off to stop reading this 80 document and reporting the issue to your vendor instead, unless you are 81 willing to install the latest Linux version yourself. Be aware the latter 82 will often be needed anyway to hunt down and fix issues. 83 84 * Perform a rough search for existing reports with your favorite internet 85 search engine; additionally, check the archives of the `Linux Kernel Mailing 86 List (LKML) <https://lore.kernel.org/lkml/>`_. If you find matching reports, 87 join the discussion instead of sending a new one. 88 89 * See if the issue you are dealing with qualifies as regression, security 90 issue, or a really severe problem: those are 'issues of high priority' that 91 need special handling in some steps that are about to follow. 92 93 * Make sure it's not the kernel's surroundings that are causing the issue 94 you face. 95 96 * Create a fresh backup and put system repair and restore tools at hand. 97 98 * Ensure your system does not enhance its kernels by building additional 99 kernel modules on-the-fly, which solutions like DKMS might be doing locally 100 without your knowledge. 101 102 * Check if your kernel was 'tainted' when the issue occurred, as the event 103 that made the kernel set this flag might be causing the issue you face. 104 105 * Write down coarsely how to reproduce the issue. If you deal with multiple 106 issues at once, create separate notes for each of them and make sure they 107 work independently on a freshly booted system. That's needed, as each issue 108 needs to get reported to the kernel developers separately, unless they are 109 strongly entangled. 110 111 * If you are facing a regression within a stable or longterm version line 112 (say something broke when updating from 5.10.4 to 5.10.5), scroll down to 113 'Dealing with regressions within a stable and longterm kernel line'. 114 115 * Locate the driver or kernel subsystem that seems to be causing the issue. 116 Find out how and where its developers expect reports. Note: most of the 117 time this won't be bugzilla.kernel.org, as issues typically need to be sent 118 by mail to a maintainer and a public mailing list. 119 120 * Search the archives of the bug tracker or mailing list in question 121 thoroughly for reports that might match your issue. If you find anything, 122 join the discussion instead of sending a new report. 123 124After these preparations you'll now enter the main part: 125 126 * Unless you are already running the latest 'mainline' Linux kernel, better 127 go and install it for the reporting process. Testing and reporting with 128 the latest 'stable' Linux can be an acceptable alternative in some 129 situations; during the merge window that actually might be even the best 130 approach, but in that development phase it can be an even better idea to 131 suspend your efforts for a few days anyway. Whatever version you choose, 132 ideally use a 'vanilla' build. Ignoring these advices will dramatically 133 increase the risk your report will be rejected or ignored. 134 135 * Ensure the kernel you just installed does not 'taint' itself when 136 running. 137 138 * Reproduce the issue with the kernel you just installed. If it doesn't show 139 up there, scroll down to the instructions for issues only happening with 140 stable and longterm kernels. 141 142 * Optimize your notes: try to find and write the most straightforward way to 143 reproduce your issue. Make sure the end result has all the important 144 details, and at the same time is easy to read and understand for others 145 that hear about it for the first time. And if you learned something in this 146 process, consider searching again for existing reports about the issue. 147 148 * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider 149 decoding the kernel log to find the line of code that triggered the error. 150 151 * If your problem is a regression, try to narrow down when the issue was 152 introduced as much as possible. 153 154 * Start to compile the report by writing a detailed description about the 155 issue. Always mention a few things: the latest kernel version you installed 156 for reproducing, the Linux Distribution used, and your notes on how to 157 reproduce the issue. Ideally, make the kernel's build configuration 158 (.config) and the output from ``dmesg`` available somewhere on the net and 159 link to it. Include or upload all other information that might be relevant, 160 like the output/screenshot of an Oops or the output from ``lspci``. Once 161 you wrote this main part, insert a normal length paragraph on top of it 162 outlining the issue and the impact quickly. On top of this add one sentence 163 that briefly describes the problem and gets people to read on. Now give the 164 thing a descriptive title or subject that yet again is shorter. Then you're 165 ready to send or file the report like the MAINTAINERS file told you, unless 166 you are dealing with one of those 'issues of high priority': they need 167 special care which is explained in 'Special handling for high priority 168 issues' below. 169 170 * Wait for reactions and keep the thing rolling until you can accept the 171 outcome in one way or the other. Thus react publicly and in a timely manner 172 to any inquiries. Test proposed fixes. Do proactive testing: retest with at 173 least every first release candidate (RC) of a new mainline version and 174 report your results. Send friendly reminders if things stall. And try to 175 help yourself, if you don't get any help or if it's unsatisfying. 176 177 178Reporting regressions within a stable and longterm kernel line 179-------------------------------------------------------------- 180 181This subsection is for you, if you followed above process and got sent here at 182the point about regression within a stable or longterm kernel version line. You 183face one of those if something breaks when updating from 5.10.4 to 5.10.5 (a 184switch from 5.9.15 to 5.10.5 does not qualify). The developers want to fix such 185regressions as quickly as possible, hence there is a streamlined process to 186report them: 187 188 * Check if the kernel developers still maintain the Linux kernel version 189 line you care about: go to the `front page of kernel.org 190 <https://kernel.org/>`_ and make sure it mentions 191 the latest release of the particular version line without an '[EOL]' tag. 192 193 * Check the archives of the `Linux stable mailing list 194 <https://lore.kernel.org/stable/>`_ for existing reports. 195 196 * Install the latest release from the particular version line as a vanilla 197 kernel. Ensure this kernel is not tainted and still shows the problem, as 198 the issue might have already been fixed there. If you first noticed the 199 problem with a vendor kernel, check a vanilla build of the last version 200 known to work performs fine as well. 201 202 * Send a short problem report to the Linux stable mailing list 203 (stable@vger.kernel.org) and CC the Linux regressions mailing list 204 (regressions@lists.linux.dev); if you suspect the cause in a particular 205 subsystem, CC its maintainer and its mailing list. Roughly describe the 206 issue and ideally explain how to reproduce it. Mention the first version 207 that shows the problem and the last version that's working fine. Then 208 wait for further instructions. 209 210The reference section below explains each of these steps in more detail. 211 212 213Reporting issues only occurring in older kernel version lines 214------------------------------------------------------------- 215 216This subsection is for you, if you tried the latest mainline kernel as outlined 217above, but failed to reproduce your issue there; at the same time you want to 218see the issue fixed in a still supported stable or longterm series or vendor 219kernels regularly rebased on those. If that is the case, follow these steps: 220 221 * Prepare yourself for the possibility that going through the next few steps 222 might not get the issue solved in older releases: the fix might be too big 223 or risky to get backported there. 224 225 * Perform the first three steps in the section "Dealing with regressions 226 within a stable and longterm kernel line" above. 227 228 * Search the Linux kernel version control system for the change that fixed 229 the issue in mainline, as its commit message might tell you if the fix is 230 scheduled for backporting already. If you don't find anything that way, 231 search the appropriate mailing lists for posts that discuss such an issue 232 or peer-review possible fixes; then check the discussions if the fix was 233 deemed unsuitable for backporting. If backporting was not considered at 234 all, join the newest discussion, asking if it's in the cards. 235 236 * One of the former steps should lead to a solution. If that doesn't work 237 out, ask the maintainers for the subsystem that seems to be causing the 238 issue for advice; CC the mailing list for the particular subsystem as well 239 as the stable mailing list. 240 241The reference section below explains each of these steps in more detail. 242 243 244Conclusion of the step-by-step guide 245------------------------------------ 246 247Did you run into trouble following the step-by-step guide not cleared up by the 248reference section below? Did you spot errors? Or do you have ideas on how to 249improve the guide? 250 251If any of that applies, please let the developers know by sending a short note 252or a patch to Thorsten Leemhuis <linux@leemhuis.info> while ideally CCing the 253public Linux docs mailing list <linux-doc@vger.kernel.org>. Such feedback is 254vital to improve this text further, which is in everybody's interest, as it will 255enable more people to master the task described here. 256 257 258Reference section: Reporting issues to the kernel maintainers 259============================================================= 260 261The step-by-step guide above outlines all the major steps in brief fashion, 262which usually covers everything required. But even experienced users will 263sometimes wonder how to actually realize some of those steps or why they are 264needed; there are also corner cases the guide ignores for readability. That is 265what the entries in this reference section are for, which provide additional 266information for each of the steps in the guide. 267 268A few words of general advice: 269 270* The Linux developers are well aware that reporting bugs to them is more 271 complicated and demanding than in other FLOSS projects. Some of it is because 272 the kernel is different, among others due to its mail-driven development 273 process and because it consists mostly of drivers. Some of it is because 274 improving things would require work in several technical areas and people 275 triaging bugs –– and nobody has stepped up to do or fund that work. 276 277* A warranty or support contract with some vendor doesn't entitle you to 278 request fixes from the upstream Linux developers: Such contracts are 279 completely outside the scope of the upstream Linux kernel, its development 280 community, and this document -- even if those handling the issue work for the 281 vendor who issued the contract. If you want to claim your rights, use the 282 vendor's support channel. 283 284* If you never reported an issue to a FLOSS project before, consider skimming 285 guides like `How to ask good questions 286 <https://jvns.ca/blog/good-questions/>`_, `How To Ask Questions The Smart Way 287 <http://www.catb.org/esr/faqs/smart-questions.html>`_, and `How to Report 288 Bugs Effectively <https://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_,. 289 290With that off the table, find below details for the steps from the detailed 291guide on reporting issues to the Linux kernel developers. 292 293 294Make sure you're using the upstream Linux kernel 295------------------------------------------------ 296 297 *Are you facing an issue with a Linux kernel a hardware or software vendor 298 provided? Then in almost all cases you are better off to stop reading this 299 document and reporting the issue to your vendor instead, unless you are 300 willing to install the latest Linux version yourself. Be aware the latter 301 will often be needed anyway to hunt down and fix issues.* 302 303Like most programmers, Linux kernel developers don't like to spend time dealing 304with reports for issues that don't even happen with their current code. It's 305just a waste everybody's time, especially yours. Unfortunately such situations 306easily happen when it comes to the kernel and often leads to frustration on both 307sides. That's because almost all Linux-based kernels pre-installed on devices 308(Computers, Laptops, Smartphones, Routers, …) and most shipped by Linux 309distributors are quite distant from the official Linux kernel as distributed by 310kernel.org: these kernels from these vendors are often ancient from the point of 311Linux development or heavily modified, often both. 312 313Most of these vendor kernels are quite unsuitable for reporting issues to the 314Linux kernel developers: an issue you face with one of them might have been 315fixed by the Linux kernel developers months or years ago already; additionally, 316the modifications and enhancements by the vendor might be causing the issue you 317face, even if they look small or totally unrelated. That's why you should report 318issues with these kernels to the vendor. Its developers should look into the 319report and, in case it turns out to be an upstream issue, fix it directly 320upstream or forward the report there. In practice that often does not work out 321or might not what you want. You thus might want to consider circumventing the 322vendor by installing the very latest Linux kernel core yourself. If that's an 323option for you move ahead in this process, as a later step in this guide will 324explain how to do that once it rules out other potential causes for your issue. 325 326Note, the previous paragraph is starting with the word 'most', as sometimes 327developers in fact are willing to handle reports about issues occurring with 328vendor kernels. If they do in the end highly depends on the developers and the 329issue in question. Your chances are quite good if the distributor applied only 330small modifications to a kernel based on a recent Linux version; that for 331example often holds true for the mainline kernels shipped by Debian GNU/Linux 332Sid or Fedora Rawhide. Some developers will also accept reports about issues 333with kernels from distributions shipping the latest stable kernel, as long as 334it's only slightly modified; that for example is often the case for Arch Linux, 335regular Fedora releases, and openSUSE Tumbleweed. But keep in mind, you better 336want to use a mainline Linux and avoid using a stable kernel for this 337process, as outlined in the section 'Install a fresh kernel for testing' in more 338detail. 339 340Obviously you are free to ignore all this advice and report problems with an old 341or heavily modified vendor kernel to the upstream Linux developers. But note, 342those often get rejected or ignored, so consider yourself warned. But it's still 343better than not reporting the issue at all: sometimes such reports directly or 344indirectly will help to get the issue fixed over time. 345 346 347Search for existing reports, first run 348-------------------------------------- 349 350 *Perform a rough search for existing reports with your favorite internet 351 search engine; additionally, check the archives of the Linux Kernel Mailing 352 List (LKML). If you find matching reports, join the discussion instead of 353 sending a new one.* 354 355Reporting an issue that someone else already brought forward is often a waste of 356time for everyone involved, especially you as the reporter. So it's in your own 357interest to thoroughly check if somebody reported the issue already. At this 358step of the process it's okay to just perform a rough search: a later step will 359tell you to perform a more detailed search once you know where your issue needs 360to be reported to. Nevertheless, do not hurry with this step of the reporting 361process, it can save you time and trouble. 362 363Simply search the internet with your favorite search engine first. Afterwards, 364search the `Linux Kernel Mailing List (LKML) archives 365<https://lore.kernel.org/lkml/>`_. 366 367If you get flooded with results consider telling your search engine to limit 368search timeframe to the past month or year. And wherever you search, make sure 369to use good search terms; vary them a few times, too. While doing so try to 370look at the issue from the perspective of someone else: that will help you to 371come up with other words to use as search terms. Also make sure not to use too 372many search terms at once. Remember to search with and without information like 373the name of the kernel driver or the name of the affected hardware component. 374But its exact brand name (say 'ASUS Red Devil Radeon RX 5700 XT Gaming OC') 375often is not much helpful, as it is too specific. Instead try search terms like 376the model line (Radeon 5700 or Radeon 5000) and the code name of the main chip 377('Navi' or 'Navi10') with and without its manufacturer ('AMD'). 378 379In case you find an existing report about your issue, join the discussion, as 380you might be able to provide valuable additional information. That can be 381important even when a fix is prepared or in its final stages already, as 382developers might look for people that can provide additional information or 383test a proposed fix. Jump to the section 'Duties after the report went out' for 384details on how to get properly involved. 385 386Note, searching `bugzilla.kernel.org <https://bugzilla.kernel.org/>`_ might also 387be a good idea, as that might provide valuable insights or turn up matching 388reports. If you find the latter, just keep in mind: most subsystems expect 389reports in different places, as described below in the section "Check where you 390need to report your issue". The developers that should take care of the issue 391thus might not even be aware of the bugzilla ticket. Hence, check the ticket if 392the issue already got reported as outlined in this document and if not consider 393doing so. 394 395 396Issue of high priority? 397----------------------- 398 399 *See if the issue you are dealing with qualifies as regression, security 400 issue, or a really severe problem: those are 'issues of high priority' that 401 need special handling in some steps that are about to follow.* 402 403Linus Torvalds and the leading Linux kernel developers want to see some issues 404fixed as soon as possible, hence there are 'issues of high priority' that get 405handled slightly differently in the reporting process. Three type of cases 406qualify: regressions, security issues, and really severe problems. 407 408You deal with a regression if some application or practical use case running 409fine with one Linux kernel works worse or not at all with a newer version 410compiled using a similar configuration. The document 411Documentation/admin-guide/reporting-regressions.rst explains this in more 412detail. It also provides a good deal of other information about regressions you 413might want to be aware of; it for example explains how to add your issue to the 414list of tracked regressions, to ensure it won't fall through the cracks. 415 416What qualifies as security issue is left to your judgment. Consider reading 417Documentation/process/security-bugs.rst before proceeding, as it 418provides additional details how to best handle security issues. 419 420An issue is a 'really severe problem' when something totally unacceptably bad 421happens. That's for example the case when a Linux kernel corrupts the data it's 422handling or damages hardware it's running on. You're also dealing with a severe 423issue when the kernel suddenly stops working with an error message ('kernel 424panic') or without any farewell note at all. Note: do not confuse a 'panic' (a 425fatal error where the kernel stop itself) with a 'Oops' (a recoverable error), 426as the kernel remains running after the latter. 427 428 429Ensure a healthy environment 430---------------------------- 431 432 *Make sure it's not the kernel's surroundings that are causing the issue 433 you face.* 434 435Problems that look a lot like a kernel issue are sometimes caused by build or 436runtime environment. It's hard to rule out that problem completely, but you 437should minimize it: 438 439 * Use proven tools when building your kernel, as bugs in the compiler or the 440 binutils can cause the resulting kernel to misbehave. 441 442 * Ensure your computer components run within their design specifications; 443 that's especially important for the main processor, the main memory, and the 444 motherboard. Therefore, stop undervolting or overclocking when facing a 445 potential kernel issue. 446 447 * Try to make sure it's not faulty hardware that is causing your issue. Bad 448 main memory for example can result in a multitude of issues that will 449 manifest itself in problems looking like kernel issues. 450 451 * If you're dealing with a filesystem issue, you might want to check the file 452 system in question with ``fsck``, as it might be damaged in a way that leads 453 to unexpected kernel behavior. 454 455 * When dealing with a regression, make sure it's not something else that 456 changed in parallel to updating the kernel. The problem for example might be 457 caused by other software that was updated at the same time. It can also 458 happen that a hardware component coincidentally just broke when you rebooted 459 into a new kernel for the first time. Updating the systems BIOS or changing 460 something in the BIOS Setup can also lead to problems that on look a lot 461 like a kernel regression. 462 463 464Prepare for emergencies 465----------------------- 466 467 *Create a fresh backup and put system repair and restore tools at hand.* 468 469Reminder, you are dealing with computers, which sometimes do unexpected things, 470especially if you fiddle with crucial parts like the kernel of its operating 471system. That's what you are about to do in this process. Thus, make sure to 472create a fresh backup; also ensure you have all tools at hand to repair or 473reinstall the operating system as well as everything you need to restore the 474backup. 475 476 477Make sure your kernel doesn't get enhanced 478------------------------------------------ 479 480 *Ensure your system does not enhance its kernels by building additional 481 kernel modules on-the-fly, which solutions like DKMS might be doing locally 482 without your knowledge.* 483 484The risk your issue report gets ignored or rejected dramatically increases if 485your kernel gets enhanced in any way. That's why you should remove or disable 486mechanisms like akmods and DKMS: those build add-on kernel modules 487automatically, for example when you install a new Linux kernel or boot it for 488the first time. Also remove any modules they might have installed. Then reboot 489before proceeding. 490 491Note, you might not be aware that your system is using one of these solutions: 492they often get set up silently when you install Nvidia's proprietary graphics 493driver, VirtualBox, or other software that requires a some support from a 494module not part of the Linux kernel. That why your might need to uninstall the 495packages with such software to get rid of any 3rd party kernel module. 496 497 498Check 'taint' flag 499------------------ 500 501 *Check if your kernel was 'tainted' when the issue occurred, as the event 502 that made the kernel set this flag might be causing the issue you face.* 503 504The kernel marks itself with a 'taint' flag when something happens that might 505lead to follow-up errors that look totally unrelated. The issue you face might 506be such an error if your kernel is tainted. That's why it's in your interest to 507rule this out early before investing more time into this process. This is the 508only reason why this step is here, as this process later will tell you to 509install the latest mainline kernel; you will need to check the taint flag again 510then, as that's when it matters because it's the kernel the report will focus 511on. 512 513On a running system is easy to check if the kernel tainted itself: if ``cat 514/proc/sys/kernel/tainted`` returns '0' then the kernel is not tainted and 515everything is fine. Checking that file is impossible in some situations; that's 516why the kernel also mentions the taint status when it reports an internal 517problem (a 'kernel bug'), a recoverable error (a 'kernel Oops') or a 518non-recoverable error before halting operation (a 'kernel panic'). Look near 519the top of the error messages printed when one of these occurs and search for a 520line starting with 'CPU:'. It should end with 'Not tainted' if the kernel was 521not tainted when it noticed the problem; it was tainted if you see 'Tainted:' 522followed by a few spaces and some letters. 523 524If your kernel is tainted, study Documentation/admin-guide/tainted-kernels.rst 525to find out why. Try to eliminate the reason. Often it's caused by one these 526three things: 527 528 1. A recoverable error (a 'kernel Oops') occurred and the kernel tainted 529 itself, as the kernel knows it might misbehave in strange ways after that 530 point. In that case check your kernel or system log and look for a section 531 that starts with this:: 532 533 Oops: 0000 [#1] SMP 534 535 That's the first Oops since boot-up, as the '#1' between the brackets shows. 536 Every Oops and any other problem that happens after that point might be a 537 follow-up problem to that first Oops, even if both look totally unrelated. 538 Rule this out by getting rid of the cause for the first Oops and reproducing 539 the issue afterwards. Sometimes simply restarting will be enough, sometimes 540 a change to the configuration followed by a reboot can eliminate the Oops. 541 But don't invest too much time into this at this point of the process, as 542 the cause for the Oops might already be fixed in the newer Linux kernel 543 version you are going to install later in this process. 544 545 2. Your system uses a software that installs its own kernel modules, for 546 example Nvidia's proprietary graphics driver or VirtualBox. The kernel 547 taints itself when it loads such module from external sources (even if 548 they are Open Source): they sometimes cause errors in unrelated kernel 549 areas and thus might be causing the issue you face. You therefore have to 550 prevent those modules from loading when you want to report an issue to the 551 Linux kernel developers. Most of the time the easiest way to do that is: 552 temporarily uninstall such software including any modules they might have 553 installed. Afterwards reboot. 554 555 3. The kernel also taints itself when it's loading a module that resides in 556 the staging tree of the Linux kernel source. That's a special area for 557 code (mostly drivers) that does not yet fulfill the normal Linux kernel 558 quality standards. When you report an issue with such a module it's 559 obviously okay if the kernel is tainted; just make sure the module in 560 question is the only reason for the taint. If the issue happens in an 561 unrelated area reboot and temporarily block the module from being loaded 562 by specifying ``foo.blacklist=1`` as kernel parameter (replace 'foo' with 563 the name of the module in question). 564 565 566Document how to reproduce issue 567------------------------------- 568 569 *Write down coarsely how to reproduce the issue. If you deal with multiple 570 issues at once, create separate notes for each of them and make sure they 571 work independently on a freshly booted system. That's needed, as each issue 572 needs to get reported to the kernel developers separately, unless they are 573 strongly entangled.* 574 575If you deal with multiple issues at once, you'll have to report each of them 576separately, as they might be handled by different developers. Describing 577various issues in one report also makes it quite difficult for others to tear 578it apart. Hence, only combine issues in one report if they are very strongly 579entangled. 580 581Additionally, during the reporting process you will have to test if the issue 582happens with other kernel versions. Therefore, it will make your work easier if 583you know exactly how to reproduce an issue quickly on a freshly booted system. 584 585Note: it's often fruitless to report issues that only happened once, as they 586might be caused by a bit flip due to cosmic radiation. That's why you should 587try to rule that out by reproducing the issue before going further. Feel free 588to ignore this advice if you are experienced enough to tell a one-time error 589due to faulty hardware apart from a kernel issue that rarely happens and thus 590is hard to reproduce. 591 592 593Regression in stable or longterm kernel? 594---------------------------------------- 595 596 *If you are facing a regression within a stable or longterm version line 597 (say something broke when updating from 5.10.4 to 5.10.5), scroll down to 598 'Dealing with regressions within a stable and longterm kernel line'.* 599 600Regression within a stable and longterm kernel version line are something the 601Linux developers want to fix badly, as such issues are even more unwanted than 602regression in the main development branch, as they can quickly affect a lot of 603people. The developers thus want to learn about such issues as quickly as 604possible, hence there is a streamlined process to report them. Note, 605regressions with newer kernel version line (say something broke when switching 606from 5.9.15 to 5.10.5) do not qualify. 607 608 609Check where you need to report your issue 610----------------------------------------- 611 612 *Locate the driver or kernel subsystem that seems to be causing the issue. 613 Find out how and where its developers expect reports. Note: most of the 614 time this won't be bugzilla.kernel.org, as issues typically need to be sent 615 by mail to a maintainer and a public mailing list.* 616 617It's crucial to send your report to the right people, as the Linux kernel is a 618big project and most of its developers are only familiar with a small subset of 619it. Quite a few programmers for example only care for just one driver, for 620example one for a WiFi chip; its developer likely will only have small or no 621knowledge about the internals of remote or unrelated "subsystems", like the TCP 622stack, the PCIe/PCI subsystem, memory management or file systems. 623 624Problem is: the Linux kernel lacks a central bug tracker where you can simply 625file your issue and make it reach the developers that need to know about it. 626That's why you have to find the right place and way to report issues yourself. 627You can do that with the help of a script (see below), but it mainly targets 628kernel developers and experts. For everybody else the MAINTAINERS file is the 629better place. 630 631How to read the MAINTAINERS file 632~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 633To illustrate how to use the :ref:`MAINTAINERS <maintainers>` file, let's assume 634the WiFi in your Laptop suddenly misbehaves after updating the kernel. In that 635case it's likely an issue in the WiFi driver. Obviously it could also be some 636code it builds upon, but unless you suspect something like that stick to the 637driver. If it's really something else, the driver's developers will get the 638right people involved. 639 640Sadly, there is no way to check which code is driving a particular hardware 641component that is both universal and easy. 642 643In case of a problem with the WiFi driver you for example might want to look at 644the output of ``lspci -k``, as it lists devices on the PCI/PCIe bus and the 645kernel module driving it:: 646 647 [user@something ~]$ lspci -k 648 [...] 649 3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32) 650 Subsystem: Bigfoot Networks, Inc. Device 1535 651 Kernel driver in use: ath10k_pci 652 Kernel modules: ath10k_pci 653 [...] 654 655But this approach won't work if your WiFi chip is connected over USB or some 656other internal bus. In those cases you might want to check your WiFi manager or 657the output of ``ip link``. Look for the name of the problematic network 658interface, which might be something like 'wlp58s0'. This name can be used like 659this to find the module driving it:: 660 661 [user@something ~]$ realpath --relative-to=/sys/module/ /sys/class/net/wlp58s0/device/driver/module 662 ath10k_pci 663 664In case tricks like these don't bring you any further, try to search the 665internet on how to narrow down the driver or subsystem in question. And if you 666are unsure which it is: just try your best guess, somebody will help you if you 667guessed poorly. 668 669Once you know the driver or subsystem, you want to search for it in the 670MAINTAINERS file. In the case of 'ath10k_pci' you won't find anything, as the 671name is too specific. Sometimes you will need to search on the net for help; 672but before doing so, try a somewhat shorted or modified name when searching the 673MAINTAINERS file, as then you might find something like this:: 674 675 QUALCOMM ATHEROS ATH10K WIRELESS DRIVER 676 Mail: A. Some Human <shuman@example.com> 677 Mailing list: ath10k@lists.infradead.org 678 Status: Supported 679 Web-page: https://wireless.wiki.kernel.org/en/users/Drivers/ath10k 680 SCM: git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git 681 Files: drivers/net/wireless/ath/ath10k/ 682 683Note: the line description will be abbreviations, if you read the plain 684MAINTAINERS file found in the root of the Linux source tree. 'Mail:' for 685example will be 'M:', 'Mailing list:' will be 'L', and 'Status:' will be 'S:'. 686A section near the top of the file explains these and other abbreviations. 687 688First look at the line 'Status'. Ideally it should be 'Supported' or 689'Maintained'. If it states 'Obsolete' then you are using some outdated approach 690that was replaced by a newer solution you need to switch to. Sometimes the code 691only has someone who provides 'Odd Fixes' when feeling motivated. And with 692'Orphan' you are totally out of luck, as nobody takes care of the code anymore. 693That only leaves these options: arrange yourself to live with the issue, fix it 694yourself, or find a programmer somewhere willing to fix it. 695 696After checking the status, look for a line starting with 'bugs:': it will tell 697you where to find a subsystem specific bug tracker to file your issue. The 698example above does not have such a line. That is the case for most sections, as 699Linux kernel development is completely driven by mail. Very few subsystems use 700a bug tracker, and only some of those rely on bugzilla.kernel.org. 701 702In this and many other cases you thus have to look for lines starting with 703'Mail:' instead. Those mention the name and the email addresses for the 704maintainers of the particular code. Also look for a line starting with 'Mailing 705list:', which tells you the public mailing list where the code is developed. 706Your report later needs to go by mail to those addresses. Additionally, for all 707issue reports sent by email, make sure to add the Linux Kernel Mailing List 708(LKML) <linux-kernel@vger.kernel.org> to CC. Don't omit either of the mailing 709lists when sending your issue report by mail later! Maintainers are busy people 710and might leave some work for other developers on the subsystem specific list; 711and LKML is important to have one place where all issue reports can be found. 712 713 714Finding the maintainers with the help of a script 715~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 716 717For people that have the Linux sources at hand there is a second option to find 718the proper place to report: the script 'scripts/get_maintainer.pl' which tries 719to find all people to contact. It queries the MAINTAINERS file and needs to be 720called with a path to the source code in question. For drivers compiled as 721module if often can be found with a command like this:: 722 723 $ modinfo ath10k_pci | grep filename | sed 's!/lib/modules/.*/kernel/!!; s!filename:!!; s!\.ko\(\|\.xz\)!!' 724 drivers/net/wireless/ath/ath10k/ath10k_pci.ko 725 726Pass parts of this to the script:: 727 728 $ ./scripts/get_maintainer.pl -f drivers/net/wireless/ath/ath10k* 729 Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER) 730 Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS) 731 ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER) 732 linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS)) 733 netdev@vger.kernel.org (open list:NETWORKING DRIVERS) 734 linux-kernel@vger.kernel.org (open list) 735 736Don't sent your report to all of them. Send it to the maintainers, which the 737script calls "supporter:"; additionally CC the most specific mailing list for 738the code as well as the Linux Kernel Mailing List (LKML). In this case you thus 739would need to send the report to 'Some Human <shuman@example.com>' with 740'ath10k@lists.infradead.org' and 'linux-kernel@vger.kernel.org' in CC. 741 742Note: in case you cloned the Linux sources with git you might want to call 743``get_maintainer.pl`` a second time with ``--git``. The script then will look 744at the commit history to find which people recently worked on the code in 745question, as they might be able to help. But use these results with care, as it 746can easily send you in a wrong direction. That for example happens quickly in 747areas rarely changed (like old or unmaintained drivers): sometimes such code is 748modified during tree-wide cleanups by developers that do not care about the 749particular driver at all. 750 751 752Search for existing reports, second run 753--------------------------------------- 754 755 *Search the archives of the bug tracker or mailing list in question 756 thoroughly for reports that might match your issue. If you find anything, 757 join the discussion instead of sending a new report.* 758 759As mentioned earlier already: reporting an issue that someone else already 760brought forward is often a waste of time for everyone involved, especially you 761as the reporter. That's why you should search for existing report again, now 762that you know where they need to be reported to. If it's mailing list, you will 763often find its archives on `lore.kernel.org <https://lore.kernel.org/>`_. 764 765But some list are hosted in different places. That for example is the case for 766the ath10k WiFi driver used as example in the previous step. But you'll often 767find the archives for these lists easily on the net. Searching for 'archive 768ath10k@lists.infradead.org' for example will lead you to the `Info page for the 769ath10k mailing list <https://lists.infradead.org/mailman/listinfo/ath10k>`_, 770which at the top links to its 771`list archives <https://lists.infradead.org/pipermail/ath10k/>`_. Sadly this and 772quite a few other lists miss a way to search the archives. In those cases use a 773regular internet search engine and add something like 774'site:lists.infradead.org/pipermail/ath10k/' to your search terms, which limits 775the results to the archives at that URL. 776 777It's also wise to check the internet, LKML and maybe bugzilla.kernel.org again 778at this point. If your report needs to be filed in a bug tracker, you may want 779to check the mailing list archives for the subsystem as well, as someone might 780have reported it only there. 781 782For details how to search and what to do if you find matching reports see 783"Search for existing reports, first run" above. 784 785Do not hurry with this step of the reporting process: spending 30 to 60 minutes 786or even more time can save you and others quite a lot of time and trouble. 787 788 789Install a fresh kernel for testing 790---------------------------------- 791 792 *Unless you are already running the latest 'mainline' Linux kernel, better 793 go and install it for the reporting process. Testing and reporting with 794 the latest 'stable' Linux can be an acceptable alternative in some 795 situations; during the merge window that actually might be even the best 796 approach, but in that development phase it can be an even better idea to 797 suspend your efforts for a few days anyway. Whatever version you choose, 798 ideally use a 'vanilla' built. Ignoring these advices will dramatically 799 increase the risk your report will be rejected or ignored.* 800 801As mentioned in the detailed explanation for the first step already: Like most 802programmers, Linux kernel developers don't like to spend time dealing with 803reports for issues that don't even happen with the current code. It's just a 804waste everybody's time, especially yours. That's why it's in everybody's 805interest that you confirm the issue still exists with the latest upstream code 806before reporting it. You are free to ignore this advice, but as outlined 807earlier: doing so dramatically increases the risk that your issue report might 808get rejected or simply ignored. 809 810In the scope of the kernel "latest upstream" normally means: 811 812 * Install a mainline kernel; the latest stable kernel can be an option, but 813 most of the time is better avoided. Longterm kernels (sometimes called 'LTS 814 kernels') are unsuitable at this point of the process. The next subsection 815 explains all of this in more detail. 816 817 * The over next subsection describes way to obtain and install such a kernel. 818 It also outlines that using a pre-compiled kernel are fine, but better are 819 vanilla, which means: it was built using Linux sources taken straight `from 820 kernel.org <https://kernel.org/>`_ and not modified or enhanced in any way. 821 822Choosing the right version for testing 823~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 824 825Head over to `kernel.org <https://kernel.org/>`_ to find out which version you 826want to use for testing. Ignore the big yellow button that says 'Latest release' 827and look a little lower at the table. At its top you'll see a line starting with 828mainline, which most of the time will point to a pre-release with a version 829number like '5.8-rc2'. If that's the case, you'll want to use this mainline 830kernel for testing, as that where all fixes have to be applied first. Do not let 831that 'rc' scare you, these 'development kernels' are pretty reliable — and you 832made a backup, as you were instructed above, didn't you? 833 834In about two out of every nine to ten weeks, mainline might point you to a 835proper release with a version number like '5.7'. If that happens, consider 836suspending the reporting process until the first pre-release of the next 837version (5.8-rc1) shows up on kernel.org. That's because the Linux development 838cycle then is in its two-week long 'merge window'. The bulk of the changes and 839all intrusive ones get merged for the next release during this time. It's a bit 840more risky to use mainline during this period. Kernel developers are also often 841quite busy then and might have no spare time to deal with issue reports. It's 842also quite possible that one of the many changes applied during the merge 843window fixes the issue you face; that's why you soon would have to retest with 844a newer kernel version anyway, as outlined below in the section 'Duties after 845the report went out'. 846 847That's why it might make sense to wait till the merge window is over. But don't 848to that if you're dealing with something that shouldn't wait. In that case 849consider obtaining the latest mainline kernel via git (see below) or use the 850latest stable version offered on kernel.org. Using that is also acceptable in 851case mainline for some reason does currently not work for you. An in general: 852using it for reproducing the issue is also better than not reporting it issue 853at all. 854 855Better avoid using the latest stable kernel outside merge windows, as all fixes 856must be applied to mainline first. That's why checking the latest mainline 857kernel is so important: any issue you want to see fixed in older version lines 858needs to be fixed in mainline first before it can get backported, which can 859take a few days or weeks. Another reason: the fix you hope for might be too 860hard or risky for backporting; reporting the issue again hence is unlikely to 861change anything. 862 863These aspects are also why longterm kernels (sometimes called "LTS kernels") 864are unsuitable for this part of the reporting process: they are to distant from 865the current code. Hence go and test mainline first and follow the process 866further: if the issue doesn't occur with mainline it will guide you how to get 867it fixed in older version lines, if that's in the cards for the fix in question. 868 869How to obtain a fresh Linux kernel 870~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 871 872**Using a pre-compiled kernel**: This is often the quickest, easiest, and safest 873way for testing — especially is you are unfamiliar with the Linux kernel. The 874problem: most of those shipped by distributors or add-on repositories are build 875from modified Linux sources. They are thus not vanilla and therefore often 876unsuitable for testing and issue reporting: the changes might cause the issue 877you face or influence it somehow. 878 879But you are in luck if you are using a popular Linux distribution: for quite a 880few of them you'll find repositories on the net that contain packages with the 881latest mainline or stable Linux built as vanilla kernel. It's totally okay to 882use these, just make sure from the repository's description they are vanilla or 883at least close to it. Additionally ensure the packages contain the latest 884versions as offered on kernel.org. The packages are likely unsuitable if they 885are older than a week, as new mainline and stable kernels typically get released 886at least once a week. 887 888Please note that you might need to build your own kernel manually later: that's 889sometimes needed for debugging or testing fixes, as described later in this 890document. Also be aware that pre-compiled kernels might lack debug symbols that 891are needed to decode messages the kernel prints when a panic, Oops, warning, or 892BUG occurs; if you plan to decode those, you might be better off compiling a 893kernel yourself (see the end of this subsection and the section titled 'Decode 894failure messages' for details). 895 896**Using git**: Developers and experienced Linux users familiar with git are 897often best served by obtaining the latest Linux kernel sources straight from the 898`official development repository on kernel.org 899<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_. 900Those are likely a bit ahead of the latest mainline pre-release. Don't worry 901about it: they are as reliable as a proper pre-release, unless the kernel's 902development cycle is currently in the middle of a merge window. But even then 903they are quite reliable. 904 905**Conventional**: People unfamiliar with git are often best served by 906downloading the sources as tarball from `kernel.org <https://kernel.org/>`_. 907 908How to actually build a kernel is not described here, as many websites explain 909the necessary steps already. If you are new to it, consider following one of 910those how-to's that suggest to use ``make localmodconfig``, as that tries to 911pick up the configuration of your current kernel and then tries to adjust it 912somewhat for your system. That does not make the resulting kernel any better, 913but quicker to compile. 914 915Note: If you are dealing with a panic, Oops, warning, or BUG from the kernel, 916please try to enable CONFIG_KALLSYMS when configuring your kernel. 917Additionally, enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the 918latter is the relevant one of those two, but can only be reached if you enable 919the former. Be aware CONFIG_DEBUG_INFO increases the storage space required to 920build a kernel by quite a bit. But that's worth it, as these options will allow 921you later to pinpoint the exact line of code that triggers your issue. The 922section 'Decode failure messages' below explains this in more detail. 923 924But keep in mind: Always keep a record of the issue encountered in case it is 925hard to reproduce. Sending an undecoded report is better than not reporting 926the issue at all. 927 928 929Check 'taint' flag 930------------------ 931 932 *Ensure the kernel you just installed does not 'taint' itself when 933 running.* 934 935As outlined above in more detail already: the kernel sets a 'taint' flag when 936something happens that can lead to follow-up errors that look totally 937unrelated. That's why you need to check if the kernel you just installed does 938not set this flag. And if it does, you in almost all the cases needs to 939eliminate the reason for it before you reporting issues that occur with it. See 940the section above for details how to do that. 941 942 943Reproduce issue with the fresh kernel 944------------------------------------- 945 946 *Reproduce the issue with the kernel you just installed. If it doesn't show 947 up there, scroll down to the instructions for issues only happening with 948 stable and longterm kernels.* 949 950Check if the issue occurs with the fresh Linux kernel version you just 951installed. If it was fixed there already, consider sticking with this version 952line and abandoning your plan to report the issue. But keep in mind that other 953users might still be plagued by it, as long as it's not fixed in either stable 954and longterm version from kernel.org (and thus vendor kernels derived from 955those). If you prefer to use one of those or just want to help their users, 956head over to the section "Details about reporting issues only occurring in 957older kernel version lines" below. 958 959 960Optimize description to reproduce issue 961--------------------------------------- 962 963 *Optimize your notes: try to find and write the most straightforward way to 964 reproduce your issue. Make sure the end result has all the important 965 details, and at the same time is easy to read and understand for others 966 that hear about it for the first time. And if you learned something in this 967 process, consider searching again for existing reports about the issue.* 968 969An unnecessarily complex report will make it hard for others to understand your 970report. Thus try to find a reproducer that's straight forward to describe and 971thus easy to understand in written form. Include all important details, but at 972the same time try to keep it as short as possible. 973 974In this in the previous steps you likely have learned a thing or two about the 975issue you face. Use this knowledge and search again for existing reports 976instead you can join. 977 978 979Decode failure messages 980----------------------- 981 982 *If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider 983 decoding the kernel log to find the line of code that triggered the error.* 984 985When the kernel detects an internal problem, it will log some information about 986the executed code. This makes it possible to pinpoint the exact line in the 987source code that triggered the issue and shows how it was called. But that only 988works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring 989your kernel. If you did so, consider to decode the information from the 990kernel's log. That will make it a lot easier to understand what lead to the 991'panic', 'Oops', 'warning', or 'BUG', which increases the chances that someone 992can provide a fix. 993 994Decoding can be done with a script you find in the Linux source tree. If you 995are running a kernel you compiled yourself earlier, call it like this:: 996 997 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux 998 999If you are running a packaged vanilla kernel, you will likely have to install 1000the corresponding packages with debug symbols. Then call the script (which you 1001might need to get from the Linux sources if your distro does not package it) 1002like this:: 1003 1004 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh \ 1005 /usr/lib/debug/lib/modules/5.10.10-4.1.x86_64/vmlinux /usr/src/kernels/5.10.10-4.1.x86_64/ 1006 1007The script will work on log lines like the following, which show the address of 1008the code the kernel was executing when the error occurred:: 1009 1010 [ 68.387301] RIP: 0010:test_module_init+0x5/0xffa [test_module] 1011 1012Once decoded, these lines will look like this:: 1013 1014 [ 68.387301] RIP: 0010:test_module_init (/home/username/linux-5.10.5/test-module/test-module.c:16) test_module 1015 1016In this case the executed code was built from the file 1017'~/linux-5.10.5/test-module/test-module.c' and the error occurred by the 1018instructions found in line '16'. 1019 1020The script will similarly decode the addresses mentioned in the section 1021starting with 'Call trace', which show the path to the function where the 1022problem occurred. Additionally, the script will show the assembler output for 1023the code section the kernel was executing. 1024 1025Note, if you can't get this to work, simply skip this step and mention the 1026reason for it in the report. If you're lucky, it might not be needed. And if it 1027is, someone might help you to get things going. Also be aware this is just one 1028of several ways to decode kernel stack traces. Sometimes different steps will 1029be required to retrieve the relevant details. Don't worry about that, if that's 1030needed in your case, developers will tell you what to do. 1031 1032 1033Special care for regressions 1034---------------------------- 1035 1036 *If your problem is a regression, try to narrow down when the issue was 1037 introduced as much as possible.* 1038 1039Linux lead developer Linus Torvalds insists that the Linux kernel never 1040worsens, that's why he deems regressions as unacceptable and wants to see them 1041fixed quickly. That's why changes that introduced a regression are often 1042promptly reverted if the issue they cause can't get solved quickly any other 1043way. Reporting a regression is thus a bit like playing a kind of trump card to 1044get something quickly fixed. But for that to happen the change that's causing 1045the regression needs to be known. Normally it's up to the reporter to track 1046down the culprit, as maintainers often won't have the time or setup at hand to 1047reproduce it themselves. 1048 1049To find the change there is a process called 'bisection' which the document 1050Documentation/admin-guide/bug-bisect.rst describes in detail. That process 1051will often require you to build about ten to twenty kernel images, trying to 1052reproduce the issue with each of them before building the next. Yes, that takes 1053some time, but don't worry, it works a lot quicker than most people assume. 1054Thanks to a 'binary search' this will lead you to the one commit in the source 1055code management system that's causing the regression. Once you find it, search 1056the net for the subject of the change, its commit id and the shortened commit id 1057(the first 12 characters of the commit id). This will lead you to existing 1058reports about it, if there are any. 1059 1060Note, a bisection needs a bit of know-how, which not everyone has, and quite a 1061bit of effort, which not everyone is willing to invest. Nevertheless, it's 1062highly recommended performing a bisection yourself. If you really can't or 1063don't want to go down that route at least find out which mainline kernel 1064introduced the regression. If something for example breaks when switching from 10655.5.15 to 5.8.4, then try at least all the mainline releases in that area (5.6, 10665.7 and 5.8) to check when it first showed up. Unless you're trying to find a 1067regression in a stable or longterm kernel, avoid testing versions which number 1068has three sections (5.6.12, 5.7.8), as that makes the outcome hard to 1069interpret, which might render your testing useless. Once you found the major 1070version which introduced the regression, feel free to move on in the reporting 1071process. But keep in mind: it depends on the issue at hand if the developers 1072will be able to help without knowing the culprit. Sometimes they might 1073recognize from the report want went wrong and can fix it; other times they will 1074be unable to help unless you perform a bisection. 1075 1076When dealing with regressions make sure the issue you face is really caused by 1077the kernel and not by something else, as outlined above already. 1078 1079In the whole process keep in mind: an issue only qualifies as regression if the 1080older and the newer kernel got built with a similar configuration. This can be 1081achieved by using ``make olddefconfig``, as explained in more detail by 1082Documentation/admin-guide/reporting-regressions.rst; that document also 1083provides a good deal of other information about regressions you might want to be 1084aware of. 1085 1086 1087Write and send the report 1088------------------------- 1089 1090 *Start to compile the report by writing a detailed description about the 1091 issue. Always mention a few things: the latest kernel version you installed 1092 for reproducing, the Linux Distribution used, and your notes on how to 1093 reproduce the issue. Ideally, make the kernel's build configuration 1094 (.config) and the output from ``dmesg`` available somewhere on the net and 1095 link to it. Include or upload all other information that might be relevant, 1096 like the output/screenshot of an Oops or the output from ``lspci``. Once 1097 you wrote this main part, insert a normal length paragraph on top of it 1098 outlining the issue and the impact quickly. On top of this add one sentence 1099 that briefly describes the problem and gets people to read on. Now give the 1100 thing a descriptive title or subject that yet again is shorter. Then you're 1101 ready to send or file the report like the MAINTAINERS file told you, unless 1102 you are dealing with one of those 'issues of high priority': they need 1103 special care which is explained in 'Special handling for high priority 1104 issues' below.* 1105 1106Now that you have prepared everything it's time to write your report. How to do 1107that is partly explained by the three documents linked to in the preface above. 1108That's why this text will only mention a few of the essentials as well as 1109things specific to the Linux kernel. 1110 1111There is one thing that fits both categories: the most crucial parts of your 1112report are the title/subject, the first sentence, and the first paragraph. 1113Developers often get quite a lot of mail. They thus often just take a few 1114seconds to skim a mail before deciding to move on or look closer. Thus: the 1115better the top section of your report, the higher are the chances that someone 1116will look into it and help you. And that is why you should ignore them for now 1117and write the detailed report first. ;-) 1118 1119Things each report should mention 1120~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1121 1122Describe in detail how your issue happens with the fresh vanilla kernel you 1123installed. Try to include the step-by-step instructions you wrote and optimized 1124earlier that outline how you and ideally others can reproduce the issue; in 1125those rare cases where that's impossible try to describe what you did to 1126trigger it. 1127 1128Also include all the relevant information others might need to understand the 1129issue and its environment. What's actually needed depends a lot on the issue, 1130but there are some things you should include always: 1131 1132 * the output from ``cat /proc/version``, which contains the Linux kernel 1133 version number and the compiler it was built with. 1134 1135 * the Linux distribution the machine is running (``hostnamectl | grep 1136 "Operating System"``) 1137 1138 * the architecture of the CPU and the operating system (``uname -mi``) 1139 1140 * if you are dealing with a regression and performed a bisection, mention the 1141 subject and the commit-id of the change that is causing it. 1142 1143In a lot of cases it's also wise to make two more things available to those 1144that read your report: 1145 1146 * the configuration used for building your Linux kernel (the '.config' file) 1147 1148 * the kernel's messages that you get from ``dmesg`` written to a file. Make 1149 sure that it starts with a line like 'Linux version 5.8-1 1150 (foobar@example.com) (gcc (GCC) 10.2.1, GNU ld version 2.34) #1 SMP Mon Aug 1151 3 14:54:37 UTC 2020' If it's missing, then important messages from the first 1152 boot phase already got discarded. In this case instead consider using 1153 ``journalctl -b 0 -k``; alternatively you can also reboot, reproduce the 1154 issue and call ``dmesg`` right afterwards. 1155 1156These two files are big, that's why it's a bad idea to put them directly into 1157your report. If you are filing the issue in a bug tracker then attach them to 1158the ticket. If you report the issue by mail do not attach them, as that makes 1159the mail too large; instead do one of these things: 1160 1161 * Upload the files somewhere public (your website, a public file paste 1162 service, a ticket created just for this purpose on `bugzilla.kernel.org 1163 <https://bugzilla.kernel.org/>`_, ...) and include a link to them in your 1164 report. Ideally use something where the files stay available for years, as 1165 they could be useful to someone many years from now; this for example can 1166 happen if five or ten years from now a developer works on some code that was 1167 changed just to fix your issue. 1168 1169 * Put the files aside and mention you will send them later in individual 1170 replies to your own mail. Just remember to actually do that once the report 1171 went out. ;-) 1172 1173Things that might be wise to provide 1174~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1175 1176Depending on the issue you might need to add more background data. Here are a 1177few suggestions what often is good to provide: 1178 1179 * If you are dealing with a 'warning', an 'OOPS' or a 'panic' from the kernel, 1180 include it. If you can't copy'n'paste it, try to capture a netconsole trace 1181 or at least take a picture of the screen. 1182 1183 * If the issue might be related to your computer hardware, mention what kind 1184 of system you use. If you for example have problems with your graphics card, 1185 mention its manufacturer, the card's model, and what chip is uses. If it's a 1186 laptop mention its name, but try to make sure it's meaningful. 'Dell XPS 13' 1187 for example is not, because it might be the one from 2012; that one looks 1188 not that different from the one sold today, but apart from that the two have 1189 nothing in common. Hence, in such cases add the exact model number, which 1190 for example are '9380' or '7390' for XPS 13 models introduced during 2019. 1191 Names like 'Lenovo Thinkpad T590' are also somewhat ambiguous: there are 1192 variants of this laptop with and without a dedicated graphics chip, so try 1193 to find the exact model name or specify the main components. 1194 1195 * Mention the relevant software in use. If you have problems with loading 1196 modules, you want to mention the versions of kmod, systemd, and udev in use. 1197 If one of the DRM drivers misbehaves, you want to state the versions of 1198 libdrm and Mesa; also specify your Wayland compositor or the X-Server and 1199 its driver. If you have a filesystem issue, mention the version of 1200 corresponding filesystem utilities (e2fsprogs, btrfs-progs, xfsprogs, ...). 1201 1202 * Gather additional information from the kernel that might be of interest. The 1203 output from ``lspci -nn`` will for example help others to identify what 1204 hardware you use. If you have a problem with hardware you even might want to 1205 make the output from ``sudo lspci -vvv`` available, as that provides 1206 insights how the components were configured. For some issues it might be 1207 good to include the contents of files like ``/proc/cpuinfo``, 1208 ``/proc/ioports``, ``/proc/iomem``, ``/proc/modules``, or 1209 ``/proc/scsi/scsi``. Some subsystem also offer tools to collect relevant 1210 information. One such tool is ``alsa-info.sh`` `which the audio/sound 1211 subsystem developers provide <https://www.alsa-project.org/wiki/AlsaInfo>`_. 1212 1213Those examples should give your some ideas of what data might be wise to 1214attach, but you have to think yourself what will be helpful for others to know. 1215Don't worry too much about forgetting something, as developers will ask for 1216additional details they need. But making everything important available from 1217the start increases the chance someone will take a closer look. 1218 1219 1220The important part: the head of your report 1221~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1222 1223Now that you have the detailed part of the report prepared let's get to the 1224most important section: the first few sentences. Thus go to the top, add 1225something like 'The detailed description:' before the part you just wrote and 1226insert two newlines at the top. Now write one normal length paragraph that 1227describes the issue roughly. Leave out all boring details and focus on the 1228crucial parts readers need to know to understand what this is all about; if you 1229think this bug affects a lot of users, mention this to get people interested. 1230 1231Once you did that insert two more lines at the top and write a one sentence 1232summary that explains quickly what the report is about. After that you have to 1233get even more abstract and write an even shorter subject/title for the report. 1234 1235Now that you have written this part take some time to optimize it, as it is the 1236most important parts of your report: a lot of people will only read this before 1237they decide if reading the rest is time well spent. 1238 1239Now send or file the report like the :ref:`MAINTAINERS <maintainers>` file told 1240you, unless it's one of those 'issues of high priority' outlined earlier: in 1241that case please read the next subsection first before sending the report on 1242its way. 1243 1244Special handling for high priority issues 1245~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1246 1247Reports for high priority issues need special handling. 1248 1249**Severe issues**: make sure the subject or ticket title as well as the first 1250paragraph makes the severeness obvious. 1251 1252**Regressions**: make the report's subject start with '[REGRESSION]'. 1253 1254In case you performed a successful bisection, use the title of the change that 1255introduced the regression as the second part of your subject. Make the report 1256also mention the commit id of the culprit. In case of an unsuccessful bisection, 1257make your report mention the latest tested version that's working fine (say 5.7) 1258and the oldest where the issue occurs (say 5.8-rc1). 1259 1260When sending the report by mail, CC the Linux regressions mailing list 1261(regressions@lists.linux.dev). In case the report needs to be filed to some web 1262tracker, proceed to do so. Once filed, forward the report by mail to the 1263regressions list; CC the maintainer and the mailing list for the subsystem in 1264question. Make sure to inline the forwarded report, hence do not attach it. 1265Also add a short note at the top where you mention the URL to the ticket. 1266 1267When mailing or forwarding the report, in case of a successful bisection add the 1268author of the culprit to the recipients; also CC everyone in the signed-off-by 1269chain, which you find at the end of its commit message. 1270 1271**Security issues**: for these issues your will have to evaluate if a 1272short-term risk to other users would arise if details were publicly disclosed. 1273If that's not the case simply proceed with reporting the issue as described. 1274For issues that bear such a risk you will need to adjust the reporting process 1275slightly: 1276 1277 * If the MAINTAINERS file instructed you to report the issue by mail, do not 1278 CC any public mailing lists. 1279 1280 * If you were supposed to file the issue in a bug tracker make sure to mark 1281 the ticket as 'private' or 'security issue'. If the bug tracker does not 1282 offer a way to keep reports private, forget about it and send your report as 1283 a private mail to the maintainers instead. 1284 1285In both cases make sure to also mail your report to the addresses the 1286MAINTAINERS file lists in the section 'security contact'. Ideally directly CC 1287them when sending the report by mail. If you filed it in a bug tracker, forward 1288the report's text to these addresses; but on top of it put a small note where 1289you mention that you filed it with a link to the ticket. 1290 1291See Documentation/process/security-bugs.rst for more information. 1292 1293 1294Duties after the report went out 1295-------------------------------- 1296 1297 *Wait for reactions and keep the thing rolling until you can accept the 1298 outcome in one way or the other. Thus react publicly and in a timely manner 1299 to any inquiries. Test proposed fixes. Do proactive testing: retest with at 1300 least every first release candidate (RC) of a new mainline version and 1301 report your results. Send friendly reminders if things stall. And try to 1302 help yourself, if you don't get any help or if it's unsatisfying.* 1303 1304If your report was good and you are really lucky then one of the developers 1305might immediately spot what's causing the issue; they then might write a patch 1306to fix it, test it, and send it straight for integration in mainline while 1307tagging it for later backport to stable and longterm kernels that need it. Then 1308all you need to do is reply with a 'Thank you very much' and switch to a version 1309with the fix once it gets released. 1310 1311But this ideal scenario rarely happens. That's why the job is only starting 1312once you got the report out. What you'll have to do depends on the situations, 1313but often it will be the things listed below. But before digging into the 1314details, here are a few important things you need to keep in mind for this part 1315of the process. 1316 1317 1318General advice for further interactions 1319~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1320 1321**Always reply in public**: When you filed the issue in a bug tracker, always 1322reply there and do not contact any of the developers privately about it. For 1323mailed reports always use the 'Reply-all' function when replying to any mails 1324you receive. That includes mails with any additional data you might want to add 1325to your report: go to your mail applications 'Sent' folder and use 'reply-all' 1326on your mail with the report. This approach will make sure the public mailing 1327list(s) and everyone else that gets involved over time stays in the loop; it 1328also keeps the mail thread intact, which among others is really important for 1329mailing lists to group all related mails together. 1330 1331There are just two situations where a comment in a bug tracker or a 'Reply-all' 1332is unsuitable: 1333 1334 * Someone tells you to send something privately. 1335 1336 * You were told to send something, but noticed it contains sensitive 1337 information that needs to be kept private. In that case it's okay to send it 1338 in private to the developer that asked for it. But note in the ticket or a 1339 mail that you did that, so everyone else knows you honored the request. 1340 1341**Do research before asking for clarifications or help**: In this part of the 1342process someone might tell you to do something that requires a skill you might 1343not have mastered yet. For example, you might be asked to use some test tools 1344you never have heard of yet; or you might be asked to apply a patch to the 1345Linux kernel sources to test if it helps. In some cases it will be fine sending 1346a reply asking for instructions how to do that. But before going that route try 1347to find the answer own your own by searching the internet; alternatively 1348consider asking in other places for advice. For example ask a friend or post 1349about it to a chatroom or forum you normally hang out. 1350 1351**Be patient**: If you are really lucky you might get a reply to your report 1352within a few hours. But most of the time it will take longer, as maintainers 1353are scattered around the globe and thus might be in a different time zone – one 1354where they already enjoy their night away from keyboard. 1355 1356In general, kernel developers will take one to five business days to respond to 1357reports. Sometimes it will take longer, as they might be busy with the merge 1358windows, other work, visiting developer conferences, or simply enjoying a long 1359summer holiday. 1360 1361The 'issues of high priority' (see above for an explanation) are an exception 1362here: maintainers should address them as soon as possible; that's why you 1363should wait a week at maximum (or just two days if it's something urgent) 1364before sending a friendly reminder. 1365 1366Sometimes the maintainer might not be responding in a timely manner; other 1367times there might be disagreements, for example if an issue qualifies as 1368regression or not. In such cases raise your concerns on the mailing list and 1369ask others for public or private replies how to move on. If that fails, it 1370might be appropriate to get a higher authority involved. In case of a WiFi 1371driver that would be the wireless maintainers; if there are no higher level 1372maintainers or all else fails, it might be one of those rare situations where 1373it's okay to get Linus Torvalds involved. 1374 1375**Proactive testing**: Every time the first pre-release (the 'rc1') of a new 1376mainline kernel version gets released, go and check if the issue is fixed there 1377or if anything of importance changed. Mention the outcome in the ticket or in a 1378mail you sent as reply to your report (make sure it has all those in the CC 1379that up to that point participated in the discussion). This will show your 1380commitment and that you are willing to help. It also tells developers if the 1381issue persists and makes sure they do not forget about it. A few other 1382occasional retests (for example with rc3, rc5 and the final) are also a good 1383idea, but only report your results if something relevant changed or if you are 1384writing something anyway. 1385 1386With all these general things off the table let's get into the details of how 1387to help to get issues resolved once they were reported. 1388 1389Inquires and testing request 1390~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1391 1392Here are your duties in case you got replies to your report: 1393 1394**Check who you deal with**: Most of the time it will be the maintainer or a 1395developer of the particular code area that will respond to your report. But as 1396issues are normally reported in public it could be anyone that's replying — 1397including people that want to help, but in the end might guide you totally off 1398track with their questions or requests. That rarely happens, but it's one of 1399many reasons why it's wise to quickly run an internet search to see who you're 1400interacting with. By doing this you also get aware if your report was heard by 1401the right people, as a reminder to the maintainer (see below) might be in order 1402later if discussion fades out without leading to a satisfying solution for the 1403issue. 1404 1405**Inquiries for data**: Often you will be asked to test something or provide 1406additional details. Try to provide the requested information soon, as you have 1407the attention of someone that might help and risk losing it the longer you 1408wait; that outcome is even likely if you do not provide the information within 1409a few business days. 1410 1411**Requests for testing**: When you are asked to test a diagnostic patch or a 1412possible fix, try to test it in timely manner, too. But do it properly and make 1413sure to not rush it: mixing things up can happen easily and can lead to a lot 1414of confusion for everyone involved. A common mistake for example is thinking a 1415proposed patch with a fix was applied, but in fact wasn't. Things like that 1416happen even to experienced testers occasionally, but they most of the time will 1417notice when the kernel with the fix behaves just as one without it. 1418 1419What to do when nothing of substance happens 1420~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1421 1422Some reports will not get any reaction from the responsible Linux kernel 1423developers; or a discussion around the issue evolved, but faded out with 1424nothing of substance coming out of it. 1425 1426In these cases wait two (better: three) weeks before sending a friendly 1427reminder: maybe the maintainer was just away from keyboard for a while when 1428your report arrived or had something more important to take care of. When 1429writing the reminder, kindly ask if anything else from your side is needed to 1430get the ball running somehow. If the report got out by mail, do that in the 1431first lines of a mail that is a reply to your initial mail (see above) which 1432includes a full quote of the original report below: that's on of those few 1433situations where such a 'TOFU' (Text Over, Fullquote Under) is the right 1434approach, as then all the recipients will have the details at hand immediately 1435in the proper order. 1436 1437After the reminder wait three more weeks for replies. If you still don't get a 1438proper reaction, you first should reconsider your approach. Did you maybe try 1439to reach out to the wrong people? Was the report maybe offensive or so 1440confusing that people decided to completely stay away from it? The best way to 1441rule out such factors: show the report to one or two people familiar with FLOSS 1442issue reporting and ask for their opinion. Also ask them for their advice how 1443to move forward. That might mean: prepare a better report and make those people 1444review it before you send it out. Such an approach is totally fine; just 1445mention that this is the second and improved report on the issue and include a 1446link to the first report. 1447 1448If the report was proper you can send a second reminder; in it ask for advice 1449why the report did not get any replies. A good moment for this second reminder 1450mail is shortly after the first pre-release (the 'rc1') of a new Linux kernel 1451version got published, as you should retest and provide a status update at that 1452point anyway (see above). 1453 1454If the second reminder again results in no reaction within a week, try to 1455contact a higher-level maintainer asking for advice: even busy maintainers by 1456then should at least have sent some kind of acknowledgment. 1457 1458Remember to prepare yourself for a disappointment: maintainers ideally should 1459react somehow to every issue report, but they are only obliged to fix those 1460'issues of high priority' outlined earlier. So don't be too devastating if you 1461get a reply along the lines of 'thanks for the report, I have more important 1462issues to deal with currently and won't have time to look into this for the 1463foreseeable future'. 1464 1465It's also possible that after some discussion in the bug tracker or on a list 1466nothing happens anymore and reminders don't help to motivate anyone to work out 1467a fix. Such situations can be devastating, but is within the cards when it 1468comes to Linux kernel development. This and several other reasons for not 1469getting help are explained in 'Why some issues won't get any reaction or remain 1470unfixed after being reported' near the end of this document. 1471 1472Don't get devastated if you don't find any help or if the issue in the end does 1473not get solved: the Linux kernel is FLOSS and thus you can still help yourself. 1474You for example could try to find others that are affected and team up with 1475them to get the issue resolved. Such a team could prepare a fresh report 1476together that mentions how many you are and why this is something that in your 1477option should get fixed. Maybe together you can also narrow down the root cause 1478or the change that introduced a regression, which often makes developing a fix 1479easier. And with a bit of luck there might be someone in the team that knows a 1480bit about programming and might be able to write a fix. 1481 1482 1483Reference for "Reporting regressions within a stable and longterm kernel line" 1484------------------------------------------------------------------------------ 1485 1486This subsection provides details for the steps you need to perform if you face 1487a regression within a stable and longterm kernel line. 1488 1489Make sure the particular version line still gets support 1490~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1491 1492 *Check if the kernel developers still maintain the Linux kernel version 1493 line you care about: go to the front page of kernel.org and make sure it 1494 mentions the latest release of the particular version line without an 1495 '[EOL]' tag.* 1496 1497Most kernel version lines only get supported for about three months, as 1498maintaining them longer is quite a lot of work. Hence, only one per year is 1499chosen and gets supported for at least two years (often six). That's why you 1500need to check if the kernel developers still support the version line you care 1501for. 1502 1503Note, if kernel.org lists two stable version lines on the front page, you 1504should consider switching to the newer one and forget about the older one: 1505support for it is likely to be abandoned soon. Then it will get a "end-of-life" 1506(EOL) stamp. Version lines that reached that point still get mentioned on the 1507kernel.org front page for a week or two, but are unsuitable for testing and 1508reporting. 1509 1510Search stable mailing list 1511~~~~~~~~~~~~~~~~~~~~~~~~~~ 1512 1513 *Check the archives of the Linux stable mailing list for existing reports.* 1514 1515Maybe the issue you face is already known and was fixed or is about to. Hence, 1516`search the archives of the Linux stable mailing list 1517<https://lore.kernel.org/stable/>`_ for reports about an issue like yours. If 1518you find any matches, consider joining the discussion, unless the fix is 1519already finished and scheduled to get applied soon. 1520 1521Reproduce issue with the newest release 1522~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1523 1524 *Install the latest release from the particular version line as a vanilla 1525 kernel. Ensure this kernel is not tainted and still shows the problem, as 1526 the issue might have already been fixed there. If you first noticed the 1527 problem with a vendor kernel, check a vanilla build of the last version 1528 known to work performs fine as well.* 1529 1530Before investing any more time in this process you want to check if the issue 1531was already fixed in the latest release of version line you're interested in. 1532This kernel needs to be vanilla and shouldn't be tainted before the issue 1533happens, as detailed outlined already above in the section "Install a fresh 1534kernel for testing". 1535 1536Did you first notice the regression with a vendor kernel? Then changes the 1537vendor applied might be interfering. You need to rule that out by performing 1538a recheck. Say something broke when you updated from 5.10.4-vendor.42 to 15395.10.5-vendor.43. Then after testing the latest 5.10 release as outlined in 1540the previous paragraph check if a vanilla build of Linux 5.10.4 works fine as 1541well. If things are broken there, the issue does not qualify as upstream 1542regression and you need switch back to the main step-by-step guide to report 1543the issue. 1544 1545Report the regression 1546~~~~~~~~~~~~~~~~~~~~~ 1547 1548 *Send a short problem report to the Linux stable mailing list 1549 (stable@vger.kernel.org) and CC the Linux regressions mailing list 1550 (regressions@lists.linux.dev); if you suspect the cause in a particular 1551 subsystem, CC its maintainer and its mailing list. Roughly describe the 1552 issue and ideally explain how to reproduce it. Mention the first version 1553 that shows the problem and the last version that's working fine. Then 1554 wait for further instructions.* 1555 1556When reporting a regression that happens within a stable or longterm kernel 1557line (say when updating from 5.10.4 to 5.10.5) a brief report is enough for 1558the start to get the issue reported quickly. Hence a rough description to the 1559stable and regressions mailing list is all it takes; but in case you suspect 1560the cause in a particular subsystem, CC its maintainers and its mailing list 1561as well, because that will speed things up. 1562 1563And note, it helps developers a great deal if you can specify the exact version 1564that introduced the problem. Hence if possible within a reasonable time frame, 1565try to find that version using vanilla kernels. Let's assume something broke when 1566your distributor released a update from Linux kernel 5.10.5 to 5.10.8. Then as 1567instructed above go and check the latest kernel from that version line, say 15685.10.9. If it shows the problem, try a vanilla 5.10.5 to ensure that no patches 1569the distributor applied interfere. If the issue doesn't manifest itself there, 1570try 5.10.7 and then (depending on the outcome) 5.10.8 or 5.10.6 to find the 1571first version where things broke. Mention it in the report and state that 5.10.9 1572is still broken. 1573 1574What the previous paragraph outlines is basically a rough manual 'bisection'. 1575Once your report is out your might get asked to do a proper one, as it allows to 1576pinpoint the exact change that causes the issue (which then can easily get 1577reverted to fix the issue quickly). Hence consider to do a proper bisection 1578right away if time permits. See the section 'Special care for regressions' and 1579the document Documentation/admin-guide/bug-bisect.rst for details how to 1580perform one. In case of a successful bisection add the author of the culprit to 1581the recipients; also CC everyone in the signed-off-by chain, which you find at 1582the end of its commit message. 1583 1584 1585Reference for "Reporting issues only occurring in older kernel version lines" 1586----------------------------------------------------------------------------- 1587 1588This section provides details for the steps you need to take if you could not 1589reproduce your issue with a mainline kernel, but want to see it fixed in older 1590version lines (aka stable and longterm kernels). 1591 1592Some fixes are too complex 1593~~~~~~~~~~~~~~~~~~~~~~~~~~ 1594 1595 *Prepare yourself for the possibility that going through the next few steps 1596 might not get the issue solved in older releases: the fix might be too big 1597 or risky to get backported there.* 1598 1599Even small and seemingly obvious code-changes sometimes introduce new and 1600totally unexpected problems. The maintainers of the stable and longterm kernels 1601are very aware of that and thus only apply changes to these kernels that are 1602within rules outlined in Documentation/process/stable-kernel-rules.rst. 1603 1604Complex or risky changes for example do not qualify and thus only get applied 1605to mainline. Other fixes are easy to get backported to the newest stable and 1606longterm kernels, but too risky to integrate into older ones. So be aware the 1607fix you are hoping for might be one of those that won't be backported to the 1608version line your care about. In that case you'll have no other choice then to 1609live with the issue or switch to a newer Linux version, unless you want to 1610patch the fix into your kernels yourself. 1611 1612Common preparations 1613~~~~~~~~~~~~~~~~~~~ 1614 1615 *Perform the first three steps in the section "Reporting issues only 1616 occurring in older kernel version lines" above.* 1617 1618You need to carry out a few steps already described in another section of this 1619guide. Those steps will let you: 1620 1621 * Check if the kernel developers still maintain the Linux kernel version line 1622 you care about. 1623 1624 * Search the Linux stable mailing list for exiting reports. 1625 1626 * Check with the latest release. 1627 1628 1629Check code history and search for existing discussions 1630~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1631 1632 *Search the Linux kernel version control system for the change that fixed 1633 the issue in mainline, as its commit message might tell you if the fix is 1634 scheduled for backporting already. If you don't find anything that way, 1635 search the appropriate mailing lists for posts that discuss such an issue 1636 or peer-review possible fixes; then check the discussions if the fix was 1637 deemed unsuitable for backporting. If backporting was not considered at 1638 all, join the newest discussion, asking if it's in the cards.* 1639 1640In a lot of cases the issue you deal with will have happened with mainline, but 1641got fixed there. The commit that fixed it would need to get backported as well 1642to get the issue solved. That's why you want to search for it or any 1643discussions abound it. 1644 1645 * First try to find the fix in the Git repository that holds the Linux kernel 1646 sources. You can do this with the web interfaces `on kernel.org 1647 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_ 1648 or its mirror `on GitHub <https://github.com/torvalds/linux>`_; if you have 1649 a local clone you alternatively can search on the command line with ``git 1650 log --grep=<pattern>``. 1651 1652 If you find the fix, look if the commit message near the end contains a 1653 'stable tag' that looks like this: 1654 1655 Cc: <stable@vger.kernel.org> # 5.4+ 1656 1657 If that's case the developer marked the fix safe for backporting to version 1658 line 5.4 and later. Most of the time it's getting applied there within two 1659 weeks, but sometimes it takes a bit longer. 1660 1661 * If the commit doesn't tell you anything or if you can't find the fix, look 1662 again for discussions about the issue. Search the net with your favorite 1663 internet search engine as well as the archives for the `Linux kernel 1664 developers mailing list <https://lore.kernel.org/lkml/>`_. Also read the 1665 section `Locate kernel area that causes the issue` above and follow the 1666 instructions to find the subsystem in question: its bug tracker or mailing 1667 list archive might have the answer you are looking for. 1668 1669 * If you see a proposed fix, search for it in the version control system as 1670 outlined above, as the commit might tell you if a backport can be expected. 1671 1672 * Check the discussions for any indicators the fix might be too risky to get 1673 backported to the version line you care about. If that's the case you have 1674 to live with the issue or switch to the kernel version line where the fix 1675 got applied. 1676 1677 * If the fix doesn't contain a stable tag and backporting was not discussed, 1678 join the discussion: mention the version where you face the issue and that 1679 you would like to see it fixed, if suitable. 1680 1681 1682Ask for advice 1683~~~~~~~~~~~~~~ 1684 1685 *One of the former steps should lead to a solution. If that doesn't work 1686 out, ask the maintainers for the subsystem that seems to be causing the 1687 issue for advice; CC the mailing list for the particular subsystem as well 1688 as the stable mailing list.* 1689 1690If the previous three steps didn't get you closer to a solution there is only 1691one option left: ask for advice. Do that in a mail you sent to the maintainers 1692for the subsystem where the issue seems to have its roots; CC the mailing list 1693for the subsystem as well as the stable mailing list (stable@vger.kernel.org). 1694 1695 1696Appendix: Why it is somewhat hard to report kernel bugs 1697======================================================= 1698 1699The Linux kernel developers are well aware that reporting bugs to them is harder 1700than in other Free/Libre Open Source Projects. Many reasons for that lie in the 1701nature of kernels, Linux' development model, and how the world uses the kernel: 1702 1703* *Most kernels of Linux distributions are totally unsuitable for reporting bugs 1704 upstream.* The reference section above already explained this in detail: 1705 outdated codebases as well as modifications and add-ons lead to kernel bugs 1706 that were fixed upstream a long time ago or never happened there in the first 1707 place. Developers of other Open Source software face these problems as well, 1708 but the situation is a lot worse when it comes to the kernel, as the changes 1709 and their impact are much more severe -- which is why many kernel developers 1710 expect reports with kernels built from fresh and nearly unmodified sources. 1711 1712* *Bugs often only occur in a special environment.* That is because Linux is 1713 mostly drivers and can be used in a multitude of ways. Developers often do not 1714 have a matching setup at hand -- and therefore frequently must rely on bug 1715 reporters for isolating a problems's cause and testing proposed fixes. 1716 1717* *The kernel has hundreds of maintainers, but all-rounders are very rare.* That 1718 again is and effect caused by the multitude of features and drivers, due to 1719 which many kernel developers know little about lower or higher layers related 1720 to their code and even less about other areas. 1721 1722* *It is hard finding where to report issues to, among others, due to the lack 1723 of a central bug tracker.* This is something even some kernel developers 1724 dislike, but that's the situation everyone has to deal with currently. 1725 1726* *Stable and longterm kernels are primarily maintained by a dedicated 'stable 1727 team', which only handles regressions introduced within stable and longterm 1728 series.* When someone reports a bug, say, using Linux 6.1.2, the team will, 1729 therefore, always ask if mainline is affected: if the bug already happened 1730 in 6.1 or occurs with latest mainline (say, 6.2-rc3), they in everybody's 1731 interest shove it to the regular developers, as those know the code best. 1732 1733* *Linux developers are free to focus on latest mainline.* Some, thus, react 1734 coldly to reports about bugs in, say, Linux 6.0 when 6.1 is already out; 1735 even the latter might not be enough once 6.2-rc1 is out. Some will also not 1736 be very welcoming to reports with 6.1.5 or 6.1.6, as the problem might be a 1737 series-specific regression the stable team (see above) caused and must fix. 1738 1739* *Sometimes there is nobody to help.* Sometimes this is due to the lack of 1740 hardware documentation -- for example, when a driver was built using reverse 1741 engineering or was taken over by spare-time developers when the hardware 1742 manufacturer left it behind. Other times there is nobody to even report bugs 1743 to: when maintainers move on without a replacement, their code often remains 1744 in the kernel as long as it's useful. 1745 1746Some of these aspects could be improved to facilitate bug reporting -- many 1747Linux kernel developers are well aware of this and would be glad if a few 1748individuals or an entity would make this their mission. 1749 1750.. 1751 end-of-content 1752.. 1753 This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If 1754 you spot a typo or small mistake, feel free to let him know directly and 1755 he'll fix it. You are free to do the same in a mostly informal way if you 1756 want to contribute changes to the text, but for copyright reasons please CC 1757 linux-doc@vger.kernel.org and "sign-off" your contribution as 1758 Documentation/process/submitting-patches.rst outlines in the section "Sign 1759 your work - the Developer's Certificate of Origin". 1760.. 1761 This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top 1762 of the file. If you want to distribute this text under CC-BY-4.0 only, 1763 please use "The Linux kernel developers" for author attribution and link 1764 this as source: 1765 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-issues.rst 1766.. 1767 Note: Only the content of this RST file as found in the Linux kernel sources 1768 is available under CC-BY-4.0, as versions of this text that were processed 1769 (for example by the kernel's build system) might contain content taken from 1770 files which use a more restrictive license. 1771