1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) 2.. 3 If you want to distribute this text under CC-BY-4.0 only, please use 'The 4 Linux kernel developers' for author attribution and link this as source: 5 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-issues.rst 6.. 7 Note: Only the content of this RST file as found in the Linux kernel sources 8 is available under CC-BY-4.0, as versions of this text that were processed 9 (for example by the kernel's build system) might contain content taken from 10 files which use a more restrictive license. 11 12 13Reporting issues 14++++++++++++++++ 15 16 17The short guide (aka TL;DR) 18=========================== 19 20Are you facing a regression with vanilla kernels from the same stable or 21longterm series? One still supported? Then search the `LKML 22<https://lore.kernel.org/lkml/>`_ and the `Linux stable mailing list 23<https://lore.kernel.org/stable/>`_ archives for matching reports to join. If 24you don't find any, install `the latest release from that series 25<https://kernel.org/>`_. If it still shows the issue, report it to the stable 26mailing list (stable@vger.kernel.org) and CC the regressions list 27(regressions@lists.linux.dev). 28 29In all other cases try your best guess which kernel part might be causing the 30issue. Check the :ref:`MAINTAINERS <maintainers>` file for how its developers 31expect to be told about problems, which most of the time will be by email with a 32mailing list in CC. Check the destination's archives for matching reports; 33search the `LKML <https://lore.kernel.org/lkml/>`_ and the web, too. If you 34don't find any to join, install `the latest mainline kernel 35<https://kernel.org/>`_. If the issue is present there, send a report. 36 37The issue was fixed there, but you would like to see it resolved in a still 38supported stable or longterm series as well? Then install its latest release. 39If it shows the problem, search for the change that fixed it in mainline and 40check if backporting is in the works or was discarded; if it's neither, ask 41those who handled the change for it. 42 43**General remarks**: When installing and testing a kernel as outlined above, 44ensure it's vanilla (IOW: not patched and not using add-on modules). Also make 45sure it's built and running in a healthy environment and not already tainted 46before the issue occurs. 47 48If you are facing multiple issues with the Linux kernel at once, report each 49separately. While writing your report, include all information relevant to the 50issue, like the kernel and the distro used. In case of a regression, CC the 51regressions mailing list (regressions@lists.linux.dev) to your report; also try 52to include the commit-id of the change causing it, which a bisection can find. 53 54Once the report is out, answer any questions that come up and help where you 55can. That includes keeping the ball rolling by occasionally retesting with newer 56releases and sending a status update afterwards. 57 58Step-by-step guide how to report issues to the kernel maintainers 59================================================================= 60 61The above TL;DR outlines roughly how to report issues to the Linux kernel 62developers. It might be all that's needed for people already familiar with 63reporting issues to Free/Libre & Open Source Software (FLOSS) projects. For 64everyone else there is this section. It is more detailed and uses a 65step-by-step approach. It still tries to be brief for readability and leaves 66out a lot of details; those are described below the step-by-step guide in a 67reference section, which explains each of the steps in more detail. 68 69Note: this section covers a few more aspects than the TL;DR and does things in 70a slightly different order. That's in your interest, to make sure you notice 71early if an issue that looks like a Linux kernel problem is actually caused by 72something else. These steps thus help to ensure the time you invest in this 73process won't feel wasted in the end: 74 75 * Are you facing an issue with a Linux kernel a hardware or software vendor 76 provided? Then in almost all cases you are better off to stop reading this 77 document and reporting the issue to your vendor instead, unless you are 78 willing to install the latest Linux version yourself. Be aware the latter 79 will often be needed anyway to hunt down and fix issues. 80 81 * Perform a rough search for existing reports with your favorite internet 82 search engine; additionally, check the archives of the `Linux Kernel Mailing 83 List (LKML) <https://lore.kernel.org/lkml/>`_. If you find matching reports, 84 join the discussion instead of sending a new one. 85 86 * See if the issue you are dealing with qualifies as regression, security 87 issue, or a really severe problem: those are 'issues of high priority' that 88 need special handling in some steps that are about to follow. 89 90 * Make sure it's not the kernel's surroundings that are causing the issue 91 you face. 92 93 * Create a fresh backup and put system repair and restore tools at hand. 94 95 * Ensure your system does not enhance its kernels by building additional 96 kernel modules on-the-fly, which solutions like DKMS might be doing locally 97 without your knowledge. 98 99 * Check if your kernel was 'tainted' when the issue occurred, as the event 100 that made the kernel set this flag might be causing the issue you face. 101 102 * Write down coarsely how to reproduce the issue. If you deal with multiple 103 issues at once, create separate notes for each of them and make sure they 104 work independently on a freshly booted system. That's needed, as each issue 105 needs to get reported to the kernel developers separately, unless they are 106 strongly entangled. 107 108 * If you are facing a regression within a stable or longterm version line 109 (say something broke when updating from 5.10.4 to 5.10.5), scroll down to 110 'Dealing with regressions within a stable and longterm kernel line'. 111 112 * Locate the driver or kernel subsystem that seems to be causing the issue. 113 Find out how and where its developers expect reports. Note: most of the 114 time this won't be bugzilla.kernel.org, as issues typically need to be sent 115 by mail to a maintainer and a public mailing list. 116 117 * Search the archives of the bug tracker or mailing list in question 118 thoroughly for reports that might match your issue. If you find anything, 119 join the discussion instead of sending a new report. 120 121After these preparations you'll now enter the main part: 122 123 * Unless you are already running the latest 'mainline' Linux kernel, better 124 go and install it for the reporting process. Testing and reporting with 125 the latest 'stable' Linux can be an acceptable alternative in some 126 situations; during the merge window that actually might be even the best 127 approach, but in that development phase it can be an even better idea to 128 suspend your efforts for a few days anyway. Whatever version you choose, 129 ideally use a 'vanilla' build. Ignoring these advices will dramatically 130 increase the risk your report will be rejected or ignored. 131 132 * Ensure the kernel you just installed does not 'taint' itself when 133 running. 134 135 * Reproduce the issue with the kernel you just installed. If it doesn't show 136 up there, scroll down to the instructions for issues only happening with 137 stable and longterm kernels. 138 139 * Optimize your notes: try to find and write the most straightforward way to 140 reproduce your issue. Make sure the end result has all the important 141 details, and at the same time is easy to read and understand for others 142 that hear about it for the first time. And if you learned something in this 143 process, consider searching again for existing reports about the issue. 144 145 * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider 146 decoding the kernel log to find the line of code that triggered the error. 147 148 * If your problem is a regression, try to narrow down when the issue was 149 introduced as much as possible. 150 151 * Start to compile the report by writing a detailed description about the 152 issue. Always mention a few things: the latest kernel version you installed 153 for reproducing, the Linux Distribution used, and your notes on how to 154 reproduce the issue. Ideally, make the kernel's build configuration 155 (.config) and the output from ``dmesg`` available somewhere on the net and 156 link to it. Include or upload all other information that might be relevant, 157 like the output/screenshot of an Oops or the output from ``lspci``. Once 158 you wrote this main part, insert a normal length paragraph on top of it 159 outlining the issue and the impact quickly. On top of this add one sentence 160 that briefly describes the problem and gets people to read on. Now give the 161 thing a descriptive title or subject that yet again is shorter. Then you're 162 ready to send or file the report like the MAINTAINERS file told you, unless 163 you are dealing with one of those 'issues of high priority': they need 164 special care which is explained in 'Special handling for high priority 165 issues' below. 166 167 * Wait for reactions and keep the thing rolling until you can accept the 168 outcome in one way or the other. Thus react publicly and in a timely manner 169 to any inquiries. Test proposed fixes. Do proactive testing: retest with at 170 least every first release candidate (RC) of a new mainline version and 171 report your results. Send friendly reminders if things stall. And try to 172 help yourself, if you don't get any help or if it's unsatisfying. 173 174 175Reporting regressions within a stable and longterm kernel line 176-------------------------------------------------------------- 177 178This subsection is for you, if you followed above process and got sent here at 179the point about regression within a stable or longterm kernel version line. You 180face one of those if something breaks when updating from 5.10.4 to 5.10.5 (a 181switch from 5.9.15 to 5.10.5 does not qualify). The developers want to fix such 182regressions as quickly as possible, hence there is a streamlined process to 183report them: 184 185 * Check if the kernel developers still maintain the Linux kernel version 186 line you care about: go to the `front page of kernel.org 187 <https://kernel.org/>`_ and make sure it mentions 188 the latest release of the particular version line without an '[EOL]' tag. 189 190 * Check the archives of the `Linux stable mailing list 191 <https://lore.kernel.org/stable/>`_ for existing reports. 192 193 * Install the latest release from the particular version line as a vanilla 194 kernel. Ensure this kernel is not tainted and still shows the problem, as 195 the issue might have already been fixed there. If you first noticed the 196 problem with a vendor kernel, check a vanilla build of the last version 197 known to work performs fine as well. 198 199 * Send a short problem report to the Linux stable mailing list 200 (stable@vger.kernel.org) and CC the Linux regressions mailing list 201 (regressions@lists.linux.dev). Roughly describe the issue and ideally 202 explain how to reproduce it. Mention the first version that shows the 203 problem and the last version that's working fine. Then wait for further 204 instructions. 205 206The reference section below explains each of these steps in more detail. 207 208 209Reporting issues only occurring in older kernel version lines 210------------------------------------------------------------- 211 212This subsection is for you, if you tried the latest mainline kernel as outlined 213above, but failed to reproduce your issue there; at the same time you want to 214see the issue fixed in a still supported stable or longterm series or vendor 215kernels regularly rebased on those. If that the case, follow these steps: 216 217 * Prepare yourself for the possibility that going through the next few steps 218 might not get the issue solved in older releases: the fix might be too big 219 or risky to get backported there. 220 221 * Perform the first three steps in the section "Dealing with regressions 222 within a stable and longterm kernel line" above. 223 224 * Search the Linux kernel version control system for the change that fixed 225 the issue in mainline, as its commit message might tell you if the fix is 226 scheduled for backporting already. If you don't find anything that way, 227 search the appropriate mailing lists for posts that discuss such an issue 228 or peer-review possible fixes; then check the discussions if the fix was 229 deemed unsuitable for backporting. If backporting was not considered at 230 all, join the newest discussion, asking if it's in the cards. 231 232 * One of the former steps should lead to a solution. If that doesn't work 233 out, ask the maintainers for the subsystem that seems to be causing the 234 issue for advice; CC the mailing list for the particular subsystem as well 235 as the stable mailing list. 236 237The reference section below explains each of these steps in more detail. 238 239 240Reference section: Reporting issues to the kernel maintainers 241============================================================= 242 243The detailed guides above outline all the major steps in brief fashion, which 244should be enough for most people. But sometimes there are situations where even 245experienced users might wonder how to actually do one of those steps. That's 246what this section is for, as it will provide a lot more details on each of the 247above steps. Consider this as reference documentation: it's possible to read it 248from top to bottom. But it's mainly meant to skim over and a place to look up 249details how to actually perform those steps. 250 251A few words of general advice before digging into the details: 252 253 * The Linux kernel developers are well aware this process is complicated and 254 demands more than other FLOSS projects. We'd love to make it simpler. But 255 that would require work in various places as well as some infrastructure, 256 which would need constant maintenance; nobody has stepped up to do that 257 work, so that's just how things are for now. 258 259 * A warranty or support contract with some vendor doesn't entitle you to 260 request fixes from developers in the upstream Linux kernel community: such 261 contracts are completely outside the scope of the Linux kernel, its 262 development community, and this document. That's why you can't demand 263 anything such a contract guarantees in this context, not even if the 264 developer handling the issue works for the vendor in question. If you want 265 to claim your rights, use the vendor's support channel instead. When doing 266 so, you might want to mention you'd like to see the issue fixed in the 267 upstream Linux kernel; motivate them by saying it's the only way to ensure 268 the fix in the end will get incorporated in all Linux distributions. 269 270 * If you never reported an issue to a FLOSS project before you should consider 271 reading `How to Report Bugs Effectively 272 <https://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_, `How To Ask 273 Questions The Smart Way 274 <http://www.catb.org/esr/faqs/smart-questions.html>`_, and `How to ask good 275 questions <https://jvns.ca/blog/good-questions/>`_. 276 277With that off the table, find below the details on how to properly report 278issues to the Linux kernel developers. 279 280 281Make sure you're using the upstream Linux kernel 282------------------------------------------------ 283 284 *Are you facing an issue with a Linux kernel a hardware or software vendor 285 provided? Then in almost all cases you are better off to stop reading this 286 document and reporting the issue to your vendor instead, unless you are 287 willing to install the latest Linux version yourself. Be aware the latter 288 will often be needed anyway to hunt down and fix issues.* 289 290Like most programmers, Linux kernel developers don't like to spend time dealing 291with reports for issues that don't even happen with their current code. It's 292just a waste everybody's time, especially yours. Unfortunately such situations 293easily happen when it comes to the kernel and often leads to frustration on both 294sides. That's because almost all Linux-based kernels pre-installed on devices 295(Computers, Laptops, Smartphones, Routers, …) and most shipped by Linux 296distributors are quite distant from the official Linux kernel as distributed by 297kernel.org: these kernels from these vendors are often ancient from the point of 298Linux development or heavily modified, often both. 299 300Most of these vendor kernels are quite unsuitable for reporting issues to the 301Linux kernel developers: an issue you face with one of them might have been 302fixed by the Linux kernel developers months or years ago already; additionally, 303the modifications and enhancements by the vendor might be causing the issue you 304face, even if they look small or totally unrelated. That's why you should report 305issues with these kernels to the vendor. Its developers should look into the 306report and, in case it turns out to be an upstream issue, fix it directly 307upstream or forward the report there. In practice that often does not work out 308or might not what you want. You thus might want to consider circumventing the 309vendor by installing the very latest Linux kernel core yourself. If that's an 310option for you move ahead in this process, as a later step in this guide will 311explain how to do that once it rules out other potential causes for your issue. 312 313Note, the previous paragraph is starting with the word 'most', as sometimes 314developers in fact are willing to handle reports about issues occurring with 315vendor kernels. If they do in the end highly depends on the developers and the 316issue in question. Your chances are quite good if the distributor applied only 317small modifications to a kernel based on a recent Linux version; that for 318example often holds true for the mainline kernels shipped by Debian GNU/Linux 319Sid or Fedora Rawhide. Some developers will also accept reports about issues 320with kernels from distributions shipping the latest stable kernel, as long as 321its only slightly modified; that for example is often the case for Arch Linux, 322regular Fedora releases, and openSUSE Tumbleweed. But keep in mind, you better 323want to use a mainline Linux and avoid using a stable kernel for this 324process, as outlined in the section 'Install a fresh kernel for testing' in more 325detail. 326 327Obviously you are free to ignore all this advice and report problems with an old 328or heavily modified vendor kernel to the upstream Linux developers. But note, 329those often get rejected or ignored, so consider yourself warned. But it's still 330better than not reporting the issue at all: sometimes such reports directly or 331indirectly will help to get the issue fixed over time. 332 333 334Search for existing reports, first run 335-------------------------------------- 336 337 *Perform a rough search for existing reports with your favorite internet 338 search engine; additionally, check the archives of the Linux Kernel Mailing 339 List (LKML). If you find matching reports, join the discussion instead of 340 sending a new one.* 341 342Reporting an issue that someone else already brought forward is often a waste of 343time for everyone involved, especially you as the reporter. So it's in your own 344interest to thoroughly check if somebody reported the issue already. At this 345step of the process it's okay to just perform a rough search: a later step will 346tell you to perform a more detailed search once you know where your issue needs 347to be reported to. Nevertheless, do not hurry with this step of the reporting 348process, it can save you time and trouble. 349 350Simply search the internet with your favorite search engine first. Afterwards, 351search the `Linux Kernel Mailing List (LKML) archives 352<https://lore.kernel.org/lkml/>`_. 353 354If you get flooded with results consider telling your search engine to limit 355search timeframe to the past month or year. And wherever you search, make sure 356to use good search terms; vary them a few times, too. While doing so try to 357look at the issue from the perspective of someone else: that will help you to 358come up with other words to use as search terms. Also make sure not to use too 359many search terms at once. Remember to search with and without information like 360the name of the kernel driver or the name of the affected hardware component. 361But its exact brand name (say 'ASUS Red Devil Radeon RX 5700 XT Gaming OC') 362often is not much helpful, as it is too specific. Instead try search terms like 363the model line (Radeon 5700 or Radeon 5000) and the code name of the main chip 364('Navi' or 'Navi10') with and without its manufacturer ('AMD'). 365 366In case you find an existing report about your issue, join the discussion, as 367you might be able to provide valuable additional information. That can be 368important even when a fix is prepared or in its final stages already, as 369developers might look for people that can provide additional information or 370test a proposed fix. Jump to the section 'Duties after the report went out' for 371details on how to get properly involved. 372 373Note, searching `bugzilla.kernel.org <https://bugzilla.kernel.org/>`_ might also 374be a good idea, as that might provide valuable insights or turn up matching 375reports. If you find the latter, just keep in mind: most subsystems expect 376reports in different places, as described below in the section "Check where you 377need to report your issue". The developers that should take care of the issue 378thus might not even be aware of the bugzilla ticket. Hence, check the ticket if 379the issue already got reported as outlined in this document and if not consider 380doing so. 381 382 383Issue of high priority? 384----------------------- 385 386 *See if the issue you are dealing with qualifies as regression, security 387 issue, or a really severe problem: those are 'issues of high priority' that 388 need special handling in some steps that are about to follow.* 389 390Linus Torvalds and the leading Linux kernel developers want to see some issues 391fixed as soon as possible, hence there are 'issues of high priority' that get 392handled slightly differently in the reporting process. Three type of cases 393qualify: regressions, security issues, and really severe problems. 394 395You deal with a 'regression' if something that worked with an older version of 396the Linux kernel does not work with a newer one or somehow works worse with it. 397It thus is a regression when a WiFi driver that did a fine job with Linux 5.7 398somehow misbehaves with 5.8 or doesn't work at all. It's also a regression if 399an application shows erratic behavior with a newer kernel, which might happen 400due to incompatible changes in the interface between the kernel and the 401userland (like procfs and sysfs). Significantly reduced performance or 402increased power consumption also qualify as regression. But keep in mind: the 403new kernel needs to be built with a configuration that is similar to the one 404from the old kernel (see below how to achieve that). That's because the kernel 405developers sometimes can not avoid incompatibilities when implementing new 406features; but to avoid regressions such features have to be enabled explicitly 407during build time configuration. 408 409What qualifies as security issue is left to your judgment. Consider reading 410'Documentation/admin-guide/security-bugs.rst' before proceeding, as it 411provides additional details how to best handle security issues. 412 413An issue is a 'really severe problem' when something totally unacceptably bad 414happens. That's for example the case when a Linux kernel corrupts the data it's 415handling or damages hardware it's running on. You're also dealing with a severe 416issue when the kernel suddenly stops working with an error message ('kernel 417panic') or without any farewell note at all. Note: do not confuse a 'panic' (a 418fatal error where the kernel stop itself) with a 'Oops' (a recoverable error), 419as the kernel remains running after the latter. 420 421 422Ensure a healthy environment 423---------------------------- 424 425 *Make sure it's not the kernel's surroundings that are causing the issue 426 you face.* 427 428Problems that look a lot like a kernel issue are sometimes caused by build or 429runtime environment. It's hard to rule out that problem completely, but you 430should minimize it: 431 432 * Use proven tools when building your kernel, as bugs in the compiler or the 433 binutils can cause the resulting kernel to misbehave. 434 435 * Ensure your computer components run within their design specifications; 436 that's especially important for the main processor, the main memory, and the 437 motherboard. Therefore, stop undervolting or overclocking when facing a 438 potential kernel issue. 439 440 * Try to make sure it's not faulty hardware that is causing your issue. Bad 441 main memory for example can result in a multitude of issues that will 442 manifest itself in problems looking like kernel issues. 443 444 * If you're dealing with a filesystem issue, you might want to check the file 445 system in question with ``fsck``, as it might be damaged in a way that leads 446 to unexpected kernel behavior. 447 448 * When dealing with a regression, make sure it's not something else that 449 changed in parallel to updating the kernel. The problem for example might be 450 caused by other software that was updated at the same time. It can also 451 happen that a hardware component coincidentally just broke when you rebooted 452 into a new kernel for the first time. Updating the systems BIOS or changing 453 something in the BIOS Setup can also lead to problems that on look a lot 454 like a kernel regression. 455 456 457Prepare for emergencies 458----------------------- 459 460 *Create a fresh backup and put system repair and restore tools at hand.* 461 462Reminder, you are dealing with computers, which sometimes do unexpected things, 463especially if you fiddle with crucial parts like the kernel of its operating 464system. That's what you are about to do in this process. Thus, make sure to 465create a fresh backup; also ensure you have all tools at hand to repair or 466reinstall the operating system as well as everything you need to restore the 467backup. 468 469 470Make sure your kernel doesn't get enhanced 471------------------------------------------ 472 473 *Ensure your system does not enhance its kernels by building additional 474 kernel modules on-the-fly, which solutions like DKMS might be doing locally 475 without your knowledge.* 476 477The risk your issue report gets ignored or rejected dramatically increases if 478your kernel gets enhanced in any way. That's why you should remove or disable 479mechanisms like akmods and DKMS: those build add-on kernel modules 480automatically, for example when you install a new Linux kernel or boot it for 481the first time. Also remove any modules they might have installed. Then reboot 482before proceeding. 483 484Note, you might not be aware that your system is using one of these solutions: 485they often get set up silently when you install Nvidia's proprietary graphics 486driver, VirtualBox, or other software that requires a some support from a 487module not part of the Linux kernel. That why your might need to uninstall the 488packages with such software to get rid of any 3rd party kernel module. 489 490 491Check 'taint' flag 492------------------ 493 494 *Check if your kernel was 'tainted' when the issue occurred, as the event 495 that made the kernel set this flag might be causing the issue you face.* 496 497The kernel marks itself with a 'taint' flag when something happens that might 498lead to follow-up errors that look totally unrelated. The issue you face might 499be such an error if your kernel is tainted. That's why it's in your interest to 500rule this out early before investing more time into this process. This is the 501only reason why this step is here, as this process later will tell you to 502install the latest mainline kernel; you will need to check the taint flag again 503then, as that's when it matters because it's the kernel the report will focus 504on. 505 506On a running system is easy to check if the kernel tainted itself: if ``cat 507/proc/sys/kernel/tainted`` returns '0' then the kernel is not tainted and 508everything is fine. Checking that file is impossible in some situations; that's 509why the kernel also mentions the taint status when it reports an internal 510problem (a 'kernel bug'), a recoverable error (a 'kernel Oops') or a 511non-recoverable error before halting operation (a 'kernel panic'). Look near 512the top of the error messages printed when one of these occurs and search for a 513line starting with 'CPU:'. It should end with 'Not tainted' if the kernel was 514not tainted when it noticed the problem; it was tainted if you see 'Tainted:' 515followed by a few spaces and some letters. 516 517If your kernel is tainted, study 'Documentation/admin-guide/tainted-kernels.rst' 518to find out why. Try to eliminate the reason. Often it's caused by one these 519three things: 520 521 1. A recoverable error (a 'kernel Oops') occurred and the kernel tainted 522 itself, as the kernel knows it might misbehave in strange ways after that 523 point. In that case check your kernel or system log and look for a section 524 that starts with this:: 525 526 Oops: 0000 [#1] SMP 527 528 That's the first Oops since boot-up, as the '#1' between the brackets shows. 529 Every Oops and any other problem that happens after that point might be a 530 follow-up problem to that first Oops, even if both look totally unrelated. 531 Rule this out by getting rid of the cause for the first Oops and reproducing 532 the issue afterwards. Sometimes simply restarting will be enough, sometimes 533 a change to the configuration followed by a reboot can eliminate the Oops. 534 But don't invest too much time into this at this point of the process, as 535 the cause for the Oops might already be fixed in the newer Linux kernel 536 version you are going to install later in this process. 537 538 2. Your system uses a software that installs its own kernel modules, for 539 example Nvidia's proprietary graphics driver or VirtualBox. The kernel 540 taints itself when it loads such module from external sources (even if 541 they are Open Source): they sometimes cause errors in unrelated kernel 542 areas and thus might be causing the issue you face. You therefore have to 543 prevent those modules from loading when you want to report an issue to the 544 Linux kernel developers. Most of the time the easiest way to do that is: 545 temporarily uninstall such software including any modules they might have 546 installed. Afterwards reboot. 547 548 3. The kernel also taints itself when it's loading a module that resides in 549 the staging tree of the Linux kernel source. That's a special area for 550 code (mostly drivers) that does not yet fulfill the normal Linux kernel 551 quality standards. When you report an issue with such a module it's 552 obviously okay if the kernel is tainted; just make sure the module in 553 question is the only reason for the taint. If the issue happens in an 554 unrelated area reboot and temporarily block the module from being loaded 555 by specifying ``foo.blacklist=1`` as kernel parameter (replace 'foo' with 556 the name of the module in question). 557 558 559Document how to reproduce issue 560------------------------------- 561 562 *Write down coarsely how to reproduce the issue. If you deal with multiple 563 issues at once, create separate notes for each of them and make sure they 564 work independently on a freshly booted system. That's needed, as each issue 565 needs to get reported to the kernel developers separately, unless they are 566 strongly entangled.* 567 568If you deal with multiple issues at once, you'll have to report each of them 569separately, as they might be handled by different developers. Describing 570various issues in one report also makes it quite difficult for others to tear 571it apart. Hence, only combine issues in one report if they are very strongly 572entangled. 573 574Additionally, during the reporting process you will have to test if the issue 575happens with other kernel versions. Therefore, it will make your work easier if 576you know exactly how to reproduce an issue quickly on a freshly booted system. 577 578Note: it's often fruitless to report issues that only happened once, as they 579might be caused by a bit flip due to cosmic radiation. That's why you should 580try to rule that out by reproducing the issue before going further. Feel free 581to ignore this advice if you are experienced enough to tell a one-time error 582due to faulty hardware apart from a kernel issue that rarely happens and thus 583is hard to reproduce. 584 585 586Regression in stable or longterm kernel? 587---------------------------------------- 588 589 *If you are facing a regression within a stable or longterm version line 590 (say something broke when updating from 5.10.4 to 5.10.5), scroll down to 591 'Dealing with regressions within a stable and longterm kernel line'.* 592 593Regression within a stable and longterm kernel version line are something the 594Linux developers want to fix badly, as such issues are even more unwanted than 595regression in the main development branch, as they can quickly affect a lot of 596people. The developers thus want to learn about such issues as quickly as 597possible, hence there is a streamlined process to report them. Note, 598regressions with newer kernel version line (say something broke when switching 599from 5.9.15 to 5.10.5) do not qualify. 600 601 602Check where you need to report your issue 603----------------------------------------- 604 605 *Locate the driver or kernel subsystem that seems to be causing the issue. 606 Find out how and where its developers expect reports. Note: most of the 607 time this won't be bugzilla.kernel.org, as issues typically need to be sent 608 by mail to a maintainer and a public mailing list.* 609 610It's crucial to send your report to the right people, as the Linux kernel is a 611big project and most of its developers are only familiar with a small subset of 612it. Quite a few programmers for example only care for just one driver, for 613example one for a WiFi chip; its developer likely will only have small or no 614knowledge about the internals of remote or unrelated "subsystems", like the TCP 615stack, the PCIe/PCI subsystem, memory management or file systems. 616 617Problem is: the Linux kernel lacks a central bug tracker where you can simply 618file your issue and make it reach the developers that need to know about it. 619That's why you have to find the right place and way to report issues yourself. 620You can do that with the help of a script (see below), but it mainly targets 621kernel developers and experts. For everybody else the MAINTAINERS file is the 622better place. 623 624How to read the MAINTAINERS file 625~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 626To illustrate how to use the :ref:`MAINTAINERS <maintainers>` file, lets assume 627the WiFi in your Laptop suddenly misbehaves after updating the kernel. In that 628case it's likely an issue in the WiFi driver. Obviously it could also be some 629code it builds upon, but unless you suspect something like that stick to the 630driver. If it's really something else, the driver's developers will get the 631right people involved. 632 633Sadly, there is no way to check which code is driving a particular hardware 634component that is both universal and easy. 635 636In case of a problem with the WiFi driver you for example might want to look at 637the output of ``lspci -k``, as it lists devices on the PCI/PCIe bus and the 638kernel module driving it:: 639 640 [user@something ~]$ lspci -k 641 [...] 642 3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32) 643 Subsystem: Bigfoot Networks, Inc. Device 1535 644 Kernel driver in use: ath10k_pci 645 Kernel modules: ath10k_pci 646 [...] 647 648But this approach won't work if your WiFi chip is connected over USB or some 649other internal bus. In those cases you might want to check your WiFi manager or 650the output of ``ip link``. Look for the name of the problematic network 651interface, which might be something like 'wlp58s0'. This name can be used like 652this to find the module driving it:: 653 654 [user@something ~]$ realpath --relative-to=/sys/module/ /sys/class/net/wlp58s0/device/driver/module 655 ath10k_pci 656 657In case tricks like these don't bring you any further, try to search the 658internet on how to narrow down the driver or subsystem in question. And if you 659are unsure which it is: just try your best guess, somebody will help you if you 660guessed poorly. 661 662Once you know the driver or subsystem, you want to search for it in the 663MAINTAINERS file. In the case of 'ath10k_pci' you won't find anything, as the 664name is too specific. Sometimes you will need to search on the net for help; 665but before doing so, try a somewhat shorted or modified name when searching the 666MAINTAINERS file, as then you might find something like this:: 667 668 QUALCOMM ATHEROS ATH10K WIRELESS DRIVER 669 Mail: A. Some Human <shuman@example.com> 670 Mailing list: ath10k@lists.infradead.org 671 Status: Supported 672 Web-page: https://wireless.wiki.kernel.org/en/users/Drivers/ath10k 673 SCM: git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git 674 Files: drivers/net/wireless/ath/ath10k/ 675 676Note: the line description will be abbreviations, if you read the plain 677MAINTAINERS file found in the root of the Linux source tree. 'Mail:' for 678example will be 'M:', 'Mailing list:' will be 'L', and 'Status:' will be 'S:'. 679A section near the top of the file explains these and other abbreviations. 680 681First look at the line 'Status'. Ideally it should be 'Supported' or 682'Maintained'. If it states 'Obsolete' then you are using some outdated approach 683that was replaced by a newer solution you need to switch to. Sometimes the code 684only has someone who provides 'Odd Fixes' when feeling motivated. And with 685'Orphan' you are totally out of luck, as nobody takes care of the code anymore. 686That only leaves these options: arrange yourself to live with the issue, fix it 687yourself, or find a programmer somewhere willing to fix it. 688 689After checking the status, look for a line starting with 'bugs:': it will tell 690you where to find a subsystem specific bug tracker to file your issue. The 691example above does not have such a line. That is the case for most sections, as 692Linux kernel development is completely driven by mail. Very few subsystems use 693a bug tracker, and only some of those rely on bugzilla.kernel.org. 694 695In this and many other cases you thus have to look for lines starting with 696'Mail:' instead. Those mention the name and the email addresses for the 697maintainers of the particular code. Also look for a line starting with 'Mailing 698list:', which tells you the public mailing list where the code is developed. 699Your report later needs to go by mail to those addresses. Additionally, for all 700issue reports sent by email, make sure to add the Linux Kernel Mailing List 701(LKML) <linux-kernel@vger.kernel.org> to CC. Don't omit either of the mailing 702lists when sending your issue report by mail later! Maintainers are busy people 703and might leave some work for other developers on the subsystem specific list; 704and LKML is important to have one place where all issue reports can be found. 705 706 707Finding the maintainers with the help of a script 708~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 709 710For people that have the Linux sources at hand there is a second option to find 711the proper place to report: the script 'scripts/get_maintainer.pl' which tries 712to find all people to contact. It queries the MAINTAINERS file and needs to be 713called with a path to the source code in question. For drivers compiled as 714module if often can be found with a command like this:: 715 716 $ modinfo ath10k_pci | grep filename | sed 's!/lib/modules/.*/kernel/!!; s!filename:!!; s!\.ko\(\|\.xz\)!!' 717 drivers/net/wireless/ath/ath10k/ath10k_pci.ko 718 719Pass parts of this to the script:: 720 721 $ ./scripts/get_maintainer.pl -f drivers/net/wireless/ath/ath10k* 722 Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER) 723 Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS) 724 ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER) 725 linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS)) 726 netdev@vger.kernel.org (open list:NETWORKING DRIVERS) 727 linux-kernel@vger.kernel.org (open list) 728 729Don't sent your report to all of them. Send it to the maintainers, which the 730script calls "supporter:"; additionally CC the most specific mailing list for 731the code as well as the Linux Kernel Mailing List (LKML). In this case you thus 732would need to send the report to 'Some Human <shuman@example.com>' with 733'ath10k@lists.infradead.org' and 'linux-kernel@vger.kernel.org' in CC. 734 735Note: in case you cloned the Linux sources with git you might want to call 736``get_maintainer.pl`` a second time with ``--git``. The script then will look 737at the commit history to find which people recently worked on the code in 738question, as they might be able to help. But use these results with care, as it 739can easily send you in a wrong direction. That for example happens quickly in 740areas rarely changed (like old or unmaintained drivers): sometimes such code is 741modified during tree-wide cleanups by developers that do not care about the 742particular driver at all. 743 744 745Search for existing reports, second run 746--------------------------------------- 747 748 *Search the archives of the bug tracker or mailing list in question 749 thoroughly for reports that might match your issue. If you find anything, 750 join the discussion instead of sending a new report.* 751 752As mentioned earlier already: reporting an issue that someone else already 753brought forward is often a waste of time for everyone involved, especially you 754as the reporter. That's why you should search for existing report again, now 755that you know where they need to be reported to. If it's mailing list, you will 756often find its archives on `lore.kernel.org <https://lore.kernel.org/>`_. 757 758But some list are hosted in different places. That for example is the case for 759the ath10k WiFi driver used as example in the previous step. But you'll often 760find the archives for these lists easily on the net. Searching for 'archive 761ath10k@lists.infradead.org' for example will lead you to the `Info page for the 762ath10k mailing list <https://lists.infradead.org/mailman/listinfo/ath10k>`_, 763which at the top links to its 764`list archives <https://lists.infradead.org/pipermail/ath10k/>`_. Sadly this and 765quite a few other lists miss a way to search the archives. In those cases use a 766regular internet search engine and add something like 767'site:lists.infradead.org/pipermail/ath10k/' to your search terms, which limits 768the results to the archives at that URL. 769 770It's also wise to check the internet, LKML and maybe bugzilla.kernel.org again 771at this point. 772 773For details how to search and what to do if you find matching reports see 774"Search for existing reports, first run" above. 775 776Do not hurry with this step of the reporting process: spending 30 to 60 minutes 777or even more time can save you and others quite a lot of time and trouble. 778 779 780Install a fresh kernel for testing 781---------------------------------- 782 783 *Unless you are already running the latest 'mainline' Linux kernel, better 784 go and install it for the reporting process. Testing and reporting with 785 the latest 'stable' Linux can be an acceptable alternative in some 786 situations; during the merge window that actually might be even the best 787 approach, but in that development phase it can be an even better idea to 788 suspend your efforts for a few days anyway. Whatever version you choose, 789 ideally use a 'vanilla' built. Ignoring these advices will dramatically 790 increase the risk your report will be rejected or ignored.* 791 792As mentioned in the detailed explanation for the first step already: Like most 793programmers, Linux kernel developers don't like to spend time dealing with 794reports for issues that don't even happen with the current code. It's just a 795waste everybody's time, especially yours. That's why it's in everybody's 796interest that you confirm the issue still exists with the latest upstream code 797before reporting it. You are free to ignore this advice, but as outlined 798earlier: doing so dramatically increases the risk that your issue report might 799get rejected or simply ignored. 800 801In the scope of the kernel "latest upstream" normally means: 802 803 * Install a mainline kernel; the latest stable kernel can be an option, but 804 most of the time is better avoided. Longterm kernels (sometimes called 'LTS 805 kernels') are unsuitable at this point of the process. The next subsection 806 explains all of this in more detail. 807 808 * The over next subsection describes way to obtain and install such a kernel. 809 It also outlines that using a pre-compiled kernel are fine, but better are 810 vanilla, which means: it was built using Linux sources taken straight `from 811 kernel.org <https://kernel.org/>`_ and not modified or enhanced in any way. 812 813Choosing the right version for testing 814~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 815 816Head over to `kernel.org <https://kernel.org/>`_ to find out which version you 817want to use for testing. Ignore the big yellow button that says 'Latest release' 818and look a little lower at the table. At its top you'll see a line starting with 819mainline, which most of the time will point to a pre-release with a version 820number like '5.8-rc2'. If that's the case, you'll want to use this mainline 821kernel for testing, as that where all fixes have to be applied first. Do not let 822that 'rc' scare you, these 'development kernels' are pretty reliable — and you 823made a backup, as you were instructed above, didn't you? 824 825In about two out of every nine to ten weeks, mainline might point you to a 826proper release with a version number like '5.7'. If that happens, consider 827suspending the reporting process until the first pre-release of the next 828version (5.8-rc1) shows up on kernel.org. That's because the Linux development 829cycle then is in its two-week long 'merge window'. The bulk of the changes and 830all intrusive ones get merged for the next release during this time. It's a bit 831more risky to use mainline during this period. Kernel developers are also often 832quite busy then and might have no spare time to deal with issue reports. It's 833also quite possible that one of the many changes applied during the merge 834window fixes the issue you face; that's why you soon would have to retest with 835a newer kernel version anyway, as outlined below in the section 'Duties after 836the report went out'. 837 838That's why it might make sense to wait till the merge window is over. But don't 839to that if you're dealing with something that shouldn't wait. In that case 840consider obtaining the latest mainline kernel via git (see below) or use the 841latest stable version offered on kernel.org. Using that is also acceptable in 842case mainline for some reason does currently not work for you. An in general: 843using it for reproducing the issue is also better than not reporting it issue 844at all. 845 846Better avoid using the latest stable kernel outside merge windows, as all fixes 847must be applied to mainline first. That's why checking the latest mainline 848kernel is so important: any issue you want to see fixed in older version lines 849needs to be fixed in mainline first before it can get backported, which can 850take a few days or weeks. Another reason: the fix you hope for might be too 851hard or risky for backporting; reporting the issue again hence is unlikely to 852change anything. 853 854These aspects are also why longterm kernels (sometimes called "LTS kernels") 855are unsuitable for this part of the reporting process: they are to distant from 856the current code. Hence go and test mainline first and follow the process 857further: if the issue doesn't occur with mainline it will guide you how to get 858it fixed in older version lines, if that's in the cards for the fix in question. 859 860How to obtain a fresh Linux kernel 861~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 862 863**Using a pre-compiled kernel**: This is often the quickest, easiest, and safest 864way for testing — especially is you are unfamiliar with the Linux kernel. The 865problem: most of those shipped by distributors or add-on repositories are build 866from modified Linux sources. They are thus not vanilla and therefore often 867unsuitable for testing and issue reporting: the changes might cause the issue 868you face or influence it somehow. 869 870But you are in luck if you are using a popular Linux distribution: for quite a 871few of them you'll find repositories on the net that contain packages with the 872latest mainline or stable Linux built as vanilla kernel. It's totally okay to 873use these, just make sure from the repository's description they are vanilla or 874at least close to it. Additionally ensure the packages contain the latest 875versions as offered on kernel.org. The packages are likely unsuitable if they 876are older than a week, as new mainline and stable kernels typically get released 877at least once a week. 878 879Please note that you might need to build your own kernel manually later: that's 880sometimes needed for debugging or testing fixes, as described later in this 881document. Also be aware that pre-compiled kernels might lack debug symbols that 882are needed to decode messages the kernel prints when a panic, Oops, warning, or 883BUG occurs; if you plan to decode those, you might be better off compiling a 884kernel yourself (see the end of this subsection and the section titled 'Decode 885failure messages' for details). 886 887**Using git**: Developers and experienced Linux users familiar with git are 888often best served by obtaining the latest Linux kernel sources straight from the 889`official development repository on kernel.org 890<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_. 891Those are likely a bit ahead of the latest mainline pre-release. Don't worry 892about it: they are as reliable as a proper pre-release, unless the kernel's 893development cycle is currently in the middle of a merge window. But even then 894they are quite reliable. 895 896**Conventional**: People unfamiliar with git are often best served by 897downloading the sources as tarball from `kernel.org <https://kernel.org/>`_. 898 899How to actually build a kernel is not described here, as many websites explain 900the necessary steps already. If you are new to it, consider following one of 901those how-to's that suggest to use ``make localmodconfig``, as that tries to 902pick up the configuration of your current kernel and then tries to adjust it 903somewhat for your system. That does not make the resulting kernel any better, 904but quicker to compile. 905 906Note: If you are dealing with a panic, Oops, warning, or BUG from the kernel, 907please try to enable CONFIG_KALLSYMS when configuring your kernel. 908Additionally, enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the 909latter is the relevant one of those two, but can only be reached if you enable 910the former. Be aware CONFIG_DEBUG_INFO increases the storage space required to 911build a kernel by quite a bit. But that's worth it, as these options will allow 912you later to pinpoint the exact line of code that triggers your issue. The 913section 'Decode failure messages' below explains this in more detail. 914 915But keep in mind: Always keep a record of the issue encountered in case it is 916hard to reproduce. Sending an undecoded report is better than not reporting 917the issue at all. 918 919 920Check 'taint' flag 921------------------ 922 923 *Ensure the kernel you just installed does not 'taint' itself when 924 running.* 925 926As outlined above in more detail already: the kernel sets a 'taint' flag when 927something happens that can lead to follow-up errors that look totally 928unrelated. That's why you need to check if the kernel you just installed does 929not set this flag. And if it does, you in almost all the cases needs to 930eliminate the reason for it before you reporting issues that occur with it. See 931the section above for details how to do that. 932 933 934Reproduce issue with the fresh kernel 935------------------------------------- 936 937 *Reproduce the issue with the kernel you just installed. If it doesn't show 938 up there, scroll down to the instructions for issues only happening with 939 stable and longterm kernels.* 940 941Check if the issue occurs with the fresh Linux kernel version you just 942installed. If it was fixed there already, consider sticking with this version 943line and abandoning your plan to report the issue. But keep in mind that other 944users might still be plagued by it, as long as it's not fixed in either stable 945and longterm version from kernel.org (and thus vendor kernels derived from 946those). If you prefer to use one of those or just want to help their users, 947head over to the section "Details about reporting issues only occurring in 948older kernel version lines" below. 949 950 951Optimize description to reproduce issue 952--------------------------------------- 953 954 *Optimize your notes: try to find and write the most straightforward way to 955 reproduce your issue. Make sure the end result has all the important 956 details, and at the same time is easy to read and understand for others 957 that hear about it for the first time. And if you learned something in this 958 process, consider searching again for existing reports about the issue.* 959 960An unnecessarily complex report will make it hard for others to understand your 961report. Thus try to find a reproducer that's straight forward to describe and 962thus easy to understand in written form. Include all important details, but at 963the same time try to keep it as short as possible. 964 965In this in the previous steps you likely have learned a thing or two about the 966issue you face. Use this knowledge and search again for existing reports 967instead you can join. 968 969 970Decode failure messages 971----------------------- 972 973 *If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider 974 decoding the kernel log to find the line of code that triggered the error.* 975 976When the kernel detects an internal problem, it will log some information about 977the executed code. This makes it possible to pinpoint the exact line in the 978source code that triggered the issue and shows how it was called. But that only 979works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring 980your kernel. If you did so, consider to decode the information from the 981kernel's log. That will make it a lot easier to understand what lead to the 982'panic', 'Oops', 'warning', or 'BUG', which increases the chances that someone 983can provide a fix. 984 985Decoding can be done with a script you find in the Linux source tree. If you 986are running a kernel you compiled yourself earlier, call it like this:: 987 988 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux 989 990If you are running a packaged vanilla kernel, you will likely have to install 991the corresponding packages with debug symbols. Then call the script (which you 992might need to get from the Linux sources if your distro does not package it) 993like this:: 994 995 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh \ 996 /usr/lib/debug/lib/modules/5.10.10-4.1.x86_64/vmlinux /usr/src/kernels/5.10.10-4.1.x86_64/ 997 998The script will work on log lines like the following, which show the address of 999the code the kernel was executing when the error occurred:: 1000 1001 [ 68.387301] RIP: 0010:test_module_init+0x5/0xffa [test_module] 1002 1003Once decoded, these lines will look like this:: 1004 1005 [ 68.387301] RIP: 0010:test_module_init (/home/username/linux-5.10.5/test-module/test-module.c:16) test_module 1006 1007In this case the executed code was built from the file 1008'~/linux-5.10.5/test-module/test-module.c' and the error occurred by the 1009instructions found in line '16'. 1010 1011The script will similarly decode the addresses mentioned in the section 1012starting with 'Call trace', which show the path to the function where the 1013problem occurred. Additionally, the script will show the assembler output for 1014the code section the kernel was executing. 1015 1016Note, if you can't get this to work, simply skip this step and mention the 1017reason for it in the report. If you're lucky, it might not be needed. And if it 1018is, someone might help you to get things going. Also be aware this is just one 1019of several ways to decode kernel stack traces. Sometimes different steps will 1020be required to retrieve the relevant details. Don't worry about that, if that's 1021needed in your case, developers will tell you what to do. 1022 1023 1024Special care for regressions 1025---------------------------- 1026 1027 *If your problem is a regression, try to narrow down when the issue was 1028 introduced as much as possible.* 1029 1030Linux lead developer Linus Torvalds insists that the Linux kernel never 1031worsens, that's why he deems regressions as unacceptable and wants to see them 1032fixed quickly. That's why changes that introduced a regression are often 1033promptly reverted if the issue they cause can't get solved quickly any other 1034way. Reporting a regression is thus a bit like playing a kind of trump card to 1035get something quickly fixed. But for that to happen the change that's causing 1036the regression needs to be known. Normally it's up to the reporter to track 1037down the culprit, as maintainers often won't have the time or setup at hand to 1038reproduce it themselves. 1039 1040To find the change there is a process called 'bisection' which the document 1041'Documentation/admin-guide/bug-bisect.rst' describes in detail. That process 1042will often require you to build about ten to twenty kernel images, trying to 1043reproduce the issue with each of them before building the next. Yes, that takes 1044some time, but don't worry, it works a lot quicker than most people assume. 1045Thanks to a 'binary search' this will lead you to the one commit in the source 1046code management system that's causing the regression. Once you find it, search 1047the net for the subject of the change, its commit id and the shortened commit id 1048(the first 12 characters of the commit id). This will lead you to existing 1049reports about it, if there are any. 1050 1051Note, a bisection needs a bit of know-how, which not everyone has, and quite a 1052bit of effort, which not everyone is willing to invest. Nevertheless, it's 1053highly recommended performing a bisection yourself. If you really can't or 1054don't want to go down that route at least find out which mainline kernel 1055introduced the regression. If something for example breaks when switching from 10565.5.15 to 5.8.4, then try at least all the mainline releases in that area (5.6, 10575.7 and 5.8) to check when it first showed up. Unless you're trying to find a 1058regression in a stable or longterm kernel, avoid testing versions which number 1059has three sections (5.6.12, 5.7.8), as that makes the outcome hard to 1060interpret, which might render your testing useless. Once you found the major 1061version which introduced the regression, feel free to move on in the reporting 1062process. But keep in mind: it depends on the issue at hand if the developers 1063will be able to help without knowing the culprit. Sometimes they might 1064recognize from the report want went wrong and can fix it; other times they will 1065be unable to help unless you perform a bisection. 1066 1067When dealing with regressions make sure the issue you face is really caused by 1068the kernel and not by something else, as outlined above already. 1069 1070In the whole process keep in mind: an issue only qualifies as regression if the 1071older and the newer kernel got built with a similar configuration. The best way 1072to archive this: copy the configuration file (``.config``) from the old working 1073kernel freshly to each newer kernel version you try. Afterwards run ``make 1074olddefconfig`` to adjust it for the needs of the new version. 1075 1076 1077Write and send the report 1078------------------------- 1079 1080 *Start to compile the report by writing a detailed description about the 1081 issue. Always mention a few things: the latest kernel version you installed 1082 for reproducing, the Linux Distribution used, and your notes on how to 1083 reproduce the issue. Ideally, make the kernel's build configuration 1084 (.config) and the output from ``dmesg`` available somewhere on the net and 1085 link to it. Include or upload all other information that might be relevant, 1086 like the output/screenshot of an Oops or the output from ``lspci``. Once 1087 you wrote this main part, insert a normal length paragraph on top of it 1088 outlining the issue and the impact quickly. On top of this add one sentence 1089 that briefly describes the problem and gets people to read on. Now give the 1090 thing a descriptive title or subject that yet again is shorter. Then you're 1091 ready to send or file the report like the MAINTAINERS file told you, unless 1092 you are dealing with one of those 'issues of high priority': they need 1093 special care which is explained in 'Special handling for high priority 1094 issues' below.* 1095 1096Now that you have prepared everything it's time to write your report. How to do 1097that is partly explained by the three documents linked to in the preface above. 1098That's why this text will only mention a few of the essentials as well as 1099things specific to the Linux kernel. 1100 1101There is one thing that fits both categories: the most crucial parts of your 1102report are the title/subject, the first sentence, and the first paragraph. 1103Developers often get quite a lot of mail. They thus often just take a few 1104seconds to skim a mail before deciding to move on or look closer. Thus: the 1105better the top section of your report, the higher are the chances that someone 1106will look into it and help you. And that is why you should ignore them for now 1107and write the detailed report first. ;-) 1108 1109Things each report should mention 1110~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1111 1112Describe in detail how your issue happens with the fresh vanilla kernel you 1113installed. Try to include the step-by-step instructions you wrote and optimized 1114earlier that outline how you and ideally others can reproduce the issue; in 1115those rare cases where that's impossible try to describe what you did to 1116trigger it. 1117 1118Also include all the relevant information others might need to understand the 1119issue and its environment. What's actually needed depends a lot on the issue, 1120but there are some things you should include always: 1121 1122 * the output from ``cat /proc/version``, which contains the Linux kernel 1123 version number and the compiler it was built with. 1124 1125 * the Linux distribution the machine is running (``hostnamectl | grep 1126 "Operating System"``) 1127 1128 * the architecture of the CPU and the operating system (``uname -mi``) 1129 1130 * if you are dealing with a regression and performed a bisection, mention the 1131 subject and the commit-id of the change that is causing it. 1132 1133In a lot of cases it's also wise to make two more things available to those 1134that read your report: 1135 1136 * the configuration used for building your Linux kernel (the '.config' file) 1137 1138 * the kernel's messages that you get from ``dmesg`` written to a file. Make 1139 sure that it starts with a line like 'Linux version 5.8-1 1140 (foobar@example.com) (gcc (GCC) 10.2.1, GNU ld version 2.34) #1 SMP Mon Aug 1141 3 14:54:37 UTC 2020' If it's missing, then important messages from the first 1142 boot phase already got discarded. In this case instead consider using 1143 ``journalctl -b 0 -k``; alternatively you can also reboot, reproduce the 1144 issue and call ``dmesg`` right afterwards. 1145 1146These two files are big, that's why it's a bad idea to put them directly into 1147your report. If you are filing the issue in a bug tracker then attach them to 1148the ticket. If you report the issue by mail do not attach them, as that makes 1149the mail too large; instead do one of these things: 1150 1151 * Upload the files somewhere public (your website, a public file paste 1152 service, a ticket created just for this purpose on `bugzilla.kernel.org 1153 <https://bugzilla.kernel.org/>`_, ...) and include a link to them in your 1154 report. Ideally use something where the files stay available for years, as 1155 they could be useful to someone many years from now; this for example can 1156 happen if five or ten years from now a developer works on some code that was 1157 changed just to fix your issue. 1158 1159 * Put the files aside and mention you will send them later in individual 1160 replies to your own mail. Just remember to actually do that once the report 1161 went out. ;-) 1162 1163Things that might be wise to provide 1164~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1165 1166Depending on the issue you might need to add more background data. Here are a 1167few suggestions what often is good to provide: 1168 1169 * If you are dealing with a 'warning', an 'OOPS' or a 'panic' from the kernel, 1170 include it. If you can't copy'n'paste it, try to capture a netconsole trace 1171 or at least take a picture of the screen. 1172 1173 * If the issue might be related to your computer hardware, mention what kind 1174 of system you use. If you for example have problems with your graphics card, 1175 mention its manufacturer, the card's model, and what chip is uses. If it's a 1176 laptop mention its name, but try to make sure it's meaningful. 'Dell XPS 13' 1177 for example is not, because it might be the one from 2012; that one looks 1178 not that different from the one sold today, but apart from that the two have 1179 nothing in common. Hence, in such cases add the exact model number, which 1180 for example are '9380' or '7390' for XPS 13 models introduced during 2019. 1181 Names like 'Lenovo Thinkpad T590' are also somewhat ambiguous: there are 1182 variants of this laptop with and without a dedicated graphics chip, so try 1183 to find the exact model name or specify the main components. 1184 1185 * Mention the relevant software in use. If you have problems with loading 1186 modules, you want to mention the versions of kmod, systemd, and udev in use. 1187 If one of the DRM drivers misbehaves, you want to state the versions of 1188 libdrm and Mesa; also specify your Wayland compositor or the X-Server and 1189 its driver. If you have a filesystem issue, mention the version of 1190 corresponding filesystem utilities (e2fsprogs, btrfs-progs, xfsprogs, ...). 1191 1192 * Gather additional information from the kernel that might be of interest. The 1193 output from ``lspci -nn`` will for example help others to identify what 1194 hardware you use. If you have a problem with hardware you even might want to 1195 make the output from ``sudo lspci -vvv`` available, as that provides 1196 insights how the components were configured. For some issues it might be 1197 good to include the contents of files like ``/proc/cpuinfo``, 1198 ``/proc/ioports``, ``/proc/iomem``, ``/proc/modules``, or 1199 ``/proc/scsi/scsi``. Some subsystem also offer tools to collect relevant 1200 information. One such tool is ``alsa-info.sh`` `which the audio/sound 1201 subsystem developers provide <https://www.alsa-project.org/wiki/AlsaInfo>`_. 1202 1203Those examples should give your some ideas of what data might be wise to 1204attach, but you have to think yourself what will be helpful for others to know. 1205Don't worry too much about forgetting something, as developers will ask for 1206additional details they need. But making everything important available from 1207the start increases the chance someone will take a closer look. 1208 1209 1210The important part: the head of your report 1211~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1212 1213Now that you have the detailed part of the report prepared let's get to the 1214most important section: the first few sentences. Thus go to the top, add 1215something like 'The detailed description:' before the part you just wrote and 1216insert two newlines at the top. Now write one normal length paragraph that 1217describes the issue roughly. Leave out all boring details and focus on the 1218crucial parts readers need to know to understand what this is all about; if you 1219think this bug affects a lot of users, mention this to get people interested. 1220 1221Once you did that insert two more lines at the top and write a one sentence 1222summary that explains quickly what the report is about. After that you have to 1223get even more abstract and write an even shorter subject/title for the report. 1224 1225Now that you have written this part take some time to optimize it, as it is the 1226most important parts of your report: a lot of people will only read this before 1227they decide if reading the rest is time well spent. 1228 1229Now send or file the report like the :ref:`MAINTAINERS <maintainers>` file told 1230you, unless it's one of those 'issues of high priority' outlined earlier: in 1231that case please read the next subsection first before sending the report on 1232its way. 1233 1234Special handling for high priority issues 1235~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1236 1237Reports for high priority issues need special handling. 1238 1239**Severe issues**: make sure the subject or ticket title as well as the first 1240paragraph makes the severeness obvious. 1241 1242**Regressions**: make the report's subject start with '[REGRESSION]'. 1243 1244In case you performed a successful bisection, use the title of the change that 1245introduced the regression as the second part of your subject. Make the report 1246also mention the commit id of the culprit. In case of an unsuccessful bisection, 1247make your report mention the latest tested version that's working fine (say 5.7) 1248and the oldest where the issue occurs (say 5.8-rc1). 1249 1250When sending the report by mail, CC the Linux regressions mailing list 1251(regressions@lists.linux.dev). In case the report needs to be filed to some web 1252tracker, proceed to do so; once filed, forward the report by mail to the 1253regressions list. Make sure to inline the forwarded report, hence do not attach 1254it. Also add a short note at the top where you mention the URL to the ticket. 1255 1256When mailing or forwarding the report, in case of a successful bisection add the 1257author of the culprit to the recipients; also CC everyone in the signed-off-by 1258chain, which you find at the end of its commit message. 1259 1260**Security issues**: for these issues your will have to evaluate if a 1261short-term risk to other users would arise if details were publicly disclosed. 1262If that's not the case simply proceed with reporting the issue as described. 1263For issues that bear such a risk you will need to adjust the reporting process 1264slightly: 1265 1266 * If the MAINTAINERS file instructed you to report the issue by mail, do not 1267 CC any public mailing lists. 1268 1269 * If you were supposed to file the issue in a bug tracker make sure to mark 1270 the ticket as 'private' or 'security issue'. If the bug tracker does not 1271 offer a way to keep reports private, forget about it and send your report as 1272 a private mail to the maintainers instead. 1273 1274In both cases make sure to also mail your report to the addresses the 1275MAINTAINERS file lists in the section 'security contact'. Ideally directly CC 1276them when sending the report by mail. If you filed it in a bug tracker, forward 1277the report's text to these addresses; but on top of it put a small note where 1278you mention that you filed it with a link to the ticket. 1279 1280See 'Documentation/admin-guide/security-bugs.rst' for more information. 1281 1282 1283Duties after the report went out 1284-------------------------------- 1285 1286 *Wait for reactions and keep the thing rolling until you can accept the 1287 outcome in one way or the other. Thus react publicly and in a timely manner 1288 to any inquiries. Test proposed fixes. Do proactive testing: retest with at 1289 least every first release candidate (RC) of a new mainline version and 1290 report your results. Send friendly reminders if things stall. And try to 1291 help yourself, if you don't get any help or if it's unsatisfying.* 1292 1293If your report was good and you are really lucky then one of the developers 1294might immediately spot what's causing the issue; they then might write a patch 1295to fix it, test it, and send it straight for integration in mainline while 1296tagging it for later backport to stable and longterm kernels that need it. Then 1297all you need to do is reply with a 'Thank you very much' and switch to a version 1298with the fix once it gets released. 1299 1300But this ideal scenario rarely happens. That's why the job is only starting 1301once you got the report out. What you'll have to do depends on the situations, 1302but often it will be the things listed below. But before digging into the 1303details, here are a few important things you need to keep in mind for this part 1304of the process. 1305 1306 1307General advice for further interactions 1308~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1309 1310**Always reply in public**: When you filed the issue in a bug tracker, always 1311reply there and do not contact any of the developers privately about it. For 1312mailed reports always use the 'Reply-all' function when replying to any mails 1313you receive. That includes mails with any additional data you might want to add 1314to your report: go to your mail applications 'Sent' folder and use 'reply-all' 1315on your mail with the report. This approach will make sure the public mailing 1316list(s) and everyone else that gets involved over time stays in the loop; it 1317also keeps the mail thread intact, which among others is really important for 1318mailing lists to group all related mails together. 1319 1320There are just two situations where a comment in a bug tracker or a 'Reply-all' 1321is unsuitable: 1322 1323 * Someone tells you to send something privately. 1324 1325 * You were told to send something, but noticed it contains sensitive 1326 information that needs to be kept private. In that case it's okay to send it 1327 in private to the developer that asked for it. But note in the ticket or a 1328 mail that you did that, so everyone else knows you honored the request. 1329 1330**Do research before asking for clarifications or help**: In this part of the 1331process someone might tell you to do something that requires a skill you might 1332not have mastered yet. For example, you might be asked to use some test tools 1333you never have heard of yet; or you might be asked to apply a patch to the 1334Linux kernel sources to test if it helps. In some cases it will be fine sending 1335a reply asking for instructions how to do that. But before going that route try 1336to find the answer own your own by searching the internet; alternatively 1337consider asking in other places for advice. For example ask a friend or post 1338about it to a chatroom or forum you normally hang out. 1339 1340**Be patient**: If you are really lucky you might get a reply to your report 1341within a few hours. But most of the time it will take longer, as maintainers 1342are scattered around the globe and thus might be in a different time zone – one 1343where they already enjoy their night away from keyboard. 1344 1345In general, kernel developers will take one to five business days to respond to 1346reports. Sometimes it will take longer, as they might be busy with the merge 1347windows, other work, visiting developer conferences, or simply enjoying a long 1348summer holiday. 1349 1350The 'issues of high priority' (see above for an explanation) are an exception 1351here: maintainers should address them as soon as possible; that's why you 1352should wait a week at maximum (or just two days if it's something urgent) 1353before sending a friendly reminder. 1354 1355Sometimes the maintainer might not be responding in a timely manner; other 1356times there might be disagreements, for example if an issue qualifies as 1357regression or not. In such cases raise your concerns on the mailing list and 1358ask others for public or private replies how to move on. If that fails, it 1359might be appropriate to get a higher authority involved. In case of a WiFi 1360driver that would be the wireless maintainers; if there are no higher level 1361maintainers or all else fails, it might be one of those rare situations where 1362it's okay to get Linus Torvalds involved. 1363 1364**Proactive testing**: Every time the first pre-release (the 'rc1') of a new 1365mainline kernel version gets released, go and check if the issue is fixed there 1366or if anything of importance changed. Mention the outcome in the ticket or in a 1367mail you sent as reply to your report (make sure it has all those in the CC 1368that up to that point participated in the discussion). This will show your 1369commitment and that you are willing to help. It also tells developers if the 1370issue persists and makes sure they do not forget about it. A few other 1371occasional retests (for example with rc3, rc5 and the final) are also a good 1372idea, but only report your results if something relevant changed or if you are 1373writing something anyway. 1374 1375With all these general things off the table let's get into the details of how 1376to help to get issues resolved once they were reported. 1377 1378Inquires and testing request 1379~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1380 1381Here are your duties in case you got replies to your report: 1382 1383**Check who you deal with**: Most of the time it will be the maintainer or a 1384developer of the particular code area that will respond to your report. But as 1385issues are normally reported in public it could be anyone that's replying — 1386including people that want to help, but in the end might guide you totally off 1387track with their questions or requests. That rarely happens, but it's one of 1388many reasons why it's wise to quickly run an internet search to see who you're 1389interacting with. By doing this you also get aware if your report was heard by 1390the right people, as a reminder to the maintainer (see below) might be in order 1391later if discussion fades out without leading to a satisfying solution for the 1392issue. 1393 1394**Inquiries for data**: Often you will be asked to test something or provide 1395additional details. Try to provide the requested information soon, as you have 1396the attention of someone that might help and risk losing it the longer you 1397wait; that outcome is even likely if you do not provide the information within 1398a few business days. 1399 1400**Requests for testing**: When you are asked to test a diagnostic patch or a 1401possible fix, try to test it in timely manner, too. But do it properly and make 1402sure to not rush it: mixing things up can happen easily and can lead to a lot 1403of confusion for everyone involved. A common mistake for example is thinking a 1404proposed patch with a fix was applied, but in fact wasn't. Things like that 1405happen even to experienced testers occasionally, but they most of the time will 1406notice when the kernel with the fix behaves just as one without it. 1407 1408What to do when nothing of substance happens 1409~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1410 1411Some reports will not get any reaction from the responsible Linux kernel 1412developers; or a discussion around the issue evolved, but faded out with 1413nothing of substance coming out of it. 1414 1415In these cases wait two (better: three) weeks before sending a friendly 1416reminder: maybe the maintainer was just away from keyboard for a while when 1417your report arrived or had something more important to take care of. When 1418writing the reminder, kindly ask if anything else from your side is needed to 1419get the ball running somehow. If the report got out by mail, do that in the 1420first lines of a mail that is a reply to your initial mail (see above) which 1421includes a full quote of the original report below: that's on of those few 1422situations where such a 'TOFU' (Text Over, Fullquote Under) is the right 1423approach, as then all the recipients will have the details at hand immediately 1424in the proper order. 1425 1426After the reminder wait three more weeks for replies. If you still don't get a 1427proper reaction, you first should reconsider your approach. Did you maybe try 1428to reach out to the wrong people? Was the report maybe offensive or so 1429confusing that people decided to completely stay away from it? The best way to 1430rule out such factors: show the report to one or two people familiar with FLOSS 1431issue reporting and ask for their opinion. Also ask them for their advice how 1432to move forward. That might mean: prepare a better report and make those people 1433review it before you send it out. Such an approach is totally fine; just 1434mention that this is the second and improved report on the issue and include a 1435link to the first report. 1436 1437If the report was proper you can send a second reminder; in it ask for advice 1438why the report did not get any replies. A good moment for this second reminder 1439mail is shortly after the first pre-release (the 'rc1') of a new Linux kernel 1440version got published, as you should retest and provide a status update at that 1441point anyway (see above). 1442 1443If the second reminder again results in no reaction within a week, try to 1444contact a higher-level maintainer asking for advice: even busy maintainers by 1445then should at least have sent some kind of acknowledgment. 1446 1447Remember to prepare yourself for a disappointment: maintainers ideally should 1448react somehow to every issue report, but they are only obliged to fix those 1449'issues of high priority' outlined earlier. So don't be too devastating if you 1450get a reply along the lines of 'thanks for the report, I have more important 1451issues to deal with currently and won't have time to look into this for the 1452foreseeable future'. 1453 1454It's also possible that after some discussion in the bug tracker or on a list 1455nothing happens anymore and reminders don't help to motivate anyone to work out 1456a fix. Such situations can be devastating, but is within the cards when it 1457comes to Linux kernel development. This and several other reasons for not 1458getting help are explained in 'Why some issues won't get any reaction or remain 1459unfixed after being reported' near the end of this document. 1460 1461Don't get devastated if you don't find any help or if the issue in the end does 1462not get solved: the Linux kernel is FLOSS and thus you can still help yourself. 1463You for example could try to find others that are affected and team up with 1464them to get the issue resolved. Such a team could prepare a fresh report 1465together that mentions how many you are and why this is something that in your 1466option should get fixed. Maybe together you can also narrow down the root cause 1467or the change that introduced a regression, which often makes developing a fix 1468easier. And with a bit of luck there might be someone in the team that knows a 1469bit about programming and might be able to write a fix. 1470 1471 1472Reference for "Reporting regressions within a stable and longterm kernel line" 1473------------------------------------------------------------------------------ 1474 1475This subsection provides details for the steps you need to perform if you face 1476a regression within a stable and longterm kernel line. 1477 1478Make sure the particular version line still gets support 1479~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1480 1481 *Check if the kernel developers still maintain the Linux kernel version 1482 line you care about: go to the front page of kernel.org and make sure it 1483 mentions the latest release of the particular version line without an 1484 '[EOL]' tag.* 1485 1486Most kernel version lines only get supported for about three months, as 1487maintaining them longer is quite a lot of work. Hence, only one per year is 1488chosen and gets supported for at least two years (often six). That's why you 1489need to check if the kernel developers still support the version line you care 1490for. 1491 1492Note, if kernel.org lists two stable version lines on the front page, you 1493should consider switching to the newer one and forget about the older one: 1494support for it is likely to be abandoned soon. Then it will get a "end-of-life" 1495(EOL) stamp. Version lines that reached that point still get mentioned on the 1496kernel.org front page for a week or two, but are unsuitable for testing and 1497reporting. 1498 1499Search stable mailing list 1500~~~~~~~~~~~~~~~~~~~~~~~~~~ 1501 1502 *Check the archives of the Linux stable mailing list for existing reports.* 1503 1504Maybe the issue you face is already known and was fixed or is about to. Hence, 1505`search the archives of the Linux stable mailing list 1506<https://lore.kernel.org/stable/>`_ for reports about an issue like yours. If 1507you find any matches, consider joining the discussion, unless the fix is 1508already finished and scheduled to get applied soon. 1509 1510Reproduce issue with the newest release 1511~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1512 1513 *Install the latest release from the particular version line as a vanilla 1514 kernel. Ensure this kernel is not tainted and still shows the problem, as 1515 the issue might have already been fixed there. If you first noticed the 1516 problem with a vendor kernel, check a vanilla build of the last version 1517 known to work performs fine as well.* 1518 1519Before investing any more time in this process you want to check if the issue 1520was already fixed in the latest release of version line you're interested in. 1521This kernel needs to be vanilla and shouldn't be tainted before the issue 1522happens, as detailed outlined already above in the section "Install a fresh 1523kernel for testing". 1524 1525Did you first notice the regression with a vendor kernel? Then changes the 1526vendor applied might be interfering. You need to rule that out by performing 1527a recheck. Say something broke when you updated from 5.10.4-vendor.42 to 15285.10.5-vendor.43. Then after testing the latest 5.10 release as outlined in 1529the previous paragraph check if a vanilla build of Linux 5.10.4 works fine as 1530well. If things are broken there, the issue does not qualify as upstream 1531regression and you need switch back to the main step-by-step guide to report 1532the issue. 1533 1534Report the regression 1535~~~~~~~~~~~~~~~~~~~~~ 1536 1537 *Send a short problem report to the Linux stable mailing list 1538 (stable@vger.kernel.org) and CC the Linux regressions mailing list 1539 (regressions@lists.linux.dev). Roughly describe the issue and ideally 1540 explain how to reproduce it. Mention the first version that shows the 1541 problem and the last version that's working fine. Then wait for further 1542 instructions.* 1543 1544When reporting a regression that happens within a stable or longterm kernel 1545line (say when updating from 5.10.4 to 5.10.5) a brief report is enough for 1546the start to get the issue reported quickly. Hence a rough description is all 1547it takes. 1548 1549But note, it helps developers a great deal if you can specify the exact version 1550that introduced the problem. Hence if possible within a reasonable time frame, 1551try to find that version using vanilla kernels. Lets assume something broke when 1552your distributor released a update from Linux kernel 5.10.5 to 5.10.8. Then as 1553instructed above go and check the latest kernel from that version line, say 15545.10.9. If it shows the problem, try a vanilla 5.10.5 to ensure that no patches 1555the distributor applied interfere. If the issue doesn't manifest itself there, 1556try 5.10.7 and then (depending on the outcome) 5.10.8 or 5.10.6 to find the 1557first version where things broke. Mention it in the report and state that 5.10.9 1558is still broken. 1559 1560What the previous paragraph outlines is basically a rough manual 'bisection'. 1561Once your report is out your might get asked to do a proper one, as it allows to 1562pinpoint the exact change that causes the issue (which then can easily get 1563reverted to fix the issue quickly). Hence consider to do a proper bisection 1564right away if time permits. See the section 'Special care for regressions' and 1565the document 'Documentation/admin-guide/bug-bisect.rst' for details how to 1566perform one. 1567 1568 1569Reference for "Reporting issues only occurring in older kernel version lines" 1570----------------------------------------------------------------------------- 1571 1572This section provides details for the steps you need to take if you could not 1573reproduce your issue with a mainline kernel, but want to see it fixed in older 1574version lines (aka stable and longterm kernels). 1575 1576Some fixes are too complex 1577~~~~~~~~~~~~~~~~~~~~~~~~~~ 1578 1579 *Prepare yourself for the possibility that going through the next few steps 1580 might not get the issue solved in older releases: the fix might be too big 1581 or risky to get backported there.* 1582 1583Even small and seemingly obvious code-changes sometimes introduce new and 1584totally unexpected problems. The maintainers of the stable and longterm kernels 1585are very aware of that and thus only apply changes to these kernels that are 1586within rules outlined in 'Documentation/process/stable-kernel-rules.rst'. 1587 1588Complex or risky changes for example do not qualify and thus only get applied 1589to mainline. Other fixes are easy to get backported to the newest stable and 1590longterm kernels, but too risky to integrate into older ones. So be aware the 1591fix you are hoping for might be one of those that won't be backported to the 1592version line your care about. In that case you'll have no other choice then to 1593live with the issue or switch to a newer Linux version, unless you want to 1594patch the fix into your kernels yourself. 1595 1596Common preparations 1597~~~~~~~~~~~~~~~~~~~ 1598 1599 *Perform the first three steps in the section "Reporting issues only 1600 occurring in older kernel version lines" above.* 1601 1602You need to carry out a few steps already described in another section of this 1603guide. Those steps will let you: 1604 1605 * Check if the kernel developers still maintain the Linux kernel version line 1606 you care about. 1607 1608 * Search the Linux stable mailing list for exiting reports. 1609 1610 * Check with the latest release. 1611 1612 1613Check code history and search for existing discussions 1614~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1615 1616 *Search the Linux kernel version control system for the change that fixed 1617 the issue in mainline, as its commit message might tell you if the fix is 1618 scheduled for backporting already. If you don't find anything that way, 1619 search the appropriate mailing lists for posts that discuss such an issue 1620 or peer-review possible fixes; then check the discussions if the fix was 1621 deemed unsuitable for backporting. If backporting was not considered at 1622 all, join the newest discussion, asking if it's in the cards.* 1623 1624In a lot of cases the issue you deal with will have happened with mainline, but 1625got fixed there. The commit that fixed it would need to get backported as well 1626to get the issue solved. That's why you want to search for it or any 1627discussions abound it. 1628 1629 * First try to find the fix in the Git repository that holds the Linux kernel 1630 sources. You can do this with the web interfaces `on kernel.org 1631 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_ 1632 or its mirror `on GitHub <https://github.com/torvalds/linux>`_; if you have 1633 a local clone you alternatively can search on the command line with ``git 1634 log --grep=<pattern>``. 1635 1636 If you find the fix, look if the commit message near the end contains a 1637 'stable tag' that looks like this: 1638 1639 Cc: <stable@vger.kernel.org> # 5.4+ 1640 1641 If that's case the developer marked the fix safe for backporting to version 1642 line 5.4 and later. Most of the time it's getting applied there within two 1643 weeks, but sometimes it takes a bit longer. 1644 1645 * If the commit doesn't tell you anything or if you can't find the fix, look 1646 again for discussions about the issue. Search the net with your favorite 1647 internet search engine as well as the archives for the `Linux kernel 1648 developers mailing list <https://lore.kernel.org/lkml/>`_. Also read the 1649 section `Locate kernel area that causes the issue` above and follow the 1650 instructions to find the subsystem in question: its bug tracker or mailing 1651 list archive might have the answer you are looking for. 1652 1653 * If you see a proposed fix, search for it in the version control system as 1654 outlined above, as the commit might tell you if a backport can be expected. 1655 1656 * Check the discussions for any indicators the fix might be too risky to get 1657 backported to the version line you care about. If that's the case you have 1658 to live with the issue or switch to the kernel version line where the fix 1659 got applied. 1660 1661 * If the fix doesn't contain a stable tag and backporting was not discussed, 1662 join the discussion: mention the version where you face the issue and that 1663 you would like to see it fixed, if suitable. 1664 1665 1666Ask for advice 1667~~~~~~~~~~~~~~ 1668 1669 *One of the former steps should lead to a solution. If that doesn't work 1670 out, ask the maintainers for the subsystem that seems to be causing the 1671 issue for advice; CC the mailing list for the particular subsystem as well 1672 as the stable mailing list.* 1673 1674If the previous three steps didn't get you closer to a solution there is only 1675one option left: ask for advice. Do that in a mail you sent to the maintainers 1676for the subsystem where the issue seems to have its roots; CC the mailing list 1677for the subsystem as well as the stable mailing list (stable@vger.kernel.org). 1678 1679 1680Why some issues won't get any reaction or remain unfixed after being reported 1681============================================================================= 1682 1683When reporting a problem to the Linux developers, be aware only 'issues of high 1684priority' (regressions, security issues, severe problems) are definitely going 1685to get resolved. The maintainers or if all else fails Linus Torvalds himself 1686will make sure of that. They and the other kernel developers will fix a lot of 1687other issues as well. But be aware that sometimes they can't or won't help; and 1688sometimes there isn't even anyone to send a report to. 1689 1690This is best explained with kernel developers that contribute to the Linux 1691kernel in their spare time. Quite a few of the drivers in the kernel were 1692written by such programmers, often because they simply wanted to make their 1693hardware usable on their favorite operating system. 1694 1695These programmers most of the time will happily fix problems other people 1696report. But nobody can force them to do, as they are contributing voluntarily. 1697 1698Then there are situations where such developers really want to fix an issue, 1699but can't: sometimes they lack hardware programming documentation to do so. 1700This often happens when the publicly available docs are superficial or the 1701driver was written with the help of reverse engineering. 1702 1703Sooner or later spare time developers will also stop caring for the driver. 1704Maybe their test hardware broke, got replaced by something more fancy, or is so 1705old that it's something you don't find much outside of computer museums 1706anymore. Sometimes developer stops caring for their code and Linux at all, as 1707something different in their life became way more important. In some cases 1708nobody is willing to take over the job as maintainer – and nobody can be forced 1709to, as contributing to the Linux kernel is done on a voluntary basis. Abandoned 1710drivers nevertheless remain in the kernel: they are still useful for people and 1711removing would be a regression. 1712 1713The situation is not that different with developers that are paid for their 1714work on the Linux kernel. Those contribute most changes these days. But their 1715employers sooner or later also stop caring for their code or make its 1716programmer focus on other things. Hardware vendors for example earn their money 1717mainly by selling new hardware; quite a few of them hence are not investing 1718much time and energy in maintaining a Linux kernel driver for something they 1719stopped selling years ago. Enterprise Linux distributors often care for a 1720longer time period, but in new versions often leave support for old and rare 1721hardware aside to limit the scope. Often spare time contributors take over once 1722a company orphans some code, but as mentioned above: sooner or later they will 1723leave the code behind, too. 1724 1725Priorities are another reason why some issues are not fixed, as maintainers 1726quite often are forced to set those, as time to work on Linux is limited. 1727That's true for spare time or the time employers grant their developers to 1728spend on maintenance work on the upstream kernel. Sometimes maintainers also 1729get overwhelmed with reports, even if a driver is working nearly perfectly. To 1730not get completely stuck, the programmer thus might have no other choice than 1731to prioritize issue reports and reject some of them. 1732 1733But don't worry too much about all of this, a lot of drivers have active 1734maintainers who are quite interested in fixing as many issues as possible. 1735 1736 1737Closing words 1738============= 1739 1740Compared with other Free/Libre & Open Source Software it's hard to report 1741issues to the Linux kernel developers: the length and complexity of this 1742document and the implications between the lines illustrate that. But that's how 1743it is for now. The main author of this text hopes documenting the state of the 1744art will lay some groundwork to improve the situation over time. 1745 1746 1747.. 1748 This text is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If you 1749 spot a typo or small mistake, feel free to let him know directly and he'll 1750 fix it. You are free to do the same in a mostly informal way if you want 1751 to contribute changes to the text, but for copyright reasons please CC 1752 linux-doc@vger.kernel.org and "sign-off" your contribution as 1753 Documentation/process/submitting-patches.rst outlines in the section "Sign 1754 your work - the Developer's Certificate of Origin". 1755