1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) 2.. 3 If you want to distribute this text under CC-BY-4.0 only, please use 'The 4 Linux kernel developers' for author attribution and link this as source: 5 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-issues.rst 6.. 7 Note: Only the content of this RST file as found in the Linux kernel sources 8 is available under CC-BY-4.0, as versions of this text that were processed 9 (for example by the kernel's build system) might contain content taken from 10 files which use a more restrictive license. 11 12 13Reporting issues 14++++++++++++++++ 15 16 17The short guide (aka TL;DR) 18=========================== 19 20Are you facing a regression with vanilla kernels from the same stable or 21longterm series? One still supported? Then search the `LKML 22<https://lore.kernel.org/lkml/>`_ and the `Linux stable mailing list 23<https://lore.kernel.org/stable/>`_ archives for matching reports to join. If 24you don't find any, install `the latest release from that series 25<https://kernel.org/>`_. If it still shows the issue, report it to the stable 26mailing list (stable@vger.kernel.org). 27 28In all other cases try your best guess which kernel part might be causing the 29issue. Check the :ref:`MAINTAINERS <maintainers>` file for how its developers 30expect to be told about problems, which most of the time will be by email with a 31mailing list in CC. Check the destination's archives for matching reports; 32search the `LKML <https://lore.kernel.org/lkml/>`_ and the web, too. If you 33don't find any to join, install `the latest mainline kernel 34<https://kernel.org/>`_. If the issue is present there, send a report. 35 36The issue was fixed there, but you would like to see it resolved in a still 37supported stable or longterm series as well? Then install its latest release. 38If it shows the problem, search for the change that fixed it in mainline and 39check if backporting is in the works or was discarded; if it's neither, ask 40those who handled the change for it. 41 42**General remarks**: When installing and testing a kernel as outlined above, 43ensure it's vanilla (IOW: not patched and not using add-on modules). Also make 44sure it's built and running in a healthy environment and not already tainted 45before the issue occurs. 46 47While writing your report, include all information relevant to the issue, like 48the kernel and the distro used. In case of a regression try to include the 49commit-id of the change causing it, which a bisection can find. If you're facing 50multiple issues with the Linux kernel at once, report each separately. 51 52Once the report is out, answer any questions that come up and help where you 53can. That includes keeping the ball rolling by occasionally retesting with newer 54releases and sending a status update afterwards. 55 56Step-by-step guide how to report issues to the kernel maintainers 57================================================================= 58 59The above TL;DR outlines roughly how to report issues to the Linux kernel 60developers. It might be all that's needed for people already familiar with 61reporting issues to Free/Libre & Open Source Software (FLOSS) projects. For 62everyone else there is this section. It is more detailed and uses a 63step-by-step approach. It still tries to be brief for readability and leaves 64out a lot of details; those are described below the step-by-step guide in a 65reference section, which explains each of the steps in more detail. 66 67Note: this section covers a few more aspects than the TL;DR and does things in 68a slightly different order. That's in your interest, to make sure you notice 69early if an issue that looks like a Linux kernel problem is actually caused by 70something else. These steps thus help to ensure the time you invest in this 71process won't feel wasted in the end: 72 73 * Are you facing an issue with a Linux kernel a hardware or software vendor 74 provided? Then in almost all cases you are better off to stop reading this 75 document and reporting the issue to your vendor instead, unless you are 76 willing to install the latest Linux version yourself. Be aware the latter 77 will often be needed anyway to hunt down and fix issues. 78 79 * Perform a rough search for existing reports with your favorite internet 80 search engine; additionally, check the archives of the `Linux Kernel Mailing 81 List (LKML) <https://lore.kernel.org/lkml/>`_. If you find matching reports, 82 join the discussion instead of sending a new one. 83 84 * See if the issue you are dealing with qualifies as regression, security 85 issue, or a really severe problem: those are 'issues of high priority' that 86 need special handling in some steps that are about to follow. 87 88 * Make sure it's not the kernel's surroundings that are causing the issue 89 you face. 90 91 * Create a fresh backup and put system repair and restore tools at hand. 92 93 * Ensure your system does not enhance its kernels by building additional 94 kernel modules on-the-fly, which solutions like DKMS might be doing locally 95 without your knowledge. 96 97 * Check if your kernel was 'tainted' when the issue occurred, as the event 98 that made the kernel set this flag might be causing the issue you face. 99 100 * Write down coarsely how to reproduce the issue. If you deal with multiple 101 issues at once, create separate notes for each of them and make sure they 102 work independently on a freshly booted system. That's needed, as each issue 103 needs to get reported to the kernel developers separately, unless they are 104 strongly entangled. 105 106 * If you are facing a regression within a stable or longterm version line 107 (say something broke when updating from 5.10.4 to 5.10.5), scroll down to 108 'Dealing with regressions within a stable and longterm kernel line'. 109 110 * Locate the driver or kernel subsystem that seems to be causing the issue. 111 Find out how and where its developers expect reports. Note: most of the 112 time this won't be bugzilla.kernel.org, as issues typically need to be sent 113 by mail to a maintainer and a public mailing list. 114 115 * Search the archives of the bug tracker or mailing list in question 116 thoroughly for reports that might match your issue. If you find anything, 117 join the discussion instead of sending a new report. 118 119After these preparations you'll now enter the main part: 120 121 * Unless you are already running the latest 'mainline' Linux kernel, better 122 go and install it for the reporting process. Testing and reporting with 123 the latest 'stable' Linux can be an acceptable alternative in some 124 situations; during the merge window that actually might be even the best 125 approach, but in that development phase it can be an even better idea to 126 suspend your efforts for a few days anyway. Whatever version you choose, 127 ideally use a 'vanilla' build. Ignoring these advices will dramatically 128 increase the risk your report will be rejected or ignored. 129 130 * Ensure the kernel you just installed does not 'taint' itself when 131 running. 132 133 * Reproduce the issue with the kernel you just installed. If it doesn't show 134 up there, scroll down to the instructions for issues only happening with 135 stable and longterm kernels. 136 137 * Optimize your notes: try to find and write the most straightforward way to 138 reproduce your issue. Make sure the end result has all the important 139 details, and at the same time is easy to read and understand for others 140 that hear about it for the first time. And if you learned something in this 141 process, consider searching again for existing reports about the issue. 142 143 * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider 144 decoding the kernel log to find the line of code that triggered the error. 145 146 * If your problem is a regression, try to narrow down when the issue was 147 introduced as much as possible. 148 149 * Start to compile the report by writing a detailed description about the 150 issue. Always mention a few things: the latest kernel version you installed 151 for reproducing, the Linux Distribution used, and your notes on how to 152 reproduce the issue. Ideally, make the kernel's build configuration 153 (.config) and the output from ``dmesg`` available somewhere on the net and 154 link to it. Include or upload all other information that might be relevant, 155 like the output/screenshot of an Oops or the output from ``lspci``. Once 156 you wrote this main part, insert a normal length paragraph on top of it 157 outlining the issue and the impact quickly. On top of this add one sentence 158 that briefly describes the problem and gets people to read on. Now give the 159 thing a descriptive title or subject that yet again is shorter. Then you're 160 ready to send or file the report like the MAINTAINERS file told you, unless 161 you are dealing with one of those 'issues of high priority': they need 162 special care which is explained in 'Special handling for high priority 163 issues' below. 164 165 * Wait for reactions and keep the thing rolling until you can accept the 166 outcome in one way or the other. Thus react publicly and in a timely manner 167 to any inquiries. Test proposed fixes. Do proactive testing: retest with at 168 least every first release candidate (RC) of a new mainline version and 169 report your results. Send friendly reminders if things stall. And try to 170 help yourself, if you don't get any help or if it's unsatisfying. 171 172 173Reporting regressions within a stable and longterm kernel line 174-------------------------------------------------------------- 175 176This subsection is for you, if you followed above process and got sent here at 177the point about regression within a stable or longterm kernel version line. You 178face one of those if something breaks when updating from 5.10.4 to 5.10.5 (a 179switch from 5.9.15 to 5.10.5 does not qualify). The developers want to fix such 180regressions as quickly as possible, hence there is a streamlined process to 181report them: 182 183 * Check if the kernel developers still maintain the Linux kernel version 184 line you care about: go to the `front page of kernel.org 185 <https://kernel.org/>`_ and make sure it mentions 186 the latest release of the particular version line without an '[EOL]' tag. 187 188 * Check the archives of the `Linux stable mailing list 189 <https://lore.kernel.org/stable/>`_ for existing reports. 190 191 * Install the latest release from the particular version line as a vanilla 192 kernel. Ensure this kernel is not tainted and still shows the problem, as 193 the issue might have already been fixed there. If you first noticed the 194 problem with a vendor kernel, check a vanilla build of the last version 195 known to work performs fine as well.* 196 197 * Send a short problem report to the Linux stable mailing list 198 (stable@vger.kernel.org). Roughly describe the issue and ideally explain 199 how to reproduce it. Mention the first version that shows the problem and 200 the last version that's working fine. Then wait for further instructions.* 201 202The reference section below explains each of these steps in more detail. 203 204 205Reporting issues only occurring in older kernel version lines 206------------------------------------------------------------- 207 208This subsection is for you, if you tried the latest mainline kernel as outlined 209above, but failed to reproduce your issue there; at the same time you want to 210see the issue fixed in a still supported stable or longterm series or vendor 211kernels regularly rebased on those. If that the case, follow these steps: 212 213 * Prepare yourself for the possibility that going through the next few steps 214 might not get the issue solved in older releases: the fix might be too big 215 or risky to get backported there. 216 217 * Perform the first three steps in the section "Dealing with regressions 218 within a stable and longterm kernel line" above. 219 220 * Search the Linux kernel version control system for the change that fixed 221 the issue in mainline, as its commit message might tell you if the fix is 222 scheduled for backporting already. If you don't find anything that way, 223 search the appropriate mailing lists for posts that discuss such an issue 224 or peer-review possible fixes; then check the discussions if the fix was 225 deemed unsuitable for backporting. If backporting was not considered at 226 all, join the newest discussion, asking if it's in the cards. 227 228 * One of the former steps should lead to a solution. If that doesn't work 229 out, ask the maintainers for the subsystem that seems to be causing the 230 issue for advice; CC the mailing list for the particular subsystem as well 231 as the stable mailing list. 232 233The reference section below explains each of these steps in more detail. 234 235 236Reference section: Reporting issues to the kernel maintainers 237============================================================= 238 239The detailed guides above outline all the major steps in brief fashion, which 240should be enough for most people. But sometimes there are situations where even 241experienced users might wonder how to actually do one of those steps. That's 242what this section is for, as it will provide a lot more details on each of the 243above steps. Consider this as reference documentation: it's possible to read it 244from top to bottom. But it's mainly meant to skim over and a place to look up 245details how to actually perform those steps. 246 247A few words of general advice before digging into the details: 248 249 * The Linux kernel developers are well aware this process is complicated and 250 demands more than other FLOSS projects. We'd love to make it simpler. But 251 that would require work in various places as well as some infrastructure, 252 which would need constant maintenance; nobody has stepped up to do that 253 work, so that's just how things are for now. 254 255 * A warranty or support contract with some vendor doesn't entitle you to 256 request fixes from developers in the upstream Linux kernel community: such 257 contracts are completely outside the scope of the Linux kernel, its 258 development community, and this document. That's why you can't demand 259 anything such a contract guarantees in this context, not even if the 260 developer handling the issue works for the vendor in question. If you want 261 to claim your rights, use the vendor's support channel instead. When doing 262 so, you might want to mention you'd like to see the issue fixed in the 263 upstream Linux kernel; motivate them by saying it's the only way to ensure 264 the fix in the end will get incorporated in all Linux distributions. 265 266 * If you never reported an issue to a FLOSS project before you should consider 267 reading `How to Report Bugs Effectively 268 <https://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_, `How To Ask 269 Questions The Smart Way 270 <http://www.catb.org/esr/faqs/smart-questions.html>`_, and `How to ask good 271 questions <https://jvns.ca/blog/good-questions/>`_. 272 273With that off the table, find below the details on how to properly report 274issues to the Linux kernel developers. 275 276 277Make sure you're using the upstream Linux kernel 278------------------------------------------------ 279 280 *Are you facing an issue with a Linux kernel a hardware or software vendor 281 provided? Then in almost all cases you are better off to stop reading this 282 document and reporting the issue to your vendor instead, unless you are 283 willing to install the latest Linux version yourself. Be aware the latter 284 will often be needed anyway to hunt down and fix issues.* 285 286Like most programmers, Linux kernel developers don't like to spend time dealing 287with reports for issues that don't even happen with their current code. It's 288just a waste everybody's time, especially yours. Unfortunately such situations 289easily happen when it comes to the kernel and often leads to frustration on both 290sides. That's because almost all Linux-based kernels pre-installed on devices 291(Computers, Laptops, Smartphones, Routers, …) and most shipped by Linux 292distributors are quite distant from the official Linux kernel as distributed by 293kernel.org: these kernels from these vendors are often ancient from the point of 294Linux development or heavily modified, often both. 295 296Most of these vendor kernels are quite unsuitable for reporting issues to the 297Linux kernel developers: an issue you face with one of them might have been 298fixed by the Linux kernel developers months or years ago already; additionally, 299the modifications and enhancements by the vendor might be causing the issue you 300face, even if they look small or totally unrelated. That's why you should report 301issues with these kernels to the vendor. Its developers should look into the 302report and, in case it turns out to be an upstream issue, fix it directly 303upstream or forward the report there. In practice that often does not work out 304or might not what you want. You thus might want to consider circumventing the 305vendor by installing the very latest Linux kernel core yourself. If that's an 306option for you move ahead in this process, as a later step in this guide will 307explain how to do that once it rules out other potential causes for your issue. 308 309Note, the previous paragraph is starting with the word 'most', as sometimes 310developers in fact are willing to handle reports about issues occurring with 311vendor kernels. If they do in the end highly depends on the developers and the 312issue in question. Your chances are quite good if the distributor applied only 313small modifications to a kernel based on a recent Linux version; that for 314example often holds true for the mainline kernels shipped by Debian GNU/Linux 315Sid or Fedora Rawhide. Some developers will also accept reports about issues 316with kernels from distributions shipping the latest stable kernel, as long as 317its only slightly modified; that for example is often the case for Arch Linux, 318regular Fedora releases, and openSUSE Tumbleweed. But keep in mind, you better 319want to use a mainline Linux and avoid using a stable kernel for this 320process, as outlined in the section 'Install a fresh kernel for testing' in more 321detail. 322 323Obviously you are free to ignore all this advice and report problems with an old 324or heavily modified vendor kernel to the upstream Linux developers. But note, 325those often get rejected or ignored, so consider yourself warned. But it's still 326better than not reporting the issue at all: sometimes such reports directly or 327indirectly will help to get the issue fixed over time. 328 329 330Search for existing reports, first run 331-------------------------------------- 332 333 *Perform a rough search for existing reports with your favorite internet 334 search engine; additionally, check the archives of the Linux Kernel Mailing 335 List (LKML). If you find matching reports, join the discussion instead of 336 sending a new one.* 337 338Reporting an issue that someone else already brought forward is often a waste of 339time for everyone involved, especially you as the reporter. So it's in your own 340interest to thoroughly check if somebody reported the issue already. At this 341step of the process it's okay to just perform a rough search: a later step will 342tell you to perform a more detailed search once you know where your issue needs 343to be reported to. Nevertheless, do not hurry with this step of the reporting 344process, it can save you time and trouble. 345 346Simply search the internet with your favorite search engine first. Afterwards, 347search the `Linux Kernel Mailing List (LKML) archives 348<https://lore.kernel.org/lkml/>`_. 349 350If you get flooded with results consider telling your search engine to limit 351search timeframe to the past month or year. And wherever you search, make sure 352to use good search terms; vary them a few times, too. While doing so try to 353look at the issue from the perspective of someone else: that will help you to 354come up with other words to use as search terms. Also make sure not to use too 355many search terms at once. Remember to search with and without information like 356the name of the kernel driver or the name of the affected hardware component. 357But its exact brand name (say 'ASUS Red Devil Radeon RX 5700 XT Gaming OC') 358often is not much helpful, as it is too specific. Instead try search terms like 359the model line (Radeon 5700 or Radeon 5000) and the code name of the main chip 360('Navi' or 'Navi10') with and without its manufacturer ('AMD'). 361 362In case you find an existing report about your issue, join the discussion, as 363you might be able to provide valuable additional information. That can be 364important even when a fix is prepared or in its final stages already, as 365developers might look for people that can provide additional information or 366test a proposed fix. Jump to the section 'Duties after the report went out' for 367details on how to get properly involved. 368 369Note, searching `bugzilla.kernel.org <https://bugzilla.kernel.org/>`_ might also 370be a good idea, as that might provide valuable insights or turn up matching 371reports. If you find the latter, just keep in mind: most subsystems expect 372reports in different places, as described below in the section "Check where you 373need to report your issue". The developers that should take care of the issue 374thus might not even be aware of the bugzilla ticket. Hence, check the ticket if 375the issue already got reported as outlined in this document and if not consider 376doing so. 377 378 379Issue of high priority? 380----------------------- 381 382 *See if the issue you are dealing with qualifies as regression, security 383 issue, or a really severe problem: those are 'issues of high priority' that 384 need special handling in some steps that are about to follow.* 385 386Linus Torvalds and the leading Linux kernel developers want to see some issues 387fixed as soon as possible, hence there are 'issues of high priority' that get 388handled slightly differently in the reporting process. Three type of cases 389qualify: regressions, security issues, and really severe problems. 390 391You deal with a 'regression' if something that worked with an older version of 392the Linux kernel does not work with a newer one or somehow works worse with it. 393It thus is a regression when a WiFi driver that did a fine job with Linux 5.7 394somehow misbehaves with 5.8 or doesn't work at all. It's also a regression if 395an application shows erratic behavior with a newer kernel, which might happen 396due to incompatible changes in the interface between the kernel and the 397userland (like procfs and sysfs). Significantly reduced performance or 398increased power consumption also qualify as regression. But keep in mind: the 399new kernel needs to be built with a configuration that is similar to the one 400from the old kernel (see below how to achieve that). That's because the kernel 401developers sometimes can not avoid incompatibilities when implementing new 402features; but to avoid regressions such features have to be enabled explicitly 403during build time configuration. 404 405What qualifies as security issue is left to your judgment. Consider reading 406'Documentation/admin-guide/security-bugs.rst' before proceeding, as it 407provides additional details how to best handle security issues. 408 409An issue is a 'really severe problem' when something totally unacceptably bad 410happens. That's for example the case when a Linux kernel corrupts the data it's 411handling or damages hardware it's running on. You're also dealing with a severe 412issue when the kernel suddenly stops working with an error message ('kernel 413panic') or without any farewell note at all. Note: do not confuse a 'panic' (a 414fatal error where the kernel stop itself) with a 'Oops' (a recoverable error), 415as the kernel remains running after the latter. 416 417 418Ensure a healthy environment 419---------------------------- 420 421 *Make sure it's not the kernel's surroundings that are causing the issue 422 you face.* 423 424Problems that look a lot like a kernel issue are sometimes caused by build or 425runtime environment. It's hard to rule out that problem completely, but you 426should minimize it: 427 428 * Use proven tools when building your kernel, as bugs in the compiler or the 429 binutils can cause the resulting kernel to misbehave. 430 431 * Ensure your computer components run within their design specifications; 432 that's especially important for the main processor, the main memory, and the 433 motherboard. Therefore, stop undervolting or overclocking when facing a 434 potential kernel issue. 435 436 * Try to make sure it's not faulty hardware that is causing your issue. Bad 437 main memory for example can result in a multitude of issues that will 438 manifest itself in problems looking like kernel issues. 439 440 * If you're dealing with a filesystem issue, you might want to check the file 441 system in question with ``fsck``, as it might be damaged in a way that leads 442 to unexpected kernel behavior. 443 444 * When dealing with a regression, make sure it's not something else that 445 changed in parallel to updating the kernel. The problem for example might be 446 caused by other software that was updated at the same time. It can also 447 happen that a hardware component coincidentally just broke when you rebooted 448 into a new kernel for the first time. Updating the systems BIOS or changing 449 something in the BIOS Setup can also lead to problems that on look a lot 450 like a kernel regression. 451 452 453Prepare for emergencies 454----------------------- 455 456 *Create a fresh backup and put system repair and restore tools at hand.* 457 458Reminder, you are dealing with computers, which sometimes do unexpected things, 459especially if you fiddle with crucial parts like the kernel of its operating 460system. That's what you are about to do in this process. Thus, make sure to 461create a fresh backup; also ensure you have all tools at hand to repair or 462reinstall the operating system as well as everything you need to restore the 463backup. 464 465 466Make sure your kernel doesn't get enhanced 467------------------------------------------ 468 469 *Ensure your system does not enhance its kernels by building additional 470 kernel modules on-the-fly, which solutions like DKMS might be doing locally 471 without your knowledge.* 472 473The risk your issue report gets ignored or rejected dramatically increases if 474your kernel gets enhanced in any way. That's why you should remove or disable 475mechanisms like akmods and DKMS: those build add-on kernel modules 476automatically, for example when you install a new Linux kernel or boot it for 477the first time. Also remove any modules they might have installed. Then reboot 478before proceeding. 479 480Note, you might not be aware that your system is using one of these solutions: 481they often get set up silently when you install Nvidia's proprietary graphics 482driver, VirtualBox, or other software that requires a some support from a 483module not part of the Linux kernel. That why your might need to uninstall the 484packages with such software to get rid of any 3rd party kernel module. 485 486 487Check 'taint' flag 488------------------ 489 490 *Check if your kernel was 'tainted' when the issue occurred, as the event 491 that made the kernel set this flag might be causing the issue you face.* 492 493The kernel marks itself with a 'taint' flag when something happens that might 494lead to follow-up errors that look totally unrelated. The issue you face might 495be such an error if your kernel is tainted. That's why it's in your interest to 496rule this out early before investing more time into this process. This is the 497only reason why this step is here, as this process later will tell you to 498install the latest mainline kernel; you will need to check the taint flag again 499then, as that's when it matters because it's the kernel the report will focus 500on. 501 502On a running system is easy to check if the kernel tainted itself: if ``cat 503/proc/sys/kernel/tainted`` returns '0' then the kernel is not tainted and 504everything is fine. Checking that file is impossible in some situations; that's 505why the kernel also mentions the taint status when it reports an internal 506problem (a 'kernel bug'), a recoverable error (a 'kernel Oops') or a 507non-recoverable error before halting operation (a 'kernel panic'). Look near 508the top of the error messages printed when one of these occurs and search for a 509line starting with 'CPU:'. It should end with 'Not tainted' if the kernel was 510not tainted when it noticed the problem; it was tainted if you see 'Tainted:' 511followed by a few spaces and some letters. 512 513If your kernel is tainted, study 'Documentation/admin-guide/tainted-kernels.rst' 514to find out why. Try to eliminate the reason. Often it's caused by one these 515three things: 516 517 1. A recoverable error (a 'kernel Oops') occurred and the kernel tainted 518 itself, as the kernel knows it might misbehave in strange ways after that 519 point. In that case check your kernel or system log and look for a section 520 that starts with this:: 521 522 Oops: 0000 [#1] SMP 523 524 That's the first Oops since boot-up, as the '#1' between the brackets shows. 525 Every Oops and any other problem that happens after that point might be a 526 follow-up problem to that first Oops, even if both look totally unrelated. 527 Rule this out by getting rid of the cause for the first Oops and reproducing 528 the issue afterwards. Sometimes simply restarting will be enough, sometimes 529 a change to the configuration followed by a reboot can eliminate the Oops. 530 But don't invest too much time into this at this point of the process, as 531 the cause for the Oops might already be fixed in the newer Linux kernel 532 version you are going to install later in this process. 533 534 2. Your system uses a software that installs its own kernel modules, for 535 example Nvidia's proprietary graphics driver or VirtualBox. The kernel 536 taints itself when it loads such module from external sources (even if 537 they are Open Source): they sometimes cause errors in unrelated kernel 538 areas and thus might be causing the issue you face. You therefore have to 539 prevent those modules from loading when you want to report an issue to the 540 Linux kernel developers. Most of the time the easiest way to do that is: 541 temporarily uninstall such software including any modules they might have 542 installed. Afterwards reboot. 543 544 3. The kernel also taints itself when it's loading a module that resides in 545 the staging tree of the Linux kernel source. That's a special area for 546 code (mostly drivers) that does not yet fulfill the normal Linux kernel 547 quality standards. When you report an issue with such a module it's 548 obviously okay if the kernel is tainted; just make sure the module in 549 question is the only reason for the taint. If the issue happens in an 550 unrelated area reboot and temporarily block the module from being loaded 551 by specifying ``foo.blacklist=1`` as kernel parameter (replace 'foo' with 552 the name of the module in question). 553 554 555Document how to reproduce issue 556------------------------------- 557 558 *Write down coarsely how to reproduce the issue. If you deal with multiple 559 issues at once, create separate notes for each of them and make sure they 560 work independently on a freshly booted system. That's needed, as each issue 561 needs to get reported to the kernel developers separately, unless they are 562 strongly entangled.* 563 564If you deal with multiple issues at once, you'll have to report each of them 565separately, as they might be handled by different developers. Describing 566various issues in one report also makes it quite difficult for others to tear 567it apart. Hence, only combine issues in one report if they are very strongly 568entangled. 569 570Additionally, during the reporting process you will have to test if the issue 571happens with other kernel versions. Therefore, it will make your work easier if 572you know exactly how to reproduce an issue quickly on a freshly booted system. 573 574Note: it's often fruitless to report issues that only happened once, as they 575might be caused by a bit flip due to cosmic radiation. That's why you should 576try to rule that out by reproducing the issue before going further. Feel free 577to ignore this advice if you are experienced enough to tell a one-time error 578due to faulty hardware apart from a kernel issue that rarely happens and thus 579is hard to reproduce. 580 581 582Regression in stable or longterm kernel? 583---------------------------------------- 584 585 *If you are facing a regression within a stable or longterm version line 586 (say something broke when updating from 5.10.4 to 5.10.5), scroll down to 587 'Dealing with regressions within a stable and longterm kernel line'.* 588 589Regression within a stable and longterm kernel version line are something the 590Linux developers want to fix badly, as such issues are even more unwanted than 591regression in the main development branch, as they can quickly affect a lot of 592people. The developers thus want to learn about such issues as quickly as 593possible, hence there is a streamlined process to report them. Note, 594regressions with newer kernel version line (say something broke when switching 595from 5.9.15 to 5.10.5) do not qualify. 596 597 598Check where you need to report your issue 599----------------------------------------- 600 601 *Locate the driver or kernel subsystem that seems to be causing the issue. 602 Find out how and where its developers expect reports. Note: most of the 603 time this won't be bugzilla.kernel.org, as issues typically need to be sent 604 by mail to a maintainer and a public mailing list.* 605 606It's crucial to send your report to the right people, as the Linux kernel is a 607big project and most of its developers are only familiar with a small subset of 608it. Quite a few programmers for example only care for just one driver, for 609example one for a WiFi chip; its developer likely will only have small or no 610knowledge about the internals of remote or unrelated "subsystems", like the TCP 611stack, the PCIe/PCI subsystem, memory management or file systems. 612 613Problem is: the Linux kernel lacks a central bug tracker where you can simply 614file your issue and make it reach the developers that need to know about it. 615That's why you have to find the right place and way to report issues yourself. 616You can do that with the help of a script (see below), but it mainly targets 617kernel developers and experts. For everybody else the MAINTAINERS file is the 618better place. 619 620How to read the MAINTAINERS file 621~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 622To illustrate how to use the :ref:`MAINTAINERS <maintainers>` file, lets assume 623the WiFi in your Laptop suddenly misbehaves after updating the kernel. In that 624case it's likely an issue in the WiFi driver. Obviously it could also be some 625code it builds upon, but unless you suspect something like that stick to the 626driver. If it's really something else, the driver's developers will get the 627right people involved. 628 629Sadly, there is no way to check which code is driving a particular hardware 630component that is both universal and easy. 631 632In case of a problem with the WiFi driver you for example might want to look at 633the output of ``lspci -k``, as it lists devices on the PCI/PCIe bus and the 634kernel module driving it:: 635 636 [user@something ~]$ lspci -k 637 [...] 638 3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32) 639 Subsystem: Bigfoot Networks, Inc. Device 1535 640 Kernel driver in use: ath10k_pci 641 Kernel modules: ath10k_pci 642 [...] 643 644But this approach won't work if your WiFi chip is connected over USB or some 645other internal bus. In those cases you might want to check your WiFi manager or 646the output of ``ip link``. Look for the name of the problematic network 647interface, which might be something like 'wlp58s0'. This name can be used like 648this to find the module driving it:: 649 650 [user@something ~]$ realpath --relative-to=/sys/module/ /sys/class/net/wlp58s0/device/driver/module 651 ath10k_pci 652 653In case tricks like these don't bring you any further, try to search the 654internet on how to narrow down the driver or subsystem in question. And if you 655are unsure which it is: just try your best guess, somebody will help you if you 656guessed poorly. 657 658Once you know the driver or subsystem, you want to search for it in the 659MAINTAINERS file. In the case of 'ath10k_pci' you won't find anything, as the 660name is too specific. Sometimes you will need to search on the net for help; 661but before doing so, try a somewhat shorted or modified name when searching the 662MAINTAINERS file, as then you might find something like this:: 663 664 QUALCOMM ATHEROS ATH10K WIRELESS DRIVER 665 Mail: A. Some Human <shuman@example.com> 666 Mailing list: ath10k@lists.infradead.org 667 Status: Supported 668 Web-page: https://wireless.wiki.kernel.org/en/users/Drivers/ath10k 669 SCM: git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git 670 Files: drivers/net/wireless/ath/ath10k/ 671 672Note: the line description will be abbreviations, if you read the plain 673MAINTAINERS file found in the root of the Linux source tree. 'Mail:' for 674example will be 'M:', 'Mailing list:' will be 'L', and 'Status:' will be 'S:'. 675A section near the top of the file explains these and other abbreviations. 676 677First look at the line 'Status'. Ideally it should be 'Supported' or 678'Maintained'. If it states 'Obsolete' then you are using some outdated approach 679that was replaced by a newer solution you need to switch to. Sometimes the code 680only has someone who provides 'Odd Fixes' when feeling motivated. And with 681'Orphan' you are totally out of luck, as nobody takes care of the code anymore. 682That only leaves these options: arrange yourself to live with the issue, fix it 683yourself, or find a programmer somewhere willing to fix it. 684 685After checking the status, look for a line starting with 'bugs:': it will tell 686you where to find a subsystem specific bug tracker to file your issue. The 687example above does not have such a line. That is the case for most sections, as 688Linux kernel development is completely driven by mail. Very few subsystems use 689a bug tracker, and only some of those rely on bugzilla.kernel.org. 690 691In this and many other cases you thus have to look for lines starting with 692'Mail:' instead. Those mention the name and the email addresses for the 693maintainers of the particular code. Also look for a line starting with 'Mailing 694list:', which tells you the public mailing list where the code is developed. 695Your report later needs to go by mail to those addresses. Additionally, for all 696issue reports sent by email, make sure to add the Linux Kernel Mailing List 697(LKML) <linux-kernel@vger.kernel.org> to CC. Don't omit either of the mailing 698lists when sending your issue report by mail later! Maintainers are busy people 699and might leave some work for other developers on the subsystem specific list; 700and LKML is important to have one place where all issue reports can be found. 701 702 703Finding the maintainers with the help of a script 704~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 705 706For people that have the Linux sources at hand there is a second option to find 707the proper place to report: the script 'scripts/get_maintainer.pl' which tries 708to find all people to contact. It queries the MAINTAINERS file and needs to be 709called with a path to the source code in question. For drivers compiled as 710module if often can be found with a command like this:: 711 712 $ modinfo ath10k_pci | grep filename | sed 's!/lib/modules/.*/kernel/!!; s!filename:!!; s!\.ko\(\|\.xz\)!!' 713 drivers/net/wireless/ath/ath10k/ath10k_pci.ko 714 715Pass parts of this to the script:: 716 717 $ ./scripts/get_maintainer.pl -f drivers/net/wireless/ath/ath10k* 718 Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER) 719 Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS) 720 ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER) 721 linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS)) 722 netdev@vger.kernel.org (open list:NETWORKING DRIVERS) 723 linux-kernel@vger.kernel.org (open list) 724 725Don't sent your report to all of them. Send it to the maintainers, which the 726script calls "supporter:"; additionally CC the most specific mailing list for 727the code as well as the Linux Kernel Mailing List (LKML). In this case you thus 728would need to send the report to 'Some Human <shuman@example.com>' with 729'ath10k@lists.infradead.org' and 'linux-kernel@vger.kernel.org' in CC. 730 731Note: in case you cloned the Linux sources with git you might want to call 732``get_maintainer.pl`` a second time with ``--git``. The script then will look 733at the commit history to find which people recently worked on the code in 734question, as they might be able to help. But use these results with care, as it 735can easily send you in a wrong direction. That for example happens quickly in 736areas rarely changed (like old or unmaintained drivers): sometimes such code is 737modified during tree-wide cleanups by developers that do not care about the 738particular driver at all. 739 740 741Search for existing reports, second run 742--------------------------------------- 743 744 *Search the archives of the bug tracker or mailing list in question 745 thoroughly for reports that might match your issue. If you find anything, 746 join the discussion instead of sending a new report.* 747 748As mentioned earlier already: reporting an issue that someone else already 749brought forward is often a waste of time for everyone involved, especially you 750as the reporter. That's why you should search for existing report again, now 751that you know where they need to be reported to. If it's mailing list, you will 752often find its archives on `lore.kernel.org <https://lore.kernel.org/>`_. 753 754But some list are hosted in different places. That for example is the case for 755the ath10k WiFi driver used as example in the previous step. But you'll often 756find the archives for these lists easily on the net. Searching for 'archive 757ath10k@lists.infradead.org' for example will lead you to the `Info page for the 758ath10k mailing list <https://lists.infradead.org/mailman/listinfo/ath10k>`_, 759which at the top links to its 760`list archives <https://lists.infradead.org/pipermail/ath10k/>`_. Sadly this and 761quite a few other lists miss a way to search the archives. In those cases use a 762regular internet search engine and add something like 763'site:lists.infradead.org/pipermail/ath10k/' to your search terms, which limits 764the results to the archives at that URL. 765 766It's also wise to check the internet, LKML and maybe bugzilla.kernel.org again 767at this point. 768 769For details how to search and what to do if you find matching reports see 770"Search for existing reports, first run" above. 771 772Do not hurry with this step of the reporting process: spending 30 to 60 minutes 773or even more time can save you and others quite a lot of time and trouble. 774 775 776Install a fresh kernel for testing 777---------------------------------- 778 779 *Unless you are already running the latest 'mainline' Linux kernel, better 780 go and install it for the reporting process. Testing and reporting with 781 the latest 'stable' Linux can be an acceptable alternative in some 782 situations; during the merge window that actually might be even the best 783 approach, but in that development phase it can be an even better idea to 784 suspend your efforts for a few days anyway. Whatever version you choose, 785 ideally use a 'vanilla' built. Ignoring these advices will dramatically 786 increase the risk your report will be rejected or ignored.* 787 788As mentioned in the detailed explanation for the first step already: Like most 789programmers, Linux kernel developers don't like to spend time dealing with 790reports for issues that don't even happen with the current code. It's just a 791waste everybody's time, especially yours. That's why it's in everybody's 792interest that you confirm the issue still exists with the latest upstream code 793before reporting it. You are free to ignore this advice, but as outlined 794earlier: doing so dramatically increases the risk that your issue report might 795get rejected or simply ignored. 796 797In the scope of the kernel "latest upstream" normally means: 798 799 * Install a mainline kernel; the latest stable kernel can be an option, but 800 most of the time is better avoided. Longterm kernels (sometimes called 'LTS 801 kernels') are unsuitable at this point of the process. The next subsection 802 explains all of this in more detail. 803 804 * The over next subsection describes way to obtain and install such a kernel. 805 It also outlines that using a pre-compiled kernel are fine, but better are 806 vanilla, which means: it was built using Linux sources taken straight `from 807 kernel.org <https://kernel.org/>`_ and not modified or enhanced in any way. 808 809Choosing the right version for testing 810~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 811 812Head over to `kernel.org <https://kernel.org/>`_ to find out which version you 813want to use for testing. Ignore the big yellow button that says 'Latest release' 814and look a little lower at the table. At its top you'll see a line starting with 815mainline, which most of the time will point to a pre-release with a version 816number like '5.8-rc2'. If that's the case, you'll want to use this mainline 817kernel for testing, as that where all fixes have to be applied first. Do not let 818that 'rc' scare you, these 'development kernels' are pretty reliable — and you 819made a backup, as you were instructed above, didn't you? 820 821In about two out of every nine to ten weeks, mainline might point you to a 822proper release with a version number like '5.7'. If that happens, consider 823suspending the reporting process until the first pre-release of the next 824version (5.8-rc1) shows up on kernel.org. That's because the Linux development 825cycle then is in its two-week long 'merge window'. The bulk of the changes and 826all intrusive ones get merged for the next release during this time. It's a bit 827more risky to use mainline during this period. Kernel developers are also often 828quite busy then and might have no spare time to deal with issue reports. It's 829also quite possible that one of the many changes applied during the merge 830window fixes the issue you face; that's why you soon would have to retest with 831a newer kernel version anyway, as outlined below in the section 'Duties after 832the report went out'. 833 834That's why it might make sense to wait till the merge window is over. But don't 835to that if you're dealing with something that shouldn't wait. In that case 836consider obtaining the latest mainline kernel via git (see below) or use the 837latest stable version offered on kernel.org. Using that is also acceptable in 838case mainline for some reason does currently not work for you. An in general: 839using it for reproducing the issue is also better than not reporting it issue 840at all. 841 842Better avoid using the latest stable kernel outside merge windows, as all fixes 843must be applied to mainline first. That's why checking the latest mainline 844kernel is so important: any issue you want to see fixed in older version lines 845needs to be fixed in mainline first before it can get backported, which can 846take a few days or weeks. Another reason: the fix you hope for might be too 847hard or risky for backporting; reporting the issue again hence is unlikely to 848change anything. 849 850These aspects are also why longterm kernels (sometimes called "LTS kernels") 851are unsuitable for this part of the reporting process: they are to distant from 852the current code. Hence go and test mainline first and follow the process 853further: if the issue doesn't occur with mainline it will guide you how to get 854it fixed in older version lines, if that's in the cards for the fix in question. 855 856How to obtain a fresh Linux kernel 857~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 858 859**Using a pre-compiled kernel**: This is often the quickest, easiest, and safest 860way for testing — especially is you are unfamiliar with the Linux kernel. The 861problem: most of those shipped by distributors or add-on repositories are build 862from modified Linux sources. They are thus not vanilla and therefore often 863unsuitable for testing and issue reporting: the changes might cause the issue 864you face or influence it somehow. 865 866But you are in luck if you are using a popular Linux distribution: for quite a 867few of them you'll find repositories on the net that contain packages with the 868latest mainline or stable Linux built as vanilla kernel. It's totally okay to 869use these, just make sure from the repository's description they are vanilla or 870at least close to it. Additionally ensure the packages contain the latest 871versions as offered on kernel.org. The packages are likely unsuitable if they 872are older than a week, as new mainline and stable kernels typically get released 873at least once a week. 874 875Please note that you might need to build your own kernel manually later: that's 876sometimes needed for debugging or testing fixes, as described later in this 877document. Also be aware that pre-compiled kernels might lack debug symbols that 878are needed to decode messages the kernel prints when a panic, Oops, warning, or 879BUG occurs; if you plan to decode those, you might be better off compiling a 880kernel yourself (see the end of this subsection and the section titled 'Decode 881failure messages' for details). 882 883**Using git**: Developers and experienced Linux users familiar with git are 884often best served by obtaining the latest Linux kernel sources straight from the 885`official development repository on kernel.org 886<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_. 887Those are likely a bit ahead of the latest mainline pre-release. Don't worry 888about it: they are as reliable as a proper pre-release, unless the kernel's 889development cycle is currently in the middle of a merge window. But even then 890they are quite reliable. 891 892**Conventional**: People unfamiliar with git are often best served by 893downloading the sources as tarball from `kernel.org <https://kernel.org/>`_. 894 895How to actually build a kernel is not described here, as many websites explain 896the necessary steps already. If you are new to it, consider following one of 897those how-to's that suggest to use ``make localmodconfig``, as that tries to 898pick up the configuration of your current kernel and then tries to adjust it 899somewhat for your system. That does not make the resulting kernel any better, 900but quicker to compile. 901 902Note: If you are dealing with a panic, Oops, warning, or BUG from the kernel, 903please try to enable CONFIG_KALLSYMS when configuring your kernel. 904Additionally, enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the 905latter is the relevant one of those two, but can only be reached if you enable 906the former. Be aware CONFIG_DEBUG_INFO increases the storage space required to 907build a kernel by quite a bit. But that's worth it, as these options will allow 908you later to pinpoint the exact line of code that triggers your issue. The 909section 'Decode failure messages' below explains this in more detail. 910 911But keep in mind: Always keep a record of the issue encountered in case it is 912hard to reproduce. Sending an undecoded report is better than not reporting 913the issue at all. 914 915 916Check 'taint' flag 917------------------ 918 919 *Ensure the kernel you just installed does not 'taint' itself when 920 running.* 921 922As outlined above in more detail already: the kernel sets a 'taint' flag when 923something happens that can lead to follow-up errors that look totally 924unrelated. That's why you need to check if the kernel you just installed does 925not set this flag. And if it does, you in almost all the cases needs to 926eliminate the reason for it before you reporting issues that occur with it. See 927the section above for details how to do that. 928 929 930Reproduce issue with the fresh kernel 931------------------------------------- 932 933 *Reproduce the issue with the kernel you just installed. If it doesn't show 934 up there, scroll down to the instructions for issues only happening with 935 stable and longterm kernels.* 936 937Check if the issue occurs with the fresh Linux kernel version you just 938installed. If it was fixed there already, consider sticking with this version 939line and abandoning your plan to report the issue. But keep in mind that other 940users might still be plagued by it, as long as it's not fixed in either stable 941and longterm version from kernel.org (and thus vendor kernels derived from 942those). If you prefer to use one of those or just want to help their users, 943head over to the section "Details about reporting issues only occurring in 944older kernel version lines" below. 945 946 947Optimize description to reproduce issue 948--------------------------------------- 949 950 *Optimize your notes: try to find and write the most straightforward way to 951 reproduce your issue. Make sure the end result has all the important 952 details, and at the same time is easy to read and understand for others 953 that hear about it for the first time. And if you learned something in this 954 process, consider searching again for existing reports about the issue.* 955 956An unnecessarily complex report will make it hard for others to understand your 957report. Thus try to find a reproducer that's straight forward to describe and 958thus easy to understand in written form. Include all important details, but at 959the same time try to keep it as short as possible. 960 961In this in the previous steps you likely have learned a thing or two about the 962issue you face. Use this knowledge and search again for existing reports 963instead you can join. 964 965 966Decode failure messages 967----------------------- 968 969 *If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider 970 decoding the kernel log to find the line of code that triggered the error.* 971 972When the kernel detects an internal problem, it will log some information about 973the executed code. This makes it possible to pinpoint the exact line in the 974source code that triggered the issue and shows how it was called. But that only 975works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring 976your kernel. If you did so, consider to decode the information from the 977kernel's log. That will make it a lot easier to understand what lead to the 978'panic', 'Oops', 'warning', or 'BUG', which increases the chances that someone 979can provide a fix. 980 981Decoding can be done with a script you find in the Linux source tree. If you 982are running a kernel you compiled yourself earlier, call it like this:: 983 984 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux 985 986If you are running a packaged vanilla kernel, you will likely have to install 987the corresponding packages with debug symbols. Then call the script (which you 988might need to get from the Linux sources if your distro does not package it) 989like this:: 990 991 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh \ 992 /usr/lib/debug/lib/modules/5.10.10-4.1.x86_64/vmlinux /usr/src/kernels/5.10.10-4.1.x86_64/ 993 994The script will work on log lines like the following, which show the address of 995the code the kernel was executing when the error occurred:: 996 997 [ 68.387301] RIP: 0010:test_module_init+0x5/0xffa [test_module] 998 999Once decoded, these lines will look like this:: 1000 1001 [ 68.387301] RIP: 0010:test_module_init (/home/username/linux-5.10.5/test-module/test-module.c:16) test_module 1002 1003In this case the executed code was built from the file 1004'~/linux-5.10.5/test-module/test-module.c' and the error occurred by the 1005instructions found in line '16'. 1006 1007The script will similarly decode the addresses mentioned in the section 1008starting with 'Call trace', which show the path to the function where the 1009problem occurred. Additionally, the script will show the assembler output for 1010the code section the kernel was executing. 1011 1012Note, if you can't get this to work, simply skip this step and mention the 1013reason for it in the report. If you're lucky, it might not be needed. And if it 1014is, someone might help you to get things going. Also be aware this is just one 1015of several ways to decode kernel stack traces. Sometimes different steps will 1016be required to retrieve the relevant details. Don't worry about that, if that's 1017needed in your case, developers will tell you what to do. 1018 1019 1020Special care for regressions 1021---------------------------- 1022 1023 *If your problem is a regression, try to narrow down when the issue was 1024 introduced as much as possible.* 1025 1026Linux lead developer Linus Torvalds insists that the Linux kernel never 1027worsens, that's why he deems regressions as unacceptable and wants to see them 1028fixed quickly. That's why changes that introduced a regression are often 1029promptly reverted if the issue they cause can't get solved quickly any other 1030way. Reporting a regression is thus a bit like playing a kind of trump card to 1031get something quickly fixed. But for that to happen the change that's causing 1032the regression needs to be known. Normally it's up to the reporter to track 1033down the culprit, as maintainers often won't have the time or setup at hand to 1034reproduce it themselves. 1035 1036To find the change there is a process called 'bisection' which the document 1037'Documentation/admin-guide/bug-bisect.rst' describes in detail. That process 1038will often require you to build about ten to twenty kernel images, trying to 1039reproduce the issue with each of them before building the next. Yes, that takes 1040some time, but don't worry, it works a lot quicker than most people assume. 1041Thanks to a 'binary search' this will lead you to the one commit in the source 1042code management system that's causing the regression. Once you find it, search 1043the net for the subject of the change, its commit id and the shortened commit id 1044(the first 12 characters of the commit id). This will lead you to existing 1045reports about it, if there are any. 1046 1047Note, a bisection needs a bit of know-how, which not everyone has, and quite a 1048bit of effort, which not everyone is willing to invest. Nevertheless, it's 1049highly recommended performing a bisection yourself. If you really can't or 1050don't want to go down that route at least find out which mainline kernel 1051introduced the regression. If something for example breaks when switching from 10525.5.15 to 5.8.4, then try at least all the mainline releases in that area (5.6, 10535.7 and 5.8) to check when it first showed up. Unless you're trying to find a 1054regression in a stable or longterm kernel, avoid testing versions which number 1055has three sections (5.6.12, 5.7.8), as that makes the outcome hard to 1056interpret, which might render your testing useless. Once you found the major 1057version which introduced the regression, feel free to move on in the reporting 1058process. But keep in mind: it depends on the issue at hand if the developers 1059will be able to help without knowing the culprit. Sometimes they might 1060recognize from the report want went wrong and can fix it; other times they will 1061be unable to help unless you perform a bisection. 1062 1063When dealing with regressions make sure the issue you face is really caused by 1064the kernel and not by something else, as outlined above already. 1065 1066In the whole process keep in mind: an issue only qualifies as regression if the 1067older and the newer kernel got built with a similar configuration. The best way 1068to archive this: copy the configuration file (``.config``) from the old working 1069kernel freshly to each newer kernel version you try. Afterwards run ``make 1070olddefconfig`` to adjust it for the needs of the new version. 1071 1072 1073Write and send the report 1074------------------------- 1075 1076 *Start to compile the report by writing a detailed description about the 1077 issue. Always mention a few things: the latest kernel version you installed 1078 for reproducing, the Linux Distribution used, and your notes on how to 1079 reproduce the issue. Ideally, make the kernel's build configuration 1080 (.config) and the output from ``dmesg`` available somewhere on the net and 1081 link to it. Include or upload all other information that might be relevant, 1082 like the output/screenshot of an Oops or the output from ``lspci``. Once 1083 you wrote this main part, insert a normal length paragraph on top of it 1084 outlining the issue and the impact quickly. On top of this add one sentence 1085 that briefly describes the problem and gets people to read on. Now give the 1086 thing a descriptive title or subject that yet again is shorter. Then you're 1087 ready to send or file the report like the MAINTAINERS file told you, unless 1088 you are dealing with one of those 'issues of high priority': they need 1089 special care which is explained in 'Special handling for high priority 1090 issues' below.* 1091 1092Now that you have prepared everything it's time to write your report. How to do 1093that is partly explained by the three documents linked to in the preface above. 1094That's why this text will only mention a few of the essentials as well as 1095things specific to the Linux kernel. 1096 1097There is one thing that fits both categories: the most crucial parts of your 1098report are the title/subject, the first sentence, and the first paragraph. 1099Developers often get quite a lot of mail. They thus often just take a few 1100seconds to skim a mail before deciding to move on or look closer. Thus: the 1101better the top section of your report, the higher are the chances that someone 1102will look into it and help you. And that is why you should ignore them for now 1103and write the detailed report first. ;-) 1104 1105Things each report should mention 1106~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1107 1108Describe in detail how your issue happens with the fresh vanilla kernel you 1109installed. Try to include the step-by-step instructions you wrote and optimized 1110earlier that outline how you and ideally others can reproduce the issue; in 1111those rare cases where that's impossible try to describe what you did to 1112trigger it. 1113 1114Also include all the relevant information others might need to understand the 1115issue and its environment. What's actually needed depends a lot on the issue, 1116but there are some things you should include always: 1117 1118 * the output from ``cat /proc/version``, which contains the Linux kernel 1119 version number and the compiler it was built with. 1120 1121 * the Linux distribution the machine is running (``hostnamectl | grep 1122 "Operating System"``) 1123 1124 * the architecture of the CPU and the operating system (``uname -mi``) 1125 1126 * if you are dealing with a regression and performed a bisection, mention the 1127 subject and the commit-id of the change that is causing it. 1128 1129In a lot of cases it's also wise to make two more things available to those 1130that read your report: 1131 1132 * the configuration used for building your Linux kernel (the '.config' file) 1133 1134 * the kernel's messages that you get from ``dmesg`` written to a file. Make 1135 sure that it starts with a line like 'Linux version 5.8-1 1136 (foobar@example.com) (gcc (GCC) 10.2.1, GNU ld version 2.34) #1 SMP Mon Aug 1137 3 14:54:37 UTC 2020' If it's missing, then important messages from the first 1138 boot phase already got discarded. In this case instead consider using 1139 ``journalctl -b 0 -k``; alternatively you can also reboot, reproduce the 1140 issue and call ``dmesg`` right afterwards. 1141 1142These two files are big, that's why it's a bad idea to put them directly into 1143your report. If you are filing the issue in a bug tracker then attach them to 1144the ticket. If you report the issue by mail do not attach them, as that makes 1145the mail too large; instead do one of these things: 1146 1147 * Upload the files somewhere public (your website, a public file paste 1148 service, a ticket created just for this purpose on `bugzilla.kernel.org 1149 <https://bugzilla.kernel.org/>`_, ...) and include a link to them in your 1150 report. Ideally use something where the files stay available for years, as 1151 they could be useful to someone many years from now; this for example can 1152 happen if five or ten years from now a developer works on some code that was 1153 changed just to fix your issue. 1154 1155 * Put the files aside and mention you will send them later in individual 1156 replies to your own mail. Just remember to actually do that once the report 1157 went out. ;-) 1158 1159Things that might be wise to provide 1160~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1161 1162Depending on the issue you might need to add more background data. Here are a 1163few suggestions what often is good to provide: 1164 1165 * If you are dealing with a 'warning', an 'OOPS' or a 'panic' from the kernel, 1166 include it. If you can't copy'n'paste it, try to capture a netconsole trace 1167 or at least take a picture of the screen. 1168 1169 * If the issue might be related to your computer hardware, mention what kind 1170 of system you use. If you for example have problems with your graphics card, 1171 mention its manufacturer, the card's model, and what chip is uses. If it's a 1172 laptop mention its name, but try to make sure it's meaningful. 'Dell XPS 13' 1173 for example is not, because it might be the one from 2012; that one looks 1174 not that different from the one sold today, but apart from that the two have 1175 nothing in common. Hence, in such cases add the exact model number, which 1176 for example are '9380' or '7390' for XPS 13 models introduced during 2019. 1177 Names like 'Lenovo Thinkpad T590' are also somewhat ambiguous: there are 1178 variants of this laptop with and without a dedicated graphics chip, so try 1179 to find the exact model name or specify the main components. 1180 1181 * Mention the relevant software in use. If you have problems with loading 1182 modules, you want to mention the versions of kmod, systemd, and udev in use. 1183 If one of the DRM drivers misbehaves, you want to state the versions of 1184 libdrm and Mesa; also specify your Wayland compositor or the X-Server and 1185 its driver. If you have a filesystem issue, mention the version of 1186 corresponding filesystem utilities (e2fsprogs, btrfs-progs, xfsprogs, ...). 1187 1188 * Gather additional information from the kernel that might be of interest. The 1189 output from ``lspci -nn`` will for example help others to identify what 1190 hardware you use. If you have a problem with hardware you even might want to 1191 make the output from ``sudo lspci -vvv`` available, as that provides 1192 insights how the components were configured. For some issues it might be 1193 good to include the contents of files like ``/proc/cpuinfo``, 1194 ``/proc/ioports``, ``/proc/iomem``, ``/proc/modules``, or 1195 ``/proc/scsi/scsi``. Some subsystem also offer tools to collect relevant 1196 information. One such tool is ``alsa-info.sh`` `which the audio/sound 1197 subsystem developers provide <https://www.alsa-project.org/wiki/AlsaInfo>`_. 1198 1199Those examples should give your some ideas of what data might be wise to 1200attach, but you have to think yourself what will be helpful for others to know. 1201Don't worry too much about forgetting something, as developers will ask for 1202additional details they need. But making everything important available from 1203the start increases the chance someone will take a closer look. 1204 1205 1206The important part: the head of your report 1207~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1208 1209Now that you have the detailed part of the report prepared let's get to the 1210most important section: the first few sentences. Thus go to the top, add 1211something like 'The detailed description:' before the part you just wrote and 1212insert two newlines at the top. Now write one normal length paragraph that 1213describes the issue roughly. Leave out all boring details and focus on the 1214crucial parts readers need to know to understand what this is all about; if you 1215think this bug affects a lot of users, mention this to get people interested. 1216 1217Once you did that insert two more lines at the top and write a one sentence 1218summary that explains quickly what the report is about. After that you have to 1219get even more abstract and write an even shorter subject/title for the report. 1220 1221Now that you have written this part take some time to optimize it, as it is the 1222most important parts of your report: a lot of people will only read this before 1223they decide if reading the rest is time well spent. 1224 1225Now send or file the report like the :ref:`MAINTAINERS <maintainers>` file told 1226you, unless it's one of those 'issues of high priority' outlined earlier: in 1227that case please read the next subsection first before sending the report on 1228its way. 1229 1230Special handling for high priority issues 1231~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1232 1233Reports for high priority issues need special handling. 1234 1235**Severe issues**: make sure the subject or ticket title as well as the first 1236paragraph makes the severeness obvious. 1237 1238**Regressions**: If the issue is a regression add [REGRESSION] to the mail's 1239subject or the title in the bug-tracker. If you did not perform a bisection 1240mention at least the latest mainline version you tested that worked fine (say 12415.7) and the oldest where the issue occurs (say 5.8). If you did a successful 1242bisection mention the commit id and subject of the change that causes the 1243regression. Also make sure to add the author of that change to your report; if 1244you need to file your bug in a bug-tracker forward the report to him in a 1245private mail and mention where your filed it. 1246 1247**Security issues**: for these issues your will have to evaluate if a 1248short-term risk to other users would arise if details were publicly disclosed. 1249If that's not the case simply proceed with reporting the issue as described. 1250For issues that bear such a risk you will need to adjust the reporting process 1251slightly: 1252 1253 * If the MAINTAINERS file instructed you to report the issue by mail, do not 1254 CC any public mailing lists. 1255 1256 * If you were supposed to file the issue in a bug tracker make sure to mark 1257 the ticket as 'private' or 'security issue'. If the bug tracker does not 1258 offer a way to keep reports private, forget about it and send your report as 1259 a private mail to the maintainers instead. 1260 1261In both cases make sure to also mail your report to the addresses the 1262MAINTAINERS file lists in the section 'security contact'. Ideally directly CC 1263them when sending the report by mail. If you filed it in a bug tracker, forward 1264the report's text to these addresses; but on top of it put a small note where 1265you mention that you filed it with a link to the ticket. 1266 1267See 'Documentation/admin-guide/security-bugs.rst' for more information. 1268 1269 1270Duties after the report went out 1271-------------------------------- 1272 1273 *Wait for reactions and keep the thing rolling until you can accept the 1274 outcome in one way or the other. Thus react publicly and in a timely manner 1275 to any inquiries. Test proposed fixes. Do proactive testing: retest with at 1276 least every first release candidate (RC) of a new mainline version and 1277 report your results. Send friendly reminders if things stall. And try to 1278 help yourself, if you don't get any help or if it's unsatisfying.* 1279 1280If your report was good and you are really lucky then one of the developers 1281might immediately spot what's causing the issue; they then might write a patch 1282to fix it, test it, and send it straight for integration in mainline while 1283tagging it for later backport to stable and longterm kernels that need it. Then 1284all you need to do is reply with a 'Thank you very much' and switch to a version 1285with the fix once it gets released. 1286 1287But this ideal scenario rarely happens. That's why the job is only starting 1288once you got the report out. What you'll have to do depends on the situations, 1289but often it will be the things listed below. But before digging into the 1290details, here are a few important things you need to keep in mind for this part 1291of the process. 1292 1293 1294General advice for further interactions 1295~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1296 1297**Always reply in public**: When you filed the issue in a bug tracker, always 1298reply there and do not contact any of the developers privately about it. For 1299mailed reports always use the 'Reply-all' function when replying to any mails 1300you receive. That includes mails with any additional data you might want to add 1301to your report: go to your mail applications 'Sent' folder and use 'reply-all' 1302on your mail with the report. This approach will make sure the public mailing 1303list(s) and everyone else that gets involved over time stays in the loop; it 1304also keeps the mail thread intact, which among others is really important for 1305mailing lists to group all related mails together. 1306 1307There are just two situations where a comment in a bug tracker or a 'Reply-all' 1308is unsuitable: 1309 1310 * Someone tells you to send something privately. 1311 1312 * You were told to send something, but noticed it contains sensitive 1313 information that needs to be kept private. In that case it's okay to send it 1314 in private to the developer that asked for it. But note in the ticket or a 1315 mail that you did that, so everyone else knows you honored the request. 1316 1317**Do research before asking for clarifications or help**: In this part of the 1318process someone might tell you to do something that requires a skill you might 1319not have mastered yet. For example, you might be asked to use some test tools 1320you never have heard of yet; or you might be asked to apply a patch to the 1321Linux kernel sources to test if it helps. In some cases it will be fine sending 1322a reply asking for instructions how to do that. But before going that route try 1323to find the answer own your own by searching the internet; alternatively 1324consider asking in other places for advice. For example ask a friend or post 1325about it to a chatroom or forum you normally hang out. 1326 1327**Be patient**: If you are really lucky you might get a reply to your report 1328within a few hours. But most of the time it will take longer, as maintainers 1329are scattered around the globe and thus might be in a different time zone – one 1330where they already enjoy their night away from keyboard. 1331 1332In general, kernel developers will take one to five business days to respond to 1333reports. Sometimes it will take longer, as they might be busy with the merge 1334windows, other work, visiting developer conferences, or simply enjoying a long 1335summer holiday. 1336 1337The 'issues of high priority' (see above for an explanation) are an exception 1338here: maintainers should address them as soon as possible; that's why you 1339should wait a week at maximum (or just two days if it's something urgent) 1340before sending a friendly reminder. 1341 1342Sometimes the maintainer might not be responding in a timely manner; other 1343times there might be disagreements, for example if an issue qualifies as 1344regression or not. In such cases raise your concerns on the mailing list and 1345ask others for public or private replies how to move on. If that fails, it 1346might be appropriate to get a higher authority involved. In case of a WiFi 1347driver that would be the wireless maintainers; if there are no higher level 1348maintainers or all else fails, it might be one of those rare situations where 1349it's okay to get Linus Torvalds involved. 1350 1351**Proactive testing**: Every time the first pre-release (the 'rc1') of a new 1352mainline kernel version gets released, go and check if the issue is fixed there 1353or if anything of importance changed. Mention the outcome in the ticket or in a 1354mail you sent as reply to your report (make sure it has all those in the CC 1355that up to that point participated in the discussion). This will show your 1356commitment and that you are willing to help. It also tells developers if the 1357issue persists and makes sure they do not forget about it. A few other 1358occasional retests (for example with rc3, rc5 and the final) are also a good 1359idea, but only report your results if something relevant changed or if you are 1360writing something anyway. 1361 1362With all these general things off the table let's get into the details of how 1363to help to get issues resolved once they were reported. 1364 1365Inquires and testing request 1366~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1367 1368Here are your duties in case you got replies to your report: 1369 1370**Check who you deal with**: Most of the time it will be the maintainer or a 1371developer of the particular code area that will respond to your report. But as 1372issues are normally reported in public it could be anyone that's replying — 1373including people that want to help, but in the end might guide you totally off 1374track with their questions or requests. That rarely happens, but it's one of 1375many reasons why it's wise to quickly run an internet search to see who you're 1376interacting with. By doing this you also get aware if your report was heard by 1377the right people, as a reminder to the maintainer (see below) might be in order 1378later if discussion fades out without leading to a satisfying solution for the 1379issue. 1380 1381**Inquiries for data**: Often you will be asked to test something or provide 1382additional details. Try to provide the requested information soon, as you have 1383the attention of someone that might help and risk losing it the longer you 1384wait; that outcome is even likely if you do not provide the information within 1385a few business days. 1386 1387**Requests for testing**: When you are asked to test a diagnostic patch or a 1388possible fix, try to test it in timely manner, too. But do it properly and make 1389sure to not rush it: mixing things up can happen easily and can lead to a lot 1390of confusion for everyone involved. A common mistake for example is thinking a 1391proposed patch with a fix was applied, but in fact wasn't. Things like that 1392happen even to experienced testers occasionally, but they most of the time will 1393notice when the kernel with the fix behaves just as one without it. 1394 1395What to do when nothing of substance happens 1396~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1397 1398Some reports will not get any reaction from the responsible Linux kernel 1399developers; or a discussion around the issue evolved, but faded out with 1400nothing of substance coming out of it. 1401 1402In these cases wait two (better: three) weeks before sending a friendly 1403reminder: maybe the maintainer was just away from keyboard for a while when 1404your report arrived or had something more important to take care of. When 1405writing the reminder, kindly ask if anything else from your side is needed to 1406get the ball running somehow. If the report got out by mail, do that in the 1407first lines of a mail that is a reply to your initial mail (see above) which 1408includes a full quote of the original report below: that's on of those few 1409situations where such a 'TOFU' (Text Over, Fullquote Under) is the right 1410approach, as then all the recipients will have the details at hand immediately 1411in the proper order. 1412 1413After the reminder wait three more weeks for replies. If you still don't get a 1414proper reaction, you first should reconsider your approach. Did you maybe try 1415to reach out to the wrong people? Was the report maybe offensive or so 1416confusing that people decided to completely stay away from it? The best way to 1417rule out such factors: show the report to one or two people familiar with FLOSS 1418issue reporting and ask for their opinion. Also ask them for their advice how 1419to move forward. That might mean: prepare a better report and make those people 1420review it before you send it out. Such an approach is totally fine; just 1421mention that this is the second and improved report on the issue and include a 1422link to the first report. 1423 1424If the report was proper you can send a second reminder; in it ask for advice 1425why the report did not get any replies. A good moment for this second reminder 1426mail is shortly after the first pre-release (the 'rc1') of a new Linux kernel 1427version got published, as you should retest and provide a status update at that 1428point anyway (see above). 1429 1430If the second reminder again results in no reaction within a week, try to 1431contact a higher-level maintainer asking for advice: even busy maintainers by 1432then should at least have sent some kind of acknowledgment. 1433 1434Remember to prepare yourself for a disappointment: maintainers ideally should 1435react somehow to every issue report, but they are only obliged to fix those 1436'issues of high priority' outlined earlier. So don't be too devastating if you 1437get a reply along the lines of 'thanks for the report, I have more important 1438issues to deal with currently and won't have time to look into this for the 1439foreseeable future'. 1440 1441It's also possible that after some discussion in the bug tracker or on a list 1442nothing happens anymore and reminders don't help to motivate anyone to work out 1443a fix. Such situations can be devastating, but is within the cards when it 1444comes to Linux kernel development. This and several other reasons for not 1445getting help are explained in 'Why some issues won't get any reaction or remain 1446unfixed after being reported' near the end of this document. 1447 1448Don't get devastated if you don't find any help or if the issue in the end does 1449not get solved: the Linux kernel is FLOSS and thus you can still help yourself. 1450You for example could try to find others that are affected and team up with 1451them to get the issue resolved. Such a team could prepare a fresh report 1452together that mentions how many you are and why this is something that in your 1453option should get fixed. Maybe together you can also narrow down the root cause 1454or the change that introduced a regression, which often makes developing a fix 1455easier. And with a bit of luck there might be someone in the team that knows a 1456bit about programming and might be able to write a fix. 1457 1458 1459Reference for "Reporting regressions within a stable and longterm kernel line" 1460------------------------------------------------------------------------------ 1461 1462This subsection provides details for the steps you need to perform if you face 1463a regression within a stable and longterm kernel line. 1464 1465Make sure the particular version line still gets support 1466~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1467 1468 *Check if the kernel developers still maintain the Linux kernel version 1469 line you care about: go to the front page of kernel.org and make sure it 1470 mentions the latest release of the particular version line without an 1471 '[EOL]' tag.* 1472 1473Most kernel version lines only get supported for about three months, as 1474maintaining them longer is quite a lot of work. Hence, only one per year is 1475chosen and gets supported for at least two years (often six). That's why you 1476need to check if the kernel developers still support the version line you care 1477for. 1478 1479Note, if kernel.org lists two stable version lines on the front page, you 1480should consider switching to the newer one and forget about the older one: 1481support for it is likely to be abandoned soon. Then it will get a "end-of-life" 1482(EOL) stamp. Version lines that reached that point still get mentioned on the 1483kernel.org front page for a week or two, but are unsuitable for testing and 1484reporting. 1485 1486Search stable mailing list 1487~~~~~~~~~~~~~~~~~~~~~~~~~~ 1488 1489 *Check the archives of the Linux stable mailing list for existing reports.* 1490 1491Maybe the issue you face is already known and was fixed or is about to. Hence, 1492`search the archives of the Linux stable mailing list 1493<https://lore.kernel.org/stable/>`_ for reports about an issue like yours. If 1494you find any matches, consider joining the discussion, unless the fix is 1495already finished and scheduled to get applied soon. 1496 1497Reproduce issue with the newest release 1498~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1499 1500 *Install the latest release from the particular version line as a vanilla 1501 kernel. Ensure this kernel is not tainted and still shows the problem, as 1502 the issue might have already been fixed there. If you first noticed the 1503 problem with a vendor kernel, check a vanilla build of the last version 1504 known to work performs fine as well.* 1505 1506Before investing any more time in this process you want to check if the issue 1507was already fixed in the latest release of version line you're interested in. 1508This kernel needs to be vanilla and shouldn't be tainted before the issue 1509happens, as detailed outlined already above in the section "Install a fresh 1510kernel for testing". 1511 1512Did you first notice the regression with a vendor kernel? Then changes the 1513vendor applied might be interfering. You need to rule that out by performing 1514a recheck. Say something broke when you updated from 5.10.4-vendor.42 to 15155.10.5-vendor.43. Then after testing the latest 5.10 release as outlined in 1516the previous paragraph check if a vanilla build of Linux 5.10.4 works fine as 1517well. If things are broken there, the issue does not qualify as upstream 1518regression and you need switch back to the main step-by-step guide to report 1519the issue. 1520 1521Report the regression 1522~~~~~~~~~~~~~~~~~~~~~ 1523 1524 *Send a short problem report to the Linux stable mailing list 1525 (stable@vger.kernel.org). Roughly describe the issue and ideally explain 1526 how to reproduce it. Mention the first version that shows the problem and 1527 the last version that's working fine. Then wait for further instructions.* 1528 1529When reporting a regression that happens within a stable or longterm kernel 1530line (say when updating from 5.10.4 to 5.10.5) a brief report is enough for 1531the start to get the issue reported quickly. Hence a rough description is all 1532it takes. 1533 1534But note, it helps developers a great deal if you can specify the exact version 1535that introduced the problem. Hence if possible within a reasonable time frame, 1536try to find that version using vanilla kernels. Lets assume something broke when 1537your distributor released a update from Linux kernel 5.10.5 to 5.10.8. Then as 1538instructed above go and check the latest kernel from that version line, say 15395.10.9. If it shows the problem, try a vanilla 5.10.5 to ensure that no patches 1540the distributor applied interfere. If the issue doesn't manifest itself there, 1541try 5.10.7 and then (depending on the outcome) 5.10.8 or 5.10.6 to find the 1542first version where things broke. Mention it in the report and state that 5.10.9 1543is still broken. 1544 1545What the previous paragraph outlines is basically a rough manual 'bisection'. 1546Once your report is out your might get asked to do a proper one, as it allows to 1547pinpoint the exact change that causes the issue (which then can easily get 1548reverted to fix the issue quickly). Hence consider to do a proper bisection 1549right away if time permits. See the section 'Special care for regressions' and 1550the document 'Documentation/admin-guide/bug-bisect.rst' for details how to 1551perform one. 1552 1553 1554Reference for "Reporting issues only occurring in older kernel version lines" 1555----------------------------------------------------------------------------- 1556 1557This section provides details for the steps you need to take if you could not 1558reproduce your issue with a mainline kernel, but want to see it fixed in older 1559version lines (aka stable and longterm kernels). 1560 1561Some fixes are too complex 1562~~~~~~~~~~~~~~~~~~~~~~~~~~ 1563 1564 *Prepare yourself for the possibility that going through the next few steps 1565 might not get the issue solved in older releases: the fix might be too big 1566 or risky to get backported there.* 1567 1568Even small and seemingly obvious code-changes sometimes introduce new and 1569totally unexpected problems. The maintainers of the stable and longterm kernels 1570are very aware of that and thus only apply changes to these kernels that are 1571within rules outlined in 'Documentation/process/stable-kernel-rules.rst'. 1572 1573Complex or risky changes for example do not qualify and thus only get applied 1574to mainline. Other fixes are easy to get backported to the newest stable and 1575longterm kernels, but too risky to integrate into older ones. So be aware the 1576fix you are hoping for might be one of those that won't be backported to the 1577version line your care about. In that case you'll have no other choice then to 1578live with the issue or switch to a newer Linux version, unless you want to 1579patch the fix into your kernels yourself. 1580 1581Common preparations 1582~~~~~~~~~~~~~~~~~~~ 1583 1584 *Perform the first three steps in the section "Reporting issues only 1585 occurring in older kernel version lines" above.* 1586 1587You need to carry out a few steps already described in another section of this 1588guide. Those steps will let you: 1589 1590 * Check if the kernel developers still maintain the Linux kernel version line 1591 you care about. 1592 1593 * Search the Linux stable mailing list for exiting reports. 1594 1595 * Check with the latest release. 1596 1597 1598Check code history and search for existing discussions 1599~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1600 1601 *Search the Linux kernel version control system for the change that fixed 1602 the issue in mainline, as its commit message might tell you if the fix is 1603 scheduled for backporting already. If you don't find anything that way, 1604 search the appropriate mailing lists for posts that discuss such an issue 1605 or peer-review possible fixes; then check the discussions if the fix was 1606 deemed unsuitable for backporting. If backporting was not considered at 1607 all, join the newest discussion, asking if it's in the cards.* 1608 1609In a lot of cases the issue you deal with will have happened with mainline, but 1610got fixed there. The commit that fixed it would need to get backported as well 1611to get the issue solved. That's why you want to search for it or any 1612discussions abound it. 1613 1614 * First try to find the fix in the Git repository that holds the Linux kernel 1615 sources. You can do this with the web interfaces `on kernel.org 1616 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_ 1617 or its mirror `on GitHub <https://github.com/torvalds/linux>`_; if you have 1618 a local clone you alternatively can search on the command line with ``git 1619 log --grep=<pattern>``. 1620 1621 If you find the fix, look if the commit message near the end contains a 1622 'stable tag' that looks like this: 1623 1624 Cc: <stable@vger.kernel.org> # 5.4+ 1625 1626 If that's case the developer marked the fix safe for backporting to version 1627 line 5.4 and later. Most of the time it's getting applied there within two 1628 weeks, but sometimes it takes a bit longer. 1629 1630 * If the commit doesn't tell you anything or if you can't find the fix, look 1631 again for discussions about the issue. Search the net with your favorite 1632 internet search engine as well as the archives for the `Linux kernel 1633 developers mailing list <https://lore.kernel.org/lkml/>`_. Also read the 1634 section `Locate kernel area that causes the issue` above and follow the 1635 instructions to find the subsystem in question: its bug tracker or mailing 1636 list archive might have the answer you are looking for. 1637 1638 * If you see a proposed fix, search for it in the version control system as 1639 outlined above, as the commit might tell you if a backport can be expected. 1640 1641 * Check the discussions for any indicators the fix might be too risky to get 1642 backported to the version line you care about. If that's the case you have 1643 to live with the issue or switch to the kernel version line where the fix 1644 got applied. 1645 1646 * If the fix doesn't contain a stable tag and backporting was not discussed, 1647 join the discussion: mention the version where you face the issue and that 1648 you would like to see it fixed, if suitable. 1649 1650 1651Ask for advice 1652~~~~~~~~~~~~~~ 1653 1654 *One of the former steps should lead to a solution. If that doesn't work 1655 out, ask the maintainers for the subsystem that seems to be causing the 1656 issue for advice; CC the mailing list for the particular subsystem as well 1657 as the stable mailing list.* 1658 1659If the previous three steps didn't get you closer to a solution there is only 1660one option left: ask for advice. Do that in a mail you sent to the maintainers 1661for the subsystem where the issue seems to have its roots; CC the mailing list 1662for the subsystem as well as the stable mailing list (stable@vger.kernel.org). 1663 1664 1665Why some issues won't get any reaction or remain unfixed after being reported 1666============================================================================= 1667 1668When reporting a problem to the Linux developers, be aware only 'issues of high 1669priority' (regressions, security issues, severe problems) are definitely going 1670to get resolved. The maintainers or if all else fails Linus Torvalds himself 1671will make sure of that. They and the other kernel developers will fix a lot of 1672other issues as well. But be aware that sometimes they can't or won't help; and 1673sometimes there isn't even anyone to send a report to. 1674 1675This is best explained with kernel developers that contribute to the Linux 1676kernel in their spare time. Quite a few of the drivers in the kernel were 1677written by such programmers, often because they simply wanted to make their 1678hardware usable on their favorite operating system. 1679 1680These programmers most of the time will happily fix problems other people 1681report. But nobody can force them to do, as they are contributing voluntarily. 1682 1683Then there are situations where such developers really want to fix an issue, 1684but can't: sometimes they lack hardware programming documentation to do so. 1685This often happens when the publicly available docs are superficial or the 1686driver was written with the help of reverse engineering. 1687 1688Sooner or later spare time developers will also stop caring for the driver. 1689Maybe their test hardware broke, got replaced by something more fancy, or is so 1690old that it's something you don't find much outside of computer museums 1691anymore. Sometimes developer stops caring for their code and Linux at all, as 1692something different in their life became way more important. In some cases 1693nobody is willing to take over the job as maintainer – and nobody can be forced 1694to, as contributing to the Linux kernel is done on a voluntary basis. Abandoned 1695drivers nevertheless remain in the kernel: they are still useful for people and 1696removing would be a regression. 1697 1698The situation is not that different with developers that are paid for their 1699work on the Linux kernel. Those contribute most changes these days. But their 1700employers sooner or later also stop caring for their code or make its 1701programmer focus on other things. Hardware vendors for example earn their money 1702mainly by selling new hardware; quite a few of them hence are not investing 1703much time and energy in maintaining a Linux kernel driver for something they 1704stopped selling years ago. Enterprise Linux distributors often care for a 1705longer time period, but in new versions often leave support for old and rare 1706hardware aside to limit the scope. Often spare time contributors take over once 1707a company orphans some code, but as mentioned above: sooner or later they will 1708leave the code behind, too. 1709 1710Priorities are another reason why some issues are not fixed, as maintainers 1711quite often are forced to set those, as time to work on Linux is limited. 1712That's true for spare time or the time employers grant their developers to 1713spend on maintenance work on the upstream kernel. Sometimes maintainers also 1714get overwhelmed with reports, even if a driver is working nearly perfectly. To 1715not get completely stuck, the programmer thus might have no other choice than 1716to prioritize issue reports and reject some of them. 1717 1718But don't worry too much about all of this, a lot of drivers have active 1719maintainers who are quite interested in fixing as many issues as possible. 1720 1721 1722Closing words 1723============= 1724 1725Compared with other Free/Libre & Open Source Software it's hard to report 1726issues to the Linux kernel developers: the length and complexity of this 1727document and the implications between the lines illustrate that. But that's how 1728it is for now. The main author of this text hopes documenting the state of the 1729art will lay some groundwork to improve the situation over time. 1730 1731 1732.. 1733 This text is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If you 1734 spot a typo or small mistake, feel free to let him know directly and he'll 1735 fix it. You are free to do the same in a mostly informal way if you want 1736 to contribute changes to the text, but for copyright reasons please CC 1737 linux-doc@vger.kernel.org and "sign-off" your contribution as 1738 Documentation/process/submitting-patches.rst outlines in the section "Sign 1739 your work - the Developer's Certificate of Origin". 1740