1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) 2.. 3 If you want to distribute this text under CC-BY-4.0 only, please use 'The 4 Linux kernel developers' for author attribution and link this as source: 5 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-issues.rst 6.. 7 Note: Only the content of this RST file as found in the Linux kernel sources 8 is available under CC-BY-4.0, as versions of this text that were processed 9 (for example by the kernel's build system) might contain content taken from 10 files which use a more restrictive license. 11 12.. important:: 13 14 This document is being prepared to replace 15 'Documentation/admin-guide/reporting-bugs.rst'. The main work is done and 16 you are already free to follow its instructions when reporting issues to the 17 Linux kernel developers. But keep in mind, below text still needs a few 18 finishing touches and review. It was merged to the Linux kernel sources at 19 this stage to make this process easier and increase the text's visibility. 20 21 Any improvements for the text or other feedback is thus very much welcome. 22 Please send it to 'Thorsten Leemhuis <linux@leemhuis.info>' and 'Jonathan 23 Corbet <corbet@lwn.net>', ideally with 'Linux kernel mailing list (LKML) 24 <linux-kernel@vger.kernel.org>' and the 'Linux Kernel Documentation List 25 <linux-doc@vger.kernel.org>' in CC. 26 27 Areas in the text that still need work or discussion contain a hint like this 28 which point out the remaining issues; all of them start with the word "FIXME" 29 to make them easy to find. 30 31 32Reporting issues 33++++++++++++++++ 34 35 36The short guide (aka TL;DR) 37=========================== 38 39If you're facing multiple issues with the Linux kernel at once, report each 40separately to its developers. Try your best guess which kernel part might be 41causing the issue. Check the :ref:`MAINTAINERS <maintainers>` file for how its 42developers expect to be told about issues. Note, it's rarely 43`bugzilla.kernel.org <https://bugzilla.kernel.org/>`_, as in almost all cases 44the report needs to be sent by email! 45 46Check the destination thoroughly for existing reports; also search the LKML 47archives and the web. Join existing discussion if you find matches. If you 48don't find any, install `the latest Linux mainline kernel 49<https://kernel.org/>`_. Make sure it's vanilla, thus is not patched or using 50add-on kernel modules. Also ensure the kernel is running in a healthy 51environment and is not already tainted before the issue occurs. 52 53If you can reproduce your issue with the mainline kernel, send a report to the 54destination you determined earlier. Make sure it includes all relevant 55information, which in case of a regression should mention the change that's 56causing it which can often can be found with a bisection. Also ensure the 57report reaches all people that need to know about it, for example the security 58team, the stable maintainers or the developers of the patch that causes a 59regression. Once the report is out, answer any questions that might be raised 60and help where you can. That includes keeping the ball rolling: every time a 61new rc1 mainline kernel is released, check if the issue is still happening 62there and attach a status update to your initial report. 63 64If you can not reproduce the issue with the mainline kernel, consider sticking 65with it; if you'd like to use an older version line and want to see it fixed 66there, first make sure it's still supported. Install its latest release as 67vanilla kernel. If you cannot reproduce the issue there, try to find the commit 68that fixed it in mainline or any discussion preceding it: those will often 69mention if backporting is planed or considered too complex. If backporting was 70not discussed, ask if it's in the cards. In case you don't find any commits or 71a preceding discussion, see the Linux-stable mailing list archives for existing 72reports, as it might be a regression specific to the version line. If it is, 73report it like you would report a problem in mainline (including the 74bisection). 75 76If you reached this point without a solution, ask for advice one the 77subsystem's mailing list. 78 79 80Step-by-step guide how to report issues to the kernel maintainers 81================================================================= 82 83The above TL;DR outlines roughly how to report issues to the Linux kernel 84developers. It might be all that's needed for people already familiar with 85reporting issues to Free/Libre & Open Source Software (FLOSS) projects. For 86everyone else there is this section. It is more detailed and uses a 87step-by-step approach. It still tries to be brief for readability and leaves 88out a lot of details; those are described below the step-by-step guide in a 89reference section, which explains each of the steps in more detail. 90 91Note: this section covers a few more aspects than the TL;DR and does things in 92a slightly different order. That's in your interest, to make sure you notice 93early if an issue that looks like a Linux kernel problem is actually caused by 94something else. These steps thus help to ensure the time you invest in this 95process won't feel wasted in the end: 96 97 * Stop reading this document and report the problem to your vendor instead, 98 unless you are running the latest mainline kernel already or are willing to 99 install it. This kernel must not be modified or enhanced in any way, and 100 thus be considered 'vanilla'. 101 102 * See if the issue you are dealing with qualifies as regression, security 103 issue, or a really severe problem: those are 'issues of high priority' that 104 need special handling in some steps that are about to follow. 105 106 * Check if your kernel was 'tainted' when the issue occurred, as the event 107 that made the kernel set this flag might be causing the issue you face. 108 109 * Locate the driver or kernel subsystem that seems to be causing the issue. 110 Find out how and where its developers expect reports. Note: most of the 111 time this won't be bugzilla.kernel.org, as issues typically need to be sent 112 by mail to a maintainer and a public mailing list. 113 114 * Search the archives of the bug tracker or mailing list in question 115 thoroughly for reports that might match your issue. Also check if you find 116 something with your favorite internet search engine or in the Linux Kernel 117 Mailing List (LKML) archives. If you find anything, join the discussion 118 instead of sending a new report. 119 120 * Create a fresh backup and put system repair and restore tools at hand. 121 122 * Ensure your system does not enhance its kernels by building additional 123 kernel modules on-the-fly, which solutions like DKMS might be doing locally 124 without your knowledge. 125 126 * Make sure it's not the kernel's surroundings that are causing the issue 127 you face. 128 129 * Write down coarsely how to reproduce the issue. If you deal with multiple 130 issues at once, create separate notes for each of them and make sure they 131 work independently on a freshly booted system. That's needed, as each issue 132 needs to get reported to the kernel developers separately, unless they are 133 strongly entangled. 134 135After these preparations you'll now enter the main part: 136 137 * Install the latest Linux mainline kernel: that's where all issues get 138 fixed first, because it's the version line the kernel developers mainly 139 care about. Testing and reporting with the latest Linux stable kernel can 140 be an acceptable alternative in some situations, for example during the 141 merge window; but during that period you might want to suspend your efforts 142 till its end anyway. 143 144 * Ensure the kernel you just installed does not 'taint' itself when 145 running. 146 147 * Reproduce the issue with the kernel you just installed. If it doesn't show 148 up there, head over to the instructions for issues only happening with 149 stable and longterm kernels. 150 151 * Optimize your notes: try to find and write the most straightforward way to 152 reproduce your issue. Make sure the end result has all the important 153 details, and at the same time is easy to read and understand for others 154 that hear about it for the first time. And if you learned something in this 155 process, consider searching again for existing reports about the issue. 156 157 * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider 158 decoding the kernel log to find the line of code that triggered the error. 159 160 * If your problem is a regression, try to narrow down when the issue was 161 introduced as much as possible. 162 163 * Start to compile the report by writing a detailed description about the 164 issue. Always mention a few things: the latest kernel version you installed 165 for reproducing, the Linux Distribution used, and your notes on how to 166 reproduce the issue. Ideally, make the kernel's build configuration 167 (.config) and the output from ``dmesg`` available somewhere on the net and 168 link to it. Include or upload all other information that might be relevant, 169 like the output/screenshot of an Oops or the output from ``lspci``. Once 170 you wrote this main part, insert a normal length paragraph on top of it 171 outlining the issue and the impact quickly. On top of this add one sentence 172 that briefly describes the problem and gets people to read on. Now give the 173 thing a descriptive title or subject that yet again is shorter. Then you're 174 ready to send or file the report like the MAINTAINERS file told you, unless 175 you are dealing with one of those 'issues of high priority': they need 176 special care which is explained in 'Special handling for high priority 177 issues' below. 178 179 * Wait for reactions and keep the thing rolling until you can accept the 180 outcome in one way or the other. Thus react publicly and in a timely manner 181 to any inquiries. Test proposed fixes. Do proactive testing: retest with at 182 least every first release candidate (RC) of a new mainline version and 183 report your results. Send friendly reminders if things stall. And try to 184 help yourself, if you don't get any help or if it's unsatisfying. 185 186 187Reporting issues only occurring in older kernel version lines 188------------------------------------------------------------- 189 190This section is for you, if you tried the latest mainline kernel as outlined 191above, but failed to reproduce your issue there; at the same time you want to 192see the issue fixed in older version lines or a vendor kernel that's regularly 193rebased on new stable or longterm releases. If that case follow these steps: 194 195 * Prepare yourself for the possibility that going through the next few steps 196 might not get the issue solved in older releases: the fix might be too big 197 or risky to get backported there. 198 199 * Check if the kernel developers still maintain the Linux kernel version 200 line you care about: go to the front page of kernel.org and make sure it 201 mentions the latest release of the particular version line without an 202 '[EOL]' tag. 203 204 * Check the archives of the Linux stable mailing list for existing reports. 205 206 * Install the latest release from the particular version line as a vanilla 207 kernel. Ensure this kernel is not tainted and still shows the problem, as 208 the issue might have already been fixed there. 209 210 * Search the Linux kernel version control system for the change that fixed 211 the issue in mainline, as its commit message might tell you if the fix is 212 scheduled for backporting already. If you don't find anything that way, 213 search the appropriate mailing lists for posts that discuss such an issue 214 or peer-review possible fixes; then check the discussions if the fix was 215 deemed unsuitable for backporting. If backporting was not considered at 216 all, join the newest discussion, asking if it's in the cards. 217 218 * Check if you're dealing with a regression that was never present in 219 mainline by installing the first release of the version line you care 220 about. If the issue doesn't show up with it, you basically need to report 221 the issue with this version like you would report a problem with mainline 222 (see above). This ideally includes a bisection followed by a search for 223 existing reports on the net; with the help of the subject and the two 224 relevant commit-ids. If that doesn't turn up anything, write the report; CC 225 or forward the report to the stable maintainers, the stable mailing list, 226 and those who authored the change. Include the shortened commit-id if you 227 found the change that causes it. 228 229 * One of the former steps should lead to a solution. If that doesn't work 230 out, ask the maintainers for the subsystem that seems to be causing the 231 issue for advice; CC the mailing list for the particular subsystem as well 232 as the stable mailing list. 233 234 235Reference section: Reporting issues to the kernel maintainers 236============================================================= 237 238The detailed guides above outline all the major steps in brief fashion, which 239should be enough for most people. But sometimes there are situations where even 240experienced users might wonder how to actually do one of those steps. That's 241what this section is for, as it will provide a lot more details on each of the 242above steps. Consider this as reference documentation: it's possible to read it 243from top to bottom. But it's mainly meant to skim over and a place to look up 244details how to actually perform those steps. 245 246A few words of general advice before digging into the details: 247 248 * The Linux kernel developers are well aware this process is complicated and 249 demands more than other FLOSS projects. We'd love to make it simpler. But 250 that would require work in various places as well as some infrastructure, 251 which would need constant maintenance; nobody has stepped up to do that 252 work, so that's just how things are for now. 253 254 * A warranty or support contract with some vendor doesn't entitle you to 255 request fixes from developers in the upstream Linux kernel community: such 256 contracts are completely outside the scope of the Linux kernel, its 257 development community, and this document. That's why you can't demand 258 anything such a contract guarantees in this context, not even if the 259 developer handling the issue works for the vendor in question. If you want 260 to claim your rights, use the vendor's support channel instead. When doing 261 so, you might want to mention you'd like to see the issue fixed in the 262 upstream Linux kernel; motivate them by saying it's the only way to ensure 263 the fix in the end will get incorporated in all Linux distributions. 264 265 * If you never reported an issue to a FLOSS project before you should consider 266 reading `How to Report Bugs Effectively 267 <https://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_, `How To Ask 268 Questions The Smart Way 269 <http://www.catb.org/esr/faqs/smart-questions.html>`_, and `How to ask good 270 questions <https://jvns.ca/blog/good-questions/>`_. 271 272With that off the table, find below the details on how to properly report 273issues to the Linux kernel developers. 274 275 276Make sure you're using the upstream Linux kernel 277------------------------------------------------ 278 279 *Stop reading this document and report the problem to your vendor instead, 280 unless you are running the latest mainline kernel already or are willing to 281 install it. This kernel must not be modified or enhanced in any way, and 282 thus be considered 'vanilla'.* 283 284Like most programmers, Linux kernel developers don't like to spend time dealing 285with reports for issues that don't even happen with the source code they 286maintain: it's just a waste everybody's time, yours included. That's why you 287later will have to test your issue with the latest 'vanilla' kernel: a kernel 288that was build using the Linux sources taken straight from `kernel.org 289<https://kernel.org/>`_ and not modified or enhanced in any way. 290 291Almost all kernels used in devices (Computers, Laptops, Smartphones, Routers, 292…) and most kernels shipped by Linux distributors are ancient from the point of 293kernel development and heavily modified. They thus do not qualify for reporting 294an issue to the Linux kernel developers: the issue you face with such a kernel 295might be fixed already or caused by the changes or additions, even if they look 296small or totally unrelated. That's why issues with such kernels need to be 297reported to the vendor that distributed it. Its developers should look into the 298report and, in case it turns out to be an upstream issue, fix it directly 299upstream or report it there. In practice that sometimes does not work out. If 300that the case, you might want to circumvent the vendor by installing the latest 301mainline kernel yourself and reporting the issue as outlined in this document; 302just make sure to use really fresh kernel (see below). 303 304 305.. note:: 306 307 FIXME: Should we accept reports for issues with kernel images that are pretty 308 close to vanilla? But when are they close enough and how to put that line in 309 words? Maybe something like this? 310 311 *Note: Some Linux kernel developers accept reports from vendor kernels that 312 are known to be close to upstream. That for example is often the case for 313 the kernels that Debian GNU/Linux Sid or Fedora Rawhide ship, which are 314 normally following mainline closely and carry only a few patches. So a 315 report with one of these might be accepted by the developers that need to 316 handle it. But if they do, depends heavily on the individual developers and 317 the issue at hand. That's why installing a mainline vanilla kernel is the 318 safe bet.* 319 320 *Arch Linux, other Fedora releases, and openSUSE Tumbleweed often use quite 321 recent stable kernels that are pretty close to upstream, too. Some 322 developers accept bugs from them as well. But note that you normally should 323 avoid stable kernels for reporting issues and use a mainline kernel instead 324 (see below).* 325 326 Are there any other major Linux distributions that should be mentioned here? 327 328 329Issue of high priority? 330----------------------- 331 332 *See if the issue you are dealing with qualifies as regression, security 333 issue, or a really severe problem: those are 'issues of high priority' that 334 need special handling in some steps that are about to follow.* 335 336Linus Torvalds and the leading Linux kernel developers want to see some issues 337fixed as soon as possible, hence there are 'issues of high priority' that get 338handled slightly differently in the reporting process. Three type of cases 339qualify: regressions, security issues, and really severe problems. 340 341You deal with a 'regression' if something that worked with an older version of 342the Linux kernel does not work with a newer one or somehow works worse with it. 343It thus is a regression when a WiFi driver that did a fine job with Linux 5.7 344somehow misbehaves with 5.8 or doesn't work at all. It's also a regression if 345an application shows erratic behavior with a newer kernel, which might happen 346due to incompatible changes in the interface between the kernel and the 347userland (like procfs and sysfs). Significantly reduced performance or 348increased power consumption also qualify as regression. But keep in mind: the 349new kernel needs to be built with a configuration that is similar to the one 350from the old kernel (see below how to achieve that). That's because the kernel 351developers sometimes can not avoid incompatibilities when implementing new 352features; but to avoid regressions such features have to be enabled explicitly 353during build time configuration. 354 355What qualifies as security issue is left to your judgment. Consider reading 356'Documentation/admin-guide/security-bugs.rst' before proceeding, as it 357provides additional details how to best handle security issues. 358 359An issue is a 'really severe problem' when something totally unacceptably bad 360happens. That's for example the case when a Linux kernel corrupts the data it's 361handling or damages hardware it's running on. You're also dealing with a severe 362issue when the kernel suddenly stops working with an error message ('kernel 363panic') or without any farewell note at all. Note: do not confuse a 'panic' (a 364fatal error where the kernel stop itself) with a 'Oops' (a recoverable error), 365as the kernel remains running after the latter. 366 367 368Check 'taint' flag 369------------------ 370 371 *Check if your kernel was 'tainted' when the issue occurred, as the event 372 that made the kernel set this flag might be causing the issue you face.* 373 374The kernel marks itself with a 'taint' flag when something happens that might 375lead to follow-up errors that look totally unrelated. The issue you face might 376be such an error if your kernel is tainted. That's why it's in your interest to 377rule this out early before investing more time into this process. This is the 378only reason why this step is here, as this process later will tell you to 379install the latest mainline kernel; you will need to check the taint flag again 380then, as that's when it matters because it's the kernel the report will focus 381on. 382 383On a running system is easy to check if the kernel tainted itself: if ``cat 384/proc/sys/kernel/tainted`` returns '0' then the kernel is not tainted and 385everything is fine. Checking that file is impossible in some situations; that's 386why the kernel also mentions the taint status when it reports an internal 387problem (a 'kernel bug'), a recoverable error (a 'kernel Oops') or a 388non-recoverable error before halting operation (a 'kernel panic'). Look near 389the top of the error messages printed when one of these occurs and search for a 390line starting with 'CPU:'. It should end with 'Not tainted' if the kernel was 391not tainted when it noticed the problem; it was tainted if you see 'Tainted:' 392followed by a few spaces and some letters. 393 394If your kernel is tainted, study 'Documentation/admin-guide/tainted-kernels.rst' 395to find out why. Try to eliminate the reason. Often it's caused by one these 396three things: 397 398 1. A recoverable error (a 'kernel Oops') occurred and the kernel tainted 399 itself, as the kernel knows it might misbehave in strange ways after that 400 point. In that case check your kernel or system log and look for a section 401 that starts with this:: 402 403 Oops: 0000 [#1] SMP 404 405 That's the first Oops since boot-up, as the '#1' between the brackets shows. 406 Every Oops and any other problem that happens after that point might be a 407 follow-up problem to that first Oops, even if both look totally unrelated. 408 Rule this out by getting rid of the cause for the first Oops and reproducing 409 the issue afterwards. Sometimes simply restarting will be enough, sometimes 410 a change to the configuration followed by a reboot can eliminate the Oops. 411 But don't invest too much time into this at this point of the process, as 412 the cause for the Oops might already be fixed in the newer Linux kernel 413 version you are going to install later in this process. 414 415 2. Your system uses a software that installs its own kernel modules, for 416 example Nvidia's proprietary graphics driver or VirtualBox. The kernel 417 taints itself when it loads such module from external sources (even if 418 they are Open Source): they sometimes cause errors in unrelated kernel 419 areas and thus might be causing the issue you face. You therefore have to 420 prevent those modules from loading when you want to report an issue to the 421 Linux kernel developers. Most of the time the easiest way to do that is: 422 temporarily uninstall such software including any modules they might have 423 installed. Afterwards reboot. 424 425 3. The kernel also taints itself when it's loading a module that resides in 426 the staging tree of the Linux kernel source. That's a special area for 427 code (mostly drivers) that does not yet fulfill the normal Linux kernel 428 quality standards. When you report an issue with such a module it's 429 obviously okay if the kernel is tainted; just make sure the module in 430 question is the only reason for the taint. If the issue happens in an 431 unrelated area reboot and temporarily block the module from being loaded 432 by specifying ``foo.blacklist=1`` as kernel parameter (replace 'foo' with 433 the name of the module in question). 434 435 436Locate kernel area that causes the issue 437---------------------------------------- 438 439 *Locate the driver or kernel subsystem that seems to be causing the issue. 440 Find out how and where its developers expect reports. Note: most of the 441 time this won't be bugzilla.kernel.org, as issues typically need to be sent 442 by mail to a maintainer and a public mailing list.* 443 444It's crucial to send your report to the right people, as the Linux kernel is a 445big project and most of its developers are only familiar with a small subset of 446it. Quite a few programmers for example only care for just one driver, for 447example one for a WiFi chip; its developer likely will only have small or no 448knowledge about the internals of remote or unrelated "subsystems", like the TCP 449stack, the PCIe/PCI subsystem, memory management or file systems. 450 451Problem is: the Linux kernel lacks a central bug tracker where you can simply 452file your issue and make it reach the developers that need to know about it. 453That's why you have to find the right place and way to report issues yourself. 454You can do that with the help of a script (see below), but it mainly targets 455kernel developers and experts. For everybody else the MAINTAINERS file is the 456better place. 457 458How to read the MAINTAINERS file 459~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 460To illustrate how to use the :ref:`MAINTAINERS <maintainers>` file, lets assume 461the WiFi in your Laptop suddenly misbehaves after updating the kernel. In that 462case it's likely an issue in the WiFi driver. Obviously it could also be some 463code it builds upon, but unless you suspect something like that stick to the 464driver. If it's really something else, the driver's developers will get the 465right people involved. 466 467Sadly, there is no way to check which code is driving a particular hardware 468component that is both universal and easy. 469 470In case of a problem with the WiFi driver you for example might want to look at 471the output of ``lspci -k``, as it lists devices on the PCI/PCIe bus and the 472kernel module driving it:: 473 474 [user@something ~]$ lspci -k 475 [...] 476 3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32) 477 Subsystem: Bigfoot Networks, Inc. Device 1535 478 Kernel driver in use: ath10k_pci 479 Kernel modules: ath10k_pci 480 [...] 481 482But this approach won't work if your WiFi chip is connected over USB or some 483other internal bus. In those cases you might want to check your WiFi manager or 484the output of ``ip link``. Look for the name of the problematic network 485interface, which might be something like 'wlp58s0'. This name can be used like 486this to find the module driving it:: 487 488 [user@something ~]$ realpath --relative-to=/sys/module/ /sys/class/net/wlp58s0/device/driver/module 489 ath10k_pci 490 491In case tricks like these don't bring you any further, try to search the 492internet on how to narrow down the driver or subsystem in question. And if you 493are unsure which it is: just try your best guess, somebody will help you if you 494guessed poorly. 495 496Once you know the driver or subsystem, you want to search for it in the 497MAINTAINERS file. In the case of 'ath10k_pci' you won't find anything, as the 498name is too specific. Sometimes you will need to search on the net for help; 499but before doing so, try a somewhat shorted or modified name when searching the 500MAINTAINERS file, as then you might find something like this:: 501 502 QUALCOMM ATHEROS ATH10K WIRELESS DRIVER 503 Mail: A. Some Human <shuman@example.com> 504 Mailing list: ath10k@lists.infradead.org 505 Status: Supported 506 Web-page: https://wireless.wiki.kernel.org/en/users/Drivers/ath10k 507 SCM: git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git 508 Files: drivers/net/wireless/ath/ath10k/ 509 510Note: the line description will be abbreviations, if you read the plain 511MAINTAINERS file found in the root of the Linux source tree. 'Mail:' for 512example will be 'M:', 'Mailing list:' will be 'L', and 'Status:' will be 'S:'. 513A section near the top of the file explains these and other abbreviations. 514 515First look at the line 'Status'. Ideally it should be 'Supported' or 516'Maintained'. If it states 'Obsolete' then you are using some outdated approach 517that was replaced by a newer solution you need to switch to. Sometimes the code 518only has someone who provides 'Odd Fixes' when feeling motivated. And with 519'Orphan' you are totally out of luck, as nobody takes care of the code anymore. 520That only leaves these options: arrange yourself to live with the issue, fix it 521yourself, or find a programmer somewhere willing to fix it. 522 523After checking the status, look for a line starting with 'bugs:': it will tell 524you where to find a subsystem specific bug tracker to file your issue. The 525example above does not have such a line. That is the case for most sections, as 526Linux kernel development is completely driven by mail. Very few subsystems use 527a bug tracker, and only some of those rely on bugzilla.kernel.org. 528 529 530.. note:: 531 532 FIXME: The old text took a totally different approach to bugzilla.kernel.org, 533 as it mentions it as the place to file issue for people that don't known how 534 to contact the appropriate people. The new one mentions it rarely; and when 535 it does like here, it warns users that it's often the wrong place to go. 536 537 This approach was chosen as the main author of this document noticed quite a 538 few users (or even a lot?) get no reply to the bugs they file in bugzilla. 539 That's kind of expected, as quite a few (many? most?) of the maintainers 540 don't even get notified when reports for their subsystem get filed there. And 541 not getting a single reply to report is something that is just annoying for 542 users and might make them angry. Improving bugzilla.k.o would be an option, 543 but on the kernel and maintainers summit 2017 it was agreed on to first go 544 this route (sorry it took so long): it's easier to achieve and less 545 controversial, as putting additional burden on already overworked maintainers 546 is unlikely to get well received. 547 548 549In this and many other cases you thus have to look for lines starting with 550'Mail:' instead. Those mention the name and the email addresses for the 551maintainers of the particular code. Also look for a line starting with 'Mailing 552list:', which tells you the public mailing list where the code is developed. 553Your report later needs to go by mail to those addresses. Additionally, for all 554issue reports sent by email, make sure to add the Linux Kernel Mailing List 555(LKML) <linux-kernel@vger.kernel.org> to CC. Don't omit either of the mailing 556lists when sending your issue report by mail later! Maintainers are busy people 557and might leave some work for other developers on the subsystem specific list; 558and LKML is important to have one place where all issue reports can be found. 559 560 561.. note:: 562 563 FIXME: Above section tells users to always CC LKML. These days it's a kind of 564 "catch-all" list anyway, which nearly nobody seems to follow closely. So it 565 seems appropriate to go "all in" and make people send their reports here, 566 too, as everything (reports, fixes, ...) then can be found in one place (at 567 least for all reports sent by mail and all subsystems that CC LKML). 568 569 Related: Should we create mailing list like 'linux-issues@vger.kernel.org' 570 and tell users above to always CC it when reporting issues? Then there would 571 be one central place reporters could search for existing reports (at least 572 for issues reported by mail) without getting regular LKML traffic mixed into 573 the results. 574 575 576Finding the maintainers with the help of a script 577~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 578 579For people that have the Linux sources at hand there is a second option to find 580the proper place to report: the script 'scripts/get_maintainer.pl' which tries 581to find all people to contact. It queries the MAINTAINERS file and needs to be 582called with a path to the source code in question. For drivers compiled as 583module if often can be found with a command like this:: 584 585 $ modinfo ath10k_pci | grep filename | sed 's!/lib/modules/.*/kernel/!!; s!filename:!!; s!\.ko\(\|\.xz\)!!' 586 drivers/net/wireless/ath/ath10k/ath10k_pci.ko 587 588Pass parts of this to the script:: 589 590 $ ./scripts/get_maintainer.pl -f drivers/net/wireless/ath/ath10k* 591 Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER) 592 Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS) 593 ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER) 594 linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS)) 595 netdev@vger.kernel.org (open list:NETWORKING DRIVERS) 596 linux-kernel@vger.kernel.org (open list) 597 598Don't sent your report to all of them. Send it to the maintainers, which the 599script calls "supporter:"; additionally CC the most specific mailing list for 600the code as well as the Linux Kernel Mailing List (LKML). In this case you thus 601would need to send the report to 'Some Human <shuman@example.com>' with 602'ath10k@lists.infradead.org' and 'linux-kernel@vger.kernel.org' in CC. 603 604Note: in case you cloned the Linux sources with git you might want to call 605``get_maintainer.pl`` a second time with ``--git``. The script then will look 606at the commit history to find which people recently worked on the code in 607question, as they might be able to help. But use these results with care, as it 608can easily send you in a wrong direction. That for example happens quickly in 609areas rarely changed (like old or unmaintained drivers): sometimes such code is 610modified during tree-wide cleanups by developers that do not care about the 611particular driver at all. 612 613 614Search for existing reports 615--------------------------- 616 617 *Search the archives of the bug tracker or mailing list in question 618 thoroughly for reports that might match your issue. Also check if you find 619 something with your favorite internet search engine or in the Linux Kernel 620 Mailing List (LKML) archives. If you find anything, join the discussion 621 instead of sending a new report.* 622 623Reporting an issue that someone else already brought forward is often a waste 624of time for everyone involved, especially you as the reporter. So it's in your 625own interest to thoroughly check if somebody reported the issue already. Thus 626do not hurry with this step of the reporting process. Spending 30 to 60 minutes 627or even more time can save you and others quite a lot of time and trouble. 628 629The best place to search is the bug tracker or the mailing list where your 630report needs to be filed. You'll find quite a few of those lists on 631`lore.kernel.org <https://lore.kernel.org/>`_, but some are hosted in 632different places. That for example is the case for the ath10k WiFi driver used 633as example in the previous step. But you'll often find the archives for these 634lists easily on the net. Searching for 'archive ath10k@lists.infradead.org' for 635example will quickly lead you to the `Info page for the ath10k mailing list 636<https://lists.infradead.org/mailman/listinfo/ath10k>`_, which at the top links 637to its `list archives <https://lists.infradead.org/pipermail/ath10k/>`_. 638 639Sadly this and quite a few other lists miss a way to search the archives. In 640those cases use a regular internet search engine and add something like 641'site:lists.infradead.org/pipermail/ath10k/' to your search terms, which limits 642the results to the archives at that URL. 643 644Additionally, search the internet and the `Linux Kernel Mailing List (LKML) 645archives <https://lore.kernel.org/lkml/>`_, as maybe the real culprit might be 646in some other subsystem. Searching in `bugzilla.kernel.org 647<https://bugzilla.kernel.org/>`_ might also be a good idea, but if you find 648anything there keep in mind: most subsystems expect reports in different 649places, hence those you find there might have not even reached the people 650responsible for the subsystem in question. Nevertheless, the data there might 651provide valuable insights. 652 653If you get flooded with results consider telling your search engine to limit 654search timeframe to the past month or year. And wherever you search, make sure 655to use good search terms; vary them a few times, too. While doing so try to 656look at the issue from the perspective of someone else: that will help you to 657come up with other words to use as search terms. Also make sure not to use too 658many search terms at once. Remember to search with and without information like 659the name of the kernel driver or the name of the affected hardware component. 660But its exact brand name (say 'ASUS Red Devil Radeon RX 5700 XT Gaming OC') 661often is not much helpful, as it is too specific. Instead try search terms like 662the model line (Radeon 5700 or Radeon 5000) and the code name of the main chip 663('Navi' or 'Navi10') with and without its manufacturer ('AMD'). 664 665In case you find an existing report about your issue, join the discussion, as 666you might be able to provide valuable additional information. That can be 667important even when a fix is prepared or in its final stages already, as 668developers might look for people that can provide additional information or 669test a proposed fix. Jump to the section 'Duties after the report went out' for 670details on how to get properly involved. 671 672 673Prepare for emergencies 674----------------------- 675 676 *Create a fresh backup and put system repair and restore tools at hand.* 677 678Reminder, you are dealing with computers, which sometimes do unexpected things, 679especially if you fiddle with crucial parts like the kernel of its operating 680system. That's what you are about to do in this process. Thus, make sure to 681create a fresh backup; also ensure you have all tools at hand to repair or 682reinstall the operating system as well as everything you need to restore the 683backup. 684 685 686Make sure your kernel doesn't get enhanced 687------------------------------------------ 688 689 *Ensure your system does not enhance its kernels by building additional 690 kernel modules on-the-fly, which solutions like DKMS might be doing locally 691 without your knowledge.* 692 693Your kernel must be 'vanilla' when reporting an issue, but stops being pure as 694soon as it loads a kernel module not built from the sources used to compile the 695kernel image itself. That's why you need to ensure your Linux kernel stays 696vanilla by removing or disabling mechanisms like akmods and DKMS: those might 697build additional kernel modules automatically, for example when your boot into 698a newly installed Linux kernel the first time. Reboot after removing them and 699any modules they installed. 700 701Note, you might not be aware that your system is using one of these solutions: 702they often get set up silently when you install Nvidia's proprietary graphics 703driver, VirtualBox, or other software that requires a some support from a 704module not part of the Linux kernel. That why your might need to uninstall the 705packages with such software to get rid of any 3rd party kernel module. 706 707 708Ensure a healthy environment 709---------------------------- 710 711 *Make sure it's not the kernel's surroundings that are causing the issue 712 you face.* 713 714Problems that look a lot like a kernel issue are sometimes caused by build or 715runtime environment. It's hard to rule out that problem completely, but you 716should minimize it: 717 718 * Use proven tools when building your kernel, as bugs in the compiler or the 719 binutils can cause the resulting kernel to misbehave. 720 721 * Ensure your computer components run within their design specifications; 722 that's especially important for the main processor, the main memory, and the 723 motherboard. Therefore, stop undervolting or overclocking when facing a 724 potential kernel issue. 725 726 * Try to make sure it's not faulty hardware that is causing your issue. Bad 727 main memory for example can result in a multitude of issues that will 728 manifest itself in problems looking like kernel issues. 729 730 * If you're dealing with a filesystem issue, you might want to check the file 731 system in question with ``fsck``, as it might be damaged in a way that leads 732 to unexpected kernel behavior. 733 734 * When dealing with a regression, make sure it's not something else that 735 changed in parallel to updating the kernel. The problem for example might be 736 caused by other software that was updated at the same time. It can also 737 happen that a hardware component coincidentally just broke when you rebooted 738 into a new kernel for the first time. Updating the systems BIOS or changing 739 something in the BIOS Setup can also lead to problems that on look a lot 740 like a kernel regression. 741 742 743Document how to reproduce issue 744------------------------------- 745 746 *Write down coarsely how to reproduce the issue. If you deal with multiple 747 issues at once, create separate notes for each of them and make sure they 748 work independently on a freshly booted system. That's needed, as each issue 749 needs to get reported to the kernel developers separately, unless they are 750 strongly entangled.* 751 752If you deal with multiple issues at once, you'll have to report each of them 753separately, as they might be handled by different developers. Describing 754various issues in one report also makes it quite difficult for others to tear 755it apart. Hence, only combine issues in one report if they are very strongly 756entangled. 757 758Additionally, during the reporting process you will have to test if the issue 759happens with other kernel versions. Therefore, it will make your work easier if 760you know exactly how to reproduce an issue quickly on a freshly booted system. 761 762Note: it's often fruitless to report issues that only happened once, as they 763might be caused by a bit flip due to cosmic radiation. That's why you should 764try to rule that out by reproducing the issue before going further. Feel free 765to ignore this advice if you are experienced enough to tell a one-time error 766due to faulty hardware apart from a kernel issue that rarely happens and thus 767is hard to reproduce. 768 769 770Install a fresh kernel for testing 771---------------------------------- 772 773 *Install the latest Linux mainline kernel: that's where all issues get 774 fixed first, because it's the version line the kernel developers mainly 775 care about. Testing and reporting with the latest Linux stable kernel can 776 be an acceptable alternative in some situations, for example during the 777 merge window; but during that period you might want to suspend your efforts 778 till its end anyway.* 779 780Reporting an issue to the Linux kernel developers they fixed weeks or months 781ago is annoying for them and wasting their and your time. That's why it's in 782everybody's interest to check if the issue occurs with the latest codebase 783before reporting it. 784 785In the scope of the Linux kernel the term 'latest' means: a kernel version 786recently created from the main line of development, as this 'mainline' tree is 787where developers first apply fixes; only after that are they are allowed to get 788backported to older, still supported version lines called 'stable' and 789'longterm' kernels. That's why you should check a recent mainline kernel, even 790if you deal with an issue you only want to see fixed in an older version line. 791Another reason: some fixes are only applied to mainline or recent version 792lines, as it's too hard or risky to backport them to older versions. If that 793the case, reporting the issue again is unlikely to change anything. 794 795Longterm kernels (sometimes called "LTS kernels") are therefore unsuitable for 796testing; they simply are too distant from current development. Even the latest 797Linux 'stable' kernel is a significant bit behind and thus better avoided. At 798least most of the time, as sometimes a stable kernel can the best choice; but 799in those situations you might want to wait a few days anyway: 800 801Choosing between mainline, stable and waiting 802~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 803 804Head over to `kernel.org <https://kernel.org/>`_ to decide which version to 805use. Ignore the big yellow button that says 'Latest release' and look a little 806lower for a table. At its top you'll see a line starting with 'mainline', which 807most of the time will point to a pre-release with a version number like 808'5.8-rc2'. If that's the case, you'll want to use this mainline kernel for 809testing. Do not let that 'rc' scare you, these 'development kernels' are pretty 810reliable — and you made a backup, as you were instructed above, didn't you? 811 812In about two out of every nine to ten weeks, 'mainline' might point you to a 813proper release with a version number like '5.7'. If that happens, consider 814suspending the reporting process until the first pre-release of the next 815version (5.8-rc1) shows up on kernel.org. That's because the Linux development 816cycle then is in its two-week long 'merge window'. The bulk of the changes and 817all intrusive ones get merged for the next release during this time. It's a bit 818more risky to use mainline during this period. Kernel developers are also often 819quite busy then and might have no spare time to deal with issue reports. It's 820also quite possible that one of the many changes applied during the merge 821window fixes the issue you face; that's why you soon would have to retest with 822a newer kernel version anyway, as outlined below in the section 'Duties after 823the report went out'. 824 825That's why it might make sense to wait till the merge window is over. But don't 826to that if you're dealing with something that shouldn't wait. In that case 827consider obtaining the latest mainline kernel via git (see below) or use the 828latest stable version offered on kernel.org. Using that is also acceptable in 829case mainline for some reason does currently not work for you. An in general: 830using it for reproducing the issue is also better than not reporting it issue 831at all. 832 833How to obtain a fresh Linux kernel 834~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 835 836You can use pre-built or self-compiled kernel for testing; if you choose the 837latter approach, you can either obtain the source code using git or download it 838as tar archive. 839 840Using a pre-compiled kernel for testing is often the quickest, easiest, and 841safest way – especially is you are unfamiliar with the Linux kernel. But it 842needs to be a vanilla kernel, which can be hard to come buy. You are in luck if 843you are using a popular Linux distribution: for quite a few of them you'll find 844repositories on the net that contain packages with the latest mainline or 845stable kernels in vanilla fashion. It's totally okay to use these, just make 846sure from the repository's documentation they are really vanilla. And ensure 847the packages contain the latest versions as offered on kernel.org; they are 848likely unsuitable if the package is older than a week, as new mainline and 849stable kernels typically get released at least once a week. And be aware that 850you might need to get build your own kernel later anyway when it comes to 851helping test fixes, as described later in this document. 852 853Developers and experienced Linux users familiar with git are often best served 854by obtaining the latest Linux kernel sources straight from the `official 855development repository on kernel.org 856<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_. 857Those are likely a bit ahead of the latest mainline pre-release. Don't worry 858about it: they are as reliable as a proper pre-release, unless the kernel's 859development cycle is currently in the middle of a merge window. But even then 860they are quite reliable. 861 862People unfamiliar with git are often best served by downloading the sources as 863tarball from `kernel.org <https://kernel.org/>`_. 864 865How to actually build a kernel isnot described here, as many websites explain 866the necessary steps already. If you are new to it, consider following one of 867those how-to's that suggest to use ``make localmodconfig``, as that tries to 868pick up the configuration of your current kernel and then tries to adjust it 869somewhat for your system. That does not make the resulting kernel any better, 870but quicker to compile. 871 872Note: If you are dealing with a panic, Oops, warning, or BUG from the kernel, 873please try to enable CONFIG_KALLSYMS when configuring your kernel. 874Additionally, enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the 875latter is the relevant one of those two, but can only be reached if you enable 876the former. Be aware CONFIG_DEBUG_INFO increases the storage space required to 877build a kernel by quite a bit. But that's worth it, as these options will allow 878you later to pinpoint the exact line of code that triggers your issue. The 879section 'Decode failure messages' below explains this in more detail. 880 881But keep in mind: Always keep a record of the issue encountered in case it is 882hard to reproduce. Sending an undecoded report is better than not reporting 883the issue at all. 884 885 886Check 'taint' flag 887------------------ 888 889 *Ensure the kernel you just installed does not 'taint' itself when 890 running.* 891 892As outlined above in more detail already: the kernel sets a 'taint' flag when 893something happens that can lead to follow-up errors that look totally 894unrelated. That's why you need to check if the kernel you just installed does 895not set this flag. And if it does, you in almost all the cases needs to 896eliminate the reason for it before you reporting issues that occur with it. See 897the section above for details how to do that. 898 899 900Reproduce issue with the fresh kernel 901------------------------------------- 902 903 *Reproduce the issue with the kernel you just installed. If it doesn't show 904 up there, head over to the instructions for issues only happening with 905 stable and longterm kernels.* 906 907Check if the issue occurs with the fresh Linux kernel version you just 908installed. If it was fixed there already, consider sticking with this version 909line and abandoning your plan to report the issue. But keep in mind that other 910users might still be plagued by it, as long as it's not fixed in either stable 911and longterm version from kernel.org (and thus vendor kernels derived from 912those). If you prefer to use one of those or just want to help their users, 913head over to the section "Details about reporting issues only occurring in 914older kernel version lines" below. 915 916 917Optimize description to reproduce issue 918--------------------------------------- 919 920 *Optimize your notes: try to find and write the most straightforward way to 921 reproduce your issue. Make sure the end result has all the important 922 details, and at the same time is easy to read and understand for others 923 that hear about it for the first time. And if you learned something in this 924 process, consider searching again for existing reports about the issue.* 925 926An unnecessarily complex report will make it hard for others to understand your 927report. Thus try to find a reproducer that's straight forward to describe and 928thus easy to understand in written form. Include all important details, but at 929the same time try to keep it as short as possible. 930 931In this in the previous steps you likely have learned a thing or two about the 932issue you face. Use this knowledge and search again for existing reports 933instead you can join. 934 935 936Decode failure messages 937----------------------- 938 939 *If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider 940 decoding the kernel log to find the line of code that triggered the error.* 941 942When the kernel detects an internal problem, it will log some information about 943the executed code. This makes it possible to pinpoint the exact line in the 944source code that triggered the issue and shows how it was called. But that only 945works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring 946your kernel. If you did so, consider to decode the information from the 947kernel's log. That will make it a lot easier to understand what lead to the 948'panic', 'Oops', 'warning', or 'BUG', which increases the chances that someone 949can provide a fix. 950 951Decoding can be done with a script you find in the Linux source tree. If you 952are running a kernel you compiled yourself earlier, call it like this:: 953 954 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux 955 956If you are running a packaged vanilla kernel, you will likely have to install 957the corresponding packages with debug symbols. Then call the script (which you 958might need to get from the Linux sources if your distro does not package it) 959like this:: 960 961 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh \ 962 /usr/lib/debug/lib/modules/5.10.10-4.1.x86_64/vmlinux /usr/src/kernels/5.10.10-4.1.x86_64/ 963 964The script will work on log lines like the following, which show the address of 965the code the kernel was executing when the error occurred:: 966 967 [ 68.387301] RIP: 0010:test_module_init+0x5/0xffa [test_module] 968 969Once decoded, these lines will look like this:: 970 971 [ 68.387301] RIP: 0010:test_module_init (/home/username/linux-5.10.5/test-module/test-module.c:16) test_module 972 973In this case the executed code was built from the file 974'~/linux-5.10.5/test-module/test-module.c' and the error occurred by the 975instructions found in line '16'. 976 977The script will similarly decode the addresses mentioned in the section 978starting with 'Call trace', which show the path to the function where the 979problem occurred. Additionally, the script will show the assembler output for 980the code section the kernel was executing. 981 982Note, if you can't get this to work, simply skip this step and mention the 983reason for it in the report. If you're lucky, it might not be needed. And if it 984is, someone might help you to get things going. Also be aware this is just one 985of several ways to decode kernel stack traces. Sometimes different steps will 986be required to retrieve the relevant details. Don't worry about that, if that's 987needed in your case, developers will tell you what to do. 988 989 990Special care for regressions 991---------------------------- 992 993 *If your problem is a regression, try to narrow down when the issue was 994 introduced as much as possible.* 995 996Linux lead developer Linus Torvalds insists that the Linux kernel never 997worsens, that's why he deems regressions as unacceptable and wants to see them 998fixed quickly. That's why changes that introduced a regression are often 999promptly reverted if the issue they cause can't get solved quickly any other 1000way. Reporting a regression is thus a bit like playing a kind of trump card to 1001get something quickly fixed. But for that to happen the change that's causing 1002the regression needs to be known. Normally it's up to the reporter to track 1003down the culprit, as maintainers often won't have the time or setup at hand to 1004reproduce it themselves. 1005 1006To find the change there is a process called 'bisection' which the document 1007'Documentation/admin-guide/bug-bisect.rst' describes in detail. That process 1008will often require you to build about ten to twenty kernel images, trying to 1009reproduce the issue with each of them before building the next. Yes, that takes 1010some time, but don't worry, it works a lot quicker than most people assume. 1011Thanks to a 'binary search' this will lead you to the one commit in the source 1012code management system that's causing the regression. Once you find it, search 1013the net for the subject of the change, its commit id and the shortened commit id 1014(the first 12 characters of the commit id). This will lead you to existing 1015reports about it, if there are any. 1016 1017Note, a bisection needs a bit of know-how, which not everyone has, and quite a 1018bit of effort, which not everyone is willing to invest. Nevertheless, it's 1019highly recommended performing a bisection yourself. If you really can't or 1020don't want to go down that route at least find out which mainline kernel 1021introduced the regression. If something for example breaks when switching from 10225.5.15 to 5.8.4, then try at least all the mainline releases in that area (5.6, 10235.7 and 5.8) to check when it first showed up. Unless you're trying to find a 1024regression in a stable or longterm kernel, avoid testing versions which number 1025has three sections (5.6.12, 5.7.8), as that makes the outcome hard to 1026interpret, which might render your testing useless. Once you found the major 1027version which introduced the regression, feel free to move on in the reporting 1028process. But keep in mind: it depends on the issue at hand if the developers 1029will be able to help without knowing the culprit. Sometimes they might 1030recognize from the report want went wrong and can fix it; other times they will 1031be unable to help unless you perform a bisection. 1032 1033When dealing with regressions make sure the issue you face is really caused by 1034the kernel and not by something else, as outlined above already. 1035 1036In the whole process keep in mind: an issue only qualifies as regression if the 1037older and the newer kernel got built with a similar configuration. The best way 1038to archive this: copy the configuration file (``.config``) from the old working 1039kernel freshly to each newer kernel version you try. Afterwards run ``make 1040oldnoconfig`` to adjust it for the needs of the new version without enabling 1041any new feature, as those are allowed to cause regressions. 1042 1043 1044Write and send the report 1045------------------------- 1046 1047 *Start to compile the report by writing a detailed description about the 1048 issue. Always mention a few things: the latest kernel version you installed 1049 for reproducing, the Linux Distribution used, and your notes on how to 1050 reproduce the issue. Ideally, make the kernel's build configuration 1051 (.config) and the output from ``dmesg`` available somewhere on the net and 1052 link to it. Include or upload all other information that might be relevant, 1053 like the output/screenshot of an Oops or the output from ``lspci``. Once 1054 you wrote this main part, insert a normal length paragraph on top of it 1055 outlining the issue and the impact quickly. On top of this add one sentence 1056 that briefly describes the problem and gets people to read on. Now give the 1057 thing a descriptive title or subject that yet again is shorter. Then you're 1058 ready to send or file the report like the MAINTAINERS file told you, unless 1059 you are dealing with one of those 'issues of high priority': they need 1060 special care which is explained in 'Special handling for high priority 1061 issues' below.* 1062 1063Now that you have prepared everything it's time to write your report. How to do 1064that is partly explained by the three documents linked to in the preface above. 1065That's why this text will only mention a few of the essentials as well as 1066things specific to the Linux kernel. 1067 1068There is one thing that fits both categories: the most crucial parts of your 1069report are the title/subject, the first sentence, and the first paragraph. 1070Developers often get quite a lot of mail. They thus often just take a few 1071seconds to skim a mail before deciding to move on or look closer. Thus: the 1072better the top section of your report, the higher are the chances that someone 1073will look into it and help you. And that is why you should ignore them for now 1074and write the detailed report first. ;-) 1075 1076Things each report should mention 1077~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1078 1079Describe in detail how your issue happens with the fresh vanilla kernel you 1080installed. Try to include the step-by-step instructions you wrote and optimized 1081earlier that outline how you and ideally others can reproduce the issue; in 1082those rare cases where that's impossible try to describe what you did to 1083trigger it. 1084 1085Also include all the relevant information others might need to understand the 1086issue and its environment. What's actually needed depends a lot on the issue, 1087but there are some things you should include always: 1088 1089 * the output from ``cat /proc/version``, which contains the Linux kernel 1090 version number and the compiler it was built with. 1091 1092 * the Linux distribution the machine is running (``hostnamectl | grep 1093 "Operating System"``) 1094 1095 * the architecture of the CPU and the operating system (``uname -mi``) 1096 1097 * if you are dealing with a regression and performed a bisection, mention the 1098 subject and the commit-id of the change that is causing it. 1099 1100In a lot of cases it's also wise to make two more things available to those 1101that read your report: 1102 1103 * the configuration used for building your Linux kernel (the '.config' file) 1104 1105 * the kernel's messages that you get from ``dmesg`` written to a file. Make 1106 sure that it starts with a line like 'Linux version 5.8-1 1107 (foobar@example.com) (gcc (GCC) 10.2.1, GNU ld version 2.34) #1 SMP Mon Aug 1108 3 14:54:37 UTC 2020' If it's missing, then important messages from the first 1109 boot phase already got discarded. In this case instead consider using 1110 ``journalctl -b 0 -k``; alternatively you can also reboot, reproduce the 1111 issue and call ``dmesg`` right afterwards. 1112 1113These two files are big, that's why it's a bad idea to put them directly into 1114your report. If you are filing the issue in a bug tracker then attach them to 1115the ticket. If you report the issue by mail do not attach them, as that makes 1116the mail too large; instead do one of these things: 1117 1118 * Upload the files somewhere public (your website, a public file paste 1119 service, a ticket created just for this purpose on `bugzilla.kernel.org 1120 <https://bugzilla.kernel.org/>`_, ...) and include a link to them in your 1121 report. Ideally use something where the files stay available for years, as 1122 they could be useful to someone many years from now; this for example can 1123 happen if five or ten years from now a developer works on some code that was 1124 changed just to fix your issue. 1125 1126 * Put the files aside and mention you will send them later in individual 1127 replies to your own mail. Just remember to actually do that once the report 1128 went out. ;-) 1129 1130Things that might be wise to provide 1131~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1132 1133Depending on the issue you might need to add more background data. Here are a 1134few suggestions what often is good to provide: 1135 1136 * If you are dealing with a 'warning', an 'OOPS' or a 'panic' from the kernel, 1137 include it. If you can't copy'n'paste it, try to capture a netconsole trace 1138 or at least take a picture of the screen. 1139 1140 * If the issue might be related to your computer hardware, mention what kind 1141 of system you use. If you for example have problems with your graphics card, 1142 mention its manufacturer, the card's model, and what chip is uses. If it's a 1143 laptop mention its name, but try to make sure it's meaningful. 'Dell XPS 13' 1144 for example is not, because it might be the one from 2012; that one looks 1145 not that different from the one sold today, but apart from that the two have 1146 nothing in common. Hence, in such cases add the exact model number, which 1147 for example are '9380' or '7390' for XPS 13 models introduced during 2019. 1148 Names like 'Lenovo Thinkpad T590' are also somewhat ambiguous: there are 1149 variants of this laptop with and without a dedicated graphics chip, so try 1150 to find the exact model name or specify the main components. 1151 1152 * Mention the relevant software in use. If you have problems with loading 1153 modules, you want to mention the versions of kmod, systemd, and udev in use. 1154 If one of the DRM drivers misbehaves, you want to state the versions of 1155 libdrm and Mesa; also specify your Wayland compositor or the X-Server and 1156 its driver. If you have a filesystem issue, mention the version of 1157 corresponding filesystem utilities (e2fsprogs, btrfs-progs, xfsprogs, ...). 1158 1159 * Gather additional information from the kernel that might be of interest. The 1160 output from ``lspci -nn`` will for example help others to identify what 1161 hardware you use. If you have a problem with hardware you even might want to 1162 make the output from ``sudo lspci -vvv`` available, as that provides 1163 insights how the components were configured. For some issues it might be 1164 good to include the contents of files like ``/proc/cpuinfo``, 1165 ``/proc/ioports``, ``/proc/iomem``, ``/proc/modules``, or 1166 ``/proc/scsi/scsi``. Some subsystem also offer tools to collect relevant 1167 information. One such tool is ``alsa-info.sh`` `which the audio/sound 1168 subsystem developers provide <https://www.alsa-project.org/wiki/AlsaInfo>`_. 1169 1170Those examples should give your some ideas of what data might be wise to 1171attach, but you have to think yourself what will be helpful for others to know. 1172Don't worry too much about forgetting something, as developers will ask for 1173additional details they need. But making everything important available from 1174the start increases the chance someone will take a closer look. 1175 1176 1177The important part: the head of your report 1178~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1179 1180Now that you have the detailed part of the report prepared let's get to the 1181most important section: the first few sentences. Thus go to the top, add 1182something like 'The detailed description:' before the part you just wrote and 1183insert two newlines at the top. Now write one normal length paragraph that 1184describes the issue roughly. Leave out all boring details and focus on the 1185crucial parts readers need to know to understand what this is all about; if you 1186think this bug affects a lot of users, mention this to get people interested. 1187 1188Once you did that insert two more lines at the top and write a one sentence 1189summary that explains quickly what the report is about. After that you have to 1190get even more abstract and write an even shorter subject/title for the report. 1191 1192Now that you have written this part take some time to optimize it, as it is the 1193most important parts of your report: a lot of people will only read this before 1194they decide if reading the rest is time well spent. 1195 1196Now send or file the report like the :ref:`MAINTAINERS <maintainers>` file told 1197you, unless it's one of those 'issues of high priority' outlined earlier: in 1198that case please read the next subsection first before sending the report on 1199its way. 1200 1201Special handling for high priority issues 1202~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1203 1204Reports for high priority issues need special handling. 1205 1206**Severe bugs**: make sure the subject or ticket title as well as the first 1207paragraph makes the severeness obvious. 1208 1209**Regressions**: If the issue is a regression add [REGRESSION] to the mail's 1210subject or the title in the bug-tracker. If you did not perform a bisection 1211mention at least the latest mainline version you tested that worked fine (say 12125.7) and the oldest where the issue occurs (say 5.8). If you did a successful 1213bisection mention the commit id and subject of the change that causes the 1214regression. Also make sure to add the author of that change to your report; if 1215you need to file your bug in a bug-tracker forward the report to him in a 1216private mail and mention where your filed it. 1217 1218**Security issues**: for these issues your will have to evaluate if a 1219short-term risk to other users would arise if details were publicly disclosed. 1220If that's not the case simply proceed with reporting the issue as described. 1221For issues that bear such a risk you will need to adjust the reporting process 1222slightly: 1223 1224 * If the MAINTAINERS file instructed you to report the issue by mail, do not 1225 CC any public mailing lists. 1226 1227 * If you were supposed to file the issue in a bug tracker make sure to mark 1228 the ticket as 'private' or 'security issue'. If the bug tracker does not 1229 offer a way to keep reports private, forget about it and send your report as 1230 a private mail to the maintainers instead. 1231 1232In both cases make sure to also mail your report to the addresses the 1233MAINTAINERS file lists in the section 'security contact'. Ideally directly CC 1234them when sending the report by mail. If you filed it in a bug tracker, forward 1235the report's text to these addresses; but on top of it put a small note where 1236you mention that you filed it with a link to the ticket. 1237 1238See 'Documentation/admin-guide/security-bugs.rst' for more information. 1239 1240 1241Duties after the report went out 1242-------------------------------- 1243 1244 *Wait for reactions and keep the thing rolling until you can accept the 1245 outcome in one way or the other. Thus react publicly and in a timely manner 1246 to any inquiries. Test proposed fixes. Do proactive testing: retest with at 1247 least every first release candidate (RC) of a new mainline version and 1248 report your results. Send friendly reminders if things stall. And try to 1249 help yourself, if you don't get any help or if it's unsatisfying.* 1250 1251If your report was good and you are really lucky then one of the developers 1252might immediately spot what's causing the issue; they then might write a patch 1253to fix it, test it, and send it straight for integration in mainline while 1254tagging it for later backport to stable and longterm kernels that need it. Then 1255all you need to do is reply with a 'Thank you very much' and switch to a version 1256with the fix once it gets released. 1257 1258But this ideal scenario rarely happens. That's why the job is only starting 1259once you got the report out. What you'll have to do depends on the situations, 1260but often it will be the things listed below. But before digging into the 1261details, here are a few important things you need to keep in mind for this part 1262of the process. 1263 1264 1265General advice for further interactions 1266~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1267 1268**Always reply in public**: When you filed the issue in a bug tracker, always 1269reply there and do not contact any of the developers privately about it. For 1270mailed reports always use the 'Reply-all' function when replying to any mails 1271you receive. That includes mails with any additional data you might want to add 1272to your report: go to your mail applications 'Sent' folder and use 'reply-all' 1273on your mail with the report. This approach will make sure the public mailing 1274list(s) and everyone else that gets involved over time stays in the loop; it 1275also keeps the mail thread intact, which among others is really important for 1276mailing lists to group all related mails together. 1277 1278There are just two situations where a comment in a bug tracker or a 'Reply-all' 1279is unsuitable: 1280 1281 * Someone tells you to send something privately. 1282 1283 * You were told to send something, but noticed it contains sensitive 1284 information that needs to be kept private. In that case it's okay to send it 1285 in private to the developer that asked for it. But note in the ticket or a 1286 mail that you did that, so everyone else knows you honored the request. 1287 1288**Do research before asking for clarifications or help**: In this part of the 1289process someone might tell you to do something that requires a skill you might 1290not have mastered yet. For example, you might be asked to use some test tools 1291you never have heard of yet; or you might be asked to apply a patch to the 1292Linux kernel sources to test if it helps. In some cases it will be fine sending 1293a reply asking for instructions how to do that. But before going that route try 1294to find the answer own your own by searching the internet; alternatively 1295consider asking in other places for advice. For example ask a fried or post 1296about it to a chatroom or forum you normally hang out. 1297 1298**Be patient**: If you are really lucky you might get a reply to your report 1299within a few hours. But most of the time it will take longer, as maintainers 1300are scattered around the globe and thus might be in a different time zone – one 1301where they already enjoy their night away from keyboard. 1302 1303In general, kernel developers will take one to five business days to respond to 1304reports. Sometimes it will take longer, as they might be busy with the merge 1305windows, other work, visiting developer conferences, or simply enjoying a long 1306summer holiday. 1307 1308The 'issues of high priority' (see above for an explanation) are an exception 1309here: maintainers should address them as soon as possible; that's why you 1310should wait a week at maximum (or just two days if it's something urgent) 1311before sending a friendly reminder. 1312 1313Sometimes the maintainer might not be responding in a timely manner; other 1314times there might be disagreements, for example if an issue qualifies as 1315regression or not. In such cases raise your concerns on the mailing list and 1316ask others for public or private replies how to move on. If that fails, it 1317might be appropriate to get a higher authority involved. In case of a WiFi 1318driver that would be the wireless maintainers; if there are no higher level 1319maintainers or all else fails, it might be one of those rare situations where 1320it's okay to get Linus Torvalds involved. 1321 1322**Proactive testing**: Every time the first pre-release (the 'rc1') of a new 1323mainline kernel version gets released, go and check if the issue is fixed there 1324or if anything of importance changed. Mention the outcome in the ticket or in a 1325mail you sent as reply to your report (make sure it has all those in the CC 1326that up to that point participated in the discussion). This will show your 1327commitment and that you are willing to help. It also tells developers if the 1328issue persists and makes sure they do not forget about it. A few other 1329occasional retests (for example with rc3, rc5 and the final) are also a good 1330idea, but only report your results if something relevant changed or if you are 1331writing something anyway. 1332 1333With all these general things off the table let's get into the details of how 1334to help to get issues resolved once they were reported. 1335 1336Inquires and testing request 1337~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1338 1339Here are your duties in case you got replies to your report: 1340 1341**Check who you deal with**: Most of the time it will be the maintainer or a 1342developer of the particular code area that will respond to your report. But as 1343issues are normally reported in public it could be anyone that's replying — 1344including people that want to help, but in the end might guide you totally off 1345track with their questions or requests. That rarely happens, but it's one of 1346many reasons why it's wise to quickly run an internet search to see who you're 1347interacting with. By doing this you also get aware if your report was heard by 1348the right people, as a reminder to the maintainer (see below) might be in order 1349later if discussion fades out without leading to a satisfying solution for the 1350issue. 1351 1352**Inquiries for data**: Often you will be asked to test something or provide 1353additional details. Try to provide the requested information soon, as you have 1354the attention of someone that might help and risk losing it the longer you 1355wait; that outcome is even likely if you do not provide the information within 1356a few business days. 1357 1358**Requests for testing**: When you are asked to test a diagnostic patch or a 1359possible fix, try to test it in timely manner, too. But do it properly and make 1360sure to not rush it: mixing things up can happen easily and can lead to a lot 1361of confusion for everyone involved. A common mistake for example is thinking a 1362proposed patch with a fix was applied, but in fact wasn't. Things like that 1363happen even to experienced testers occasionally, but they most of the time will 1364notice when the kernel with the fix behaves just as one without it. 1365 1366What to do when nothing of substance happens 1367~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1368 1369Some reports will not get any reaction from the responsible Linux kernel 1370developers; or a discussion around the issue evolved, but faded out with 1371nothing of substance coming out of it. 1372 1373In these cases wait two (better: three) weeks before sending a friendly 1374reminder: maybe the maintainer was just away from keyboard for a while when 1375your report arrived or had something more important to take care of. When 1376writing the reminder, kindly ask if anything else from your side is needed to 1377get the ball running somehow. If the report got out by mail, do that in the 1378first lines of a mail that is a reply to your initial mail (see above) which 1379includes a full quote of the original report below: that's on of those few 1380situations where such a 'TOFU' (Text Over, Fullquote Under) is the right 1381approach, as then all the recipients will have the details at hand immediately 1382in the proper order. 1383 1384After the reminder wait three more weeks for replies. If you still don't get a 1385proper reaction, you first should reconsider your approach. Did you maybe try 1386to reach out to the wrong people? Was the report maybe offensive or so 1387confusing that people decided to completely stay away from it? The best way to 1388rule out such factors: show the report to one or two people familiar with FLOSS 1389issue reporting and ask for their opinion. Also ask them for their advice how 1390to move forward. That might mean: prepare a better report and make those people 1391review it before you send it out. Such an approach is totally fine; just 1392mention that this is the second and improved report on the issue and include a 1393link to the first report. 1394 1395If the report was proper you can send a second reminder; in it ask for advice 1396why the report did not get any replies. A good moment for this second reminder 1397mail is shortly after the first pre-release (the 'rc1') of a new Linux kernel 1398version got published, as you should retest and provide a status update at that 1399point anyway (see above). 1400 1401If the second reminder again results in no reaction within a week, try to 1402contact a higher-level maintainer asking for advice: even busy maintainers by 1403then should at least have sent some kind of acknowledgment. 1404 1405Remember to prepare yourself for a disappointment: maintainers ideally should 1406react somehow to every issue report, but they are only obliged to fix those 1407'issues of high priority' outlined earlier. So don't be too devastating if you 1408get a reply along the lines of 'thanks for the report, I have more important 1409issues to deal with currently and won't have time to look into this for the 1410foreseeable future'. 1411 1412It's also possible that after some discussion in the bug tracker or on a list 1413nothing happens anymore and reminders don't help to motivate anyone to work out 1414a fix. Such situations can be devastating, but is within the cards when it 1415comes to Linux kernel development. This and several other reasons for not 1416getting help are explained in 'Why some issues won't get any reaction or remain 1417unfixed after being reported' near the end of this document. 1418 1419Don't get devastated if you don't find any help or if the issue in the end does 1420not get solved: the Linux kernel is FLOSS and thus you can still help yourself. 1421You for example could try to find others that are affected and team up with 1422them to get the issue resolved. Such a team could prepare a fresh report 1423together that mentions how many you are and why this is something that in your 1424option should get fixed. Maybe together you can also narrow down the root cause 1425or the change that introduced a regression, which often makes developing a fix 1426easier. And with a bit of luck there might be someone in the team that knows a 1427bit about programming and might be able to write a fix. 1428 1429 1430Details about reporting issues only occurring in older kernel version lines 1431--------------------------------------------------------------------------- 1432 1433This subsection provides details for steps you need to take if you could not 1434reproduce your issue with a mainline kernel, but want to see it fixed in older 1435version lines (aka stable and longterm kernels). 1436 1437Some fixes are too complex 1438~~~~~~~~~~~~~~~~~~~~~~~~~~ 1439 1440 *Prepare yourself for the possibility that going through the next few steps 1441 might not get the issue solved in older releases: the fix might be too big 1442 or risky to get backported there.* 1443 1444Even small and seemingly obvious code-changes sometimes introduce new and 1445totally unexpected problems. The maintainers of the stable and longterm kernels 1446are very aware of that and thus only apply changes to these kernels that are 1447within rules outlined in 'Documentation/process/stable-kernel-rules.rst'. 1448 1449Complex or risky changes for example do not qualify and thus only get applied 1450to mainline. Other fixes are easy to get backported to the newest stable and 1451longterm kernels, but too risky to integrate into older ones. So be aware the 1452fix you are hoping for might be one of those that won't be backported to the 1453version line your care about. In that case you'll have no other choice then to 1454live with the issue or switch to a newer Linux version, unless you want to 1455patch the fix into your kernels yourself. 1456 1457Make sure the particular version line still gets support 1458~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1459 1460 *Check if the kernel developers still maintain the Linux kernel version 1461 line you care about: go to the front page of kernel.org and make sure it 1462 mentions the latest release of the particular version line without an 1463 '[EOL]' tag.* 1464 1465Most kernel version lines only get supported for about three months, as 1466maintaining them longer is quite a lot of work. Hence, only one per year is 1467chosen and gets supported for at least two years (often six). That's why you 1468need to check if the kernel developers still support the version line you care 1469for. 1470 1471Note, if kernel.org lists two 'stable' version lines on the front page, you 1472should consider switching to the newer one and forget about the older one: 1473support for it is likely to be abandoned soon. Then it will get a "end-of-life" 1474(EOL) stamp. Version lines that reached that point still get mentioned on the 1475kernel.org front page for a week or two, but are unsuitable for testing and 1476reporting. 1477 1478Search stable mailing list 1479~~~~~~~~~~~~~~~~~~~~~~~~~~ 1480 1481 *Check the archives of the Linux stable mailing list for existing reports.* 1482 1483Maybe the issue you face is already known and was fixed or is about to. Hence, 1484`search the archives of the Linux stable mailing list 1485<https://lore.kernel.org/stable/>`_ for reports about an issue like yours. If 1486you find any matches, consider joining the discussion, unless the fix is 1487already finished and scheduled to get applied soon. 1488 1489Reproduce issue with the newest release 1490~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1491 1492 *Install the latest release from the particular version line as a vanilla 1493 kernel. Ensure this kernel is not tainted and still shows the problem, as 1494 the issue might have already been fixed there.* 1495 1496Before investing any more time in this process you want to check if the issue 1497was already fixed in the latest release of version line you're interested in. 1498This kernel needs to be vanilla and shouldn't be tainted before the issue 1499happens, as detailed outlined already above in the process of testing mainline. 1500 1501Check code history and search for existing discussions 1502~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1503 1504 *Search the Linux kernel version control system for the change that fixed 1505 the issue in mainline, as its commit message might tell you if the fix is 1506 scheduled for backporting already. If you don't find anything that way, 1507 search the appropriate mailing lists for posts that discuss such an issue 1508 or peer-review possible fixes; then check the discussions if the fix was 1509 deemed unsuitable for backporting. If backporting was not considered at 1510 all, join the newest discussion, asking if it's in the cards.* 1511 1512In a lot of cases the issue you deal with will have happened with mainline, but 1513got fixed there. The commit that fixed it would need to get backported as well 1514to get the issue solved. That's why you want to search for it or any 1515discussions abound it. 1516 1517 * First try to find the fix in the Git repository that holds the Linux kernel 1518 sources. You can do this with the web interfaces `on kernel.org 1519 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_ 1520 or its mirror `on GitHub <https://github.com/torvalds/linux>`_; if you have 1521 a local clone you alternatively can search on the command line with ``git 1522 log --grep=<pattern>``. 1523 1524 If you find the fix, look if the commit message near the end contains a 1525 'stable tag' that looks like this: 1526 1527 Cc: <stable@vger.kernel.org> # 5.4+ 1528 1529 If that's case the developer marked the fix safe for backporting to version 1530 line 5.4 and later. Most of the time it's getting applied there within two 1531 weeks, but sometimes it takes a bit longer. 1532 1533 * If the commit doesn't tell you anything or if you can't find the fix, look 1534 again for discussions about the issue. Search the net with your favorite 1535 internet search engine as well as the archives for the `Linux kernel 1536 developers mailing list <https://lore.kernel.org/lkml/>`_. Also read the 1537 section `Locate kernel area that causes the issue` above and follow the 1538 instructions to find the subsystem in question: its bug tracker or mailing 1539 list archive might have the answer you are looking for. 1540 1541 * If you see a proposed fix, search for it in the version control system as 1542 outlined above, as the commit might tell you if a backport can be expected. 1543 1544 * Check the discussions for any indicators the fix might be too risky to get 1545 backported to the version line you care about. If that's the case you have 1546 to live with the issue or switch to the kernel version line where the fix 1547 got applied. 1548 1549 * If the fix doesn't contain a stable tag and backporting was not discussed, 1550 join the discussion: mention the version where you face the issue and that 1551 you would like to see it fixed, if suitable. 1552 1553Check if it's a regression specific to stable or longterm kernels 1554~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1555 1556 *Check if you're dealing with a regression that was never present in 1557 mainline by installing the first release of the version line you care 1558 about. If the issue doesn't show up with it, you basically need to report 1559 the issue with this version like you would report a problem with mainline 1560 (see above). This ideally includes a bisection followed by a search for 1561 existing reports on the net; with the help of the subject and the two 1562 relevant commit-ids. If that doesn't turn up anything, write the report; CC 1563 or forward the report to the stable maintainers, the stable mailing list, 1564 and those who authored the change. Include the shortened commit-id if you 1565 found the change that causes it.* 1566 1567Sometimes you won't find anything in the previous step: the issue you face 1568might have never occurred in mainline, as it is caused by some change that is 1569incomplete or not correctly applied. To check this, install the first release 1570from version line you care about, e.g., if you care about 5.4.x, install 5.4. 1571 1572If the issue doesn't show itself there, it's a regression specific to the 1573particular version line. In that case you need to report it like an issue 1574happening in mainline, like the last few steps in the main section in the above 1575outline. 1576 1577One of them suggests doing a bisection, which you are strongly advised to do in 1578this case. After finding the culprit, search the net for existing reports 1579again: not only search for the exact subject and the commit-id (proper and 1580shortened to twelve characters) of the change, but also for the commit-id 1581(proper and shortened) mentioned as 'Upstream commit' in the commit message. 1582 1583Write the report; just keep a few specialties in mind: CC or forward the report 1584to the stable maintainers, the stable mailing list, which the :ref:`MAINTAINERS 1585<maintainers>` file mentions in the section "STABLE BRANCH". If you performed a 1586successful bisection, CC the author of the change and include its subject and 1587the shortened commit-id. 1588 1589Ask for advice 1590~~~~~~~~~~~~~~ 1591 1592 *One of the former steps should lead to a solution. If that doesn't work 1593 out, ask the maintainers for the subsystem that seems to be causing the 1594 issue for advice; CC the mailing list for the particular subsystem as well 1595 as the stable mailing list.* 1596 1597If the previous three steps didn't get you closer to a solution there is only 1598one option left: ask for advice. Do that in a mail you sent to the maintainers 1599for the subsystem where the issue seems to have its roots; CC the mailing list 1600for the subsystem as well as the stable mailing list the :ref:`MAINTAINERS 1601<maintainers>` file mention in the section "STABLE BRANCH". 1602 1603 1604Why some issues won't get any reaction or remain unfixed after being reported 1605============================================================================= 1606 1607When reporting a problem to the Linux developers, be aware only 'issues of high 1608priority' (regressions, security issues, severe problems) are definitely going 1609to get resolved. The maintainers or if all else fails Linus Torvalds himself 1610will make sure of that. They and the other kernel developers will fix a lot of 1611other issues as well. But be aware that sometimes they can't or won't help; and 1612sometimes there isn't even anyone to send a report to. 1613 1614This is best explained with kernel developers that contribute to the Linux 1615kernel in their spare time. Quite a few of the drivers in the kernel were 1616written by such programmers, often because they simply wanted to make their 1617hardware usable on their favorite operating system. 1618 1619These programmers most of the time will happily fix problems other people 1620report. But nobody can force them to do, as they are contributing voluntarily. 1621 1622Then there are situations where such developers really want to fix an issue, 1623but can't: sometimes they lack hardware programming documentation to do so. 1624This often happens when the publicly available docs are superficial or the 1625driver was written with the help of reverse engineering. 1626 1627Sooner or later spare time developers will also stop caring for the driver. 1628Maybe their test hardware broke, got replaced by something more fancy, or is so 1629old that it's something you don't find much outside of computer museums 1630anymore. Sometimes developer stops caring for their code and Linux at all, as 1631something different in their life became way more important. In some cases 1632nobody is willing to take over the job as maintainer – and nobody can be forced 1633to, as contributing to the Linux kernel is done on a voluntary basis. Abandoned 1634drivers nevertheless remain in the kernel: they are still useful for people and 1635removing would be a regression. 1636 1637The situation is not that different with developers that are paid for their 1638work on the Linux kernel. Those contribute most changes these days. But their 1639employers sooner or later also stop caring for their code or make its 1640programmer focus on other things. Hardware vendors for example earn their money 1641mainly by selling new hardware; quite a few of them hence are not investing 1642much time and energy in maintaining a Linux kernel driver for something they 1643stopped selling years ago. Enterprise Linux distributors often care for a 1644longer time period, but in new versions often leave support for old and rare 1645hardware aside to limit the scope. Often spare time contributors take over once 1646a company orphans some code, but as mentioned above: sooner or later they will 1647leave the code behind, too. 1648 1649Priorities are another reason why some issues are not fixed, as maintainers 1650quite often are forced to set those, as time to work on Linux is limited. 1651That's true for spare time or the time employers grant their developers to 1652spend on maintenance work on the upstream kernel. Sometimes maintainers also 1653get overwhelmed with reports, even if a driver is working nearly perfectly. To 1654not get completely stuck, the programmer thus might have no other choice than 1655to prioritize issue reports and reject some of them. 1656 1657But don't worry too much about all of this, a lot of drivers have active 1658maintainers who are quite interested in fixing as many issues as possible. 1659 1660 1661Closing words 1662============= 1663 1664Compared with other Free/Libre & Open Source Software it's hard to report 1665issues to the Linux kernel developers: the length and complexity of this 1666document and the implications between the lines illustrate that. But that's how 1667it is for now. The main author of this text hopes documenting the state of the 1668art will lay some groundwork to improve the situation over time. 1669