1.. SPDX-License-Identifier: GPL-2.0 2.. include:: <isonum.txt> 3 4=========================================================== 5The PCI Express Advanced Error Reporting Driver Guide HOWTO 6=========================================================== 7 8:Authors: - T. Long Nguyen <tom.l.nguyen@intel.com> 9 - Yanmin Zhang <yanmin.zhang@intel.com> 10 11:Copyright: |copy| 2006 Intel Corporation 12 13Overview 14=========== 15 16About this guide 17---------------- 18 19This guide describes the basics of the PCI Express (PCIe) Advanced Error 20Reporting (AER) driver and provides information on how to use it, as 21well as how to enable the drivers of Endpoint devices to conform with 22the PCIe AER driver. 23 24 25What is the PCIe AER Driver? 26---------------------------- 27 28PCIe error signaling can occur on the PCIe link itself 29or on behalf of transactions initiated on the link. PCIe 30defines two error reporting paradigms: the baseline capability and 31the Advanced Error Reporting capability. The baseline capability is 32required of all PCIe components providing a minimum defined 33set of error reporting requirements. Advanced Error Reporting 34capability is implemented with a PCIe Advanced Error Reporting 35extended capability structure providing more robust error reporting. 36 37The PCIe AER driver provides the infrastructure to support PCIe Advanced 38Error Reporting capability. The PCIe AER driver provides three basic 39functions: 40 41 - Gathers the comprehensive error information if errors occurred. 42 - Reports error to the users. 43 - Performs error recovery actions. 44 45The AER driver only attaches to Root Ports and RCECs that support the PCIe 46AER capability. 47 48 49User Guide 50========== 51 52Include the PCIe AER Root Driver into the Linux Kernel 53------------------------------------------------------ 54 55The PCIe AER driver is a Root Port service driver attached 56via the PCIe Port Bus driver. If a user wants to use it, the driver 57must be compiled. It is enabled with CONFIG_PCIEAER, which 58depends on CONFIG_PCIEPORTBUS. 59 60Load PCIe AER Root Driver 61------------------------- 62 63Some systems have AER support in firmware. Enabling Linux AER support at 64the same time the firmware handles AER would result in unpredictable 65behavior. Therefore, Linux does not handle AER events unless the firmware 66grants AER control to the OS via the ACPI _OSC method. See the PCI Firmware 67Specification for details regarding _OSC usage. 68 69AER error output 70---------------- 71 72When a PCIe AER error is captured, an error message will be output to 73console. If it's a correctable error, it is output as an info message. 74Otherwise, it is printed as an error. So users could choose different 75log level to filter out correctable error messages. 76 77Below shows an example:: 78 79 0000:50:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, id=0500(Requester ID) 80 0000:50:00.0: device [8086:0329] error status/mask=00100000/00000000 81 0000:50:00.0: [20] Unsupported Request (First) 82 0000:50:00.0: TLP Header: 04000001 00200a03 05010000 00050100 83 84In the example, 'Requester ID' means the ID of the device that sent 85the error message to the Root Port. Please refer to PCIe specs for other 86fields. 87 88AER Statistics / Counters 89------------------------- 90 91When PCIe AER errors are captured, the counters / statistics are also exposed 92in the form of sysfs attributes which are documented at 93Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats 94 95Developer Guide 96=============== 97 98To enable error recovery, a software driver must provide callbacks. 99 100To support AER better, developers need to understand how AER works. 101 102PCIe errors are classified into two types: correctable errors 103and uncorrectable errors. This classification is based on the impact 104of those errors, which may result in degraded performance or function 105failure. 106 107Correctable errors pose no impacts on the functionality of the 108interface. The PCIe protocol can recover without any software 109intervention or any loss of data. These errors are detected and 110corrected by hardware. 111 112Unlike correctable errors, uncorrectable 113errors impact functionality of the interface. Uncorrectable errors 114can cause a particular transaction or a particular PCIe link 115to be unreliable. Depending on those error conditions, uncorrectable 116errors are further classified into non-fatal errors and fatal errors. 117Non-fatal errors cause the particular transaction to be unreliable, 118but the PCIe link itself is fully functional. Fatal errors, on 119the other hand, cause the link to be unreliable. 120 121When PCIe error reporting is enabled, a device will automatically send an 122error message to the Root Port above it when it captures 123an error. The Root Port, upon receiving an error reporting message, 124internally processes and logs the error message in its AER 125Capability structure. Error information being logged includes storing 126the error reporting agent's requestor ID into the Error Source 127Identification Registers and setting the error bits of the Root Error 128Status Register accordingly. If AER error reporting is enabled in the Root 129Error Command Register, the Root Port generates an interrupt when an 130error is detected. 131 132Note that the errors as described above are related to the PCIe 133hierarchy and links. These errors do not include any device specific 134errors because device specific errors will still get sent directly to 135the device driver. 136 137Provide callbacks 138----------------- 139 140callback reset_link to reset PCIe link 141~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 142 143This callback is used to reset the PCIe physical link when a 144fatal error happens. The Root Port AER service driver provides a 145default reset_link function, but different Upstream Ports might 146have different specifications to reset the PCIe link, so 147Upstream Port drivers may provide their own reset_link functions. 148 149Section 3.2.2.2 provides more detailed info on when to call 150reset_link. 151 152PCI error-recovery callbacks 153~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 154 155The PCIe AER Root driver uses error callbacks to coordinate 156with downstream device drivers associated with a hierarchy in question 157when performing error recovery actions. 158 159Data struct pci_driver has a pointer, err_handler, to point to 160pci_error_handlers who consists of a couple of callback function 161pointers. The AER driver follows the rules defined in 162pci-error-recovery.rst except PCIe-specific parts (e.g. 163reset_link). Please refer to pci-error-recovery.rst for detailed 164definitions of the callbacks. 165 166The sections below specify when to call the error callback functions. 167 168Correctable errors 169~~~~~~~~~~~~~~~~~~ 170 171Correctable errors pose no impacts on the functionality of 172the interface. The PCIe protocol can recover without any 173software intervention or any loss of data. These errors do not 174require any recovery actions. The AER driver clears the device's 175correctable error status register accordingly and logs these errors. 176 177Non-correctable (non-fatal and fatal) errors 178~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 179 180If an error message indicates a non-fatal error, performing link reset 181at upstream is not required. The AER driver calls error_detected(dev, 182pci_channel_io_normal) to all drivers associated within a hierarchy in 183question. For example:: 184 185 Endpoint <==> Downstream Port B <==> Upstream Port A <==> Root Port 186 187If Upstream Port A captures an AER error, the hierarchy consists of 188Downstream Port B and Endpoint. 189 190A driver may return PCI_ERS_RESULT_CAN_RECOVER, 191PCI_ERS_RESULT_DISCONNECT, or PCI_ERS_RESULT_NEED_RESET, depending on 192whether it can recover or the AER driver calls mmio_enabled as next. 193 194If an error message indicates a fatal error, kernel will broadcast 195error_detected(dev, pci_channel_io_frozen) to all drivers within 196a hierarchy in question. Then, performing link reset at upstream is 197necessary. As different kinds of devices might use different approaches 198to reset link, AER port service driver is required to provide the 199function to reset link via callback parameter of pcie_do_recovery() 200function. If reset_link is not NULL, recovery function will use it 201to reset the link. If error_detected returns PCI_ERS_RESULT_CAN_RECOVER 202and reset_link returns PCI_ERS_RESULT_RECOVERED, the error handling goes 203to mmio_enabled. 204 205Frequent Asked Questions 206------------------------ 207 208Q: 209 What happens if a PCIe device driver does not provide an 210 error recovery handler (pci_driver->err_handler is equal to NULL)? 211 212A: 213 The devices attached with the driver won't be recovered. If the 214 error is fatal, kernel will print out warning messages. Please refer 215 to section 3 for more information. 216 217Q: 218 What happens if an upstream port service driver does not provide 219 callback reset_link? 220 221A: 222 Fatal error recovery will fail if the errors are reported by the 223 upstream ports who are attached by the service driver. 224 225 226Software error injection 227======================== 228 229Debugging PCIe AER error recovery code is quite difficult because it 230is hard to trigger real hardware errors. Software based error 231injection can be used to fake various kinds of PCIe errors. 232 233First you should enable PCIe AER software error injection in kernel 234configuration, that is, following item should be in your .config. 235 236CONFIG_PCIEAER_INJECT=y or CONFIG_PCIEAER_INJECT=m 237 238After reboot with new kernel or insert the module, a device file named 239/dev/aer_inject should be created. 240 241Then, you need a user space tool named aer-inject, which can be gotten 242from: 243 244 https://github.com/intel/aer-inject.git 245 246More information about aer-inject can be found in the document in 247its source code. 248