1.. SPDX-License-Identifier: GPL-2.0 2 3====================================== 4HiSilicon PCIe Tune and Trace device 5====================================== 6 7Introduction 8============ 9 10HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex 11integrated Endpoint (RCiEP) device, providing the capability 12to dynamically monitor and tune the PCIe link's events (tune), 13and trace the TLP headers (trace). The two functions are independent, 14but is recommended to use them together to analyze and enhance the 15PCIe link's performance. 16 17On Kunpeng 930 SoC, the PCIe Root Complex is composed of several 18PCIe cores. Each PCIe core includes several Root Ports and a PTT 19RCiEP, like below. The PTT device is capable of tuning and 20tracing the links of the PCIe core. 21:: 22 23 +--------------Core 0-------+ 24 | | [ PTT ] | 25 | | [Root Port]---[Endpoint] 26 | | [Root Port]---[Endpoint] 27 | | [Root Port]---[Endpoint] 28 Root Complex |------Core 1-------+ 29 | | [ PTT ] | 30 | | [Root Port]---[ Switch ]---[Endpoint] 31 | | [Root Port]---[Endpoint] `-[Endpoint] 32 | | [Root Port]---[Endpoint] 33 +---------------------------+ 34 35The PTT device driver registers one PMU device for each PTT device. 36The name of each PTT device is composed of 'hisi_ptt' prefix with 37the id of the SICL and the Core where it locates. The Kunpeng 930 38SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and 39IO dies (SICL, Super I/O Cluster), where there's one PCIe Root 40Complex for each SICL. 41:: 42 43 /sys/devices/hisi_ptt<sicl_id>_<core_id> 44 45Tune 46==== 47 48PTT tune is designed for monitoring and adjusting PCIe link parameters (events). 49Currently we support events in 2 classes. The scope of the events 50covers the PCIe core to which the PTT device belongs. 51 52Each event is presented as a file under $(PTT PMU dir)/tune, and 53a simple open/read/write/close cycle will be used to tune the event. 54:: 55 56 $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune 57 $ ls 58 qos_tx_cpl qos_tx_np qos_tx_p 59 tx_path_rx_req_alloc_buf_level 60 tx_path_tx_req_alloc_buf_level 61 $ cat qos_tx_dp 62 1 63 $ echo 2 > qos_tx_dp 64 $ cat qos_tx_dp 65 2 66 67Current value (numerical value) of the event can be simply read 68from the file, and the desired value written to the file to tune. 69 701. Tx Path QoS Control 71------------------------ 72 73The following files are provided to tune the QoS of the tx path of 74the PCIe core. 75 76- qos_tx_cpl: weight of Tx completion TLPs 77- qos_tx_np: weight of Tx non-posted TLPs 78- qos_tx_p: weight of Tx posted TLPs 79 80The weight influences the proportion of certain packets on the PCIe link. 81For example, for the storage scenario, increase the proportion 82of the completion packets on the link to enhance the performance as 83more completions are consumed. 84 85The available tune data of these events is [0, 1, 2]. 86Writing a negative value will return an error, and out of range 87values will be converted to 2. Note that the event value just 88indicates a probable level, but is not precise. 89 902. Tx Path Buffer Control 91------------------------- 92 93Following files are provided to tune the buffer of tx path of the PCIe core. 94 95- rx_alloc_buf_level: watermark of Rx requested 96- tx_alloc_buf_level: watermark of Tx requested 97 98These events influence the watermark of the buffer allocated for each 99type. Rx means the inbound while Tx means outbound. The packets will 100be stored in the buffer first and then transmitted either when the 101watermark reached or when timed out. For a busy direction, you should 102increase the related buffer watermark to avoid frequently posting and 103thus enhance the performance. In most cases just keep the default value. 104 105The available tune data of above events is [0, 1, 2]. 106Writing a negative value will return an error, and out of range 107values will be converted to 2. Note that the event value just 108indicates a probable level, but is not precise. 109 110Trace 111===== 112 113PTT trace is designed for dumping the TLP headers to the memory, which 114can be used to analyze the transactions and usage condition of the PCIe 115Link. You can choose to filter the traced headers by either Requester ID, 116or those downstream of a set of Root Ports on the same core of the PTT 117device. It's also supported to trace the headers of certain type and of 118certain direction. 119 120You can use the perf command `perf record` to set the parameters, start 121trace and get the data. It's also supported to decode the trace 122data with `perf report`. The control parameters for trace is inputted 123as event code for each events, which will be further illustrated later. 124An example usage is like 125:: 126 127 $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1, 128 format=1/ -- sleep 5 129 130This will trace the TLP headers downstream root port 0000:00:10.1 (event 131code for event 'filter' is 0x80001) with type of posted TLP requests, 132direction of inbound and traced data format of 8DW. 133 1341. Filter 135--------- 136 137The TLP headers to trace can be filtered by the Root Ports or the Requester ID 138of the Endpoint, which are located on the same core of the PTT device. You can 139set the filter by specifying the `filter` parameter which is required to start 140the trace. The parameter value is 20 bit. Bit 19 indicates the filter type. 1411 for Root Port filter and 0 for Requester filter. Bit[15:0] indicates the 142filter value. The value for a Root Port is a mask of the core port id which is 143calculated from its PCI Slot ID as (slotid & 7) * 2. The value for a Requester 144is the Requester ID (Device ID of the PCIe function). Bit[18:16] is currently 145reserved for extension. 146 147For example, if the desired filter is Endpoint function 0000:01:00.1 the filter 148value will be 0x00101. If the desired filter is Root Port 0000:00:10.0 then 149then filter value is calculated as 0x80001. 150 151Note that multiple Root Ports can be specified at one time, but only one 152Endpoint function can be specified in one trace. Specifying both Root Port 153and function at the same time is not supported. Driver maintains a list of 154available filters and will check the invalid inputs. 155 156Currently the available filters are detected in driver's probe. If the supported 157devices are removed/added after probe, you may need to reload the driver to update 158the filters. 159 1602. Type 161------- 162 163You can trace the TLP headers of certain types by specifying the `type` 164parameter, which is required to start the trace. The parameter value is 1658 bit. Current supported types and related values are shown below: 166 167- 8'b00000001: posted requests (P) 168- 8'b00000010: non-posted requests (NP) 169- 8'b00000100: completions (CPL) 170 171You can specify multiple types when tracing inbound TLP headers, but can only 172specify one when tracing outbound TLP headers. 173 1743. Direction 175------------ 176 177You can trace the TLP headers from certain direction, which is relative 178to the Root Port or the PCIe core, by specifying the `direction` parameter. 179This is optional and the default parameter is inbound. The parameter value 180is 4 bit. When the desired format is 4DW, directions and related values 181supported are shown below: 182 183- 4'b0000: inbound TLPs (P, NP, CPL) 184- 4'b0001: outbound TLPs (P, NP, CPL) 185- 4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B) 186- 4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A) 187 188When the desired format is 8DW, directions and related values supported are 189shown below: 190 191- 4'b0000: reserved 192- 4'b0001: outbound TLPs (P, NP, CPL) 193- 4'b0010: inbound TLPs (P, NP, CPL B) 194- 4'b0011: inbound TLPs (CPL A) 195 196Inbound completions are classified into two types: 197 198- completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B 199- completion B (CPL B): completion of DMA remote2local and P2P non-posted requests 200 2014. Format 202-------------- 203 204You can change the format of the traced TLP headers by specifying the 205`format` parameter. The default format is 4DW. The parameter value is 4 bit. 206Current supported formats and related values are shown below: 207 208- 4'b0000: 4DW length per TLP header 209- 4'b0001: 8DW length per TLP header 210 211The traced TLP header format is different from the PCIe standard. 212 213When using the 8DW data format, the entire TLP header is logged 214(Header DW0-3 shown below). For example, the TLP header for Memory 215Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17; 216the header for Configuration Requests is shown in Figure 2.20, etc. 217 218In addition, 8DW trace buffer entries contain a timestamp and 219possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0). 220Otherwise this field will be all 0. 221 222The bit[31:11] of DW0 is always 0x1fffff, which can be 223used to distinguish the data format. 8DW format is like 224:: 225 226 bits [ 31:11 ][ 10:0 ] 227 |---------------------------------------|-------------------| 228 DW0 [ 0x1fffff ][ Reserved (0x7ff) ] 229 DW1 [ Prefix ] 230 DW2 [ Header DW0 ] 231 DW3 [ Header DW1 ] 232 DW4 [ Header DW2 ] 233 DW5 [ Header DW3 ] 234 DW6 [ Reserved (0x0) ] 235 DW7 [ Time ] 236 237When using the 4DW data format, DW0 of the trace buffer entry 238contains selected fields of DW0 of the TLP, together with a 239timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3 240directly from the TLP header. 241 2424DW format is like 243:: 244 245 bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ] 246 |-----|---------|---|---|---|---|-------------|-------------| 247 DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ] 248 DW1 [ Header DW1 ] 249 DW2 [ Header DW2 ] 250 DW3 [ Header DW3 ] 251 2525. Memory Management 253-------------------- 254 255The traced TLP headers will be written to the memory allocated 256by the driver. The hardware accepts 4 DMA address with same size, 257and writes the buffer sequentially like below. If DMA addr 3 is 258finished and the trace is still on, it will return to addr 0. 259:: 260 261 +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+ 262 +---------------------------------------------------------+ 263 264Driver will allocate each DMA buffer of 4MiB. The finished buffer 265will be copied to the perf AUX buffer allocated by the perf core. 266Once the AUX buffer is full while the trace is still on, driver 267will commit the AUX buffer first and then apply for a new one with 268the same size. The size of AUX buffer is default to 16MiB. User can 269adjust the size by specifying the `-m` parameter of the perf command. 270 2716. Decoding 272----------- 273 274You can decode the traced data with `perf report -D` command (currently 275only support to dump the raw trace data). The traced data will be decoded 276according to the format described previously (take 8DW as an example): 277:: 278 279 [...perf headers and other information] 280 . ... HISI PTT data: size 4194304 bytes 281 . 00000000: 00 00 00 00 Prefix 282 . 00000004: 01 00 00 60 Header DW0 283 . 00000008: 0f 1e 00 01 Header DW1 284 . 0000000c: 04 00 00 00 Header DW2 285 . 00000010: 40 00 81 02 Header DW3 286 . 00000014: 33 c0 04 00 Time 287 . 00000020: 00 00 00 00 Prefix 288 . 00000024: 01 00 00 60 Header DW0 289 . 00000028: 0f 1e 00 01 Header DW1 290 . 0000002c: 04 00 00 00 Header DW2 291 . 00000030: 40 00 81 02 Header DW3 292 . 00000034: 02 00 00 00 Time 293 . 00000040: 00 00 00 00 Prefix 294 . 00000044: 01 00 00 60 Header DW0 295 . 00000048: 0f 1e 00 01 Header DW1 296 . 0000004c: 04 00 00 00 Header DW2 297 . 00000050: 40 00 81 02 Header DW3 298 [...] 299