1.. SPDX-License-Identifier: GPL-2.0+ 2 3===================================== 4Meta Platforms Host Network Interface 5===================================== 6 7Firmware Versions 8----------------- 9 10fbnic has three components stored on the flash which are provided in one PLDM 11image: 12 131. fw - The control firmware used to view and modify firmware settings, request 14 firmware actions, and retrieve firmware counters outside of the data path. 15 This is the firmware which fbnic_fw.c interacts with. 162. bootloader - The firmware which validate firmware security and control basic 17 operations including loading and updating the firmware. This is also known 18 as the cmrt firmware. 193. undi - This is the UEFI driver which is based on the Linux driver. 20 21fbnic stores two copies of these three components on flash. This allows fbnic 22to fall back to an older version of firmware automatically in case firmware 23fails to boot. Version information for both is provided as running and stored. 24The undi is only provided in stored as it is not actively running once the Linux 25driver takes over. 26 27devlink dev info provides version information for all three components. In 28addition to the version the hg commit hash of the build is included as a 29separate entry. 30 31Configuration 32------------- 33 34Ringparams (ethtool -g / -G) 35~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 36 37fbnic has two submission (host -> device) rings for every completion 38(device -> host) ring. The three ring objects together form a single 39"queue" as used by higher layer software (a Rx, or a Tx queue). 40 41For Rx the two submission rings are used to pass empty pages to the NIC. 42Ring 0 is the Header Page Queue (HPQ), NIC will use its pages to place 43L2-L4 headers (or full frames if frame is not header-data split). 44Ring 1 is the Payload Page Queue (PPQ) and used for packet payloads. 45The completion ring is used to receive packet notifications / metadata. 46ethtool ``rx`` ringparam maps to the size of the completion ring, 47``rx-mini`` to the HPQ, and ``rx-jumbo`` to the PPQ. 48 49For Tx both submission rings can be used to submit packets, the completion 50ring carries notifications for both. fbnic uses one of the submission 51rings for normal traffic from the stack and the second one for XDP frames. 52ethtool ``tx`` ringparam controls both the size of the submission rings 53and the completion ring. 54 55Every single entry on the HPQ and PPQ (``rx-mini``, ``rx-jumbo``) 56corresponds to 4kB of allocated memory, while entries on the remaining 57rings are in units of descriptors (8B). The ideal ratio of submission 58and completion ring sizes will depend on the workload, as for small packets 59multiple packets will fit into a single page. 60 61Upgrading Firmware 62------------------ 63 64fbnic supports updating firmware using signed PLDM images with devlink dev 65flash. PLDM images are written into the flash. Flashing does not interrupt 66the operation of the device. 67 68On host boot the latest UEFI driver is always used, no explicit activation 69is required. Firmware activation is required to run new control firmware. cmrt 70firmware can only be activated by power cycling the NIC. 71 72Statistics 73---------- 74 75TX MAC Interface 76~~~~~~~~~~~~~~~~ 77 78 - ``ptp_illegal_req``: packets sent to the NIC with PTP request bit set but routed to BMC/FW 79 - ``ptp_good_ts``: packets successfully routed to MAC with PTP request bit set 80 - ``ptp_bad_ts``: packets destined for MAC with PTP request bit set but aborted because of some error (e.g., DMA read error) 81 82TX Extension (TEI) Interface (TTI) 83~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 84 85 - ``tti_cm_drop``: control messages dropped at the TX Extension (TEI) Interface because of credit starvation 86 - ``tti_frame_drop``: packets dropped at the TX Extension (TEI) Interface because of credit starvation 87 - ``tti_tbi_drop``: packets dropped at the TX BMC Interface (TBI) because of credit starvation 88 89RXB (RX Buffer) Enqueue 90~~~~~~~~~~~~~~~~~~~~~~~ 91 92 - ``rxb_integrity_err[i]``: frames enqueued with integrity errors (e.g., multi-bit ECC errors) on RXB input i 93 - ``rxb_mac_err[i]``: frames enqueued with MAC end-of-frame errors (e.g., bad FCS) on RXB input i 94 - ``rxb_parser_err[i]``: frames experienced RPC parser errors 95 - ``rxb_frm_err[i]``: frames experienced signaling errors (e.g., missing end-of-packet/start-of-packet) on RXB input i 96 - ``rxb_drbo[i]_frames``: frames received at RXB input i 97 - ``rxb_drbo[i]_bytes``: bytes received at RXB input i 98 99RXB (RX Buffer) FIFO 100~~~~~~~~~~~~~~~~~~~~ 101 102 - ``rxb_fifo[i]_drop``: transitions into the drop state on RXB pool i 103 - ``rxb_fifo[i]_dropped_frames``: frames dropped on RXB pool i 104 - ``rxb_fifo[i]_ecn``: transitions into the ECN mark state on RXB pool i 105 - ``rxb_fifo[i]_level``: current occupancy of RXB pool i 106 107RXB (RX Buffer) Dequeue 108~~~~~~~~~~~~~~~~~~~~~~~ 109 110 - ``rxb_intf[i]_frames``: frames sent to the output i 111 - ``rxb_intf[i]_bytes``: bytes sent to the output i 112 - ``rxb_pbuf[i]_frames``: frames sent to output i from the perspective of internal packet buffer 113 - ``rxb_pbuf[i]_bytes``: bytes sent to output i from the perspective of internal packet buffer 114 115RPC (Rx parser) 116~~~~~~~~~~~~~~~ 117 118 - ``rpc_unkn_etype``: frames containing unknown EtherType 119 - ``rpc_unkn_ext_hdr``: frames containing unknown IPv6 extension header 120 - ``rpc_ipv4_frag``: frames containing IPv4 fragment 121 - ``rpc_ipv6_frag``: frames containing IPv6 fragment 122 - ``rpc_ipv4_esp``: frames with IPv4 ESP encapsulation 123 - ``rpc_ipv6_esp``: frames with IPv6 ESP encapsulation 124 - ``rpc_tcp_opt_err``: frames which encountered TCP option parsing error 125 - ``rpc_out_of_hdr_err``: frames where header was larger than parsable region 126 - ``ovr_size_err``: oversized frames 127 128Hardware Queues 129~~~~~~~~~~~~~~~ 130 1311. RX DMA Engine: 132 133 - ``rde_[i]_pkt_err``: packets with MAC EOP, RPC parser, RXB truncation, or RDE frame truncation errors. These error are flagged in the packet metadata because of cut-through support but the actual drop happens once PCIE/RDE is reached. 134 - ``rde_[i]_pkt_cq_drop``: packets dropped because RCQ is full 135 - ``rde_[i]_pkt_bdq_drop``: packets dropped because HPQ or PPQ ran out of host buffer 136 137PCIe 138~~~~ 139 140The fbnic driver exposes PCIe hardware performance statistics through debugfs 141(``pcie_stats``). These statistics provide insights into PCIe transaction 142behavior and potential performance bottlenecks. 143 1441. PCIe Transaction Counters: 145 146 These counters track PCIe transaction activity: 147 - ``pcie_ob_rd_tlp``: Outbound read Transaction Layer Packets count 148 - ``pcie_ob_rd_dword``: DWORDs transferred in outbound read transactions 149 - ``pcie_ob_wr_tlp``: Outbound write Transaction Layer Packets count 150 - ``pcie_ob_wr_dword``: DWORDs transferred in outbound write 151 transactions 152 - ``pcie_ob_cpl_tlp``: Outbound completion TLP count 153 - ``pcie_ob_cpl_dword``: DWORDs transferred in outbound completion TLPs 154 1552. PCIe Resource Monitoring: 156 157 These counters indicate PCIe resource exhaustion events: 158 - ``pcie_ob_rd_no_tag``: Read requests dropped due to tag unavailability 159 - ``pcie_ob_rd_no_cpl_cred``: Read requests dropped due to completion 160 credit exhaustion 161 - ``pcie_ob_rd_no_np_cred``: Read requests dropped due to non-posted 162 credit exhaustion 163