1.. SPDX-License-Identifier: GPL-2.0 2 3============ 4Devlink Trap 5============ 6 7Background 8========== 9 10Devices capable of offloading the kernel's datapath and perform functions such 11as bridging and routing must also be able to send specific packets to the 12kernel (i.e., the CPU) for processing. 13 14For example, a device acting as a multicast-aware bridge must be able to send 15IGMP membership reports to the kernel for processing by the bridge module. 16Without processing such packets, the bridge module could never populate its 17MDB. 18 19As another example, consider a device acting as router which has received an IP 20packet with a TTL of 1. Upon routing the packet the device must send it to the 21kernel so that it will route it as well and generate an ICMP Time Exceeded 22error datagram. Without letting the kernel route such packets itself, utilities 23such as ``traceroute`` could never work. 24 25The fundamental ability of sending certain packets to the kernel for processing 26is called "packet trapping". 27 28Overview 29======== 30 31The ``devlink-trap`` mechanism allows capable device drivers to register their 32supported packet traps with ``devlink`` and report trapped packets to 33``devlink`` for further analysis. 34 35Upon receiving trapped packets, ``devlink`` will perform a per-trap packets and 36bytes accounting and potentially report the packet to user space via a netlink 37event along with all the provided metadata (e.g., trap reason, timestamp, input 38port). This is especially useful for drop traps (see :ref:`Trap-Types`) 39as it allows users to obtain further visibility into packet drops that would 40otherwise be invisible. 41 42The following diagram provides a general overview of ``devlink-trap``:: 43 44 Netlink event: Packet w/ metadata 45 Or a summary of recent drops 46 ^ 47 | 48 Userspace | 49 +---------------------------------------------------+ 50 Kernel | 51 | 52 +-------+--------+ 53 | | 54 | drop_monitor | 55 | | 56 +-------^--------+ 57 | 58 | Non-control traps 59 | 60 +----+----+ 61 | | Kernel's Rx path 62 | devlink | (non-drop traps) 63 | | 64 +----^----+ ^ 65 | | 66 +-----------+ 67 | 68 +-------+-------+ 69 | | 70 | Device driver | 71 | | 72 +-------^-------+ 73 Kernel | 74 +---------------------------------------------------+ 75 Hardware | 76 | Trapped packet 77 | 78 +--+---+ 79 | | 80 | ASIC | 81 | | 82 +------+ 83 84.. _Trap-Types: 85 86Trap Types 87========== 88 89The ``devlink-trap`` mechanism supports the following packet trap types: 90 91 * ``drop``: Trapped packets were dropped by the underlying device. Packets 92 are only processed by ``devlink`` and not injected to the kernel's Rx path. 93 The trap action (see :ref:`Trap-Actions`) can be changed. 94 * ``exception``: Trapped packets were not forwarded as intended by the 95 underlying device due to an exception (e.g., TTL error, missing neighbour 96 entry) and trapped to the control plane for resolution. Packets are 97 processed by ``devlink`` and injected to the kernel's Rx path. Changing the 98 action of such traps is not allowed, as it can easily break the control 99 plane. 100 * ``control``: Trapped packets were trapped by the device because these are 101 control packets required for the correct functioning of the control plane. 102 For example, ARP request and IGMP query packets. Packets are injected to 103 the kernel's Rx path, but not reported to the kernel's drop monitor. 104 Changing the action of such traps is not allowed, as it can easily break 105 the control plane. 106 107.. _Trap-Actions: 108 109Trap Actions 110============ 111 112The ``devlink-trap`` mechanism supports the following packet trap actions: 113 114 * ``trap``: The sole copy of the packet is sent to the CPU. 115 * ``drop``: The packet is dropped by the underlying device and a copy is not 116 sent to the CPU. 117 * ``mirror``: The packet is forwarded by the underlying device and a copy is 118 sent to the CPU. 119 120Generic Packet Traps 121==================== 122 123Generic packet traps are used to describe traps that trap well-defined packets 124or packets that are trapped due to well-defined conditions (e.g., TTL error). 125Such traps can be shared by multiple device drivers and their description must 126be added to the following table: 127 128.. list-table:: List of Generic Packet Traps 129 :widths: 5 5 90 130 131 * - Name 132 - Type 133 - Description 134 * - ``source_mac_is_multicast`` 135 - ``drop`` 136 - Traps incoming packets that the device decided to drop because of a 137 multicast source MAC 138 * - ``vlan_tag_mismatch`` 139 - ``drop`` 140 - Traps incoming packets that the device decided to drop in case of VLAN 141 tag mismatch: The ingress bridge port is not configured with a PVID and 142 the packet is untagged or prio-tagged 143 * - ``ingress_vlan_filter`` 144 - ``drop`` 145 - Traps incoming packets that the device decided to drop in case they are 146 tagged with a VLAN that is not configured on the ingress bridge port 147 * - ``ingress_spanning_tree_filter`` 148 - ``drop`` 149 - Traps incoming packets that the device decided to drop in case the STP 150 state of the ingress bridge port is not "forwarding" 151 * - ``port_list_is_empty`` 152 - ``drop`` 153 - Traps packets that the device decided to drop in case they need to be 154 flooded (e.g., unknown unicast, unregistered multicast) and there are 155 no ports the packets should be flooded to 156 * - ``port_loopback_filter`` 157 - ``drop`` 158 - Traps packets that the device decided to drop in case after layer 2 159 forwarding the only port from which they should be transmitted through 160 is the port from which they were received 161 * - ``blackhole_route`` 162 - ``drop`` 163 - Traps packets that the device decided to drop in case they hit a 164 blackhole route 165 * - ``ttl_value_is_too_small`` 166 - ``exception`` 167 - Traps unicast packets that should be forwarded by the device whose TTL 168 was decremented to 0 or less 169 * - ``tail_drop`` 170 - ``drop`` 171 - Traps packets that the device decided to drop because they could not be 172 enqueued to a transmission queue which is full 173 * - ``non_ip`` 174 - ``drop`` 175 - Traps packets that the device decided to drop because they need to 176 undergo a layer 3 lookup, but are not IP or MPLS packets 177 * - ``uc_dip_over_mc_dmac`` 178 - ``drop`` 179 - Traps packets that the device decided to drop because they need to be 180 routed and they have a unicast destination IP and a multicast destination 181 MAC 182 * - ``dip_is_loopback_address`` 183 - ``drop`` 184 - Traps packets that the device decided to drop because they need to be 185 routed and their destination IP is the loopback address (i.e., 127.0.0.0/8 186 and ::1/128) 187 * - ``sip_is_mc`` 188 - ``drop`` 189 - Traps packets that the device decided to drop because they need to be 190 routed and their source IP is multicast (i.e., 224.0.0.0/8 and ff::/8) 191 * - ``sip_is_loopback_address`` 192 - ``drop`` 193 - Traps packets that the device decided to drop because they need to be 194 routed and their source IP is the loopback address (i.e., 127.0.0.0/8 and ::1/128) 195 * - ``ip_header_corrupted`` 196 - ``drop`` 197 - Traps packets that the device decided to drop because they need to be 198 routed and their IP header is corrupted: wrong checksum, wrong IP version 199 or too short Internet Header Length (IHL) 200 * - ``ipv4_sip_is_limited_bc`` 201 - ``drop`` 202 - Traps packets that the device decided to drop because they need to be 203 routed and their source IP is limited broadcast (i.e., 255.255.255.255/32) 204 * - ``ipv6_mc_dip_reserved_scope`` 205 - ``drop`` 206 - Traps IPv6 packets that the device decided to drop because they need to 207 be routed and their IPv6 multicast destination IP has a reserved scope 208 (i.e., ffx0::/16) 209 * - ``ipv6_mc_dip_interface_local_scope`` 210 - ``drop`` 211 - Traps IPv6 packets that the device decided to drop because they need to 212 be routed and their IPv6 multicast destination IP has an interface-local scope 213 (i.e., ffx1::/16) 214 * - ``mtu_value_is_too_small`` 215 - ``exception`` 216 - Traps packets that should have been routed by the device, but were bigger 217 than the MTU of the egress interface 218 * - ``unresolved_neigh`` 219 - ``exception`` 220 - Traps packets that did not have a matching IP neighbour after routing 221 * - ``mc_reverse_path_forwarding`` 222 - ``exception`` 223 - Traps multicast IP packets that failed reverse-path forwarding (RPF) 224 check during multicast routing 225 * - ``reject_route`` 226 - ``exception`` 227 - Traps packets that hit reject routes (i.e., "unreachable", "prohibit") 228 * - ``ipv4_lpm_miss`` 229 - ``exception`` 230 - Traps unicast IPv4 packets that did not match any route 231 * - ``ipv6_lpm_miss`` 232 - ``exception`` 233 - Traps unicast IPv6 packets that did not match any route 234 * - ``non_routable_packet`` 235 - ``drop`` 236 - Traps packets that the device decided to drop because they are not 237 supposed to be routed. For example, IGMP queries can be flooded by the 238 device in layer 2 and reach the router. Such packets should not be 239 routed and instead dropped 240 * - ``decap_error`` 241 - ``exception`` 242 - Traps NVE and IPinIP packets that the device decided to drop because of 243 failure during decapsulation (e.g., packet being too short, reserved 244 bits set in VXLAN header) 245 * - ``overlay_smac_is_mc`` 246 - ``drop`` 247 - Traps NVE packets that the device decided to drop because their overlay 248 source MAC is multicast 249 * - ``ingress_flow_action_drop`` 250 - ``drop`` 251 - Traps packets dropped during processing of ingress flow action drop 252 * - ``egress_flow_action_drop`` 253 - ``drop`` 254 - Traps packets dropped during processing of egress flow action drop 255 * - ``stp`` 256 - ``control`` 257 - Traps STP packets 258 * - ``lacp`` 259 - ``control`` 260 - Traps LACP packets 261 * - ``lldp`` 262 - ``control`` 263 - Traps LLDP packets 264 * - ``igmp_query`` 265 - ``control`` 266 - Traps IGMP Membership Query packets 267 * - ``igmp_v1_report`` 268 - ``control`` 269 - Traps IGMP Version 1 Membership Report packets 270 * - ``igmp_v2_report`` 271 - ``control`` 272 - Traps IGMP Version 2 Membership Report packets 273 * - ``igmp_v3_report`` 274 - ``control`` 275 - Traps IGMP Version 3 Membership Report packets 276 * - ``igmp_v2_leave`` 277 - ``control`` 278 - Traps IGMP Version 2 Leave Group packets 279 * - ``mld_query`` 280 - ``control`` 281 - Traps MLD Multicast Listener Query packets 282 * - ``mld_v1_report`` 283 - ``control`` 284 - Traps MLD Version 1 Multicast Listener Report packets 285 * - ``mld_v2_report`` 286 - ``control`` 287 - Traps MLD Version 2 Multicast Listener Report packets 288 * - ``mld_v1_done`` 289 - ``control`` 290 - Traps MLD Version 1 Multicast Listener Done packets 291 * - ``ipv4_dhcp`` 292 - ``control`` 293 - Traps IPv4 DHCP packets 294 * - ``ipv6_dhcp`` 295 - ``control`` 296 - Traps IPv6 DHCP packets 297 * - ``arp_request`` 298 - ``control`` 299 - Traps ARP request packets 300 * - ``arp_response`` 301 - ``control`` 302 - Traps ARP response packets 303 * - ``arp_overlay`` 304 - ``control`` 305 - Traps NVE-decapsulated ARP packets that reached the overlay network. 306 This is required, for example, when the address that needs to be 307 resolved is a local address 308 * - ``ipv6_neigh_solicit`` 309 - ``control`` 310 - Traps IPv6 Neighbour Solicitation packets 311 * - ``ipv6_neigh_advert`` 312 - ``control`` 313 - Traps IPv6 Neighbour Advertisement packets 314 * - ``ipv4_bfd`` 315 - ``control`` 316 - Traps IPv4 BFD packets 317 * - ``ipv6_bfd`` 318 - ``control`` 319 - Traps IPv6 BFD packets 320 * - ``ipv4_ospf`` 321 - ``control`` 322 - Traps IPv4 OSPF packets 323 * - ``ipv6_ospf`` 324 - ``control`` 325 - Traps IPv6 OSPF packets 326 * - ``ipv4_bgp`` 327 - ``control`` 328 - Traps IPv4 BGP packets 329 * - ``ipv6_bgp`` 330 - ``control`` 331 - Traps IPv6 BGP packets 332 * - ``ipv4_vrrp`` 333 - ``control`` 334 - Traps IPv4 VRRP packets 335 * - ``ipv6_vrrp`` 336 - ``control`` 337 - Traps IPv6 VRRP packets 338 * - ``ipv4_pim`` 339 - ``control`` 340 - Traps IPv4 PIM packets 341 * - ``ipv6_pim`` 342 - ``control`` 343 - Traps IPv6 PIM packets 344 * - ``uc_loopback`` 345 - ``control`` 346 - Traps unicast packets that need to be routed through the same layer 3 347 interface from which they were received. Such packets are routed by the 348 kernel, but also cause it to potentially generate ICMP redirect packets 349 * - ``local_route`` 350 - ``control`` 351 - Traps unicast packets that hit a local route and need to be locally 352 delivered 353 * - ``external_route`` 354 - ``control`` 355 - Traps packets that should be routed through an external interface (e.g., 356 management interface) that does not belong to the same device (e.g., 357 switch ASIC) as the ingress interface 358 * - ``ipv6_uc_dip_link_local_scope`` 359 - ``control`` 360 - Traps unicast IPv6 packets that need to be routed and have a destination 361 IP address with a link-local scope (i.e., fe80::/10). The trap allows 362 device drivers to avoid programming link-local routes, but still receive 363 packets for local delivery 364 * - ``ipv6_dip_all_nodes`` 365 - ``control`` 366 - Traps IPv6 packets that their destination IP address is the "All Nodes 367 Address" (i.e., ff02::1) 368 * - ``ipv6_dip_all_routers`` 369 - ``control`` 370 - Traps IPv6 packets that their destination IP address is the "All Routers 371 Address" (i.e., ff02::2) 372 * - ``ipv6_router_solicit`` 373 - ``control`` 374 - Traps IPv6 Router Solicitation packets 375 * - ``ipv6_router_advert`` 376 - ``control`` 377 - Traps IPv6 Router Advertisement packets 378 * - ``ipv6_redirect`` 379 - ``control`` 380 - Traps IPv6 Redirect Message packets 381 * - ``ipv4_router_alert`` 382 - ``control`` 383 - Traps IPv4 packets that need to be routed and include the Router Alert 384 option. Such packets need to be locally delivered to raw sockets that 385 have the IP_ROUTER_ALERT socket option set 386 * - ``ipv6_router_alert`` 387 - ``control`` 388 - Traps IPv6 packets that need to be routed and include the Router Alert 389 option in their Hop-by-Hop extension header. Such packets need to be 390 locally delivered to raw sockets that have the IPV6_ROUTER_ALERT socket 391 option set 392 * - ``ptp_event`` 393 - ``control`` 394 - Traps PTP time-critical event messages (Sync, Delay_req, Pdelay_Req and 395 Pdelay_Resp) 396 * - ``ptp_general`` 397 - ``control`` 398 - Traps PTP general messages (Announce, Follow_Up, Delay_Resp, 399 Pdelay_Resp_Follow_Up, management and signaling) 400 * - ``flow_action_sample`` 401 - ``control`` 402 - Traps packets sampled during processing of flow action sample (e.g., via 403 tc's sample action) 404 * - ``flow_action_trap`` 405 - ``control`` 406 - Traps packets logged during processing of flow action trap (e.g., via 407 tc's trap action) 408 * - ``early_drop`` 409 - ``drop`` 410 - Traps packets dropped due to the RED (Random Early Detection) algorithm 411 (i.e., early drops) 412 * - ``vxlan_parsing`` 413 - ``drop`` 414 - Traps packets dropped due to an error in the VXLAN header parsing which 415 might be because of packet truncation or the I flag is not set. 416 * - ``llc_snap_parsing`` 417 - ``drop`` 418 - Traps packets dropped due to an error in the LLC+SNAP header parsing 419 * - ``vlan_parsing`` 420 - ``drop`` 421 - Traps packets dropped due to an error in the VLAN header parsing. Could 422 include unexpected packet truncation. 423 * - ``pppoe_ppp_parsing`` 424 - ``drop`` 425 - Traps packets dropped due to an error in the PPPoE+PPP header parsing. 426 This could include finding a session ID of 0xFFFF (which is reserved and 427 not for use), a PPPoE length which is larger than the frame received or 428 any common error on this type of header 429 * - ``mpls_parsing`` 430 - ``drop`` 431 - Traps packets dropped due to an error in the MPLS header parsing which 432 could include unexpected header truncation 433 * - ``arp_parsing`` 434 - ``drop`` 435 - Traps packets dropped due to an error in the ARP header parsing 436 * - ``ip_1_parsing`` 437 - ``drop`` 438 - Traps packets dropped due to an error in the first IP header parsing. 439 This packet trap could include packets which do not pass an IP checksum 440 check, a header length check (a minimum of 20 bytes), which might suffer 441 from packet truncation thus the total length field exceeds the received 442 packet length etc 443 * - ``ip_n_parsing`` 444 - ``drop`` 445 - Traps packets dropped due to an error in the parsing of the last IP 446 header (the inner one in case of an IP over IP tunnel). The same common 447 error checking is performed here as for the ip_1_parsing trap 448 * - ``gre_parsing`` 449 - ``drop`` 450 - Traps packets dropped due to an error in the GRE header parsing 451 * - ``udp_parsing`` 452 - ``drop`` 453 - Traps packets dropped due to an error in the UDP header parsing. 454 This packet trap could include checksum errorrs, an improper UDP 455 length detected (smaller than 8 bytes) or detection of header 456 truncation. 457 * - ``tcp_parsing`` 458 - ``drop`` 459 - Traps packets dropped due to an error in the TCP header parsing. 460 This could include TCP checksum errors, improper combination of SYN, FIN 461 and/or RESET etc. 462 * - ``ipsec_parsing`` 463 - ``drop`` 464 - Traps packets dropped due to an error in the IPSEC header parsing 465 * - ``sctp_parsing`` 466 - ``drop`` 467 - Traps packets dropped due to an error in the SCTP header parsing. 468 This would mean that port number 0 was used or that the header is 469 truncated. 470 * - ``dccp_parsing`` 471 - ``drop`` 472 - Traps packets dropped due to an error in the DCCP header parsing 473 * - ``gtp_parsing`` 474 - ``drop`` 475 - Traps packets dropped due to an error in the GTP header parsing 476 * - ``esp_parsing`` 477 - ``drop`` 478 - Traps packets dropped due to an error in the ESP header parsing 479 * - ``blackhole_nexthop`` 480 - ``drop`` 481 - Traps packets that the device decided to drop in case they hit a 482 blackhole nexthop 483 484Driver-specific Packet Traps 485============================ 486 487Device drivers can register driver-specific packet traps, but these must be 488clearly documented. Such traps can correspond to device-specific exceptions and 489help debug packet drops caused by these exceptions. The following list includes 490links to the description of driver-specific traps registered by various device 491drivers: 492 493 * :doc:`netdevsim` 494 * :doc:`mlxsw` 495 496.. _Generic-Packet-Trap-Groups: 497 498Generic Packet Trap Groups 499========================== 500 501Generic packet trap groups are used to aggregate logically related packet 502traps. These groups allow the user to batch operations such as setting the trap 503action of all member traps. In addition, ``devlink-trap`` can report aggregated 504per-group packets and bytes statistics, in case per-trap statistics are too 505narrow. The description of these groups must be added to the following table: 506 507.. list-table:: List of Generic Packet Trap Groups 508 :widths: 10 90 509 510 * - Name 511 - Description 512 * - ``l2_drops`` 513 - Contains packet traps for packets that were dropped by the device during 514 layer 2 forwarding (i.e., bridge) 515 * - ``l3_drops`` 516 - Contains packet traps for packets that were dropped by the device during 517 layer 3 forwarding 518 * - ``l3_exceptions`` 519 - Contains packet traps for packets that hit an exception (e.g., TTL 520 error) during layer 3 forwarding 521 * - ``buffer_drops`` 522 - Contains packet traps for packets that were dropped by the device due to 523 an enqueue decision 524 * - ``tunnel_drops`` 525 - Contains packet traps for packets that were dropped by the device during 526 tunnel encapsulation / decapsulation 527 * - ``acl_drops`` 528 - Contains packet traps for packets that were dropped by the device during 529 ACL processing 530 * - ``stp`` 531 - Contains packet traps for STP packets 532 * - ``lacp`` 533 - Contains packet traps for LACP packets 534 * - ``lldp`` 535 - Contains packet traps for LLDP packets 536 * - ``mc_snooping`` 537 - Contains packet traps for IGMP and MLD packets required for multicast 538 snooping 539 * - ``dhcp`` 540 - Contains packet traps for DHCP packets 541 * - ``neigh_discovery`` 542 - Contains packet traps for neighbour discovery packets (e.g., ARP, IPv6 543 ND) 544 * - ``bfd`` 545 - Contains packet traps for BFD packets 546 * - ``ospf`` 547 - Contains packet traps for OSPF packets 548 * - ``bgp`` 549 - Contains packet traps for BGP packets 550 * - ``vrrp`` 551 - Contains packet traps for VRRP packets 552 * - ``pim`` 553 - Contains packet traps for PIM packets 554 * - ``uc_loopback`` 555 - Contains a packet trap for unicast loopback packets (i.e., 556 ``uc_loopback``). This trap is singled-out because in cases such as 557 one-armed router it will be constantly triggered. To limit the impact on 558 the CPU usage, a packet trap policer with a low rate can be bound to the 559 group without affecting other traps 560 * - ``local_delivery`` 561 - Contains packet traps for packets that should be locally delivered after 562 routing, but do not match more specific packet traps (e.g., 563 ``ipv4_bgp``) 564 * - ``external_delivery`` 565 - Contains packet traps for packets that should be routed through an 566 external interface (e.g., management interface) that does not belong to 567 the same device (e.g., switch ASIC) as the ingress interface 568 * - ``ipv6`` 569 - Contains packet traps for various IPv6 control packets (e.g., Router 570 Advertisements) 571 * - ``ptp_event`` 572 - Contains packet traps for PTP time-critical event messages (Sync, 573 Delay_req, Pdelay_Req and Pdelay_Resp) 574 * - ``ptp_general`` 575 - Contains packet traps for PTP general messages (Announce, Follow_Up, 576 Delay_Resp, Pdelay_Resp_Follow_Up, management and signaling) 577 * - ``acl_sample`` 578 - Contains packet traps for packets that were sampled by the device during 579 ACL processing 580 * - ``acl_trap`` 581 - Contains packet traps for packets that were trapped (logged) by the 582 device during ACL processing 583 * - ``parser_error_drops`` 584 - Contains packet traps for packets that were marked by the device during 585 parsing as erroneous 586 587Packet Trap Policers 588==================== 589 590As previously explained, the underlying device can trap certain packets to the 591CPU for processing. In most cases, the underlying device is capable of handling 592packet rates that are several orders of magnitude higher compared to those that 593can be handled by the CPU. 594 595Therefore, in order to prevent the underlying device from overwhelming the CPU, 596devices usually include packet trap policers that are able to police the 597trapped packets to rates that can be handled by the CPU. 598 599The ``devlink-trap`` mechanism allows capable device drivers to register their 600supported packet trap policers with ``devlink``. The device driver can choose 601to associate these policers with supported packet trap groups (see 602:ref:`Generic-Packet-Trap-Groups`) during its initialization, thereby exposing 603its default control plane policy to user space. 604 605Device drivers should allow user space to change the parameters of the policers 606(e.g., rate, burst size) as well as the association between the policers and 607trap groups by implementing the relevant callbacks. 608 609If possible, device drivers should implement a callback that allows user space 610to retrieve the number of packets that were dropped by the policer because its 611configured policy was violated. 612 613Testing 614======= 615 616See ``tools/testing/selftests/drivers/net/netdevsim/devlink_trap.sh`` for a 617test covering the core infrastructure. Test cases should be added for any new 618functionality. 619 620Device drivers should focus their tests on device-specific functionality, such 621as the triggering of supported packet traps. 622