1.. SPDX-License-Identifier: GPL-2.0 2.. include:: <isonum.txt> 3 4=============================================== 5Ethernet switch device driver model (switchdev) 6=============================================== 7 8Copyright |copy| 2014 Jiri Pirko <jiri@resnulli.us> 9 10Copyright |copy| 2014-2015 Scott Feldman <sfeldma@gmail.com> 11 12 13The Ethernet switch device driver model (switchdev) is an in-kernel driver 14model for switch devices which offload the forwarding (data) plane from the 15kernel. 16 17Figure 1 is a block diagram showing the components of the switchdev model for 18an example setup using a data-center-class switch ASIC chip. Other setups 19with SR-IOV or soft switches, such as OVS, are possible. 20 21:: 22 23 24 User-space tools 25 26 user space | 27 +-------------------------------------------------------------------+ 28 kernel | Netlink 29 | 30 +--------------+-------------------------------+ 31 | Network stack | 32 | (Linux) | 33 | | 34 +----------------------------------------------+ 35 36 sw1p2 sw1p4 sw1p6 37 sw1p1 + sw1p3 + sw1p5 + eth1 38 + | + | + | + 39 | | | | | | | 40 +--+----+----+----+----+----+---+ +-----+-----+ 41 | Switch driver | | mgmt | 42 | (this document) | | driver | 43 | | | | 44 +--------------+----------------+ +-----------+ 45 | 46 kernel | HW bus (eg PCI) 47 +-------------------------------------------------------------------+ 48 hardware | 49 +--------------+----------------+ 50 | Switch device (sw1) | 51 | +----+ +--------+ 52 | | v offloaded data path | mgmt port 53 | | | | 54 +--|----|----+----+----+----+---+ 55 | | | | | | 56 + + + + + + 57 p1 p2 p3 p4 p5 p6 58 59 front-panel ports 60 61 62 Fig 1. 63 64 65Include Files 66------------- 67 68:: 69 70 #include <linux/netdevice.h> 71 #include <net/switchdev.h> 72 73 74Configuration 75------------- 76 77Use "depends NET_SWITCHDEV" in driver's Kconfig to ensure switchdev model 78support is built for driver. 79 80 81Switch Ports 82------------ 83 84On switchdev driver initialization, the driver will allocate and register a 85struct net_device (using register_netdev()) for each enumerated physical switch 86port, called the port netdev. A port netdev is the software representation of 87the physical port and provides a conduit for control traffic to/from the 88controller (the kernel) and the network, as well as an anchor point for higher 89level constructs such as bridges, bonds, VLANs, tunnels, and L3 routers. Using 90standard netdev tools (iproute2, ethtool, etc), the port netdev can also 91provide to the user access to the physical properties of the switch port such 92as PHY link state and I/O statistics. 93 94There is (currently) no higher-level kernel object for the switch beyond the 95port netdevs. All of the switchdev driver ops are netdev ops or switchdev ops. 96 97A switch management port is outside the scope of the switchdev driver model. 98Typically, the management port is not participating in offloaded data plane and 99is loaded with a different driver, such as a NIC driver, on the management port 100device. 101 102Switch ID 103^^^^^^^^^ 104 105The switchdev driver must implement the net_device operation 106ndo_get_port_parent_id for each port netdev, returning the same physical ID for 107each port of a switch. The ID must be unique between switches on the same 108system. The ID does not need to be unique between switches on different 109systems. 110 111The switch ID is used to locate ports on a switch and to know if aggregated 112ports belong to the same switch. 113 114Port Netdev Naming 115^^^^^^^^^^^^^^^^^^ 116 117Udev rules should be used for port netdev naming, using some unique attribute 118of the port as a key, for example the port MAC address or the port PHYS name. 119Hard-coding of kernel netdev names within the driver is discouraged; let the 120kernel pick the default netdev name, and let udev set the final name based on a 121port attribute. 122 123Using port PHYS name (ndo_get_phys_port_name) for the key is particularly 124useful for dynamically-named ports where the device names its ports based on 125external configuration. For example, if a physical 40G port is split logically 126into 4 10G ports, resulting in 4 port netdevs, the device can give a unique 127name for each port using port PHYS name. The udev rule would be:: 128 129 SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}=="<phys_switch_id>", \ 130 ATTR{phys_port_name}!="", NAME="swX$attr{phys_port_name}" 131 132Suggested naming convention is "swXpYsZ", where X is the switch name or ID, Y 133is the port name or ID, and Z is the sub-port name or ID. For example, sw1p1s0 134would be sub-port 0 on port 1 on switch 1. 135 136Port Features 137^^^^^^^^^^^^^ 138 139NETIF_F_NETNS_LOCAL 140 141If the switchdev driver (and device) only supports offloading of the default 142network namespace (netns), the driver should set this feature flag to prevent 143the port netdev from being moved out of the default netns. A netns-aware 144driver/device would not set this flag and be responsible for partitioning 145hardware to preserve netns containment. This means hardware cannot forward 146traffic from a port in one namespace to another port in another namespace. 147 148Port Topology 149^^^^^^^^^^^^^ 150 151The port netdevs representing the physical switch ports can be organized into 152higher-level switching constructs. The default construct is a standalone 153router port, used to offload L3 forwarding. Two or more ports can be bonded 154together to form a LAG. Two or more ports (or LAGs) can be bridged to bridge 155L2 networks. VLANs can be applied to sub-divide L2 networks. L2-over-L3 156tunnels can be built on ports. These constructs are built using standard Linux 157tools such as the bridge driver, the bonding/team drivers, and netlink-based 158tools such as iproute2. 159 160The switchdev driver can know a particular port's position in the topology by 161monitoring NETDEV_CHANGEUPPER notifications. For example, a port moved into a 162bond will see it's upper master change. If that bond is moved into a bridge, 163the bond's upper master will change. And so on. The driver will track such 164movements to know what position a port is in in the overall topology by 165registering for netdevice events and acting on NETDEV_CHANGEUPPER. 166 167L2 Forwarding Offload 168--------------------- 169 170The idea is to offload the L2 data forwarding (switching) path from the kernel 171to the switchdev device by mirroring bridge FDB entries down to the device. An 172FDB entry is the {port, MAC, VLAN} tuple forwarding destination. 173 174To offloading L2 bridging, the switchdev driver/device should support: 175 176 - Static FDB entries installed on a bridge port 177 - Notification of learned/forgotten src mac/vlans from device 178 - STP state changes on the port 179 - VLAN flooding of multicast/broadcast and unknown unicast packets 180 181Static FDB Entries 182^^^^^^^^^^^^^^^^^^ 183 184The switchdev driver should implement ndo_fdb_add, ndo_fdb_del and ndo_fdb_dump 185to support static FDB entries installed to the device. Static bridge FDB 186entries are installed, for example, using iproute2 bridge cmd:: 187 188 bridge fdb add ADDR dev DEV [vlan VID] [self] 189 190The driver should use the helper switchdev_port_fdb_xxx ops for ndo_fdb_xxx 191ops, and handle add/delete/dump of SWITCHDEV_OBJ_ID_PORT_FDB object using 192switchdev_port_obj_xxx ops. 193 194XXX: what should be done if offloading this rule to hardware fails (for 195example, due to full capacity in hardware tables) ? 196 197Note: by default, the bridge does not filter on VLAN and only bridges untagged 198traffic. To enable VLAN support, turn on VLAN filtering:: 199 200 echo 1 >/sys/class/net/<bridge>/bridge/vlan_filtering 201 202Notification of Learned/Forgotten Source MAC/VLANs 203^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 204 205The switch device will learn/forget source MAC address/VLAN on ingress packets 206and notify the switch driver of the mac/vlan/port tuples. The switch driver, 207in turn, will notify the bridge driver using the switchdev notifier call:: 208 209 err = call_switchdev_notifiers(val, dev, info, extack); 210 211Where val is SWITCHDEV_FDB_ADD when learning and SWITCHDEV_FDB_DEL when 212forgetting, and info points to a struct switchdev_notifier_fdb_info. On 213SWITCHDEV_FDB_ADD, the bridge driver will install the FDB entry into the 214bridge's FDB and mark the entry as NTF_EXT_LEARNED. The iproute2 bridge 215command will label these entries "offload":: 216 217 $ bridge fdb 218 52:54:00:12:35:01 dev sw1p1 master br0 permanent 219 00:02:00:00:02:00 dev sw1p1 master br0 offload 220 00:02:00:00:02:00 dev sw1p1 self 221 52:54:00:12:35:02 dev sw1p2 master br0 permanent 222 00:02:00:00:03:00 dev sw1p2 master br0 offload 223 00:02:00:00:03:00 dev sw1p2 self 224 33:33:00:00:00:01 dev eth0 self permanent 225 01:00:5e:00:00:01 dev eth0 self permanent 226 33:33:ff:00:00:00 dev eth0 self permanent 227 01:80:c2:00:00:0e dev eth0 self permanent 228 33:33:00:00:00:01 dev br0 self permanent 229 01:00:5e:00:00:01 dev br0 self permanent 230 33:33:ff:12:35:01 dev br0 self permanent 231 232Learning on the port should be disabled on the bridge using the bridge command:: 233 234 bridge link set dev DEV learning off 235 236Learning on the device port should be enabled, as well as learning_sync:: 237 238 bridge link set dev DEV learning on self 239 bridge link set dev DEV learning_sync on self 240 241Learning_sync attribute enables syncing of the learned/forgotten FDB entry to 242the bridge's FDB. It's possible, but not optimal, to enable learning on the 243device port and on the bridge port, and disable learning_sync. 244 245To support learning, the driver implements switchdev op 246switchdev_port_attr_set for SWITCHDEV_ATTR_PORT_ID_{PRE}_BRIDGE_FLAGS. 247 248FDB Ageing 249^^^^^^^^^^ 250 251The bridge will skip ageing FDB entries marked with NTF_EXT_LEARNED and it is 252the responsibility of the port driver/device to age out these entries. If the 253port device supports ageing, when the FDB entry expires, it will notify the 254driver which in turn will notify the bridge with SWITCHDEV_FDB_DEL. If the 255device does not support ageing, the driver can simulate ageing using a 256garbage collection timer to monitor FDB entries. Expired entries will be 257notified to the bridge using SWITCHDEV_FDB_DEL. See rocker driver for 258example of driver running ageing timer. 259 260To keep an NTF_EXT_LEARNED entry "alive", the driver should refresh the FDB 261entry by calling call_switchdev_notifiers(SWITCHDEV_FDB_ADD, ...). The 262notification will reset the FDB entry's last-used time to now. The driver 263should rate limit refresh notifications, for example, no more than once a 264second. (The last-used time is visible using the bridge -s fdb option). 265 266STP State Change on Port 267^^^^^^^^^^^^^^^^^^^^^^^^ 268 269Internally or with a third-party STP protocol implementation (e.g. mstpd), the 270bridge driver maintains the STP state for ports, and will notify the switch 271driver of STP state change on a port using the switchdev op 272switchdev_attr_port_set for SWITCHDEV_ATTR_PORT_ID_STP_UPDATE. 273 274State is one of BR_STATE_*. The switch driver can use STP state updates to 275update ingress packet filter list for the port. For example, if port is 276DISABLED, no packets should pass, but if port moves to BLOCKED, then STP BPDUs 277and other IEEE 01:80:c2:xx:xx:xx link-local multicast packets can pass. 278 279Note that STP BDPUs are untagged and STP state applies to all VLANs on the port 280so packet filters should be applied consistently across untagged and tagged 281VLANs on the port. 282 283Flooding L2 domain 284^^^^^^^^^^^^^^^^^^ 285 286For a given L2 VLAN domain, the switch device should flood multicast/broadcast 287and unknown unicast packets to all ports in domain, if allowed by port's 288current STP state. The switch driver, knowing which ports are within which 289vlan L2 domain, can program the switch device for flooding. The packet may 290be sent to the port netdev for processing by the bridge driver. The 291bridge should not reflood the packet to the same ports the device flooded, 292otherwise there will be duplicate packets on the wire. 293 294To avoid duplicate packets, the switch driver should mark a packet as already 295forwarded by setting the skb->offload_fwd_mark bit. The bridge driver will mark 296the skb using the ingress bridge port's mark and prevent it from being forwarded 297through any bridge port with the same mark. 298 299It is possible for the switch device to not handle flooding and push the 300packets up to the bridge driver for flooding. This is not ideal as the number 301of ports scale in the L2 domain as the device is much more efficient at 302flooding packets that software. 303 304If supported by the device, flood control can be offloaded to it, preventing 305certain netdevs from flooding unicast traffic for which there is no FDB entry. 306 307IGMP Snooping 308^^^^^^^^^^^^^ 309 310In order to support IGMP snooping, the port netdevs should trap to the bridge 311driver all IGMP join and leave messages. 312The bridge multicast module will notify port netdevs on every multicast group 313changed whether it is static configured or dynamically joined/leave. 314The hardware implementation should be forwarding all registered multicast 315traffic groups only to the configured ports. 316 317L3 Routing Offload 318------------------ 319 320Offloading L3 routing requires that device be programmed with FIB entries from 321the kernel, with the device doing the FIB lookup and forwarding. The device 322does a longest prefix match (LPM) on FIB entries matching route prefix and 323forwards the packet to the matching FIB entry's nexthop(s) egress ports. 324 325To program the device, the driver has to register a FIB notifier handler 326using register_fib_notifier. The following events are available: 327 328=================== =================================================== 329FIB_EVENT_ENTRY_ADD used for both adding a new FIB entry to the device, 330 or modifying an existing entry on the device. 331FIB_EVENT_ENTRY_DEL used for removing a FIB entry 332FIB_EVENT_RULE_ADD, 333FIB_EVENT_RULE_DEL used to propagate FIB rule changes 334=================== =================================================== 335 336FIB_EVENT_ENTRY_ADD and FIB_EVENT_ENTRY_DEL events pass:: 337 338 struct fib_entry_notifier_info { 339 struct fib_notifier_info info; /* must be first */ 340 u32 dst; 341 int dst_len; 342 struct fib_info *fi; 343 u8 tos; 344 u8 type; 345 u32 tb_id; 346 u32 nlflags; 347 }; 348 349to add/modify/delete IPv4 dst/dest_len prefix on table tb_id. The ``*fi`` 350structure holds details on the route and route's nexthops. ``*dev`` is one 351of the port netdevs mentioned in the route's next hop list. 352 353Routes offloaded to the device are labeled with "offload" in the ip route 354listing:: 355 356 $ ip route show 357 default via 192.168.0.2 dev eth0 358 11.0.0.0/30 dev sw1p1 proto kernel scope link src 11.0.0.2 offload 359 11.0.0.4/30 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload 360 11.0.0.8/30 dev sw1p2 proto kernel scope link src 11.0.0.10 offload 361 11.0.0.12/30 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload 362 12.0.0.2 proto zebra metric 30 offload 363 nexthop via 11.0.0.1 dev sw1p1 weight 1 364 nexthop via 11.0.0.9 dev sw1p2 weight 1 365 12.0.0.3 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload 366 12.0.0.4 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload 367 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.15 368 369The "offload" flag is set in case at least one device offloads the FIB entry. 370 371XXX: add/mod/del IPv6 FIB API 372 373Nexthop Resolution 374^^^^^^^^^^^^^^^^^^ 375 376The FIB entry's nexthop list contains the nexthop tuple (gateway, dev), but for 377the switch device to forward the packet with the correct dst mac address, the 378nexthop gateways must be resolved to the neighbor's mac address. Neighbor mac 379address discovery comes via the ARP (or ND) process and is available via the 380arp_tbl neighbor table. To resolve the routes nexthop gateways, the driver 381should trigger the kernel's neighbor resolution process. See the rocker 382driver's rocker_port_ipv4_resolve() for an example. 383 384The driver can monitor for updates to arp_tbl using the netevent notifier 385NETEVENT_NEIGH_UPDATE. The device can be programmed with resolved nexthops 386for the routes as arp_tbl updates. The driver implements ndo_neigh_destroy 387to know when arp_tbl neighbor entries are purged from the port. 388