1.. SPDX-License-Identifier: GPL-2.0 2 3======================================= 4DSA switch configuration from userspace 5======================================= 6 7The DSA switch configuration is not integrated into the main userspace 8network configuration suites by now and has to be performed manually. 9 10.. _dsa-config-showcases: 11 12Configuration showcases 13----------------------- 14 15To configure a DSA switch a couple of commands need to be executed. In this 16documentation some common configuration scenarios are handled as showcases: 17 18*single port* 19 Every switch port acts as a different configurable Ethernet port 20 21*bridge* 22 Every switch port is part of one configurable Ethernet bridge 23 24*gateway* 25 Every switch port except one upstream port is part of a configurable 26 Ethernet bridge. 27 The upstream port acts as different configurable Ethernet port. 28 29All configurations are performed with tools from iproute2, which is available 30at https://www.kernel.org/pub/linux/utils/net/iproute2/ 31 32Through DSA every port of a switch is handled like a normal linux Ethernet 33interface. The CPU port is the switch port connected to an Ethernet MAC chip. 34The corresponding linux Ethernet interface is called the master interface. 35All other corresponding linux interfaces are called slave interfaces. 36 37The slave interfaces depend on the master interface being up in order for them 38to send or receive traffic. Prior to kernel v5.12, the state of the master 39interface had to be managed explicitly by the user. Starting with kernel v5.12, 40the behavior is as follows: 41 42- when a DSA slave interface is brought up, the master interface is 43 automatically brought up. 44- when the master interface is brought down, all DSA slave interfaces are 45 automatically brought down. 46 47In this documentation the following Ethernet interfaces are used: 48 49*eth0* 50 the master interface 51 52*eth1* 53 another master interface 54 55*lan1* 56 a slave interface 57 58*lan2* 59 another slave interface 60 61*lan3* 62 a third slave interface 63 64*wan* 65 A slave interface dedicated for upstream traffic 66 67Further Ethernet interfaces can be configured similar. 68The configured IPs and networks are: 69 70*single port* 71 * lan1: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3) 72 * lan2: 192.0.2.5/30 (192.0.2.4 - 192.0.2.7) 73 * lan3: 192.0.2.9/30 (192.0.2.8 - 192.0.2.11) 74 75*bridge* 76 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255) 77 78*gateway* 79 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255) 80 * wan: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3) 81 82.. _dsa-tagged-configuration: 83 84Configuration with tagging support 85---------------------------------- 86 87The tagging based configuration is desired and supported by the majority of 88DSA switches. These switches are capable to tag incoming and outgoing traffic 89without using a VLAN based configuration. 90 91*single port* 92 .. code-block:: sh 93 94 # configure each interface 95 ip addr add 192.0.2.1/30 dev lan1 96 ip addr add 192.0.2.5/30 dev lan2 97 ip addr add 192.0.2.9/30 dev lan3 98 99 # For kernels earlier than v5.12, the master interface needs to be 100 # brought up manually before the slave ports. 101 ip link set eth0 up 102 103 # bring up the slave interfaces 104 ip link set lan1 up 105 ip link set lan2 up 106 ip link set lan3 up 107 108*bridge* 109 .. code-block:: sh 110 111 # For kernels earlier than v5.12, the master interface needs to be 112 # brought up manually before the slave ports. 113 ip link set eth0 up 114 115 # bring up the slave interfaces 116 ip link set lan1 up 117 ip link set lan2 up 118 ip link set lan3 up 119 120 # create bridge 121 ip link add name br0 type bridge 122 123 # add ports to bridge 124 ip link set dev lan1 master br0 125 ip link set dev lan2 master br0 126 ip link set dev lan3 master br0 127 128 # configure the bridge 129 ip addr add 192.0.2.129/25 dev br0 130 131 # bring up the bridge 132 ip link set dev br0 up 133 134*gateway* 135 .. code-block:: sh 136 137 # For kernels earlier than v5.12, the master interface needs to be 138 # brought up manually before the slave ports. 139 ip link set eth0 up 140 141 # bring up the slave interfaces 142 ip link set wan up 143 ip link set lan1 up 144 ip link set lan2 up 145 146 # configure the upstream port 147 ip addr add 192.0.2.1/30 dev wan 148 149 # create bridge 150 ip link add name br0 type bridge 151 152 # add ports to bridge 153 ip link set dev lan1 master br0 154 ip link set dev lan2 master br0 155 156 # configure the bridge 157 ip addr add 192.0.2.129/25 dev br0 158 159 # bring up the bridge 160 ip link set dev br0 up 161 162.. _dsa-vlan-configuration: 163 164Configuration without tagging support 165------------------------------------- 166 167A minority of switches are not capable to use a taging protocol 168(DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based 169configuration. 170 171*single port* 172 The configuration can only be set up via VLAN tagging and bridge setup. 173 174 .. code-block:: sh 175 176 # tag traffic on CPU port 177 ip link add link eth0 name eth0.1 type vlan id 1 178 ip link add link eth0 name eth0.2 type vlan id 2 179 ip link add link eth0 name eth0.3 type vlan id 3 180 181 # For kernels earlier than v5.12, the master interface needs to be 182 # brought up manually before the slave ports. 183 ip link set eth0 up 184 ip link set eth0.1 up 185 ip link set eth0.2 up 186 ip link set eth0.3 up 187 188 # bring up the slave interfaces 189 ip link set lan1 up 190 ip link set lan2 up 191 ip link set lan3 up 192 193 # create bridge 194 ip link add name br0 type bridge 195 196 # activate VLAN filtering 197 ip link set dev br0 type bridge vlan_filtering 1 198 199 # add ports to bridges 200 ip link set dev lan1 master br0 201 ip link set dev lan2 master br0 202 ip link set dev lan3 master br0 203 204 # tag traffic on ports 205 bridge vlan add dev lan1 vid 1 pvid untagged 206 bridge vlan add dev lan2 vid 2 pvid untagged 207 bridge vlan add dev lan3 vid 3 pvid untagged 208 209 # configure the VLANs 210 ip addr add 192.0.2.1/30 dev eth0.1 211 ip addr add 192.0.2.5/30 dev eth0.2 212 ip addr add 192.0.2.9/30 dev eth0.3 213 214 # bring up the bridge devices 215 ip link set br0 up 216 217 218*bridge* 219 .. code-block:: sh 220 221 # tag traffic on CPU port 222 ip link add link eth0 name eth0.1 type vlan id 1 223 224 # For kernels earlier than v5.12, the master interface needs to be 225 # brought up manually before the slave ports. 226 ip link set eth0 up 227 ip link set eth0.1 up 228 229 # bring up the slave interfaces 230 ip link set lan1 up 231 ip link set lan2 up 232 ip link set lan3 up 233 234 # create bridge 235 ip link add name br0 type bridge 236 237 # activate VLAN filtering 238 ip link set dev br0 type bridge vlan_filtering 1 239 240 # add ports to bridge 241 ip link set dev lan1 master br0 242 ip link set dev lan2 master br0 243 ip link set dev lan3 master br0 244 ip link set eth0.1 master br0 245 246 # tag traffic on ports 247 bridge vlan add dev lan1 vid 1 pvid untagged 248 bridge vlan add dev lan2 vid 1 pvid untagged 249 bridge vlan add dev lan3 vid 1 pvid untagged 250 251 # configure the bridge 252 ip addr add 192.0.2.129/25 dev br0 253 254 # bring up the bridge 255 ip link set dev br0 up 256 257*gateway* 258 .. code-block:: sh 259 260 # tag traffic on CPU port 261 ip link add link eth0 name eth0.1 type vlan id 1 262 ip link add link eth0 name eth0.2 type vlan id 2 263 264 # For kernels earlier than v5.12, the master interface needs to be 265 # brought up manually before the slave ports. 266 ip link set eth0 up 267 ip link set eth0.1 up 268 ip link set eth0.2 up 269 270 # bring up the slave interfaces 271 ip link set wan up 272 ip link set lan1 up 273 ip link set lan2 up 274 275 # create bridge 276 ip link add name br0 type bridge 277 278 # activate VLAN filtering 279 ip link set dev br0 type bridge vlan_filtering 1 280 281 # add ports to bridges 282 ip link set dev wan master br0 283 ip link set eth0.1 master br0 284 ip link set dev lan1 master br0 285 ip link set dev lan2 master br0 286 287 # tag traffic on ports 288 bridge vlan add dev lan1 vid 1 pvid untagged 289 bridge vlan add dev lan2 vid 1 pvid untagged 290 bridge vlan add dev wan vid 2 pvid untagged 291 292 # configure the VLANs 293 ip addr add 192.0.2.1/30 dev eth0.2 294 ip addr add 192.0.2.129/25 dev br0 295 296 # bring up the bridge devices 297 ip link set br0 up 298 299Forwarding database (FDB) management 300------------------------------------ 301 302The existing DSA switches do not have the necessary hardware support to keep 303the software FDB of the bridge in sync with the hardware tables, so the two 304tables are managed separately (``bridge fdb show`` queries both, and depending 305on whether the ``self`` or ``master`` flags are being used, a ``bridge fdb 306add`` or ``bridge fdb del`` command acts upon entries from one or both tables). 307 308Up until kernel v4.14, DSA only supported user space management of bridge FDB 309entries using the bridge bypass operations (which do not update the software 310FDB, just the hardware one) using the ``self`` flag (which is optional and can 311be omitted). 312 313 .. code-block:: sh 314 315 bridge fdb add dev swp0 00:01:02:03:04:05 self static 316 # or shorthand 317 bridge fdb add dev swp0 00:01:02:03:04:05 static 318 319Due to a bug, the bridge bypass FDB implementation provided by DSA did not 320distinguish between ``static`` and ``local`` FDB entries (``static`` are meant 321to be forwarded, while ``local`` are meant to be locally terminated, i.e. sent 322to the host port). Instead, all FDB entries with the ``self`` flag (implicit or 323explicit) are treated by DSA as ``static`` even if they are ``local``. 324 325 .. code-block:: sh 326 327 # This command: 328 bridge fdb add dev swp0 00:01:02:03:04:05 static 329 # behaves the same for DSA as this command: 330 bridge fdb add dev swp0 00:01:02:03:04:05 local 331 # or shorthand, because the 'local' flag is implicit if 'static' is not 332 # specified, it also behaves the same as: 333 bridge fdb add dev swp0 00:01:02:03:04:05 334 335The last command is an incorrect way of adding a static bridge FDB entry to a 336DSA switch using the bridge bypass operations, and works by mistake. Other 337drivers will treat an FDB entry added by the same command as ``local`` and as 338such, will not forward it, as opposed to DSA. 339 340Between kernel v4.14 and v5.14, DSA has supported in parallel two modes of 341adding a bridge FDB entry to the switch: the bridge bypass discussed above, as 342well as a new mode using the ``master`` flag which installs FDB entries in the 343software bridge too. 344 345 .. code-block:: sh 346 347 bridge fdb add dev swp0 00:01:02:03:04:05 master static 348 349Since kernel v5.14, DSA has gained stronger integration with the bridge's 350software FDB, and the support for its bridge bypass FDB implementation (using 351the ``self`` flag) has been removed. This results in the following changes: 352 353 .. code-block:: sh 354 355 # This is the only valid way of adding an FDB entry that is supported, 356 # compatible with v4.14 kernels and later: 357 bridge fdb add dev swp0 00:01:02:03:04:05 master static 358 # This command is no longer buggy and the entry is properly treated as 359 # 'local' instead of being forwarded: 360 bridge fdb add dev swp0 00:01:02:03:04:05 361 # This command no longer installs a static FDB entry to hardware: 362 bridge fdb add dev swp0 00:01:02:03:04:05 static 363 364Script writers are therefore encouraged to use the ``master static`` set of 365flags when working with bridge FDB entries on DSA switch interfaces. 366 367Affinity of user ports to CPU ports 368----------------------------------- 369 370Typically, DSA switches are attached to the host via a single Ethernet 371interface, but in cases where the switch chip is discrete, the hardware design 372may permit the use of 2 or more ports connected to the host, for an increase in 373termination throughput. 374 375DSA can make use of multiple CPU ports in two ways. First, it is possible to 376statically assign the termination traffic associated with a certain user port 377to be processed by a certain CPU port. This way, user space can implement 378custom policies of static load balancing between user ports, by spreading the 379affinities according to the available CPU ports. 380 381Secondly, it is possible to perform load balancing between CPU ports on a per 382packet basis, rather than statically assigning user ports to CPU ports. 383This can be achieved by placing the DSA masters under a LAG interface (bonding 384or team). DSA monitors this operation and creates a mirror of this software LAG 385on the CPU ports facing the physical DSA masters that constitute the LAG slave 386devices. 387 388To make use of multiple CPU ports, the firmware (device tree) description of 389the switch must mark all the links between CPU ports and their DSA masters 390using the ``ethernet`` reference/phandle. At startup, only a single CPU port 391and DSA master will be used - the numerically first port from the firmware 392description which has an ``ethernet`` property. It is up to the user to 393configure the system for the switch to use other masters. 394 395DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow 396changing the DSA master of a user port. The ``IFLA_DSA_MASTER`` u32 netlink 397attribute contains the ifindex of the master device that handles each slave 398device. The DSA master must be a valid candidate based on firmware node 399information, or a LAG interface which contains only slaves which are valid 400candidates. 401 402Using iproute2, the following manipulations are possible: 403 404 .. code-block:: sh 405 406 # See the DSA master in current use 407 ip -d link show dev swp0 408 (...) 409 dsa master eth0 410 411 # Static CPU port distribution 412 ip link set swp0 type dsa master eth1 413 ip link set swp1 type dsa master eth0 414 ip link set swp2 type dsa master eth1 415 ip link set swp3 type dsa master eth0 416 417 # CPU ports in LAG, using explicit assignment of the DSA master 418 ip link add bond0 type bond mode balance-xor && ip link set bond0 up 419 ip link set eth1 down && ip link set eth1 master bond0 420 ip link set swp0 type dsa master bond0 421 ip link set swp1 type dsa master bond0 422 ip link set swp2 type dsa master bond0 423 ip link set swp3 type dsa master bond0 424 ip link set eth0 down && ip link set eth0 master bond0 425 ip -d link show dev swp0 426 (...) 427 dsa master bond0 428 429 # CPU ports in LAG, relying on implicit migration of the DSA master 430 ip link add bond0 type bond mode balance-xor && ip link set bond0 up 431 ip link set eth0 down && ip link set eth0 master bond0 432 ip link set eth1 down && ip link set eth1 master bond0 433 ip -d link show dev swp0 434 (...) 435 dsa master bond0 436 437Notice that in the case of CPU ports under a LAG, the use of the 438``IFLA_DSA_MASTER`` netlink attribute is not strictly needed, but rather, DSA 439reacts to the ``IFLA_MASTER`` attribute change of its present master (``eth0``) 440and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly, 441when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports 442that were assigned to this interface to the first physical DSA master which is 443eligible, based on the firmware description (it effectively reverts to the 444startup configuration). 445 446In a setup with more than 2 physical CPU ports, it is therefore possible to mix 447static user to CPU port assignment with LAG between DSA masters. It is not 448possible to statically assign a user port towards a DSA master that has any 449upper interfaces (this includes LAG devices - the master must always be the LAG 450in this case). 451 452Live changing of the DSA master (and thus CPU port) affinity of a user port is 453permitted, in order to allow dynamic redistribution in response to traffic. 454 455Physical DSA masters are allowed to join and leave at any time a LAG interface 456used as a DSA master; however, DSA will reject a LAG interface as a valid 457candidate for being a DSA master unless it has at least one physical DSA master 458as a slave device. 459