1.. SPDX-License-Identifier: GPL-2.0 2 3==================================== 4Virtual Routing and Forwarding (VRF) 5==================================== 6 7The VRF Device 8============== 9 10The VRF device combined with ip rules provides the ability to create virtual 11routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the 12Linux network stack. One use case is the multi-tenancy problem where each 13tenant has their own unique routing tables and in the very least need 14different default gateways. 15 16Processes can be "VRF aware" by binding a socket to the VRF device. Packets 17through the socket then use the routing table associated with the VRF 18device. An important feature of the VRF device implementation is that it 19impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected 20(ie., they do not need to be run in each VRF). The design also allows 21the use of higher priority ip rules (Policy Based Routing, PBR) to take 22precedence over the VRF device rules directing specific traffic as desired. 23 24In addition, VRF devices allow VRFs to be nested within namespaces. For 25example network namespaces provide separation of network interfaces at the 26device layer, VLANs on the interfaces within a namespace provide L2 separation 27and then VRF devices provide L3 separation. 28 29Design 30------ 31A VRF device is created with an associated route table. Network interfaces 32are then enslaved to a VRF device:: 33 34 +-----------------------------+ 35 | vrf-blue | ===> route table 10 36 +-----------------------------+ 37 | | | 38 +------+ +------+ +-------------+ 39 | eth1 | | eth2 | ... | bond1 | 40 +------+ +------+ +-------------+ 41 | | 42 +------+ +------+ 43 | eth8 | | eth9 | 44 +------+ +------+ 45 46Packets received on an enslaved device and are switched to the VRF device 47in the IPv4 and IPv6 processing stacks giving the impression that packets 48flow through the VRF device. Similarly on egress routing rules are used to 49send packets to the VRF device driver before getting sent out the actual 50interface. This allows tcpdump on a VRF device to capture all packets into 51and out of the VRF as a whole\ [1]_. Similarly, netfilter\ [2]_ and tc rules 52can be applied using the VRF device to specify rules that apply to the VRF 53domain as a whole. 54 55.. [1] Packets in the forwarded state do not flow through the device, so those 56 packets are not seen by tcpdump. Will revisit this limitation in a 57 future release. 58 59.. [2] Iptables on ingress supports PREROUTING with skb->dev set to the real 60 ingress device and both INPUT and PREROUTING rules with skb->dev set to 61 the VRF device. For egress POSTROUTING and OUTPUT rules can be written 62 using either the VRF device or real egress device. 63 64Setup 65----- 661. VRF device is created with an association to a FIB table. 67 e.g,:: 68 69 ip link add vrf-blue type vrf table 10 70 ip link set dev vrf-blue up 71 722. An l3mdev FIB rule directs lookups to the table associated with the device. 73 A single l3mdev rule is sufficient for all VRFs. The VRF device adds the 74 l3mdev rule for IPv4 and IPv6 when the first device is created with a 75 default preference of 1000. Users may delete the rule if desired and add 76 with a different priority or install per-VRF rules. 77 78 Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:: 79 80 ip ru add oif vrf-blue table 10 81 ip ru add iif vrf-blue table 10 82 833. Set the default route for the table (and hence default route for the VRF):: 84 85 ip route add table 10 unreachable default metric 4278198272 86 87 This high metric value ensures that the default unreachable route can 88 be overridden by a routing protocol suite. FRRouting interprets 89 kernel metrics as a combined admin distance (upper byte) and priority 90 (lower 3 bytes). Thus the above metric translates to [255/8192]. 91 924. Enslave L3 interfaces to a VRF device:: 93 94 ip link set dev eth1 master vrf-blue 95 96 Local and connected routes for enslaved devices are automatically moved to 97 the table associated with VRF device. Any additional routes depending on 98 the enslaved device are dropped and will need to be reinserted to the VRF 99 FIB table following the enslavement. 100 101 The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global 102 addresses as VRF enslavement changes:: 103 104 sysctl -w net.ipv6.conf.all.keep_addr_on_down=1 105 1065. Additional VRF routes are added to associated table:: 107 108 ip route add table 10 ... 109 110 111Applications 112------------ 113Applications that are to work within a VRF need to bind their socket to the 114VRF device:: 115 116 setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1); 117 118or to specify the output device using cmsg and IP_PKTINFO. 119 120By default the scope of the port bindings for unbound sockets is 121limited to the default VRF. That is, it will not be matched by packets 122arriving on interfaces enslaved to an l3mdev and processes may bind to 123the same port if they bind to an l3mdev. 124 125TCP & UDP services running in the default VRF context (ie., not bound 126to any VRF device) can work across all VRF domains by enabling the 127tcp_l3mdev_accept and udp_l3mdev_accept sysctl options:: 128 129 sysctl -w net.ipv4.tcp_l3mdev_accept=1 130 sysctl -w net.ipv4.udp_l3mdev_accept=1 131 132These options are disabled by default so that a socket in a VRF is only 133selected for packets in that VRF. There is a similar option for RAW 134sockets, which is enabled by default for reasons of backwards compatibility. 135This is so as to specify the output device with cmsg and IP_PKTINFO, but 136using a socket not bound to the corresponding VRF. This allows e.g. older ping 137implementations to be run with specifying the device but without executing it 138in the VRF. This option can be disabled so that packets received in a VRF 139context are only handled by a raw socket bound to the VRF, and packets in the 140default VRF are only handled by a socket not bound to any VRF:: 141 142 sysctl -w net.ipv4.raw_l3mdev_accept=0 143 144netfilter rules on the VRF device can be used to limit access to services 145running in the default VRF context as well. 146 147-------------------------------------------------------------------------------- 148 149Using iproute2 for VRFs 150======================= 151iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this 152section lists both commands where appropriate -- with the vrf keyword and the 153older form without it. 154 1551. Create a VRF 156 157 To instantiate a VRF device and associate it with a table:: 158 159 $ ip link add dev NAME type vrf table ID 160 161 As of v4.8 the kernel supports the l3mdev FIB rule where a single rule 162 covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first 163 device create. 164 1652. List VRFs 166 167 To list VRFs that have been created:: 168 169 $ ip [-d] link show type vrf 170 NOTE: The -d option is needed to show the table id 171 172 For example:: 173 174 $ ip -d link show type vrf 175 11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 176 link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0 177 vrf table 1 addrgenmode eui64 178 12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 179 link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0 180 vrf table 10 addrgenmode eui64 181 13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 182 link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0 183 vrf table 66 addrgenmode eui64 184 14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 185 link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0 186 vrf table 81 addrgenmode eui64 187 188 189 Or in brief output:: 190 191 $ ip -br link show type vrf 192 mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP> 193 red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP> 194 blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP> 195 green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP> 196 197 1983. Assign a Network Interface to a VRF 199 200 Network interfaces are assigned to a VRF by enslaving the netdevice to a 201 VRF device:: 202 203 $ ip link set dev NAME master NAME 204 205 On enslavement connected and local routes are automatically moved to the 206 table associated with the VRF device. 207 208 For example:: 209 210 $ ip link set dev eth0 master mgmt 211 212 2134. Show Devices Assigned to a VRF 214 215 To show devices that have been assigned to a specific VRF add the master 216 option to the ip command:: 217 218 $ ip link show vrf NAME 219 $ ip link show master NAME 220 221 For example:: 222 223 $ ip link show vrf red 224 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000 225 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff 226 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000 227 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff 228 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000 229 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff 230 231 232 Or using the brief output:: 233 234 $ ip -br link show vrf red 235 eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP> 236 eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP> 237 eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST> 238 239 2405. Show Neighbor Entries for a VRF 241 242 To list neighbor entries associated with devices enslaved to a VRF device 243 add the master option to the ip command:: 244 245 $ ip [-6] neigh show vrf NAME 246 $ ip [-6] neigh show master NAME 247 248 For example:: 249 250 $ ip neigh show vrf red 251 10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE 252 10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE 253 254 $ ip -6 neigh show vrf red 255 2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE 256 257 2586. Show Addresses for a VRF 259 260 To show addresses for interfaces associated with a VRF add the master 261 option to the ip command:: 262 263 $ ip addr show vrf NAME 264 $ ip addr show master NAME 265 266 For example:: 267 268 $ ip addr show vrf red 269 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000 270 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff 271 inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1 272 valid_lft forever preferred_lft forever 273 inet6 2002:1::2/120 scope global 274 valid_lft forever preferred_lft forever 275 inet6 fe80::ff:fe00:202/64 scope link 276 valid_lft forever preferred_lft forever 277 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000 278 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff 279 inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2 280 valid_lft forever preferred_lft forever 281 inet6 2002:2::2/120 scope global 282 valid_lft forever preferred_lft forever 283 inet6 fe80::ff:fe00:203/64 scope link 284 valid_lft forever preferred_lft forever 285 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000 286 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff 287 288 Or in brief format:: 289 290 $ ip -br addr show vrf red 291 eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64 292 eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64 293 eth5 DOWN 294 295 2967. Show Routes for a VRF 297 298 To show routes for a VRF use the ip command to display the table associated 299 with the VRF device:: 300 301 $ ip [-6] route show vrf NAME 302 $ ip [-6] route show table ID 303 304 For example:: 305 306 $ ip route show vrf red 307 unreachable default metric 4278198272 308 broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2 309 10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2 310 local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2 311 broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2 312 broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2 313 10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2 314 local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2 315 broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2 316 317 $ ip -6 route show vrf red 318 local 2002:1:: dev lo proto none metric 0 pref medium 319 local 2002:1::2 dev lo proto none metric 0 pref medium 320 2002:1::/120 dev eth1 proto kernel metric 256 pref medium 321 local 2002:2:: dev lo proto none metric 0 pref medium 322 local 2002:2::2 dev lo proto none metric 0 pref medium 323 2002:2::/120 dev eth2 proto kernel metric 256 pref medium 324 local fe80:: dev lo proto none metric 0 pref medium 325 local fe80:: dev lo proto none metric 0 pref medium 326 local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium 327 local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium 328 fe80::/64 dev eth1 proto kernel metric 256 pref medium 329 fe80::/64 dev eth2 proto kernel metric 256 pref medium 330 ff00::/8 dev red metric 256 pref medium 331 ff00::/8 dev eth1 metric 256 pref medium 332 ff00::/8 dev eth2 metric 256 pref medium 333 unreachable default dev lo metric 4278198272 error -101 pref medium 334 3358. Route Lookup for a VRF 336 337 A test route lookup can be done for a VRF:: 338 339 $ ip [-6] route get vrf NAME ADDRESS 340 $ ip [-6] route get oif NAME ADDRESS 341 342 For example:: 343 344 $ ip route get 10.2.1.40 vrf red 345 10.2.1.40 dev eth1 table red src 10.2.1.2 346 cache 347 348 $ ip -6 route get 2002:1::32 vrf red 349 2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium 350 351 3529. Removing Network Interface from a VRF 353 354 Network interfaces are removed from a VRF by breaking the enslavement to 355 the VRF device:: 356 357 $ ip link set dev NAME nomaster 358 359 Connected routes are moved back to the default table and local entries are 360 moved to the local table. 361 362 For example:: 363 364 $ ip link set dev eth0 nomaster 365 366-------------------------------------------------------------------------------- 367 368Commands used in this example:: 369 370 cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF 371 1 mgmt 372 10 red 373 66 blue 374 81 green 375 EOF 376 377 function vrf_create 378 { 379 VRF=$1 380 TBID=$2 381 382 # create VRF device 383 ip link add ${VRF} type vrf table ${TBID} 384 385 if [ "${VRF}" != "mgmt" ]; then 386 ip route add table ${TBID} unreachable default metric 4278198272 387 fi 388 ip link set dev ${VRF} up 389 } 390 391 vrf_create mgmt 1 392 ip link set dev eth0 master mgmt 393 394 vrf_create red 10 395 ip link set dev eth1 master red 396 ip link set dev eth2 master red 397 ip link set dev eth5 master red 398 399 vrf_create blue 66 400 ip link set dev eth3 master blue 401 402 vrf_create green 81 403 ip link set dev eth4 master green 404 405 406 Interface addresses from /etc/network/interfaces: 407 auto eth0 408 iface eth0 inet static 409 address 10.0.0.2 410 netmask 255.255.255.0 411 gateway 10.0.0.254 412 413 iface eth0 inet6 static 414 address 2000:1::2 415 netmask 120 416 417 auto eth1 418 iface eth1 inet static 419 address 10.2.1.2 420 netmask 255.255.255.0 421 422 iface eth1 inet6 static 423 address 2002:1::2 424 netmask 120 425 426 auto eth2 427 iface eth2 inet static 428 address 10.2.2.2 429 netmask 255.255.255.0 430 431 iface eth2 inet6 static 432 address 2002:2::2 433 netmask 120 434 435 auto eth3 436 iface eth3 inet static 437 address 10.2.3.2 438 netmask 255.255.255.0 439 440 iface eth3 inet6 static 441 address 2002:3::2 442 netmask 120 443 444 auto eth4 445 iface eth4 inet static 446 address 10.2.4.2 447 netmask 255.255.255.0 448 449 iface eth4 inet6 static 450 address 2002:4::2 451 netmask 120 452