1.. SPDX-License-Identifier: GPL-2.0 2 3====== 4AF_XDP 5====== 6 7Overview 8======== 9 10AF_XDP is an address family that is optimized for high performance 11packet processing. 12 13This document assumes that the reader is familiar with BPF and XDP. If 14not, the Cilium project has an excellent reference guide at 15http://cilium.readthedocs.io/en/latest/bpf/. 16 17Using the XDP_REDIRECT action from an XDP program, the program can 18redirect ingress frames to other XDP enabled netdevs, using the 19bpf_redirect_map() function. AF_XDP sockets enable the possibility for 20XDP programs to redirect frames to a memory buffer in a user-space 21application. 22 23An AF_XDP socket (XSK) is created with the normal socket() 24syscall. Associated with each XSK are two rings: the RX ring and the 25TX ring. A socket can receive packets on the RX ring and it can send 26packets on the TX ring. These rings are registered and sized with the 27setsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatory 28to have at least one of these rings for each socket. An RX or TX 29descriptor ring points to a data buffer in a memory area called a 30UMEM. RX and TX can share the same UMEM so that a packet does not have 31to be copied between RX and TX. Moreover, if a packet needs to be kept 32for a while due to a possible retransmit, the descriptor that points 33to that packet can be changed to point to another and reused right 34away. This again avoids copying data. 35 36The UMEM consists of a number of equally sized chunks. A descriptor in 37one of the rings references a frame by referencing its addr. The addr 38is simply an offset within the entire UMEM region. The user space 39allocates memory for this UMEM using whatever means it feels is most 40appropriate (malloc, mmap, huge pages, etc). This memory area is then 41registered with the kernel using the new setsockopt XDP_UMEM_REG. The 42UMEM also has two rings: the FILL ring and the COMPLETION ring. The 43FILL ring is used by the application to send down addr for the kernel 44to fill in with RX packet data. References to these frames will then 45appear in the RX ring once each packet has been received. The 46COMPLETION ring, on the other hand, contains frame addr that the 47kernel has transmitted completely and can now be used again by user 48space, for either TX or RX. Thus, the frame addrs appearing in the 49COMPLETION ring are addrs that were previously transmitted using the 50TX ring. In summary, the RX and FILL rings are used for the RX path 51and the TX and COMPLETION rings are used for the TX path. 52 53The socket is then finally bound with a bind() call to a device and a 54specific queue id on that device, and it is not until bind is 55completed that traffic starts to flow. 56 57The UMEM can be shared between processes, if desired. If a process 58wants to do this, it simply skips the registration of the UMEM and its 59corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind 60call and submits the XSK of the process it would like to share UMEM 61with as well as its own newly created XSK socket. The new process will 62then receive frame addr references in its own RX ring that point to 63this shared UMEM. Note that since the ring structures are 64single-consumer / single-producer (for performance reasons), the new 65process has to create its own socket with associated RX and TX rings, 66since it cannot share this with the other process. This is also the 67reason that there is only one set of FILL and COMPLETION rings per 68UMEM. It is the responsibility of a single process to handle the UMEM. 69 70How is then packets distributed from an XDP program to the XSKs? There 71is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The 72user-space application can place an XSK at an arbitrary place in this 73map. The XDP program can then redirect a packet to a specific index in 74this map and at this point XDP validates that the XSK in that map was 75indeed bound to that device and ring number. If not, the packet is 76dropped. If the map is empty at that index, the packet is also 77dropped. This also means that it is currently mandatory to have an XDP 78program loaded (and one XSK in the XSKMAP) to be able to get any 79traffic to user space through the XSK. 80 81AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the 82driver does not have support for XDP, or XDP_SKB is explicitly chosen 83when loading the XDP program, XDP_SKB mode is employed that uses SKBs 84together with the generic XDP support and copies out the data to user 85space. A fallback mode that works for any network device. On the other 86hand, if the driver has support for XDP, it will be used by the AF_XDP 87code to provide better performance, but there is still a copy of the 88data into user space. 89 90Concepts 91======== 92 93In order to use an AF_XDP socket, a number of associated objects need 94to be setup. These objects and their options are explained in the 95following sections. 96 97For an overview on how AF_XDP works, you can also take a look at the 98Linux Plumbers paper from 2018 on the subject: 99http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. Do 100NOT consult the paper from 2017 on "AF_PACKET v4", the first attempt 101at AF_XDP. Nearly everything changed since then. Jonathan Corbet has 102also written an excellent article on LWN, "Accelerating networking 103with AF_XDP". It can be found at https://lwn.net/Articles/750845/. 104 105UMEM 106---- 107 108UMEM is a region of virtual contiguous memory, divided into 109equal-sized frames. An UMEM is associated to a netdev and a specific 110queue id of that netdev. It is created and configured (chunk size, 111headroom, start address and size) by using the XDP_UMEM_REG setsockopt 112system call. A UMEM is bound to a netdev and queue id, via the bind() 113system call. 114 115An AF_XDP is socket linked to a single UMEM, but one UMEM can have 116multiple AF_XDP sockets. To share an UMEM created via one socket A, 117the next socket B can do this by setting the XDP_SHARED_UMEM flag in 118struct sockaddr_xdp member sxdp_flags, and passing the file descriptor 119of A to struct sockaddr_xdp member sxdp_shared_umem_fd. 120 121The UMEM has two single-producer/single-consumer rings that are used 122to transfer ownership of UMEM frames between the kernel and the 123user-space application. 124 125Rings 126----- 127 128There are a four different kind of rings: FILL, COMPLETION, RX and 129TX. All rings are single-producer/single-consumer, so the user-space 130application need explicit synchronization of multiple 131processes/threads are reading/writing to them. 132 133The UMEM uses two rings: FILL and COMPLETION. Each socket associated 134with the UMEM must have an RX queue, TX queue or both. Say, that there 135is a setup with four sockets (all doing TX and RX). Then there will be 136one FILL ring, one COMPLETION ring, four TX rings and four RX rings. 137 138The rings are head(producer)/tail(consumer) based rings. A producer 139writes the data ring at the index pointed out by struct xdp_ring 140producer member, and increasing the producer index. A consumer reads 141the data ring at the index pointed out by struct xdp_ring consumer 142member, and increasing the consumer index. 143 144The rings are configured and created via the _RING setsockopt system 145calls and mmapped to user-space using the appropriate offset to mmap() 146(XDP_PGOFF_RX_RING, XDP_PGOFF_TX_RING, XDP_UMEM_PGOFF_FILL_RING and 147XDP_UMEM_PGOFF_COMPLETION_RING). 148 149The size of the rings need to be of size power of two. 150 151UMEM Fill Ring 152~~~~~~~~~~~~~~ 153 154The FILL ring is used to transfer ownership of UMEM frames from 155user-space to kernel-space. The UMEM addrs are passed in the ring. As 156an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has 15716 chunks and can pass addrs between 0 and 64k. 158 159Frames passed to the kernel are used for the ingress path (RX rings). 160 161The user application produces UMEM addrs to this ring. Note that, if 162running the application with aligned chunk mode, the kernel will mask 163the incoming addr. E.g. for a chunk size of 2k, the log2(2048) LSB of 164the addr will be masked off, meaning that 2048, 2050 and 3000 refers 165to the same chunk. If the user application is run in the unaligned 166chunks mode, then the incoming addr will be left untouched. 167 168 169UMEM Completion Ring 170~~~~~~~~~~~~~~~~~~~~ 171 172The COMPLETION Ring is used transfer ownership of UMEM frames from 173kernel-space to user-space. Just like the FILL ring, UMEM indices are 174used. 175 176Frames passed from the kernel to user-space are frames that has been 177sent (TX ring) and can be used by user-space again. 178 179The user application consumes UMEM addrs from this ring. 180 181 182RX Ring 183~~~~~~~ 184 185The RX ring is the receiving side of a socket. Each entry in the ring 186is a struct xdp_desc descriptor. The descriptor contains UMEM offset 187(addr) and the length of the data (len). 188 189If no frames have been passed to kernel via the FILL ring, no 190descriptors will (or can) appear on the RX ring. 191 192The user application consumes struct xdp_desc descriptors from this 193ring. 194 195TX Ring 196~~~~~~~ 197 198The TX ring is used to send frames. The struct xdp_desc descriptor is 199filled (index, length and offset) and passed into the ring. 200 201To start the transfer a sendmsg() system call is required. This might 202be relaxed in the future. 203 204The user application produces struct xdp_desc descriptors to this 205ring. 206 207Libbpf 208====== 209 210Libbpf is a helper library for eBPF and XDP that makes using these 211technologies a lot simpler. It also contains specific helper functions 212in tools/lib/bpf/xsk.h for facilitating the use of AF_XDP. It 213contains two types of functions: those that can be used to make the 214setup of AF_XDP socket easier and ones that can be used in the data 215plane to access the rings safely and quickly. To see an example on how 216to use this API, please take a look at the sample application in 217samples/bpf/xdpsock_usr.c which uses libbpf for both setup and data 218plane operations. 219 220We recommend that you use this library unless you have become a power 221user. It will make your program a lot simpler. 222 223XSKMAP / BPF_MAP_TYPE_XSKMAP 224============================ 225 226On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) that 227is used in conjunction with bpf_redirect_map() to pass the ingress 228frame to a socket. 229 230The user application inserts the socket into the map, via the bpf() 231system call. 232 233Note that if an XDP program tries to redirect to a socket that does 234not match the queue configuration and netdev, the frame will be 235dropped. E.g. an AF_XDP socket is bound to netdev eth0 and 236queue 17. Only the XDP program executing for eth0 and queue 17 will 237successfully pass data to the socket. Please refer to the sample 238application (samples/bpf/) in for an example. 239 240Configuration Flags and Socket Options 241====================================== 242 243These are the various configuration flags that can be used to control 244and monitor the behavior of AF_XDP sockets. 245 246XDP_COPY and XDP_ZERO_COPY bind flags 247------------------------------------- 248 249When you bind to a socket, the kernel will first try to use zero-copy 250copy. If zero-copy is not supported, it will fall back on using copy 251mode, i.e. copying all packets out to user space. But if you would 252like to force a certain mode, you can use the following flags. If you 253pass the XDP_COPY flag to the bind call, the kernel will force the 254socket into copy mode. If it cannot use copy mode, the bind call will 255fail with an error. Conversely, the XDP_ZERO_COPY flag will force the 256socket into zero-copy mode or fail. 257 258XDP_SHARED_UMEM bind flag 259------------------------- 260 261This flag enables you to bind multiple sockets to the same UMEM, but 262only if they share the same queue id. In this mode, each socket has 263their own RX and TX rings, but the UMEM (tied to the fist socket 264created) only has a single FILL ring and a single COMPLETION 265ring. To use this mode, create the first socket and bind it in the normal 266way. Create a second socket and create an RX and a TX ring, or at 267least one of them, but no FILL or COMPLETION rings as the ones from 268the first socket will be used. In the bind call, set he 269XDP_SHARED_UMEM option and provide the initial socket's fd in the 270sxdp_shared_umem_fd field. You can attach an arbitrary number of extra 271sockets this way. 272 273What socket will then a packet arrive on? This is decided by the XDP 274program. Put all the sockets in the XSK_MAP and just indicate which 275index in the array you would like to send each packet to. A simple 276round-robin example of distributing packets is shown below: 277 278.. code-block:: c 279 280 #include <linux/bpf.h> 281 #include "bpf_helpers.h" 282 283 #define MAX_SOCKS 16 284 285 struct { 286 __uint(type, BPF_MAP_TYPE_XSKMAP); 287 __uint(max_entries, MAX_SOCKS); 288 __uint(key_size, sizeof(int)); 289 __uint(value_size, sizeof(int)); 290 } xsks_map SEC(".maps"); 291 292 static unsigned int rr; 293 294 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) 295 { 296 rr = (rr + 1) & (MAX_SOCKS - 1); 297 298 return bpf_redirect_map(&xsks_map, rr, XDP_DROP); 299 } 300 301Note, that since there is only a single set of FILL and COMPLETION 302rings, and they are single producer, single consumer rings, you need 303to make sure that multiple processes or threads do not use these rings 304concurrently. There are no synchronization primitives in the 305libbpf code that protects multiple users at this point in time. 306 307Libbpf uses this mode if you create more than one socket tied to the 308same umem. However, note that you need to supply the 309XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf_flag with the 310xsk_socket__create calls and load your own XDP program as there is no 311built in one in libbpf that will route the traffic for you. 312 313XDP_USE_NEED_WAKEUP bind flag 314----------------------------- 315 316This option adds support for a new flag called need_wakeup that is 317present in the FILL ring and the TX ring, the rings for which user 318space is a producer. When this option is set in the bind call, the 319need_wakeup flag will be set if the kernel needs to be explicitly 320woken up by a syscall to continue processing packets. If the flag is 321zero, no syscall is needed. 322 323If the flag is set on the FILL ring, the application needs to call 324poll() to be able to continue to receive packets on the RX ring. This 325can happen, for example, when the kernel has detected that there are no 326more buffers on the FILL ring and no buffers left on the RX HW ring of 327the NIC. In this case, interrupts are turned off as the NIC cannot 328receive any packets (as there are no buffers to put them in), and the 329need_wakeup flag is set so that user space can put buffers on the 330FILL ring and then call poll() so that the kernel driver can put these 331buffers on the HW ring and start to receive packets. 332 333If the flag is set for the TX ring, it means that the application 334needs to explicitly notify the kernel to send any packets put on the 335TX ring. This can be accomplished either by a poll() call, as in the 336RX path, or by calling sendto(). 337 338An example of how to use this flag can be found in 339samples/bpf/xdpsock_user.c. An example with the use of libbpf helpers 340would look like this for the TX path: 341 342.. code-block:: c 343 344 if (xsk_ring_prod__needs_wakeup(&my_tx_ring)) 345 sendto(xsk_socket__fd(xsk_handle), NULL, 0, MSG_DONTWAIT, NULL, 0); 346 347I.e., only use the syscall if the flag is set. 348 349We recommend that you always enable this mode as it usually leads to 350better performance especially if you run the application and the 351driver on the same core, but also if you use different cores for the 352application and the kernel driver, as it reduces the number of 353syscalls needed for the TX path. 354 355XDP_{RX|TX|UMEM_FILL|UMEM_COMPLETION}_RING setsockopts 356------------------------------------------------------ 357 358These setsockopts sets the number of descriptors that the RX, TX, 359FILL, and COMPLETION rings respectively should have. It is mandatory 360to set the size of at least one of the RX and TX rings. If you set 361both, you will be able to both receive and send traffic from your 362application, but if you only want to do one of them, you can save 363resources by only setting up one of them. Both the FILL ring and the 364COMPLETION ring are mandatory as you need to have a UMEM tied to your 365socket. But if the XDP_SHARED_UMEM flag is used, any socket after the 366first one does not have a UMEM and should in that case not have any 367FILL or COMPLETION rings created as the ones from the shared umem will 368be used. Note, that the rings are single-producer single-consumer, so 369do not try to access them from multiple processes at the same 370time. See the XDP_SHARED_UMEM section. 371 372In libbpf, you can create Rx-only and Tx-only sockets by supplying 373NULL to the rx and tx arguments, respectively, to the 374xsk_socket__create function. 375 376If you create a Tx-only socket, we recommend that you do not put any 377packets on the fill ring. If you do this, drivers might think you are 378going to receive something when you in fact will not, and this can 379negatively impact performance. 380 381XDP_UMEM_REG setsockopt 382----------------------- 383 384This setsockopt registers a UMEM to a socket. This is the area that 385contain all the buffers that packet can recide in. The call takes a 386pointer to the beginning of this area and the size of it. Moreover, it 387also has parameter called chunk_size that is the size that the UMEM is 388divided into. It can only be 2K or 4K at the moment. If you have an 389UMEM area that is 128K and a chunk size of 2K, this means that you 390will be able to hold a maximum of 128K / 2K = 64 packets in your UMEM 391area and that your largest packet size can be 2K. 392 393There is also an option to set the headroom of each single buffer in 394the UMEM. If you set this to N bytes, it means that the packet will 395start N bytes into the buffer leaving the first N bytes for the 396application to use. The final option is the flags field, but it will 397be dealt with in separate sections for each UMEM flag. 398 399XDP_STATISTICS getsockopt 400------------------------- 401 402Gets drop statistics of a socket that can be useful for debug 403purposes. The supported statistics are shown below: 404 405.. code-block:: c 406 407 struct xdp_statistics { 408 __u64 rx_dropped; /* Dropped for reasons other than invalid desc */ 409 __u64 rx_invalid_descs; /* Dropped due to invalid descriptor */ 410 __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */ 411 }; 412 413XDP_OPTIONS getsockopt 414---------------------- 415 416Gets options from an XDP socket. The only one supported so far is 417XDP_OPTIONS_ZEROCOPY which tells you if zero-copy is on or not. 418 419Usage 420===== 421 422In order to use AF_XDP sockets two parts are needed. The 423user-space application and the XDP program. For a complete setup and 424usage example, please refer to the sample application. The user-space 425side is xdpsock_user.c and the XDP side is part of libbpf. 426 427The XDP code sample included in tools/lib/bpf/xsk.c is the following: 428 429.. code-block:: c 430 431 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) 432 { 433 int index = ctx->rx_queue_index; 434 435 // A set entry here means that the corresponding queue_id 436 // has an active AF_XDP socket bound to it. 437 if (bpf_map_lookup_elem(&xsks_map, &index)) 438 return bpf_redirect_map(&xsks_map, index, 0); 439 440 return XDP_PASS; 441 } 442 443A simple but not so performance ring dequeue and enqueue could look 444like this: 445 446.. code-block:: c 447 448 // struct xdp_rxtx_ring { 449 // __u32 *producer; 450 // __u32 *consumer; 451 // struct xdp_desc *desc; 452 // }; 453 454 // struct xdp_umem_ring { 455 // __u32 *producer; 456 // __u32 *consumer; 457 // __u64 *desc; 458 // }; 459 460 // typedef struct xdp_rxtx_ring RING; 461 // typedef struct xdp_umem_ring RING; 462 463 // typedef struct xdp_desc RING_TYPE; 464 // typedef __u64 RING_TYPE; 465 466 int dequeue_one(RING *ring, RING_TYPE *item) 467 { 468 __u32 entries = *ring->producer - *ring->consumer; 469 470 if (entries == 0) 471 return -1; 472 473 // read-barrier! 474 475 *item = ring->desc[*ring->consumer & (RING_SIZE - 1)]; 476 (*ring->consumer)++; 477 return 0; 478 } 479 480 int enqueue_one(RING *ring, const RING_TYPE *item) 481 { 482 u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer); 483 484 if (free_entries == 0) 485 return -1; 486 487 ring->desc[*ring->producer & (RING_SIZE - 1)] = *item; 488 489 // write-barrier! 490 491 (*ring->producer)++; 492 return 0; 493 } 494 495But please use the libbpf functions as they are optimized and ready to 496use. Will make your life easier. 497 498Sample application 499================== 500 501There is a xdpsock benchmarking/test application included that 502demonstrates how to use AF_XDP sockets with private UMEMs. Say that 503you would like your UDP traffic from port 4242 to end up in queue 16, 504that we will enable AF_XDP on. Here, we use ethtool for this:: 505 506 ethtool -N p3p2 rx-flow-hash udp4 fn 507 ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \ 508 action 16 509 510Running the rxdrop benchmark in XDP_DRV mode can then be done 511using:: 512 513 samples/bpf/xdpsock -i p3p2 -q 16 -r -N 514 515For XDP_SKB mode, use the switch "-S" instead of "-N" and all options 516can be displayed with "-h", as usual. 517 518This sample application uses libbpf to make the setup and usage of 519AF_XDP simpler. If you want to know how the raw uapi of AF_XDP is 520really used to make something more advanced, take a look at the libbpf 521code in tools/lib/bpf/xsk.[ch]. 522 523FAQ 524======= 525 526Q: I am not seeing any traffic on the socket. What am I doing wrong? 527 528A: When a netdev of a physical NIC is initialized, Linux usually 529 allocates one RX and TX queue pair per core. So on a 8 core system, 530 queue ids 0 to 7 will be allocated, one per core. In the AF_XDP 531 bind call or the xsk_socket__create libbpf function call, you 532 specify a specific queue id to bind to and it is only the traffic 533 towards that queue you are going to get on you socket. So in the 534 example above, if you bind to queue 0, you are NOT going to get any 535 traffic that is distributed to queues 1 through 7. If you are 536 lucky, you will see the traffic, but usually it will end up on one 537 of the queues you have not bound to. 538 539 There are a number of ways to solve the problem of getting the 540 traffic you want to the queue id you bound to. If you want to see 541 all the traffic, you can force the netdev to only have 1 queue, queue 542 id 0, and then bind to queue 0. You can use ethtool to do this:: 543 544 sudo ethtool -L <interface> combined 1 545 546 If you want to only see part of the traffic, you can program the 547 NIC through ethtool to filter out your traffic to a single queue id 548 that you can bind your XDP socket to. Here is one example in which 549 UDP traffic to and from port 4242 are sent to queue 2:: 550 551 sudo ethtool -N <interface> rx-flow-hash udp4 fn 552 sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \ 553 4242 action 2 554 555 A number of other ways are possible all up to the capabilities of 556 the NIC you have. 557 558Q: Can I use the XSKMAP to implement a switch betwen different umems 559 in copy mode? 560 561A: The short answer is no, that is not supported at the moment. The 562 XSKMAP can only be used to switch traffic coming in on queue id X 563 to sockets bound to the same queue id X. The XSKMAP can contain 564 sockets bound to different queue ids, for example X and Y, but only 565 traffic goming in from queue id Y can be directed to sockets bound 566 to the same queue id Y. In zero-copy mode, you should use the 567 switch, or other distribution mechanism, in your NIC to direct 568 traffic to the correct queue id and socket. 569 570Credits 571======= 572 573- Björn Töpel (AF_XDP core) 574- Magnus Karlsson (AF_XDP core) 575- Alexander Duyck 576- Alexei Starovoitov 577- Daniel Borkmann 578- Jesper Dangaard Brouer 579- John Fastabend 580- Jonathan Corbet (LWN coverage) 581- Michael S. Tsirkin 582- Qi Z Zhang 583- Willem de Bruijn 584