Lines Matching +full:one +full:- +full:to +full:- +full:one
1 .. SPDX-License-Identifier: GPL-2.0
18 redirect ingress frames to other XDP enabled netdevs, using the
20 XDP programs to redirect frames to a memory buffer in a user-space
28 to have at least one of these rings for each socket. An RX or TX
29 descriptor ring points to a data buffer in a memory area called a
31 to be copied between RX and TX. Moreover, if a packet needs to be kept
32 for a while due to a possible retransmit, the descriptor that points
33 to that packet can be changed to point to another and reused right
37 one of the rings references a frame by referencing its addr. The addr
43 FILL ring is used by the application to send down addr for the kernel
44 to fill in with RX packet data. References to these frames will then
53 The socket is then finally bound with a bind() call to a device and a
55 completed that traffic starts to flow.
58 wants to do this, it simply skips the registration of the UMEM and its
60 call and submits the XSK of the process it would like to share UMEM
62 then receive frame addr references in its own RX ring that point to
64 single-consumer / single-producer (for performance reasons), the new
65 process has to create its own socket with associated RX and TX rings,
67 reason that there is only one set of FILL and COMPLETION rings per
68 UMEM. It is the responsibility of a single process to handle the UMEM.
70 How is then packets distributed from an XDP program to the XSKs? There
72 user-space application can place an XSK at an arbitrary place in this
73 map. The XDP program can then redirect a packet to a specific index in
75 indeed bound to that device and ring number. If not, the packet is
77 dropped. This also means that it is currently mandatory to have an XDP
78 program loaded (and one XSK in the XSKMAP) to be able to get any
79 traffic to user space through the XSK.
84 together with the generic XDP support and copies out the data to user
87 code to provide better performance, but there is still a copy of the
93 In order to use an AF_XDP socket, a number of associated objects need
94 to be setup. These objects and their options are explained in the
99 http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. Do
106 ----
109 equal-sized frames. An UMEM is associated to a netdev and a specific
112 system call. A UMEM is bound to a netdev and queue id, via the bind()
115 An AF_XDP is socket linked to a single UMEM, but one UMEM can have
116 multiple AF_XDP sockets. To share an UMEM created via one socket A,
119 of A to struct sockaddr_xdp member sxdp_shared_umem_fd.
121 The UMEM has two single-producer/single-consumer rings that are used
122 to transfer ownership of UMEM frames between the kernel and the
123 user-space application.
126 -----
129 TX. All rings are single-producer/single-consumer, so the user-space
131 processes/threads are reading/writing to them.
136 one FILL ring, one COMPLETION ring, four TX rings and four RX rings.
145 calls and mmapped to user-space using the appropriate offset to mmap()
149 The size of the rings need to be of size power of two.
154 The FILL ring is used to transfer ownership of UMEM frames from
155 user-space to kernel-space. The UMEM addrs are passed in the ring. As
159 Frames passed to the kernel are used for the ingress path (RX rings).
161 The user application produces UMEM addrs to this ring. Note that, if
165 to the same chunk. If the user application is run in the unaligned
173 kernel-space to user-space. Just like the FILL ring, UMEM indices are
176 Frames passed from the kernel to user-space are frames that has been
177 sent (TX ring) and can be used by user-space again.
189 If no frames have been passed to kernel via the FILL ring, no
198 The TX ring is used to send frames. The struct xdp_desc descriptor is
201 To start the transfer a sendmsg() system call is required. This might
204 The user application produces struct xdp_desc descriptors to this
213 AF_XDP. It contains two types of functions: those that can be used to
215 data plane to access the rings safely and quickly.
224 is used in conjunction with bpf_redirect_map() to pass the ingress
225 frame to a socket.
230 Note that if an XDP program tries to redirect to a socket that does
232 dropped. E.g. an AF_XDP socket is bound to netdev eth0 and
234 successfully pass data to the socket. Please refer to the sample
240 These are the various configuration flags that can be used to control
244 ------------------------------------
246 When you bind to a socket, the kernel will first try to use zero-copy
247 copy. If zero-copy is not supported, it will fall back on using copy
248 mode, i.e. copying all packets out to user space. But if you would
249 like to force a certain mode, you can use the following flags. If you
250 pass the XDP_COPY flag to the bind call, the kernel will force the
253 socket into zero-copy mode or fail.
256 -------------------------
258 This flag enables you to bind multiple sockets to the same UMEM. It
261 rings as usual, but you are going to have one or more FILL and
262 COMPLETION ring pairs. You have to create one of these pairs per
263 unique netdev and queue id tuple that you bind to.
265 Starting with the case were we would like to share a UMEM between
266 sockets bound to the same netdev and queue id. The UMEM (tied to the
269 we have bound to. To use this mode, create the first socket and bind
271 ring, or at least one of them, but no FILL or COMPLETION rings as the
279 index in the array you would like to send each packet to. A simple
280 round-robin example of distributing packets is shown below:
282 .. code-block:: c
300 rr = (rr + 1) & (MAX_SOCKS - 1);
307 to make sure that multiple processes or threads do not use these rings
311 Libbpf uses this mode if you create more than one socket tied to the
312 same UMEM. However, note that you need to supply the
315 built in one in libbpf that will route the traffic for you.
318 bound to different queue ids and/or netdevs. In this case you have to
319 create one FILL ring and one COMPLETION ring for each unique
320 netdev,queue_id pair. Let us say you want to create two sockets bound
321 to two different queue ids on the same netdev. Create the first socket
323 and a TX ring, or at least one of them, and then one FILL and
327 socket. These two sockets will now share one and the same UMEM.
329 There is no need to supply an XDP program like the one in the previous
330 case where sockets were bound to the same queue id and
331 device. Instead, use the NIC's packet steering capabilities to steer
332 the packets to the right queue. In the previous example, there is only
333 one queue shared among sockets, so the NIC cannot do this steering. It
336 In libbpf, you need to use the xsk_socket__create_shared() API as it
337 takes a reference to a FILL ring and a COMPLETION ring that will be
338 created for you and bound to the shared UMEM. You can use this
341 one. Both methods yield the same result.
348 -----------------------------
353 need_wakeup flag will be set if the kernel needs to be explicitly
354 woken up by a syscall to continue processing packets. If the flag is
357 If the flag is set on the FILL ring, the application needs to call
358 poll() to be able to continue to receive packets on the RX ring. This
362 receive any packets (as there are no buffers to put them in), and the
365 buffers on the HW ring and start to receive packets.
368 needs to explicitly notify the kernel to send any packets put on the
375 .. code-block:: c
382 We recommend that you always enable this mode as it usually leads to
389 ------------------------------------------------------
393 to set the size of at least one of the RX and TX rings. If you set
394 both, you will be able to both receive and send traffic from your
395 application, but if you only want to do one of them, you can save
396 resources by only setting up one of them. Both the FILL ring and the
397 COMPLETION ring are mandatory as you need to have a UMEM tied to your
399 first one does not have a UMEM and should in that case not have any
401 be used. Note, that the rings are single-producer single-consumer, so
402 do not try to access them from multiple processes at the same
405 In libbpf, you can create Rx-only and Tx-only sockets by supplying
406 NULL to the rx and tx arguments, respectively, to the
409 If you create a Tx-only socket, we recommend that you do not put any
411 going to receive something when you in fact will not, and this can
415 -----------------------
417 This setsockopt registers a UMEM to a socket. This is the area that
419 pointer to the beginning of this area and the size of it. Moreover, it
423 will be able to hold a maximum of 128K / 2K = 64 packets in your UMEM
426 There is also an option to set the headroom of each single buffer in
427 the UMEM. If you set this to N bytes, it means that the packet will
429 application to use. The final option is the flags field, but it will
433 --------------------------
435 This is a generic SOL_SOCKET option that can be used to tie AF_XDP
436 socket to a particular network interface. It is useful when a socket
437 is created by a privileged process and passed to a non-privileged one.
438 Once the option is set, kernel will refuse attempts to bind that socket
439 to a different interface. Updating the value requires CAP_NET_RAW.
442 --------------------------------
445 and passed to the driver at one send syscall. It is applied in the copy
446 mode to allow application to tune the per-socket maximum iteration for
448 Allowed range is [32, xs->tx->nentries].
451 -------------------------
456 .. code-block:: c
460 __u64 rx_invalid_descs; /* Dropped due to invalid descriptor */
461 __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
465 ----------------------
467 Gets options from an XDP socket. The only one supported so far is
468 XDP_OPTIONS_ZEROCOPY which tells you if zero-copy is on or not.
470 Multi-Buffer Support
473 With multi-buffer support, programs using AF_XDP sockets can receive
475 zero-copy mode. For example, a packet can consist of two
476 frames/buffers, one with the header and the other one with the data,
482 * A packet consists of one or more frames
484 * A descriptor in one of the AF_XDP rings always refers to a single
486 descriptor refers to the whole packet.
488 To enable multi-buffer support for an AF_XDP socket, use the new bind
489 flag XDP_USE_SG. If this is not provided, all multi-buffer packets
491 needs to be in multi-buffer mode. This can be accomplished by using
494 To represent a packet consisting of multiple frames, a new flag called
498 of the packet. Why the reverse logic of end-of-packet (eop) flag found
499 in many NICs? Just to preserve compatibility with non-multi-buffer
500 applications that have this bit set to false for all packets on Rx,
501 and the apps set the options field to zero for Tx, as anything else
517 equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all
519 invalid. To produce an application that will work on any system
520 regardless of this config setting, limit the number of frags to 18,
523 * For zero-copy mode, the limit is up to what the NIC HW
525 consciously chose to not enforce a rigid limit (such as
526 CONFIG_MAX_SKB_FRAGS + 1) for zero-copy mode, as it would have
527 resulted in copy actions under the hood to fit into what limit the
528 NIC supports. Kind of defeats the purpose of zero-copy mode. How to
529 probe for this limit is explained in the "probe for multi-buffer
532 On the Rx path in copy-mode, the xsk core copies the XDP data into
534 detailed before. Zero-copy mode works the same, though the data is not
536 flag set to one, it means that the packet consists of multiple buffers
540 that only a complete packet (all frames in the packet) is sent to the
551 An example program each for Rx and Tx multi-buffer support can be found
555 -----
557 In order to use AF_XDP sockets two parts are needed. The user-space
559 please refer to the xdp-project at
560 https://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-example.
564 .. code-block:: c
568 int index = ctx->rx_queue_index;
571 // has an active AF_XDP socket bound to it.
581 .. code-block:: c
603 __u32 entries = *ring->producer - *ring->consumer;
606 return -1;
608 // read-barrier!
610 *item = ring->desc[*ring->consumer & (RING_SIZE - 1)];
611 (*ring->consumer)++;
617 u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer);
620 return -1;
622 ring->desc[*ring->producer & (RING_SIZE - 1)] = *item;
624 // write-barrier!
626 (*ring->producer)++;
630 But please use the libbpf functions as they are optimized and ready to
633 Usage Multi-Buffer Rx
634 ---------------------
636 Here is a simple Rx path pseudo-code example (using libxdp interfaces
637 for simplicity). Error paths have been excluded to keep it short:
639 .. code-block:: c
647 int rcvd = xsk_ring_cons__peek(&xsk->rx, opt_batch_size, &idx_rx);
649 xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq);
652 struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++);
653 char *frag = xsk_umem__get_data(xsk->umem->buffer, desc->addr);
654 bool eop = !(desc->options & XDP_PKT_CONTD);
666 *xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = desc->addr;
669 xsk_ring_prod__submit(&xsk->umem->fq, rcvd);
670 xsk_ring_cons__release(&xsk->rx, rcvd);
673 Usage Multi-Buffer Tx
674 ---------------------
676 Here is an example Tx path pseudo-code (using libxdp interfaces for
678 eventually will run out of packets to send. Also assumes pkts.addr
679 points to a valid location in the umem.
681 .. code-block:: c
688 xsk_ring_prod__reserve(&xsk->tx, batch_size, &idx);
697 tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx + i++);
698 tx_desc->addr = addr;
701 tx_desc->len = xsk_frame_size;
702 tx_desc->options = XDP_PKT_CONTD;
704 tx_desc->len = len;
705 tx_desc->options = 0;
708 len -= tx_desc->len;
720 xsk_ring_prod__submit(&xsk->tx, i);
723 Probing for Multi-Buffer Support
724 --------------------------------
726 To discover if a driver supports multi-buffer AF_XDP in SKB or DRV
727 mode, use the XDP_FEATURES feature of netlink in linux/netdev.h to
729 querying for XDP multi-buffer support. If XDP supports multi-buffer in
732 To discover if a driver supports multi-buffer AF_XDP in zero-copy
734 flag. If it is set, it means that at least zero-copy is supported and
738 supported by this device in zero-copy mode. These are the possible
741 1: Multi-buffer for zero-copy is not supported by this device, as max
742 one fragment supported means that multi-buffer is not possible.
744 >=2: Multi-buffer is supported in zero-copy mode for this device. The
750 Multi-Buffer Support for Zero-Copy Drivers
751 ------------------------------------------
753 Zero-copy drivers usually use the batched APIs for Rx and Tx
756 to facilitate extending a zero-copy driver with multi-buffer support.
761 https://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-example
762 that demonstrates how to use AF_XDP sockets with private
763 UMEMs. Say that you would like your UDP traffic from port 4242 to end
767 ethtool -N p3p2 rx-flow-hash udp4 fn
768 ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \
774 samples/bpf/xdpsock -i p3p2 -q 16 -r -N
776 For XDP_SKB mode, use the switch "-S" instead of "-N" and all options
777 can be displayed with "-h", as usual.
779 This sample application uses libbpf to make the setup and usage of
780 AF_XDP simpler. If you want to know how the raw uapi of AF_XDP is
781 really used to make something more advanced, take a look at the libbpf
790 allocates one RX and TX queue pair per core. So on a 8 core system,
791 queue ids 0 to 7 will be allocated, one per core. In the AF_XDP
793 specify a specific queue id to bind to and it is only the traffic
794 towards that queue you are going to get on you socket. So in the
795 example above, if you bind to queue 0, you are NOT going to get any
796 traffic that is distributed to queues 1 through 7. If you are
797 lucky, you will see the traffic, but usually it will end up on one
798 of the queues you have not bound to.
800 There are a number of ways to solve the problem of getting the
801 traffic you want to the queue id you bound to. If you want to see
802 all the traffic, you can force the netdev to only have 1 queue, queue
803 id 0, and then bind to queue 0. You can use ethtool to do this::
805 sudo ethtool -L <interface> combined 1
807 If you want to only see part of the traffic, you can program the
808 NIC through ethtool to filter out your traffic to a single queue id
809 that you can bind your XDP socket to. Here is one example in which
810 UDP traffic to and from port 4242 are sent to queue 2::
812 sudo ethtool -N <interface> rx-flow-hash udp4 fn
813 sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \
816 A number of other ways are possible all up to the capabilities of
819 Q: Can I use the XSKMAP to implement a switch between different umems
823 XSKMAP can only be used to switch traffic coming in on queue id X
824 to sockets bound to the same queue id X. The XSKMAP can contain
825 sockets bound to different queue ids, for example X and Y, but only
826 traffic goming in from queue id Y can be directed to sockets bound
827 to the same queue id Y. In zero-copy mode, you should use the
828 switch, or other distribution mechanism, in your NIC to direct
829 traffic to the correct queue id and socket.
833 A: Care has to be taken not to feed the same buffer in the UMEM into
834 more than one ring at the same time. If you for example feed the
837 sending it. This will cause some packets to become corrupted. Same
839 belonging to different queue ids or netdevs bound with the
845 - Björn Töpel (AF_XDP core)
846 - Magnus Karlsson (AF_XDP core)
847 - Alexander Duyck
848 - Alexei Starovoitov
849 - Daniel Borkmann
850 - Jesper Dangaard Brouer
851 - John Fastabend
852 - Jonathan Corbet (LWN coverage)
853 - Michael S. Tsirkin
854 - Qi Z Zhang
855 - Willem de Bruijn