Lines Matching +full:queue +full:- +full:pkt +full:- +full:tx
1 .. SPDX-License-Identifier: GPL-2.0
20 XDP programs to redirect frames to a memory buffer in a user-space
25 TX ring. A socket can receive packets on the RX ring and it can send
26 packets on the TX ring. These rings are registered and sized with the
28 to have at least one of these rings for each socket. An RX or TX
30 UMEM. RX and TX can share the same UMEM so that a packet does not have
31 to be copied between RX and TX. Moreover, if a packet needs to be kept
48 space, for either TX or RX. Thus, the frame addrs appearing in the
50 TX ring. In summary, the RX and FILL rings are used for the RX path
51 and the TX and COMPLETION rings are used for the TX path.
54 specific queue id on that device, and it is not until bind is
64 single-consumer / single-producer (for performance reasons), the new
65 process has to create its own socket with associated RX and TX rings,
72 user-space application can place an XSK at an arbitrary place in this
99 http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. Do
106 ----
109 equal-sized frames. An UMEM is associated to a netdev and a specific
110 queue id of that netdev. It is created and configured (chunk size,
112 system call. A UMEM is bound to a netdev and queue id, via the bind()
121 The UMEM has two single-producer/single-consumer rings that are used
123 user-space application.
126 -----
129 TX. All rings are single-producer/single-consumer, so the user-space
134 with the UMEM must have an RX queue, TX queue or both. Say, that there
135 is a setup with four sockets (all doing TX and RX). Then there will be
136 one FILL ring, one COMPLETION ring, four TX rings and four RX rings.
145 calls and mmapped to user-space using the appropriate offset to mmap()
155 user-space to kernel-space. The UMEM addrs are passed in the ring. As
173 kernel-space to user-space. Just like the FILL ring, UMEM indices are
176 Frames passed from the kernel to user-space are frames that has been
177 sent (TX ring) and can be used by user-space again.
195 TX Ring
198 The TX ring is used to send frames. The struct xdp_desc descriptor is
231 not match the queue configuration and netdev, the frame will be
233 queue 17. Only the XDP program executing for eth0 and queue 17 will
244 ------------------------------------
246 When you bind to a socket, the kernel will first try to use zero-copy
247 copy. If zero-copy is not supported, it will fall back on using copy
253 socket into zero-copy mode or fail.
256 -------------------------
259 works on the same queue id, between queue ids and between
260 netdevs/devices. In this mode, each socket has their own RX and TX
263 unique netdev and queue id tuple that you bind to.
266 sockets bound to the same netdev and queue id. The UMEM (tied to the
270 it in the normal way. Create a second socket and create an RX and a TX
280 round-robin example of distributing packets is shown below:
282 .. code-block:: c
300 rr = (rr + 1) & (MAX_SOCKS - 1);
318 bound to different queue ids and/or netdevs. In this case you have to
321 to two different queue ids on the same netdev. Create the first socket
323 and a TX ring, or at least one of them, and then one FILL and
330 case where sockets were bound to the same queue id and
332 the packets to the right queue. In the previous example, there is only
333 one queue shared among sockets, so the NIC cannot do this steering. It
343 Note that a UMEM can be shared between sockets on the same queue id
348 -----------------------------
351 present in the FILL ring and the TX ring, the rings for which user
367 If the flag is set for the TX ring, it means that the application
369 TX ring. This can be accomplished either by a poll() call, as in the
373 TX path:
375 .. code-block:: c
386 syscalls needed for the TX path.
388 XDP_{RX|TX|UMEM_FILL|UMEM_COMPLETION}_RING setsockopts
389 ------------------------------------------------------
391 These setsockopts sets the number of descriptors that the RX, TX,
393 to set the size of at least one of the RX and TX rings. If you set
401 be used. Note, that the rings are single-producer single-consumer, so
405 In libbpf, you can create Rx-only and Tx-only sockets by supplying
406 NULL to the rx and tx arguments, respectively, to the
409 If you create a Tx-only socket, we recommend that you do not put any
415 -----------------------
433 --------------------------
437 is created by a privileged process and passed to a non-privileged one.
442 --------------------------------
446 mode to allow application to tune the per-socket maximum iteration for
448 Allowed range is [32, xs->tx->nentries].
451 -------------------------
456 .. code-block:: c
465 ----------------------
468 XDP_OPTIONS_ZEROCOPY which tells you if zero-copy is on or not.
470 Multi-Buffer Support
473 With multi-buffer support, programs using AF_XDP sockets can receive
475 zero-copy mode. For example, a packet can consist of two
488 To enable multi-buffer support for an AF_XDP socket, use the new bind
489 flag XDP_USE_SG. If this is not provided, all multi-buffer packets
491 needs to be in multi-buffer mode. This can be accomplished by using
495 XDP_PKT_CONTD is introduced in the options field of the Rx and Tx
498 of the packet. Why the reverse logic of end-of-packet (eop) flag found
499 in many NICs? Just to preserve compatibility with non-multi-buffer
501 and the apps set the options field to zero for Tx, as anything else
504 These are the semantics for producing packets onto AF_XDP Tx ring
523 * For zero-copy mode, the limit is up to what the NIC HW
526 CONFIG_MAX_SKB_FRAGS + 1) for zero-copy mode, as it would have
528 NIC supports. Kind of defeats the purpose of zero-copy mode. How to
529 probe for this limit is explained in the "probe for multi-buffer
532 On the Rx path in copy-mode, the xsk core copies the XDP data into
534 detailed before. Zero-copy mode works the same, though the data is not
551 An example program each for Rx and Tx multi-buffer support can be found
555 -----
557 In order to use AF_XDP sockets two parts are needed. The user-space
559 please refer to the xdp-project at
560 https://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-example.
564 .. code-block:: c
568 int index = ctx->rx_queue_index;
581 .. code-block:: c
603 __u32 entries = *ring->producer - *ring->consumer;
606 return -1;
608 // read-barrier!
610 *item = ring->desc[*ring->consumer & (RING_SIZE - 1)];
611 (*ring->consumer)++;
617 u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer);
620 return -1;
622 ring->desc[*ring->producer & (RING_SIZE - 1)] = *item;
624 // write-barrier!
626 (*ring->producer)++;
633 Usage Multi-Buffer Rx
634 ---------------------
636 Here is a simple Rx path pseudo-code example (using libxdp interfaces
639 .. code-block:: c
645 static char *pkt;
647 int rcvd = xsk_ring_cons__peek(&xsk->rx, opt_batch_size, &idx_rx);
649 xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq);
652 struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++);
653 char *frag = xsk_umem__get_data(xsk->umem->buffer, desc->addr);
654 bool eop = !(desc->options & XDP_PKT_CONTD);
657 pkt = frag;
659 add_frag_to_pkt(pkt, frag);
662 process_pkt(pkt);
666 *xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = desc->addr;
669 xsk_ring_prod__submit(&xsk->umem->fq, rcvd);
670 xsk_ring_cons__release(&xsk->rx, rcvd);
673 Usage Multi-Buffer Tx
674 ---------------------
676 Here is an example Tx path pseudo-code (using libxdp interfaces for
681 .. code-block:: c
683 void tx_packets(struct xsk_socket_info *xsk, struct pkt *pkts,
688 xsk_ring_prod__reserve(&xsk->tx, batch_size, &idx);
697 tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx + i++);
698 tx_desc->addr = addr;
701 tx_desc->len = xsk_frame_size;
702 tx_desc->options = XDP_PKT_CONTD;
704 tx_desc->len = len;
705 tx_desc->options = 0;
708 len -= tx_desc->len;
720 xsk_ring_prod__submit(&xsk->tx, i);
723 Probing for Multi-Buffer Support
724 --------------------------------
726 To discover if a driver supports multi-buffer AF_XDP in SKB or DRV
729 querying for XDP multi-buffer support. If XDP supports multi-buffer in
732 To discover if a driver supports multi-buffer AF_XDP in zero-copy
734 flag. If it is set, it means that at least zero-copy is supported and
738 supported by this device in zero-copy mode. These are the possible
741 1: Multi-buffer for zero-copy is not supported by this device, as max
742 one fragment supported means that multi-buffer is not possible.
744 >=2: Multi-buffer is supported in zero-copy mode for this device. The
750 Multi-Buffer Support for Zero-Copy Drivers
751 ------------------------------------------
753 Zero-copy drivers usually use the batched APIs for Rx and Tx
754 processing. Note that the Tx batch API guarantees that it will provide
755 a batch of Tx descriptors that ends with full packet at the end. This
756 to facilitate extending a zero-copy driver with multi-buffer support.
761 https://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-example
764 up in queue 16, that we will enable AF_XDP on. Here, we use ethtool
767 ethtool -N p3p2 rx-flow-hash udp4 fn
768 ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \
774 samples/bpf/xdpsock -i p3p2 -q 16 -r -N
776 For XDP_SKB mode, use the switch "-S" instead of "-N" and all options
777 can be displayed with "-h", as usual.
790 allocates one RX and TX queue pair per core. So on a 8 core system,
791 queue ids 0 to 7 will be allocated, one per core. In the AF_XDP
793 specify a specific queue id to bind to and it is only the traffic
794 towards that queue you are going to get on you socket. So in the
795 example above, if you bind to queue 0, you are NOT going to get any
801 traffic you want to the queue id you bound to. If you want to see
802 all the traffic, you can force the netdev to only have 1 queue, queue
803 id 0, and then bind to queue 0. You can use ethtool to do this::
805 sudo ethtool -L <interface> combined 1
808 NIC through ethtool to filter out your traffic to a single queue id
810 UDP traffic to and from port 4242 are sent to queue 2::
812 sudo ethtool -N <interface> rx-flow-hash udp4 fn
813 sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \
823 XSKMAP can only be used to switch traffic coming in on queue id X
824 to sockets bound to the same queue id X. The XSKMAP can contain
825 sockets bound to different queue ids, for example X and Y, but only
826 traffic goming in from queue id Y can be directed to sockets bound
827 to the same queue id Y. In zero-copy mode, you should use the
829 traffic to the correct queue id and socket.
835 same buffer into the FILL ring and the TX ring at the same time, the
839 belonging to different queue ids or netdevs bound with the
845 - Björn Töpel (AF_XDP core)
846 - Magnus Karlsson (AF_XDP core)
847 - Alexander Duyck
848 - Alexei Starovoitov
849 - Daniel Borkmann
850 - Jesper Dangaard Brouer
851 - John Fastabend
852 - Jonathan Corbet (LWN coverage)
853 - Michael S. Tsirkin
854 - Qi Z Zhang
855 - Willem de Bruijn