1.. SPDX-License-Identifier: GPL-2.0 2 3============= 4Page Pool API 5============= 6 7.. kernel-doc:: include/net/page_pool/helpers.h 8 :doc: page_pool allocator 9 10Architecture overview 11===================== 12 13.. code-block:: none 14 15 +------------------+ 16 | Driver | 17 +------------------+ 18 ^ 19 | 20 | 21 | 22 v 23 +--------------------------------------------+ 24 | request memory | 25 +--------------------------------------------+ 26 ^ ^ 27 | | 28 | Pool empty | Pool has entries 29 | | 30 v v 31 +-----------------------+ +------------------------+ 32 | alloc (and map) pages | | get page from cache | 33 +-----------------------+ +------------------------+ 34 ^ ^ 35 | | 36 | cache available | No entries, refill 37 | | from ptr-ring 38 | | 39 v v 40 +-----------------+ +------------------+ 41 | Fast cache | | ptr-ring cache | 42 +-----------------+ +------------------+ 43 44Monitoring 45========== 46Information about allocated page pools, their memory use, recycling statistics 47etc. can be accessed via the netdev genetlink family 48(see Documentation/netlink/specs/netdev.yaml). 49 50Statistics 51---------- 52 53.. kernel-doc:: include/net/page_pool/types.h 54 :identifiers: struct page_pool_recycle_stats 55 struct page_pool_alloc_stats 56 struct page_pool_stats 57 58API interface 59============= 60The number of pools created **must** match the number of NAPI contexts / queues 61unless hardware restrictions make that impossible. This would otherwise beat the 62purpose of page pool, which is allocate pages fast from cache without locking. 63This lockless guarantee naturally comes from running under a NAPI softirq. 64The protection doesn't strictly have to be NAPI, any guarantee that allocating 65a page will cause no race conditions is enough. 66 67If ``params.napi`` is set, the NAPI instance must be the sole consumer 68context for pages allocated from the pool. In other words, when running in 69that NAPI context, the page pool may safely access consumer-side resources 70**without any additional locking**. 71 72.. kernel-doc:: net/core/page_pool.c 73 :identifiers: page_pool_create 74 75.. kernel-doc:: include/net/page_pool/types.h 76 :identifiers: struct page_pool_params 77 78.. kernel-doc:: include/net/page_pool/helpers.h 79 :identifiers: page_pool_put_page page_pool_put_full_page 80 page_pool_recycle_direct page_pool_free_va 81 page_pool_dev_alloc_pages page_pool_dev_alloc_frag 82 page_pool_dev_alloc page_pool_dev_alloc_va 83 page_pool_get_dma_addr page_pool_get_dma_dir 84 85.. kernel-doc:: net/core/page_pool.c 86 :identifiers: page_pool_put_page_bulk 87 88DMA sync 89-------- 90Driver is always responsible for syncing the pages for the CPU. 91Drivers may choose to take care of syncing for the device as well 92or set the ``PP_FLAG_DMA_SYNC_DEV`` flag to request that pages 93allocated from the page pool are already synced for the device. 94 95If ``PP_FLAG_DMA_SYNC_DEV`` is set, the driver must inform the core what portion 96of the buffer has to be synced. This allows the core to avoid syncing the entire 97page when the drivers knows that the device only accessed a portion of the page. 98 99Most drivers will reserve headroom in front of the frame. This part 100of the buffer is not touched by the device, so to avoid syncing 101it drivers can set the ``offset`` field in struct page_pool_params 102appropriately. 103 104For pages recycled on the XDP xmit and skb paths the page pool will 105use the ``max_len`` member of struct page_pool_params to decide how 106much of the page needs to be synced (starting at ``offset``). 107When directly freeing pages in the driver (page_pool_put_page()) 108the ``dma_sync_size`` argument specifies how much of the buffer needs 109to be synced. 110 111If in doubt set ``offset`` to 0, ``max_len`` to ``PAGE_SIZE`` and 112pass -1 as ``dma_sync_size``. That combination of arguments is always 113correct. 114 115Note that the syncing parameters are for the **entire page**, even if 116the driver allocates fragments (e.g. via ``page_pool_dev_alloc_frag()``). 117Unless the driver author really understands page pool internals 118it's recommended to always use ``offset = 0``, ``max_len = PAGE_SIZE`` 119with fragmented page pools. 120 121Coding examples 122=============== 123 124Registration 125------------ 126 127.. code-block:: c 128 129 /* Page pool registration */ 130 struct page_pool_params pp_params = { 0 }; 131 struct xdp_rxq_info xdp_rxq; 132 int err; 133 134 pp_params.order = 0; 135 /* internal DMA mapping in page_pool */ 136 pp_params.flags = PP_FLAG_DMA_MAP; 137 pp_params.pool_size = DESC_NUM; 138 pp_params.nid = NUMA_NO_NODE; 139 pp_params.dev = priv->dev; 140 pp_params.napi = napi; /* only if this NAPI is the sole consumer, see above */ 141 pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE; 142 page_pool = page_pool_create(&pp_params); 143 144 err = xdp_rxq_info_reg(&xdp_rxq, ndev, 0); 145 if (err) 146 goto err_out; 147 148 err = xdp_rxq_info_reg_mem_model(&xdp_rxq, MEM_TYPE_PAGE_POOL, page_pool); 149 if (err) 150 goto err_out; 151 152NAPI poller 153----------- 154 155 156.. code-block:: c 157 158 /* NAPI Rx poller */ 159 enum dma_data_direction dma_dir; 160 161 dma_dir = page_pool_get_dma_dir(dring->page_pool); 162 while (done < budget) { 163 if (some error) 164 page_pool_recycle_direct(page_pool, page); 165 if (packet_is_xdp) { 166 if XDP_DROP: 167 page_pool_recycle_direct(page_pool, page); 168 } else (packet_is_skb) { 169 skb_mark_for_recycle(skb); 170 new_page = page_pool_dev_alloc_pages(page_pool); 171 } 172 } 173 174Driver unload 175------------- 176 177.. code-block:: c 178 179 /* Driver unload */ 180 page_pool_put_full_page(page_pool, page, false); 181 xdp_rxq_info_unreg(&xdp_rxq); 182