1.. SPDX-License-Identifier: GPL-2.0 2 3============= 4Page Pool API 5============= 6 7The page_pool allocator is optimized for the XDP mode that uses one frame 8per-page, but it can fallback on the regular page allocator APIs. 9 10Basic use involves replacing alloc_pages() calls with the 11page_pool_alloc_pages() call. Drivers should use page_pool_dev_alloc_pages() 12replacing dev_alloc_pages(). 13 14API keeps track of in-flight pages, in order to let API user know 15when it is safe to free a page_pool object. Thus, API users 16must run page_pool_release_page() when a page is leaving the page_pool or 17call page_pool_put_page() where appropriate in order to maintain correct 18accounting. 19 20API user must call page_pool_put_page() once on a page, as it 21will either recycle the page, or in case of refcnt > 1, it will 22release the DMA mapping and in-flight state accounting. 23 24Architecture overview 25===================== 26 27.. code-block:: none 28 29 +------------------+ 30 | Driver | 31 +------------------+ 32 ^ 33 | 34 | 35 | 36 v 37 +--------------------------------------------+ 38 | request memory | 39 +--------------------------------------------+ 40 ^ ^ 41 | | 42 | Pool empty | Pool has entries 43 | | 44 v v 45 +-----------------------+ +------------------------+ 46 | alloc (and map) pages | | get page from cache | 47 +-----------------------+ +------------------------+ 48 ^ ^ 49 | | 50 | cache available | No entries, refill 51 | | from ptr-ring 52 | | 53 v v 54 +-----------------+ +------------------+ 55 | Fast cache | | ptr-ring cache | 56 +-----------------+ +------------------+ 57 58API interface 59============= 60The number of pools created **must** match the number of hardware queues 61unless hardware restrictions make that impossible. This would otherwise beat the 62purpose of page pool, which is allocate pages fast from cache without locking. 63This lockless guarantee naturally comes from running under a NAPI softirq. 64The protection doesn't strictly have to be NAPI, any guarantee that allocating 65a page will cause no race conditions is enough. 66 67* page_pool_create(): Create a pool. 68 * flags: PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV 69 * order: 2^order pages on allocation 70 * pool_size: size of the ptr_ring 71 * nid: preferred NUMA node for allocation 72 * dev: struct device. Used on DMA operations 73 * dma_dir: DMA direction 74 * max_len: max DMA sync memory size 75 * offset: DMA address offset 76 77* page_pool_put_page(): The outcome of this depends on the page refcnt. If the 78 driver bumps the refcnt > 1 this will unmap the page. If the page refcnt is 1 79 the allocator owns the page and will try to recycle it in one of the pool 80 caches. If PP_FLAG_DMA_SYNC_DEV is set, the page will be synced for_device 81 using dma_sync_single_range_for_device(). 82 83* page_pool_put_full_page(): Similar to page_pool_put_page(), but will DMA sync 84 for the entire memory area configured in area pool->max_len. 85 86* page_pool_recycle_direct(): Similar to page_pool_put_full_page() but caller 87 must guarantee safe context (e.g NAPI), since it will recycle the page 88 directly into the pool fast cache. 89 90* page_pool_release_page(): Unmap the page (if mapped) and account for it on 91 in-flight counters. 92 93* page_pool_dev_alloc_pages(): Get a page from the page allocator or page_pool 94 caches. 95 96* page_pool_get_dma_addr(): Retrieve the stored DMA address. 97 98* page_pool_get_dma_dir(): Retrieve the stored DMA direction. 99 100* page_pool_put_page_bulk(): Tries to refill a number of pages into the 101 ptr_ring cache holding ptr_ring producer lock. If the ptr_ring is full, 102 page_pool_put_page_bulk() will release leftover pages to the page allocator. 103 page_pool_put_page_bulk() is suitable to be run inside the driver NAPI tx 104 completion loop for the XDP_REDIRECT use case. 105 Please note the caller must not use data area after running 106 page_pool_put_page_bulk(), as this function overwrites it. 107 108* page_pool_get_stats(): Retrieve statistics about the page_pool. This API 109 is only available if the kernel has been configured with 110 ``CONFIG_PAGE_POOL_STATS=y``. A pointer to a caller allocated ``struct 111 page_pool_stats`` structure is passed to this API which is filled in. The 112 caller can then report those stats to the user (perhaps via ethtool, 113 debugfs, etc.). See below for an example usage of this API. 114 115Stats API and structures 116------------------------ 117If the kernel is configured with ``CONFIG_PAGE_POOL_STATS=y``, the API 118``page_pool_get_stats()`` and structures described below are available. It 119takes a pointer to a ``struct page_pool`` and a pointer to a ``struct 120page_pool_stats`` allocated by the caller. 121 122The API will fill in the provided ``struct page_pool_stats`` with 123statistics about the page_pool. 124 125The stats structure has the following fields:: 126 127 struct page_pool_stats { 128 struct page_pool_alloc_stats alloc_stats; 129 struct page_pool_recycle_stats recycle_stats; 130 }; 131 132 133The ``struct page_pool_alloc_stats`` has the following fields: 134 * ``fast``: successful fast path allocations 135 * ``slow``: slow path order-0 allocations 136 * ``slow_high_order``: slow path high order allocations 137 * ``empty``: ptr ring is empty, so a slow path allocation was forced. 138 * ``refill``: an allocation which triggered a refill of the cache 139 * ``waive``: pages obtained from the ptr ring that cannot be added to 140 the cache due to a NUMA mismatch. 141 142The ``struct page_pool_recycle_stats`` has the following fields: 143 * ``cached``: recycling placed page in the page pool cache 144 * ``cache_full``: page pool cache was full 145 * ``ring``: page placed into the ptr ring 146 * ``ring_full``: page released from page pool because the ptr ring was full 147 * ``released_refcnt``: page released (and not recycled) because refcnt > 1 148 149Coding examples 150=============== 151 152Registration 153------------ 154 155.. code-block:: c 156 157 /* Page pool registration */ 158 struct page_pool_params pp_params = { 0 }; 159 struct xdp_rxq_info xdp_rxq; 160 int err; 161 162 pp_params.order = 0; 163 /* internal DMA mapping in page_pool */ 164 pp_params.flags = PP_FLAG_DMA_MAP; 165 pp_params.pool_size = DESC_NUM; 166 pp_params.nid = NUMA_NO_NODE; 167 pp_params.dev = priv->dev; 168 pp_params.napi = napi; /* only if locking is tied to NAPI */ 169 pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE; 170 page_pool = page_pool_create(&pp_params); 171 172 err = xdp_rxq_info_reg(&xdp_rxq, ndev, 0); 173 if (err) 174 goto err_out; 175 176 err = xdp_rxq_info_reg_mem_model(&xdp_rxq, MEM_TYPE_PAGE_POOL, page_pool); 177 if (err) 178 goto err_out; 179 180NAPI poller 181----------- 182 183 184.. code-block:: c 185 186 /* NAPI Rx poller */ 187 enum dma_data_direction dma_dir; 188 189 dma_dir = page_pool_get_dma_dir(dring->page_pool); 190 while (done < budget) { 191 if (some error) 192 page_pool_recycle_direct(page_pool, page); 193 if (packet_is_xdp) { 194 if XDP_DROP: 195 page_pool_recycle_direct(page_pool, page); 196 } else (packet_is_skb) { 197 page_pool_release_page(page_pool, page); 198 new_page = page_pool_dev_alloc_pages(page_pool); 199 } 200 } 201 202Stats 203----- 204 205.. code-block:: c 206 207 #ifdef CONFIG_PAGE_POOL_STATS 208 /* retrieve stats */ 209 struct page_pool_stats stats = { 0 }; 210 if (page_pool_get_stats(page_pool, &stats)) { 211 /* perhaps the driver reports statistics with ethool */ 212 ethtool_print_allocation_stats(&stats.alloc_stats); 213 ethtool_print_recycle_stats(&stats.recycle_stats); 214 } 215 #endif 216 217Driver unload 218------------- 219 220.. code-block:: c 221 222 /* Driver unload */ 223 page_pool_put_full_page(page_pool, page, false); 224 xdp_rxq_info_unreg(&xdp_rxq); 225