21  * Allocation is done by offsets into a unit's address space.  Ie., an
29  * the static percpu variables in the kernel image as allocation services
40  * memcg-awareness.  To make a percpu allocation memcg-aware the __GFP_ACCOUNT
46  * is managed by a bitmap with metadata blocks.  The allocation map is updated
47  * on every allocation and free to reflect the current state while the boundary
48  * map is only updated on allocation.  Each metadata block contains
152 /* group information, used for vm allocation */
310  * @bits: size of allocation
313  * Check to see if the allocation can fit in the block's contig hint.
329  * @alloc_bits: size of allocation
334  * cannot fulfill an allocation, we can begin scanning from there knowing
343 	 * contig_hint == scan_hint).  Third, is the allocation request  in pcpu_next_hint()
407  * pcpu_next_fit_region - finds fit areas for a given allocation request
409  * @alloc_bits: size of allocation
416  * allocation.  block->first_free is returned if the allocation request fits
475  * a fit is found for the allocation request.
494  * @gfp: allocation flags
547  * This function is called after an allocation or free changed @chunk.
702  * Finding the final allocation spot first goes through pcpu_find_block_fit()
703  * to find a block that can hold the allocation and then pcpu_alloc_area()
739  * A full scan can be avoided on the allocation path as this is triggered
799  * pcpu_block_update_hint_alloc - update hint on allocation path
804  * Updates metadata for the allocation path.  The metadata only has to be
814 	int s_index, e_index;	/* block indexes of the freed allocation */  in pcpu_block_update_hint_alloc()
815 	int s_off, e_off;	/* block offsets of the freed allocation */  in pcpu_block_update_hint_alloc()
838 	 * block->first_free must be updated if the allocation takes its place.  in pcpu_block_update_hint_alloc()
839 	 * If the allocation breaks the contig_hint, a scan is required to  in pcpu_block_update_hint_alloc()
881 		 * When the allocation is across blocks, the end is along  in pcpu_block_update_hint_alloc()
917 	 * If the allocation is not atomic, some blocks may not be  in pcpu_block_update_hint_alloc()
951  * Updates metadata for the allocation path.  This avoids a blind block
968 	int s_index, e_index;	/* block indexes of the freed allocation */  in pcpu_block_update_hint_free()
969 	int s_off, e_off;	/* block offsets of the freed allocation */  in pcpu_block_update_hint_free()
1094  * @alloc_bits: size of request in allocation units
1098  * Given a chunk and an allocation spec, find the offset to begin searching
1101  * not quite first fit as if the allocation does not fit in the contig hint
1118 	 * allocation cannot fit in the global hint, there is memory pressure  in pcpu_find_block_fit()
1173 	/* Align allocation */  in pcpu_find_zero_area()
1200  * @alloc_bits: size of request in allocation units
1205  * allocation of @alloc_bits with alignment @align.  It needs to scan
1206  * the allocation map because if it fits within the block's contig hint,
1208  * allocation prior to breaking the contig hint.  The allocation and
1270  * This function determines the size of an allocation to free using
1271  * the boundary bitmap and clears the allocation map.
1720  * @gfp: allocation flags
1723  * contain %GFP_KERNEL, the allocation is atomic. If @gfp has __GFP_NOWARN
1724  * then no warning will be triggered on invalid or failed allocation
1752 	 * There is now a minimum allocation size of PCPU_MIN_ALLOC_SIZE,  in pcpu_alloc_noprof()
1754 	 * An allocation may have internal fragmentation from rounding up  in pcpu_alloc_noprof()
1766 		WARN(do_warn, "illegal size (%zu) or align (%zu) for percpu allocation\n",  in pcpu_alloc_noprof()
1911 			pr_warn("allocation failed, size=%zu align=%zu atomic=%d, %s\n",  in pcpu_alloc_noprof()
1993  * allocation causes the failure as it is possible that requests can be
2011 	 * allocs don't increase fragmentation.  If atomic allocation  in pcpu_balance_populated()
2386  * pcpu_alloc_alloc_info - allocate percpu allocation info
2430  * pcpu_free_alloc_info - free percpu allocation info
2443  * @ai: allocation info to dump
2514  * percpu allocation.  This is primarily used to serve module percpu
2520  * allocation in the first chunk.  The area between @ai->static_size +
2527  * @ai->atom_size is the allocation atom size and used as alignment
2530  * @ai->alloc_size is the allocation size and always multiple of
2547  * share the same vm, but use offset regions in the area allocation map.
2549  * and available for dynamic allocation like any other chunk.
2682 	 * minimum allocation size as this offsets the reserved and  in pcpu_setup_first_chunk()
2774  * @dyn_size: minimum free size for dynamic allocation in bytes
2775  * @atom_size: allocation atom size
2779  * and other parameters considering needed percpu size, allocation
2827 	/* determine the maximum # of units that can fit in an allocation */  in pcpu_build_alloc_info()
2919 		 * reflect actual allocation.  in pcpu_build_alloc_info()
2975  * @dyn_size: minimum free size for dynamic allocation in bytes
2976  * @atom_size: allocation atom size