vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj (previous version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056e3987693f247b2ec0433.

show more ...

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

vm_map: convert several bool members into flags

Extend flags to u_int.
Move system_map and needs_wakeup bools into flags.

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision

vm_map: convert several bool members into flags

Extend flags to u_int.
Move system_map and needs_wakeup bools into flags.

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47934

show more ...

vm_map: overlap system map mutex and user man sx

This saves 616-584 = 32 bytes per struct vmspace on amd64, which allows
to pack 7 vmspaces per page vs. 6 for non-overlapping layout.

I used anonymo

vm_map: overlap system map mutex and user man sx

This saves 616-584 = 32 bytes per struct vmspace on amd64, which allows
to pack 7 vmspaces per page vs. 6 for non-overlapping layout.

I used anonymous union member feature to avoid too much churn.

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D47934

show more ...

vm_page: pass page to iter_free

Pass the to-be-freed page to vm_page_iter_free as a parameter, rather
than computing it from the iterator parameter, to improve performance.

Sort declarations of pag

vm_page: pass page to iter_free

Pass the to-be-freed page to vm_page_iter_free as a parameter, rather
than computing it from the iterator parameter, to improve performance.

Sort declarations of page_iter functions in vm_page.h.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47727

show more ...

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_r

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_remove_radixdone, and remove
from it the responsibility for removing a page from its radix tree,
and pass that responsibility on to its callers.

For one of those callers, vm_page_rename, pass a pages pctrie_iter,
rather than a page, and use the iterator to remove the page from its
radix tree.

Define functions vm_page_iter_remove() and vm_page_iter_free() that
are like vm_page_remove() and vm_page_free(), respectively, except
that they take an iterator as parameter rather than a page, and use
the iterator to remove the page from the radix tree instead of
searching the radix tree. Function vm_page_iter_free() assumes that
the page is associated with an object, and calls
vm_page_free_object_prep to do the part of vm_page_free_prep that is
object-related.

In functions vm_object_split and vm_object_collapse_scan, use a
pctrie_iter to walk over the pages of the object, and use
vm_page_rename and vm_radix_iter_remove modify the radix tree without
searching for pages. In vm_object_page_remove and _kmem_unback, use a
pctrie_iter and vm_page_iter_free to remove the page from the radix
tree.

Reviewed by: markj (prevoius version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D46724

show more ...

vm_kern: Use VM_ALLOC_NOFREE when allocating 'zero_region' page

Allocate the 'zero_region' page using VM_ALLOC_NOFREE since
it never gets released.

Differential Revision: https://reviews.freebs

vm_kern: Use VM_ALLOC_NOFREE when allocating 'zero_region' page

Allocate the 'zero_region' page using VM_ALLOC_NOFREE since
it never gets released.

Differential Revision: https://reviews.freebsd.org/D46885
Reviewed by: alc, markj, kib

show more ...

vm: Add a KVA arena for M_NEVERFREED allocations

This patch adds a new KVA arena for separating M_NEVERFREED allocations.
Separating KVAs for pages that are never freed should facilitate
superpage p

vm: Add a KVA arena for M_NEVERFREED allocations

This patch adds a new KVA arena for separating M_NEVERFREED allocations.
Separating KVAs for pages that are never freed should facilitate
superpage promotion in the kernel.

Differential Revision: https://reviews.freebsd.org/D45997
Reviewed by: alc, kib, markj
Tested by: alc

show more ...

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM i

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM indicates
lack of available memory to reclaim, with any other error (currently
only ERANGE) indicating that reclamation is impossible for the
specified address range. Change all callers to only follow
up with vm_page_wait* in the ENOMEM case.

- Introduce vm_domainset_iter_ignore(), which marks the specified
domain as unavailable for further use by the iterator. Use this
function to ignore domains that can't possibly satisfy a physical
allocation request. Since WAITOK allocations run the iterators
repeatedly, this avoids the possibility of infinitely spinning
in domain iteration if no available domain can satisfy the
allocation request.

PR: 274252
Reported by: kevans
Tested by: kevans
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D42706

show more ...

vm: Add kva_alloc_aligned

Add a function like kva_alloc that allows us to specify the alignment
of the virtual address space returned.

Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differentia

vm: Add kva_alloc_aligned

Add a function like kva_alloc that allows us to specify the alignment
of the virtual address space returned.

Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42788

show more ...

vm: Use vmem_xalloc in kva_alloc

The kernel_arena used in kva_alloc has the qcache disabled. vmem_alloc
will first try to use the qcache before falling back to vmem_xalloc.

Rather than trying to us

vm: Use vmem_xalloc in kva_alloc

The kernel_arena used in kva_alloc has the qcache disabled. vmem_alloc
will first try to use the qcache before falling back to vmem_xalloc.

Rather than trying to use the qcache in vmem_alloc just call
vmem_xalloc directly.

Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42831

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

add pmap_active_cpus()

For amd64, i386, arm, and riscv, i.e. all architectures except arm64,
the custom implementation is provided since we maintain the bitmask of
active CPUs anyway.

Arm64 uses so

add pmap_active_cpus()

For amd64, i386, arm, and riscv, i.e. all architectures except arm64,
the custom implementation is provided since we maintain the bitmask of
active CPUs anyway.

Arm64 uses somewhat naive iteration over CPUs and match current vmspace'
pmap with the argument. It is not guaranteed that vmspace->pmap is the
same as the active pmap, but the inaccuracy should be toleratable.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32360

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

tslog: Annotate parts of SYSINIT cpu

Booting an amd64 kernel on Firecracker with 1 CPU and 128 MB of RAM,
SYSINIT cpu takes roughly 2770 us:
* 2280 us in vm_ksubmap_init
* 535 us in kmem_malloc

tslog: Annotate parts of SYSINIT cpu

Booting an amd64 kernel on Firecracker with 1 CPU and 128 MB of RAM,
SYSINIT cpu takes roughly 2770 us:
* 2280 us in vm_ksubmap_init
* 535 us in kmem_malloc
* 450 us in pmap_zero_page
* 1720 us in pmap_growkernel
* 1620 us in pmap_zero_page
* 80 us in bufinit
* 480 us in cpu_setregs
* 430 us in cpu_setregs calling load_cr0

Much of this is hypervisor overhead: load_cr0 is slow because it traps
to the hypervisor, and 99% of the time in pmap_zero_page is spent when
we first touch the page, presumably due to the host Linux kernel
faulting in backing pages one by one.

Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D40327

show more ...

kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36549

sys/vm: Add TSLOG to some functions

The functions pbuf_init, kva_alloc, and keg_alloc_slab are significant
contributors to the kernel boot time when FreeBSD boots inside the
Firecracker VMM. Instru

sys/vm: Add TSLOG to some functions

The functions pbuf_init, kva_alloc, and keg_alloc_slab are significant
contributors to the kernel boot time when FreeBSD boots inside the
Firecracker VMM. Instrument them so they show up on flamecharts.

show more ...

vm_kern: Update KMSAN shadow maps when allocating kmem memory

This addresses a couple of false positive reports for memory returned by
malloc_large().

Sponsored by: The FreeBSD Foundation

vm_page: Handle VM_ALLOC_NORECLAIM in the contiguous page allocator

We added _NORECLAIM to request that kmem_alloc_contig_pages() not spend
time scanning physical memory for candidates to reclaim.

vm_page: Handle VM_ALLOC_NORECLAIM in the contiguous page allocator

We added _NORECLAIM to request that kmem_alloc_contig_pages() not spend
time scanning physical memory for candidates to reclaim. In some
situations the scanning can induce large amounts of undesirable latency,
and it's less important that the request be satisfied than it is that we
not spend many milliseconds scanning.

The problem extends to vm_reserv_reclaim_contig(), which unlike
vm_reserv_reclaim() may have to scan the entire list of partially
populated reservations. Use VM_ALLOC_NORECLAIM to request that this
scan not be executed.[1]

As a side effect, this fixes a regression in 02fb0585e7b3 ("vm_page:
Drop handling of VM_ALLOC_NOOBJ in vm_page_alloc_contig_domain()")
where VM_ALLOC_CONTIG was not included in VPAC_FLAGS or VPANC_FLAGS even
though it is not masked by kmem_alloc_contig_pages().[2]

Reported by: gallatin [1], glebius [2]
Reviewed by: alc, glebius, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32899

show more ...

Convert vm_page_alloc() callers to use vm_page_alloc_noobj().

Remove page zeroing code from consumers and stop specifying
VM_ALLOC_NOOBJ. In a few places, also convert an allocation loop to
simply

Convert vm_page_alloc() callers to use vm_page_alloc_noobj().

Remove page zeroing code from consumers and stop specifying
VM_ALLOC_NOOBJ. In a few places, also convert an allocation loop to
simply use VM_ALLOC_WAITOK.

Similarly, convert vm_page_alloc_domain() callers.

Note that callers are now responsible for assigning the pindex.

Reviewed by: alc, hselasky, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31986

show more ...

uma: Introduce per-domain reclamation functions

Make it possible to reclaim items from a specific NUMA domain.

- Add uma_zone_reclaim_domain() and uma_reclaim_domain().
- Permit parallel reclamatio

uma: Introduce per-domain reclamation functions

Make it possible to reclaim items from a specific NUMA domain.

- Add uma_zone_reclaim_domain() and uma_reclaim_domain().
- Permit parallel reclamations. Use a counter instead of a flag to
synchronize with zone_dtor().
- Use the zone lock to protect cache_shrink() now that parallel reclaims
can happen.
- Add a sysctl that can be used to trigger reclamation from a specific
domain.

Currently the new KPIs are unused, so there should be no functional
change.

Reviewed by: mav
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29685

show more ...

kmem: Add KASAN state transitions

Memory allocated with kmem_* is unmapped upon free, so KASAN doesn't
provide a lot of benefit, but since allocations are always a multiple of
the page size we can c

kmem: Add KASAN state transitions

Memory allocated with kmem_* is unmapped upon free, so KASAN doesn't
provide a lot of benefit, but since allocations are always a multiple of
the page size we can create a redzone when the allocation request size
is not a multiple of the page size.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29458

show more ...

Add sysctl debug.uma_reclaim

Sponsored by: The FreeBSD Foundation
MFC after: 1 week