vm_page: drop page_iter_lookup

The functions vm_page_iter_lookup and vm_page_iter_lookup_ge are just
wrappers around vm_radix_iter_lookup and vm_radix_iter_lookup_ge,
respectively. They server no re

vm_page: drop page_iter_lookup

The functions vm_page_iter_lookup and vm_page_iter_lookup_ge are just
wrappers around vm_radix_iter_lookup and vm_radix_iter_lookup_ge,
respectively. They server no real purpose, so drop them and use the
vm_radix versions everywhere.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49203

show more ...

vm_page: expose page_alloc_after

vm_page_alloc() just calls vm_page_alloc_after(), after it has found
the predecessor of a page parameter. Many callers of vm_page_alloc()
already know that predecess

vm_page: expose page_alloc_after

vm_page_alloc() just calls vm_page_alloc_after(), after it has found
the predecessor of a page parameter. Many callers of vm_page_alloc()
already know that predecessor. Letting them pass that to
vm_page_alloc_after() directly could save a little redundant
calculation.

Reviewed by: alc
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D49103

show more ...

vm_page: define partial page invalidate

Two different functions in different files do the same thing - fill a
partial page with zeroes. Add that functionality to vm_page.c and
remove it elsewhere to

vm_page: define partial page invalidate

Two different functions in different files do the same thing - fill a
partial page with zeroes. Add that functionality to vm_page.c and
remove it elsewhere to avoid code duplication.

Reviewed by: markj, kib
Differential Revision: https://reviews.freebsd.org/D49096

show more ...

Revert "vm_page: define partial page invalidate"

A negative review arrived as this was being committed, so undo and
reevaluate.

This reverts commit 5611a38d818587b307e1fb110f72d2996c170035.

vm_page: define partial page invalidate

Two different functions in different files do the same thing - fill a
partial page with zeroes. Add that functionality to vm_page.c and
remove it elsewhere to

vm_page: define partial page invalidate

Two different functions in different files do the same thing - fill a
partial page with zeroes. Add that functionality to vm_page.c and
remove it elsewhere to avoid code duplication.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49096

show more ...

vm_page: pass page to iter_remove

Pass the to-be-freed page to vm_page_iter_remove as a parameter,
rather than computing it from the iterator parameter, to improve
performance.

Reviewed by: alc
Dif

vm_page: pass page to iter_remove

Pass the to-be-freed page to vm_page_iter_remove as a parameter,
rather than computing it from the iterator parameter, to improve
performance.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47730

show more ...

vm: Optimize page rename

Rename vm_page_rename() to vm_page_iter_rename() to reflect its
reimplementation using iterators, and pass the page to this function
rather than spending clock cycles lookin

vm: Optimize page rename

Rename vm_page_rename() to vm_page_iter_rename() to reflect its
reimplementation using iterators, and pass the page to this function
rather than spending clock cycles looking it up. Change its return
value from 0/1 to a bool.

Reviewed by: dougm, markj
Differential Revision: https://reviews.freebsd.org/D47829

show more ...

vm_page: pass page to iter_free

Pass the to-be-freed page to vm_page_iter_free as a parameter, rather
than computing it from the iterator parameter, to improve performance.

Sort declarations of pag

vm_page: pass page to iter_free

Pass the to-be-freed page to vm_page_iter_free as a parameter, rather
than computing it from the iterator parameter, to improve performance.

Sort declarations of page_iter functions in vm_page.h.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47727

show more ...

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_r

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_remove_radixdone, and remove
from it the responsibility for removing a page from its radix tree,
and pass that responsibility on to its callers.

For one of those callers, vm_page_rename, pass a pages pctrie_iter,
rather than a page, and use the iterator to remove the page from its
radix tree.

Define functions vm_page_iter_remove() and vm_page_iter_free() that
are like vm_page_remove() and vm_page_free(), respectively, except
that they take an iterator as parameter rather than a page, and use
the iterator to remove the page from the radix tree instead of
searching the radix tree. Function vm_page_iter_free() assumes that
the page is associated with an object, and calls
vm_page_free_object_prep to do the part of vm_page_free_prep that is
object-related.

In functions vm_object_split and vm_object_collapse_scan, use a
pctrie_iter to walk over the pages of the object, and use
vm_page_rename and vm_radix_iter_remove modify the radix tree without
searching for pages. In vm_object_page_remove and _kmem_unback, use a
pctrie_iter and vm_page_iter_free to remove the page from the radix
tree.

Reviewed by: markj (prevoius version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D46724

show more ...

vm: static-ize vm_page_alloc_after()

This function is only intended for the internal use of the VM system.

Reviewed by: dougm, kib, markj
Differential Revision: https://reviews.freebsd.org/D47644

vm_glue: use vm_page_alloc_domain_after

Drop the function vm_page_alloc_domain, used only in
vm_thread_stack_back, and replace it with vm_page_alloc_domain_after
there, with the extra mpred argument

vm_glue: use vm_page_alloc_domain_after

Drop the function vm_page_alloc_domain, used only in
vm_thread_stack_back, and replace it with vm_page_alloc_domain_after
there, with the extra mpred argument either computed on the first
iteration or retrieved from previous iterations. Define a function
vm_page_mpred() for computing that first mpred argument.

Reviewed by: bnovkov
Differential Revision: https://reviews.freebsd.org/D47054

show more ...

vm_page_free_pages_toq(): return the count of freed pages

Reviewed by: bnovkov
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision

vm_page_free_pages_toq(): return the count of freed pages

Reviewed by: bnovkov
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D46820

show more ...

vm_radix: offer pctrie_iterator access

Add to the vm_radix and vm_page interfaces methods to use pctrie
iterators with vm_radix tries.

Reviewed by: markj
Differential Revision: https://reviews.free

vm_radix: offer pctrie_iterator access

Add to the vm_radix and vm_page interfaces methods to use pctrie
iterators with vm_radix tries.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D46663

show more ...

vm: Introduce VM_ALLOC_NOFREE and PG_NOFREE

This patch adds two additional vm_page flags to distinguish pages that
never get released while the system is running (e.g. UMA_ZONE_NOFREE slabs).

Diffe

vm: Introduce VM_ALLOC_NOFREE and PG_NOFREE

This patch adds two additional vm_page flags to distinguish pages that
never get released while the system is running (e.g. UMA_ZONE_NOFREE slabs).

Differential Revision: https://reviews.freebsd.org/D45970
Reviewed by: alc, kib, markj
Tested by: alc

show more ...

vm: Retire vm_page_alloc_freelist{,_domain}()

Once upon a time, I created vm_page_alloc_freelist{,_domain}() to
support faster allocation of pages that were mapped by the partial
direct map on 32-bi

vm: Retire vm_page_alloc_freelist{,_domain}()

Once upon a time, I created vm_page_alloc_freelist{,_domain}() to
support faster allocation of pages that were mapped by the partial
direct map on 32-bit MIPS. At the time, I expected that these
functions might find other uses too, but those other uses never
materialized. So, these functions have not been used for some time
now. Instead, people use the more general vm_page_alloc_contig().

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D46063

show more ...

vm_page: add vm_page_clearref() helper

It is supposed to be used for ref_count manipulations when the pages
are owned by an object, but ref_count is used for something else than
the wiring, e.g. PTE

vm_page: add vm_page_clearref() helper

It is supposed to be used for ref_count manipulations when the pages
are owned by an object, but ref_count is used for something else than
the wiring, e.g. PTE population count on the page table page.

Reviewed by: markj
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D45910

show more ...

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

vm_page: Let vm_page_init_page() take a pool parameter

This is useful for a subsequent patch which implements lazy
initialization of vm_page structures using a dedicate vm_phys free page
pool.

No f

vm_page: Let vm_page_init_page() take a pool parameter

This is useful for a subsequent patch which implements lazy
initialization of vm_page structures using a dedicate vm_phys free page
pool.

No functional change intended.

Reviewed by: alc, kib, emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D40399

show more ...

vm: add macro to mark arguments used when NUMA is defined

This fixes compiler warnings when -Wunused-arguments is enabled and
not quieted.

Reviewed by: kib, markj
Obtained from: Juniper Networks, I

vm: add macro to mark arguments used when NUMA is defined

This fixes compiler warnings when -Wunused-arguments is enabled and
not quieted.

Reviewed by: kib, markj
Obtained from: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D44623

show more ...

vm_page.h: tweak page-busied assertion macros

Fix incorrect macro name and include the value of curthread in the
panic message where relevant.

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM i

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM indicates
lack of available memory to reclaim, with any other error (currently
only ERANGE) indicating that reclamation is impossible for the
specified address range. Change all callers to only follow
up with vm_page_wait* in the ENOMEM case.

- Introduce vm_domainset_iter_ignore(), which marks the specified
domain as unavailable for further use by the iterator. Use this
function to ignore domains that can't possibly satisfy a physical
allocation request. Since WAITOK allocations run the iterators
repeatedly, this avoids the possibility of infinitely spinning
in domain iteration if no available domain can satisfy the
allocation request.

PR: 274252
Reported by: kevans
Tested by: kevans
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D42706

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

vm_phys: add binary segment search

Replace several sequential searches for a segment that contains a
phyiscal address with a call to a function that does it by binary
search. In vm_page_reclaim_con

vm_phys: add binary segment search

Replace several sequential searches for a segment that contains a
phyiscal address with a call to a function that does it by binary
search. In vm_page_reclaim_contig_domain_ext, find the first segment
to reclaim from, and reclaim from each subsequent appropriate segment.
Eliminate vm_phys_scan_contig.

Reviewed by: alc, markj
Differential Revision: https://reviews.freebsd.org/D40058

show more ...

vm: implement vm_page_reclaim_contig_domain_ext()

Implement vm_page_reclaim_contig_domain_ext() to reclaim multiple
contiguous regions at once. This makes it more efficient for users
that need mult

vm: implement vm_page_reclaim_contig_domain_ext()

Implement vm_page_reclaim_contig_domain_ext() to reclaim multiple
contiguous regions at once. This makes it more efficient for users
that need multiple contiguous regions to reclaim those regions
efficiently.

This is needed because callers like ktls may need to reclaim many
contiguous regions, and each scan of physical memory can take
multiple seconds on a large memory machine (order of 100GB of
RMA). Rather than modifying the core algorithm, I extended
vm_page_reclaim_contig_domain() to take a "desired_runs" argument to
allow the caller to request that it reclaim more than just a single
run. There is no functional change intended for all existing
callers.

The first user for this interface is the ktls code
(https://reviews.freebsd.org/D39421). By reclaiming multiple runs,
ktls goes from consuming hours of CPU to refill its buffer zone to
just seconds or minutes.

Differential Revision: https://reviews.freebsd.org/D39739
Sponsored by: Netflix
Reviewed by: alc, jhb, markj

show more ...