vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_r

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_remove_radixdone, and remove
from it the responsibility for removing a page from its radix tree,
and pass that responsibility on to its callers.

For one of those callers, vm_page_rename, pass a pages pctrie_iter,
rather than a page, and use the iterator to remove the page from its
radix tree.

Define functions vm_page_iter_remove() and vm_page_iter_free() that
are like vm_page_remove() and vm_page_free(), respectively, except
that they take an iterator as parameter rather than a page, and use
the iterator to remove the page from the radix tree instead of
searching the radix tree. Function vm_page_iter_free() assumes that
the page is associated with an object, and calls
vm_page_free_object_prep to do the part of vm_page_free_prep that is
object-related.

In functions vm_object_split and vm_object_collapse_scan, use a
pctrie_iter to walk over the pages of the object, and use
vm_page_rename and vm_radix_iter_remove modify the radix tree without
searching for pages. In vm_object_page_remove and _kmem_unback, use a
pctrie_iter and vm_page_iter_free to remove the page from the radix
tree.

Reviewed by: markj (prevoius version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D46724

show more ...

vm: static-ize vm_page_alloc_after()

This function is only intended for the internal use of the VM system.

Reviewed by: dougm, kib, markj
Differential Revision: https://reviews.freebsd.org/D47644

vm_glue: use vm_page_alloc_domain_after

Drop the function vm_page_alloc_domain, used only in
vm_thread_stack_back, and replace it with vm_page_alloc_domain_after
there, with the extra mpred argument

vm_glue: use vm_page_alloc_domain_after

Drop the function vm_page_alloc_domain, used only in
vm_thread_stack_back, and replace it with vm_page_alloc_domain_after
there, with the extra mpred argument either computed on the first
iteration or retrieved from previous iterations. Define a function
vm_page_mpred() for computing that first mpred argument.

Reviewed by: bnovkov
Differential Revision: https://reviews.freebsd.org/D47054

show more ...

vm_page_free_pages_toq(): return the count of freed pages

Reviewed by: bnovkov
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision

vm_page_free_pages_toq(): return the count of freed pages

Reviewed by: bnovkov
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D46820

show more ...

vm_radix: offer pctrie_iterator access

Add to the vm_radix and vm_page interfaces methods to use pctrie
iterators with vm_radix tries.

Reviewed by: markj
Differential Revision: https://reviews.free

vm_radix: offer pctrie_iterator access

Add to the vm_radix and vm_page interfaces methods to use pctrie
iterators with vm_radix tries.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D46663

show more ...

vm: Introduce VM_ALLOC_NOFREE and PG_NOFREE

This patch adds two additional vm_page flags to distinguish pages that
never get released while the system is running (e.g. UMA_ZONE_NOFREE slabs).

Diffe

vm: Introduce VM_ALLOC_NOFREE and PG_NOFREE

This patch adds two additional vm_page flags to distinguish pages that
never get released while the system is running (e.g. UMA_ZONE_NOFREE slabs).

Differential Revision: https://reviews.freebsd.org/D45970
Reviewed by: alc, kib, markj
Tested by: alc

show more ...

vm: Retire vm_page_alloc_freelist{,_domain}()

Once upon a time, I created vm_page_alloc_freelist{,_domain}() to
support faster allocation of pages that were mapped by the partial
direct map on 32-bi

vm: Retire vm_page_alloc_freelist{,_domain}()

Once upon a time, I created vm_page_alloc_freelist{,_domain}() to
support faster allocation of pages that were mapped by the partial
direct map on 32-bit MIPS. At the time, I expected that these
functions might find other uses too, but those other uses never
materialized. So, these functions have not been used for some time
now. Instead, people use the more general vm_page_alloc_contig().

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D46063

show more ...

vm_page: add vm_page_clearref() helper

It is supposed to be used for ref_count manipulations when the pages
are owned by an object, but ref_count is used for something else than
the wiring, e.g. PTE

vm_page: add vm_page_clearref() helper

It is supposed to be used for ref_count manipulations when the pages
are owned by an object, but ref_count is used for something else than
the wiring, e.g. PTE population count on the page table page.

Reviewed by: markj
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D45910

show more ...

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

vm_page: Let vm_page_init_page() take a pool parameter

This is useful for a subsequent patch which implements lazy
initialization of vm_page structures using a dedicate vm_phys free page
pool.

No f

vm_page: Let vm_page_init_page() take a pool parameter

This is useful for a subsequent patch which implements lazy
initialization of vm_page structures using a dedicate vm_phys free page
pool.

No functional change intended.

Reviewed by: alc, kib, emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D40399

show more ...

vm: add macro to mark arguments used when NUMA is defined

This fixes compiler warnings when -Wunused-arguments is enabled and
not quieted.

Reviewed by: kib, markj
Obtained from: Juniper Networks, I

vm: add macro to mark arguments used when NUMA is defined

This fixes compiler warnings when -Wunused-arguments is enabled and
not quieted.

Reviewed by: kib, markj
Obtained from: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D44623

show more ...

vm_page.h: tweak page-busied assertion macros

Fix incorrect macro name and include the value of curthread in the
panic message where relevant.

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM i

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM indicates
lack of available memory to reclaim, with any other error (currently
only ERANGE) indicating that reclamation is impossible for the
specified address range. Change all callers to only follow
up with vm_page_wait* in the ENOMEM case.

- Introduce vm_domainset_iter_ignore(), which marks the specified
domain as unavailable for further use by the iterator. Use this
function to ignore domains that can't possibly satisfy a physical
allocation request. Since WAITOK allocations run the iterators
repeatedly, this avoids the possibility of infinitely spinning
in domain iteration if no available domain can satisfy the
allocation request.

PR: 274252
Reported by: kevans
Tested by: kevans
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D42706

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

vm_phys: add binary segment search

Replace several sequential searches for a segment that contains a
phyiscal address with a call to a function that does it by binary
search. In vm_page_reclaim_con

vm_phys: add binary segment search

Replace several sequential searches for a segment that contains a
phyiscal address with a call to a function that does it by binary
search. In vm_page_reclaim_contig_domain_ext, find the first segment
to reclaim from, and reclaim from each subsequent appropriate segment.
Eliminate vm_phys_scan_contig.

Reviewed by: alc, markj
Differential Revision: https://reviews.freebsd.org/D40058

show more ...

vm: implement vm_page_reclaim_contig_domain_ext()

Implement vm_page_reclaim_contig_domain_ext() to reclaim multiple
contiguous regions at once. This makes it more efficient for users
that need mult

vm: implement vm_page_reclaim_contig_domain_ext()

Implement vm_page_reclaim_contig_domain_ext() to reclaim multiple
contiguous regions at once. This makes it more efficient for users
that need multiple contiguous regions to reclaim those regions
efficiently.

This is needed because callers like ktls may need to reclaim many
contiguous regions, and each scan of physical memory can take
multiple seconds on a large memory machine (order of 100GB of
RMA). Rather than modifying the core algorithm, I extended
vm_page_reclaim_contig_domain() to take a "desired_runs" argument to
allow the caller to request that it reclaim more than just a single
run. There is no functional change intended for all existing
callers.

The first user for this interface is the ktls code
(https://reviews.freebsd.org/D39421). By reclaiming multiple runs,
ktls goes from consuming hours of CPU to refill its buffer zone to
just seconds or minutes.

Differential Revision: https://reviews.freebsd.org/D39739
Sponsored by: Netflix
Reviewed by: alc, jhb, markj

show more ...

Add vm_page_any_valid()

Use it and several other vm_page_*_valid() functions in more places.

Suggested and reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
D

Add vm_page_any_valid()

Use it and several other vm_page_*_valid() functions in more places.

Suggested and reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D37024

show more ...

vm/vm_extern.h, vm/vm_page.h: use sys/kassert.h

instead of fatty sys/systm.h.

Suggested by: jhb
Reviewed by: alc, imp, jhb (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

vm/vm_extern.h, vm/vm_page.h: use sys/kassert.h

instead of fatty sys/systm.h.

Suggested by: jhb
Reviewed by: alc, imp, jhb (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D34089

show more ...

vm_page: Remove vm_page_sbusy() and vm_page_xbusy()

They are unused today and cannot be safely used in the face of unlocked
lookup, in which pages may be busied without the object lock held.

Obtain

vm_page: Remove vm_page_sbusy() and vm_page_xbusy()

They are unused today and cannot be safely used in the face of unlocked
lookup, in which pages may be busied without the object lock held.

Obtained from: jeff (object_concurrency patches)
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D32948

show more ...

vm_page: Consolidate page busy sleep mechanisms

- Modify vm_page_busy_sleep() and vm_page_busy_sleep_unlocked() to take
a VM_ALLOC_* flag indicating whether to sleep on shared-busy, and fix
up c

vm_page: Consolidate page busy sleep mechanisms

- Modify vm_page_busy_sleep() and vm_page_busy_sleep_unlocked() to take
a VM_ALLOC_* flag indicating whether to sleep on shared-busy, and fix
up callers.
- Modify vm_page_busy_sleep() to return a status indicating whether the
object lock was dropped, and fix up callers.
- Convert callers of vm_page_sleep_if_busy() to use vm_page_busy_sleep()
instead.
- Remove vm_page_sleep_if_(x)busy().

No functional change intended.

Obtained from: jeff (object_concurrency patches)
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D32947

show more ...

Remove some remaining references to VM_ALLOC_NOOBJ

Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32037

vm_page: Stop handling VM_ALLOC_NOOBJ in vm_page_alloc_domain_after()

This makes the allocator simpler since it can assume object != NULL.
Also modify the function to unconditionally preserve PG_ZER

vm_page: Stop handling VM_ALLOC_NOOBJ in vm_page_alloc_domain_after()

This makes the allocator simpler since it can assume object != NULL.
Also modify the function to unconditionally preserve PG_ZERO, so
VM_ALLOC_ZERO is effectively ignored (and still must be implemented by
the caller for now).

Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32033

show more ...

Introduce vm_page_alloc_noobj_contig()

This is the same as vm_page_alloc_noobj(), but allocates physically
contiguous runs of memory. For now it is implemented in terms of
vm_page_alloc_contig(), w

Introduce vm_page_alloc_noobj_contig()

This is the same as vm_page_alloc_noobj(), but allocates physically
contiguous runs of memory. For now it is implemented in terms of
vm_page_alloc_contig(), with the difference that
vm_page_alloc_noobj_contig() implements VM_ALLOC_ZERO by zeroing the
page.

Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32005

show more ...

vm_page: Add a new page allocator interface for unnamed pages

The diff adds vm_page_alloc_noobj() and vm_page_alloc_noobj_domain().
These mostly correspond to vm_page_alloc() and vm_page_alloc_domai

vm_page: Add a new page allocator interface for unnamed pages

The diff adds vm_page_alloc_noobj() and vm_page_alloc_noobj_domain().
These mostly correspond to vm_page_alloc() and vm_page_alloc_domain()
when no VM object is specified, with the exception that they handle
VM_ALLOC_ZERO by zeroing the page, rather than by preserving PG_ZERO.

This simplifies callers and will permit simplification of the
vm_page_alloc_domain() definition.

Since the new allocator variant is similar to vm_page_alloc_freelist(),
implement both of them using a common backend allocator function. No
functional change intended.

Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31985

show more ...