vm_page: drop page_iter_lookup

The functions vm_page_iter_lookup and vm_page_iter_lookup_ge are just
wrappers around vm_radix_iter_lookup and vm_radix_iter_lookup_ge,
respectively. They server no re

vm_page: drop page_iter_lookup

The functions vm_page_iter_lookup and vm_page_iter_lookup_ge are just
wrappers around vm_radix_iter_lookup and vm_radix_iter_lookup_ge,
respectively. They server no real purpose, so drop them and use the
vm_radix versions everywhere.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49203

show more ...

vm_page: expose page_alloc_after

vm_page_alloc() just calls vm_page_alloc_after(), after it has found
the predecessor of a page parameter. Many callers of vm_page_alloc()
already know that predecess

vm_page: expose page_alloc_after

vm_page_alloc() just calls vm_page_alloc_after(), after it has found
the predecessor of a page parameter. Many callers of vm_page_alloc()
already know that predecessor. Letting them pass that to
vm_page_alloc_after() directly could save a little redundant
calculation.

Reviewed by: alc
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D49103

show more ...

vm_page: define partial page invalidate

Two different functions in different files do the same thing - fill a
partial page with zeroes. Add that functionality to vm_page.c and
remove it elsewhere to

vm_page: define partial page invalidate

Two different functions in different files do the same thing - fill a
partial page with zeroes. Add that functionality to vm_page.c and
remove it elsewhere to avoid code duplication.

Reviewed by: markj, kib
Differential Revision: https://reviews.freebsd.org/D49096

show more ...

Revert "vm_page: define partial page invalidate"

A negative review arrived as this was being committed, so undo and
reevaluate.

This reverts commit 5611a38d818587b307e1fb110f72d2996c170035.

vm_page: define partial page invalidate

Two different functions in different files do the same thing - fill a
partial page with zeroes. Add that functionality to vm_page.c and
remove it elsewhere to

vm_page: define partial page invalidate

Two different functions in different files do the same thing - fill a
partial page with zeroes. Add that functionality to vm_page.c and
remove it elsewhere to avoid code duplication.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49096

show more ...

vm_page_startup(): Clarify memory lowest, highest and size computation

Change the comment before this block of code, and separate the latter
from the preceding one by an empty line.

Move the loop o

vm_page_startup(): Clarify memory lowest, highest and size computation

Change the comment before this block of code, and separate the latter
from the preceding one by an empty line.

Move the loop on phys_avail[] to compute the minimum and maximum memory
physical addresses closer to the initialization of 'low_avail' and
'high_avail', so that it's immediately clear why the loop starts at
2 (and remove the related comment).

While here, fuse the additional loop in the VM_PHYSSEG_DENSE case that
is used to compute the exact physical memory size.

This change suppresses one occurence of detecting whether at least one
of VM_PHYSSEG_DENSE or VM_PHYSSEG_SPARSE is defined at compile time, but
there is still another one in PHYS_TO_VM_PAGE().

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48632

show more ...

vm_phys_add_seg(): Check for bad segments, allow empty ones

A bad specification is if 'start' is strictly greater than 'end', or
bounds are not page aligned.

The latter was already tested under INV

vm_phys_add_seg(): Check for bad segments, allow empty ones

A bad specification is if 'start' is strictly greater than 'end', or
bounds are not page aligned.

The latter was already tested under INVARIANTS, but now will be also on
production kernels. The reason is that vm_phys_early_startup() pours
early segments into the final phys_segs[] array via vm_phys_add_seg(),
but vm_phys_early_add_seg() did not check their validity. Checking
segments once and for all in vm_phys_add_seg() avoids duplicating
validity tests and is possible since early segments are not used before
being poured into phys_segs[]. Finally, vm_phys_add_seg() is not
performance critical.

Allow empty segments and discard them (silently, unless 'bootverbose' is
true), as vm_page_startup() was testing for this case before calling
vm_phys_add_seg(), and we felt the same test in vm_phys_early_startup()
was due before calling vm_phys_add_seg(). As a consequence, remove the
empty segment test from vm_page_startup().

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48627

show more ...

vm: Fix overflow issues in vm_page_startup

Firstly, pagecount is a u_long so we should ensure j is the same for the
sake of 64-bit systems. Secondly, ptoa is just a macro, and does not
cast its argu

vm: Fix overflow issues in vm_page_startup

Firstly, pagecount is a u_long so we should ensure j is the same for the
sake of 64-bit systems. Secondly, ptoa is just a macro, and does not
cast its argument, so in order to handle PAE systems correctly we need
to cast j to vm_paddr_t (the type of startp).

Fixes: 0078df5f0258 ("vm_phys: reduce touching of page->pool fields")

show more ...

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj (previous version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056e3987693f247b2ec0433.

show more ...

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

vm_pageout: Add a chicken switch for multithreaded PQ_INACTIVE scanning

Right now we have the vm.pageout_cpus_per_thread tunable which controls
the number of threads to start up per CPU per NUMA dom

vm_pageout: Add a chicken switch for multithreaded PQ_INACTIVE scanning

Right now we have the vm.pageout_cpus_per_thread tunable which controls
the number of threads to start up per CPU per NUMA domain, but after
booting, it's not possible to disable multi-threaded scanning.

There is at least one workload where this mechanism doesn't work well;
let's make it possible to disable it without a reboot, to simplify
troubleshooting.

Reviewed by: dougm, kib
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D48377

show more ...

vm_pageout: Make vmd_oom a bool

No functional change intended.

Reviewed by: dougm, kib
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Modirum MDPay
Differential Revision: https://reviews

vm_pageout: Make vmd_oom a bool

No functional change intended.

Reviewed by: dougm, kib
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D48376

show more ...

vm_page: pass page to iter_remove

Pass the to-be-freed page to vm_page_iter_remove as a parameter,
rather than computing it from the iterator parameter, to improve
performance.

Reviewed by: alc
Dif

vm_page: pass page to iter_remove

Pass the to-be-freed page to vm_page_iter_remove as a parameter,
rather than computing it from the iterator parameter, to improve
performance.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47730

show more ...

vm: Optimize page rename

Rename vm_page_rename() to vm_page_iter_rename() to reflect its
reimplementation using iterators, and pass the page to this function
rather than spending clock cycles lookin

vm: Optimize page rename

Rename vm_page_rename() to vm_page_iter_rename() to reflect its
reimplementation using iterators, and pass the page to this function
rather than spending clock cycles looking it up. Change its return
value from 0/1 to a bool.

Reviewed by: dougm, markj
Differential Revision: https://reviews.freebsd.org/D47829

show more ...

vm_page: pass page to iter_free

Pass the to-be-freed page to vm_page_iter_free as a parameter, rather
than computing it from the iterator parameter, to improve performance.

Sort declarations of pag

vm_page: pass page to iter_free

Pass the to-be-freed page to vm_page_iter_free as a parameter, rather
than computing it from the iterator parameter, to improve performance.

Sort declarations of page_iter functions in vm_page.h.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47727

show more ...

vm_page.c: remove transiently defined vm_page_free_toq_impl() prototype

Sponsored by: The FreeBSD Foundation

vm_page: correct page iterator patch

The previous change committed a preliminary version of the change to
use iterators to free page sequences. This updates to what was
intended to be the final ver

vm_page: correct page iterator patch

The previous change committed a preliminary version of the change to
use iterators to free page sequences. This updates to what was
intended to be the final version.

Reviewed by: markj (previous version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D46724

show more ...

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_r

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_remove_radixdone, and remove
from it the responsibility for removing a page from its radix tree,
and pass that responsibility on to its callers.

For one of those callers, vm_page_rename, pass a pages pctrie_iter,
rather than a page, and use the iterator to remove the page from its
radix tree.

Define functions vm_page_iter_remove() and vm_page_iter_free() that
are like vm_page_remove() and vm_page_free(), respectively, except
that they take an iterator as parameter rather than a page, and use
the iterator to remove the page from the radix tree instead of
searching the radix tree. Function vm_page_iter_free() assumes that
the page is associated with an object, and calls
vm_page_free_object_prep to do the part of vm_page_free_prep that is
object-related.

In functions vm_object_split and vm_object_collapse_scan, use a
pctrie_iter to walk over the pages of the object, and use
vm_page_rename and vm_radix_iter_remove modify the radix tree without
searching for pages. In vm_object_page_remove and _kmem_unback, use a
pctrie_iter and vm_page_iter_free to remove the page from the radix
tree.

Reviewed by: markj (prevoius version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D46724

show more ...

vm: static-ize vm_page_alloc_after()

This function is only intended for the internal use of the VM system.

Reviewed by: dougm, kib, markj
Differential Revision: https://reviews.freebsd.org/D47644

vm_page: use iterators in alloc_contig_domain

Restructure a bit of code to allow vm_page_alloc_contig_domain to use
pctrie iterators for lookup and insertion into the object radix tree,
to improve p

vm_page: use iterators in alloc_contig_domain

Restructure a bit of code to allow vm_page_alloc_contig_domain to use
pctrie iterators for lookup and insertion into the object radix tree,
to improve performance.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47036

show more ...

vm: Eliminate unnecessary lock asserts

There is no actual need for the VM object to be locked when initializing
a VM page iterator.

Reviewed by: dougm
Differential Revision: https://reviews.freebsd

vm: Eliminate unnecessary lock asserts

There is no actual need for the VM object to be locked when initializing
a VM page iterator.

Reviewed by: dougm
Differential Revision: https://reviews.freebsd.org/D47298

show more ...

vm_glue: use vm_page_alloc_domain_after

Drop the function vm_page_alloc_domain, used only in
vm_thread_stack_back, and replace it with vm_page_alloc_domain_after
there, with the extra mpred argument

vm_glue: use vm_page_alloc_domain_after

Drop the function vm_page_alloc_domain, used only in
vm_thread_stack_back, and replace it with vm_page_alloc_domain_after
there, with the extra mpred argument either computed on the first
iteration or retrieved from previous iterations. Define a function
vm_page_mpred() for computing that first mpred argument.

Reviewed by: bnovkov
Differential Revision: https://reviews.freebsd.org/D47054

show more ...

vm_page: move tailq validation to grab_pages

Function vm_page_acquire_unlocked both looks up pages and validates
them. Much of it serves the needs of only one caller,
vm_page_grab_pages_unlocked, b

vm_page: move tailq validation to grab_pages

Function vm_page_acquire_unlocked both looks up pages and validates
them. Much of it serves the needs of only one caller,
vm_page_grab_pages_unlocked, by checking the validity of checking
tailq links. Extract from that function the parts that serve only
vm_page_grab_pages_unlocked, and move them into that function.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D47001

show more ...

vm_page: Fix a logic bug in vm_page_unwire_managed()

When releasing a page reference, we have logic for various cases, based
on the value of the counter. But, the implementation fails to take into

vm_page: Fix a logic bug in vm_page_unwire_managed()

When releasing a page reference, we have logic for various cases, based
on the value of the counter. But, the implementation fails to take into
account the possibility that the VPRC_BLOCKED flag is set, which is ORed
into the counter for short windows when removing mappings of a page. If
the flag is set while the last reference is being released, we may fail
to add the page to a page queue when the last wiring reference is
released.

Fix the problem by performing comparisons with VPRC_BLOCKED masked off.
While here, add a related assertion.

Reviewed by: dougm, kib
Tested by: pho
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D46944

show more ...