vm: Fix overflow issues in vm_page_startup

Firstly, pagecount is a u_long so we should ensure j is the same for the
sake of 64-bit systems. Secondly, ptoa is just a macro, and does not
cast its argu

vm: Fix overflow issues in vm_page_startup

Firstly, pagecount is a u_long so we should ensure j is the same for the
sake of 64-bit systems. Secondly, ptoa is just a macro, and does not
cast its argument, so in order to handle PAE systems correctly we need
to cast j to vm_paddr_t (the type of startp).

Fixes: 0078df5f0258 ("vm_phys: reduce touching of page->pool fields")

show more ...

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj (previous version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056e3987693f247b2ec0433.

show more ...

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

vm_pageout: Add a chicken switch for multithreaded PQ_INACTIVE scanning

Right now we have the vm.pageout_cpus_per_thread tunable which controls
the number of threads to start up per CPU per NUMA dom

vm_pageout: Add a chicken switch for multithreaded PQ_INACTIVE scanning

Right now we have the vm.pageout_cpus_per_thread tunable which controls
the number of threads to start up per CPU per NUMA domain, but after
booting, it's not possible to disable multi-threaded scanning.

There is at least one workload where this mechanism doesn't work well;
let's make it possible to disable it without a reboot, to simplify
troubleshooting.

Reviewed by: dougm, kib
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D48377

show more ...

vm_pageout: Make vmd_oom a bool

No functional change intended.

Reviewed by: dougm, kib
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Modirum MDPay
Differential Revision: https://reviews

vm_pageout: Make vmd_oom a bool

No functional change intended.

Reviewed by: dougm, kib
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D48376

show more ...

vm_page: pass page to iter_remove

Pass the to-be-freed page to vm_page_iter_remove as a parameter,
rather than computing it from the iterator parameter, to improve
performance.

Reviewed by: alc
Dif

vm_page: pass page to iter_remove

Pass the to-be-freed page to vm_page_iter_remove as a parameter,
rather than computing it from the iterator parameter, to improve
performance.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47730

show more ...

vm: Optimize page rename

Rename vm_page_rename() to vm_page_iter_rename() to reflect its
reimplementation using iterators, and pass the page to this function
rather than spending clock cycles lookin

vm: Optimize page rename

Rename vm_page_rename() to vm_page_iter_rename() to reflect its
reimplementation using iterators, and pass the page to this function
rather than spending clock cycles looking it up. Change its return
value from 0/1 to a bool.

Reviewed by: dougm, markj
Differential Revision: https://reviews.freebsd.org/D47829

show more ...

vm_page: pass page to iter_free

Pass the to-be-freed page to vm_page_iter_free as a parameter, rather
than computing it from the iterator parameter, to improve performance.

Sort declarations of pag

vm_page: pass page to iter_free

Pass the to-be-freed page to vm_page_iter_free as a parameter, rather
than computing it from the iterator parameter, to improve performance.

Sort declarations of page_iter functions in vm_page.h.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47727

show more ...

vm_page.c: remove transiently defined vm_page_free_toq_impl() prototype

Sponsored by: The FreeBSD Foundation

vm_page: correct page iterator patch

The previous change committed a preliminary version of the change to
use iterators to free page sequences. This updates to what was
intended to be the final ver

vm_page: correct page iterator patch

The previous change committed a preliminary version of the change to
use iterators to free page sequences. This updates to what was
intended to be the final version.

Reviewed by: markj (previous version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D46724

show more ...

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_r

vm_page: remove pages with iterators

Use pctrie iterators for removing some page sequences from radix
trees, to avoid repeated searches from the tree root.

Rename vm_page_object_remove to vm_page_remove_radixdone, and remove
from it the responsibility for removing a page from its radix tree,
and pass that responsibility on to its callers.

For one of those callers, vm_page_rename, pass a pages pctrie_iter,
rather than a page, and use the iterator to remove the page from its
radix tree.

Define functions vm_page_iter_remove() and vm_page_iter_free() that
are like vm_page_remove() and vm_page_free(), respectively, except
that they take an iterator as parameter rather than a page, and use
the iterator to remove the page from the radix tree instead of
searching the radix tree. Function vm_page_iter_free() assumes that
the page is associated with an object, and calls
vm_page_free_object_prep to do the part of vm_page_free_prep that is
object-related.

In functions vm_object_split and vm_object_collapse_scan, use a
pctrie_iter to walk over the pages of the object, and use
vm_page_rename and vm_radix_iter_remove modify the radix tree without
searching for pages. In vm_object_page_remove and _kmem_unback, use a
pctrie_iter and vm_page_iter_free to remove the page from the radix
tree.

Reviewed by: markj (prevoius version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D46724

show more ...

vm: static-ize vm_page_alloc_after()

This function is only intended for the internal use of the VM system.

Reviewed by: dougm, kib, markj
Differential Revision: https://reviews.freebsd.org/D47644

vm_page: use iterators in alloc_contig_domain

Restructure a bit of code to allow vm_page_alloc_contig_domain to use
pctrie iterators for lookup and insertion into the object radix tree,
to improve p

vm_page: use iterators in alloc_contig_domain

Restructure a bit of code to allow vm_page_alloc_contig_domain to use
pctrie iterators for lookup and insertion into the object radix tree,
to improve performance.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47036

show more ...

vm: Eliminate unnecessary lock asserts

There is no actual need for the VM object to be locked when initializing
a VM page iterator.

Reviewed by: dougm
Differential Revision: https://reviews.freebsd

vm: Eliminate unnecessary lock asserts

There is no actual need for the VM object to be locked when initializing
a VM page iterator.

Reviewed by: dougm
Differential Revision: https://reviews.freebsd.org/D47298

show more ...

vm_glue: use vm_page_alloc_domain_after

Drop the function vm_page_alloc_domain, used only in
vm_thread_stack_back, and replace it with vm_page_alloc_domain_after
there, with the extra mpred argument

vm_glue: use vm_page_alloc_domain_after

Drop the function vm_page_alloc_domain, used only in
vm_thread_stack_back, and replace it with vm_page_alloc_domain_after
there, with the extra mpred argument either computed on the first
iteration or retrieved from previous iterations. Define a function
vm_page_mpred() for computing that first mpred argument.

Reviewed by: bnovkov
Differential Revision: https://reviews.freebsd.org/D47054

show more ...

vm_page: move tailq validation to grab_pages

Function vm_page_acquire_unlocked both looks up pages and validates
them. Much of it serves the needs of only one caller,
vm_page_grab_pages_unlocked, b

vm_page: move tailq validation to grab_pages

Function vm_page_acquire_unlocked both looks up pages and validates
them. Much of it serves the needs of only one caller,
vm_page_grab_pages_unlocked, by checking the validity of checking
tailq links. Extract from that function the parts that serve only
vm_page_grab_pages_unlocked, and move them into that function.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D47001

show more ...

vm_page: Fix a logic bug in vm_page_unwire_managed()

When releasing a page reference, we have logic for various cases, based
on the value of the counter. But, the implementation fails to take into

vm_page: Fix a logic bug in vm_page_unwire_managed()

When releasing a page reference, we have logic for various cases, based
on the value of the counter. But, the implementation fails to take into
account the possibility that the VPRC_BLOCKED flag is set, which is ORed
into the counter for short windows when removing mappings of a page. If
the flag is set while the last reference is being released, we may fail
to add the page to a page queue when the last wiring reference is
released.

Fix the problem by performing comparisons with VPRC_BLOCKED masked off.
While here, add a related assertion.

Reviewed by: dougm, kib
Tested by: pho
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D46944

show more ...

vm_page: Use atomic loads for cmpset loops

Make sure that the compiler loads the initial value value only once.
Because atomic_fcmpset is used to load the value for subsequent
iterations, this is pr

vm_page: Use atomic loads for cmpset loops

Make sure that the compiler loads the initial value value only once.
Because atomic_fcmpset is used to load the value for subsequent
iterations, this is probably not needed, but we should not rely on that.

I verified that code generated for an amd64 GENERIC kernel does not
change.

Reviewed by: dougm, alc, kib
Tested by: pho
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D46943

show more ...

vm_meter: Add counter for NOFREE pages

This change adds a new counter that tracks the total number
of permanently allocated pages.

Differential Revision: https://reviews.freebsd.org/D46978
Reviewed

vm_meter: Add counter for NOFREE pages

This change adds a new counter that tracks the total number
of permanently allocated pages.

Differential Revision: https://reviews.freebsd.org/D46978
Reviewed by: alc, markj

show more ...

vm_page_free_pages_toq(): return the count of freed pages

Reviewed by: bnovkov
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision

vm_page_free_pages_toq(): return the count of freed pages

Reviewed by: bnovkov
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D46820

show more ...

vm: Use VM_ALLOC_NOFREE when allocating bogus_page

Allocate the 'bogus_page' page using VM_ALLOC_NOFREE since
it never gets released.

Differential Revision: https://reviews.freebsd.org/D46699
Revie

vm: Use VM_ALLOC_NOFREE when allocating bogus_page

Allocate the 'bogus_page' page using VM_ALLOC_NOFREE since
it never gets released.

Differential Revision: https://reviews.freebsd.org/D46699
Reviewed by: alc, markj, kib

show more ...

vm_radix: offer pctrie_iterator access

Add to the vm_radix and vm_page interfaces methods to use pctrie
iterators with vm_radix tries.

Reviewed by: markj
Differential Revision: https://reviews.free

vm_radix: offer pctrie_iterator access

Add to the vm_radix and vm_page interfaces methods to use pctrie
iterators with vm_radix tries.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D46663

show more ...

vm_page: fuse two allocation loops

Combine two loops, each iterating over the same array of pages to
initialize them, into a single loop.

Reviewed by: kib
Differential Revision: https://reviews.fre

vm_page: fuse two allocation loops

Combine two loops, each iterating over the same array of pages to
initialize them, into a single loop.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46609

show more ...

vm_page: Fix inconsistent use of VM_NOFREE_IMPORT_ORDER in vm_page_alloc_nofree_domain

Pass VM_NOFREE_IMPORT_ORDER to vm_phys_alloc_pages instead of
VM_LEVEL_0_ORDER when allocating a higher-order p

vm_page: Fix inconsistent use of VM_NOFREE_IMPORT_ORDER in vm_page_alloc_nofree_domain

Pass VM_NOFREE_IMPORT_ORDER to vm_phys_alloc_pages instead of
VM_LEVEL_0_ORDER when allocating a higher-order page for
the NOFREE page allocator.

Reported by: alc
Fixes: a8693e8

show more ...