vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj (previous version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056e3987693f247b2ec0433.

show more ...

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

vm: Optimize page rename

Rename vm_page_rename() to vm_page_iter_rename() to reflect its
reimplementation using iterators, and pass the page to this function
rather than spending clock cycles lookin

vm: Optimize page rename

Rename vm_page_rename() to vm_page_iter_rename() to reflect its
reimplementation using iterators, and pass the page to this function
rather than spending clock cycles looking it up. Change its return
value from 0/1 to a bool.

Reviewed by: dougm, markj
Differential Revision: https://reviews.freebsd.org/D47829

show more ...

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

Revert "subr_pctrie: use ilog2(x) instead of fls(x)-1"

This reverts commit 574ef650695088d56ea12df7da76155370286f9f.

subr_pctrie: use ilog2(x) instead of fls(x)-1

In three instances where fls(x)-1 is used, the compiler does not know
that x is nonzero and so adds needless zero checks. Using ilog(x)
instead saves,

subr_pctrie: use ilog2(x) instead of fls(x)-1

In three instances where fls(x)-1 is used, the compiler does not know
that x is nonzero and so adds needless zero checks. Using ilog(x)
instead saves, in each instance, about 4 instructions, including a
conditional, and 16 or so bytes, on an amd64 build.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D45330

show more ...

vm_reserv_reclaim_contig: Return NULL not false

Reviewed by: dougm, zlei
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D44667

vm_reserv: Add vm_reserv_is_populated

Add a function to check whether an aligned block of vm pages are
allocated, for use with impending changes to arm64 superpage
managment.

Reviewed by: alc
Diffe

vm_reserv: Add vm_reserv_is_populated

Add a function to check whether an aligned block of vm pages are
allocated, for use with impending changes to arm64 superpage
managment.

Reviewed by: alc
Differential Revision: http://reviews.freebsd.org/D44575

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

vm_reserv: use enhanced bitstring for popmaps

vm_reserv.c uses its own bitstring implemenation for popmaps. Using
the bitstring_t type from a standard header eliminates the code
duplication, allows

vm_reserv: use enhanced bitstring for popmaps

vm_reserv.c uses its own bitstring implemenation for popmaps. Using
the bitstring_t type from a standard header eliminates the code
duplication, allows some bit-at-a-time operations to be replaced with
more efficient bitstring range operations, and, in
vm_reserv_test_contig, allows bit_ffc_area_at to more efficiently
search for a big-enough set of consecutive zero-bits.

Make bitstring changes improve the vm_reserv code. Define a bit_ntest
method to test whether a range of bits is all set, or all clear.
Define bit_ff_at and bit_ff_area_at to implement the ffs and ffc
versions with a parameter to choose between set- and clear- bits.
Improve the area_at implementation. Modify the bit_nset and
bit_nclear implementations to allow code optimization in the cases
when start or end are multiples of _BITSTR_BITS.

Add a few new cases to bitstring_test.

Discussed with: alc
Reviewed by: markj
Tested by: pho (earlier version)
Differential Revision: https://reviews.freebsd.org/D33312

show more ...

vm_reserv: #include vm_extern.h explicitly, for arm.

Fixes: c606ab59e7f9 vm_extern: use standard address checkers everywhere

vm_extern: use standard address checkers everywhere

Define simple functions for alignment and boundary checks and use them
everywhere instead of having slightly different implementations
scattered a

vm_extern: use standard address checkers everywhere

Define simple functions for alignment and boundary checks and use them
everywhere instead of having slightly different implementations
scattered about. Define them in vm_extern.h and use them where
possible where vm_extern.h is included.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D33685

show more ...

vm_reserv: fix zero-boundary error

Handle specially the boundary==0 case of vm_reserv_reclaim_config,
by turning off boundary adjustment in that case.

Reviewed by: alc
Tested by: pho, madpilot

vm: alloc pages from reserv before breaking it

Function vm_reserv_reclaim_contig breaks a reservation with enough
free space to satisfy an allocation request and returns the free space
to the buddy

vm: alloc pages from reserv before breaking it

Function vm_reserv_reclaim_contig breaks a reservation with enough
free space to satisfy an allocation request and returns the free space
to the buddy allocator. Change the function to allocate the request
memory from the reservation before breaking it, and return that memory
to the caller. That avoids a second call to the buddy allocator and
guarantees successful allocation after breaking the reservation, where
that success is not currently guaranteed.

Reviewed by: alc, kib (previous version)
Differential Revision: https://reviews.freebsd.org/D33644

show more ...

Correct type size format error in KASSERT.
Reported by: jenkins
Fixes: 6f1c8908272f vm: Don't break vm reserv that can't meet align reqs

vm: Don't break vm reserv that can't meet align reqs

Function vm_reserv_test_contig has incorrectly used its alignment
and boundary parameters to find a well-positioned range of empty pages
in a res

vm: Don't break vm reserv that can't meet align reqs

Function vm_reserv_test_contig has incorrectly used its alignment
and boundary parameters to find a well-positioned range of empty pages
in a reservation. Consequently, a reservation could be broken
mistakenly when it was unable to provide a satisfactory set of pages.

Rename the function, correct the errors, and add assertions to detect
the error in case it appears again.

Reviewed by: alc, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33344

show more ...

Set uninitialized popmap bits in vm_reserv_init

In vm_reserv_init, set all the marker popmap bits in vm_reserv_init,
and not just the bits of the first popmap entry.

Reviewed by: markj
MFC after: 1

Set uninitialized popmap bits in vm_reserv_init

In vm_reserv_init, set all the marker popmap bits in vm_reserv_init,
and not just the bits of the first popmap entry.

Reviewed by: markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33258

show more ...

vm_reserv: Fix list locking in vm_reserv_reclaim_contig()

The per-domain partpop queue is locked by the combination of the
per-domain lock and individual reservation mutexes.
vm_reserv_reclaim_conti

vm_reserv: Fix list locking in vm_reserv_reclaim_contig()

The per-domain partpop queue is locked by the combination of the
per-domain lock and individual reservation mutexes.
vm_reserv_reclaim_contig() scans the queue looking for partially
populated reservations that can be reclaimed in order to satisfy the
caller's allocation.

During the scan, we drop the per-domain lock. At this point, the rvn
pointer may be invalidated. Take care to load rvn after re-acquiring
the per-domain lock.

While here, simplify the condition used to check whether a reservation
was dequeued while the per-domain lock was dropped.

Reviewed by: alc, kib
Reported by: gallatin
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29203

show more ...

vm_phys: Try to clean up NUMA KPIs

It can useful for code outside the VM system to look up the NUMA domain
of a page backing a virtual or physical address, specifically when
creating NUMA-aware data

vm_phys: Try to clean up NUMA KPIs

It can useful for code outside the VM system to look up the NUMA domain
of a page backing a virtual or physical address, specifically when
creating NUMA-aware data structures. We have _vm_phys_domain() for
this, but the leading underscore implies that it's an internal function,
and vm_phys.h has dependencies on a number of other headers.

Rename vm_phys_domain() to vm_page_domain(), and _vm_phys_domain() to
vm_phys_domain(). Make the latter an inline function.

Add _vm_phys.h and define struct vm_phys_seg there so that it's easier
to use in other headers. Include it from vm_page.h so that
vm_page_domain() can be defined there.

Include machine/vmparam.h from _vm_phys.h since it depends directly on
some constants defined there.

Reviewed by: alc
Reviewed by: dougm, kib (earlier versions)
Differential Revision: https://reviews.freebsd.org/D27207

show more ...

Flag vm_reserv and vm_phys sysctls as MPSAFE.

Nothing in these subsystems relies on Giant.

MFC after: 1 week

vm_reserv: Sparsify the vm_reserv_array when VM_PHYSSEG_SPARSE

On an Ampere Altra system, the physical memory is populated
sparsely within the physical address space, with only about 0.4%
of physica

vm_reserv: Sparsify the vm_reserv_array when VM_PHYSSEG_SPARSE

On an Ampere Altra system, the physical memory is populated
sparsely within the physical address space, with only about 0.4%
of physical addresses backed by RAM in the range [0, last_pa].

This is causing the vm_reserv_array to be over-sized by a few
orders of magnitude, wasting roughly 5 GiB on a system with
256 GiB of RAM.

The sparse allocation of vm_reserv_array is controlled by defining
VM_PHYSSEG_SPARSE, with the dense allocation still remaining for
platforms with VM_PHYSSEG_DENSE.

Reviewed by: markj, alc, kib
Approved by: scottl (implicit)
MFC after: 1 week
Sponsored by: Ampere Computing, Inc.
Differential Revision: https://reviews.freebsd.org/D26130

show more ...

Merge ^/head r358712 through r358730.

Use COUNTER_U64_DEFINE_EARLY() in places where it simplifies things.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23978