vm_phys_early_startup(): Panic if phys_avail[] is empty

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48631

vm_phys_avail_split(): Tolerate split requests at boundaries

Previously, such requests would lead to a panic. The only caller so far
(vm_phys_early_startup()) actually faces the case where some add

vm_phys_avail_split(): Tolerate split requests at boundaries

Previously, such requests would lead to a panic. The only caller so far
(vm_phys_early_startup()) actually faces the case where some address can
be one of the chunk's boundaries and has to test it by hand. Moreover,
a later commit will introduce vm_phys_early_alloc_ex(), which will also
have to deal with such boundary cases.

Consequently, make this function handle boundaries by not splitting the
chunk and returning EJUSTRETURN instead of 0 to distinguish this case
from the "was split" result.

While here, expand the panic message when the address to split is not in
the passed chunk with available details.

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48630

show more ...

vm_phys_avail_count(): Fix out-of-bounds accesses

On improper termination of phys_avail[] (two consecutive 0 starting at
an even index), this function would (unnecessarily) continue searching
for th

vm_phys_avail_count(): Fix out-of-bounds accesses

On improper termination of phys_avail[] (two consecutive 0 starting at
an even index), this function would (unnecessarily) continue searching
for the termination markers even if the index was out of bounds.

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48629

show more ...

vm_phys: Check for overlap when adding a segment

Segments are passed by machine-dependent routines, so explicit checks
will make debugging much easier on very weird machines or when someone
is tweak

vm_phys: Check for overlap when adding a segment

Segments are passed by machine-dependent routines, so explicit checks
will make debugging much easier on very weird machines or when someone
is tweaking these machine-dependent routines. Additionally, this
operation is not performance-sensitive.

For the same reasons, test that we don't reach the maximum number of
physical segments (the compile-time of the internal storage) in
production kernels (replaces the existing KASSERT()).

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48628

show more ...

vm_phys_add_seg(): Check for bad segments, allow empty ones

A bad specification is if 'start' is strictly greater than 'end', or
bounds are not page aligned.

The latter was already tested under INV

vm_phys_add_seg(): Check for bad segments, allow empty ones

A bad specification is if 'start' is strictly greater than 'end', or
bounds are not page aligned.

The latter was already tested under INVARIANTS, but now will be also on
production kernels. The reason is that vm_phys_early_startup() pours
early segments into the final phys_segs[] array via vm_phys_add_seg(),
but vm_phys_early_add_seg() did not check their validity. Checking
segments once and for all in vm_phys_add_seg() avoids duplicating
validity tests and is possible since early segments are not used before
being poured into phys_segs[]. Finally, vm_phys_add_seg() is not
performance critical.

Allow empty segments and discard them (silently, unless 'bootverbose' is
true), as vm_page_startup() was testing for this case before calling
vm_phys_add_seg(), and we felt the same test in vm_phys_early_startup()
was due before calling vm_phys_add_seg(). As a consequence, remove the
empty segment test from vm_page_startup().

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48627

show more ...

vm_phys_avail_check(): Check index parity, fix panic messages

The passed index must be the start of a chunk in phys_avail[], so must
be even. Test for that and print a separate panic message.

Whil

vm_phys_avail_check(): Check index parity, fix panic messages

The passed index must be the start of a chunk in phys_avail[], so must
be even. Test for that and print a separate panic message.

While here, fix panic messages: In one, the wrong chunk boundary was
printed, and in another, the desired but not the actual condition was
printed, possibly leading to confusion.

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48626

show more ...

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj (previous version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056

Revert "vm_phys: reduce touching of page->pool fields". Pho reports, and I have
verified, that it sometimes crashes the kernel on the mmap41.sh stress test.

This reverts commit c669b08bd834553ec056e3987693f247b2ec0433.

show more ...

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the pa

vm_phys: reduce touching of page->pool fields

Change the usage of the pool field in vm_page structs.

Currently, every page belongs to a pool, and the pool field identifies
that pool, whether the page is allocated or free.

With this change, the pool field of the first page of a free block is
used by the buddy allocator to identify its pool, but the buddy
allocator makes no guarantees about the pool field value for allocated
pages. The buddy allocator requires that a pool parameter be passed as
part of freeing memory. A function that allocates memory may use the
pool field of a page to record what pool to pass as that parameter
when the memory is freed, but might not need to do so for every
allocated page.

Suggested by: alc
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D45409

show more ...

vm_phys: hide alloc_freelist_pages

Make vm_phys_alloc_freelist_pages static. There are no longer any
callers outside of vm_phys.c.

Reviewed by: alc, markj
Differential Revision: https://reviews.fr

vm_phys: hide alloc_freelist_pages

Make vm_phys_alloc_freelist_pages static. There are no longer any
callers outside of vm_phys.c.

Reviewed by: alc, markj
Differential Revision: https://reviews.freebsd.org/D46539

show more ...

vm_phys: Make sure that vm_phys_enq_chunk() stays in bounds

vm_phys_enq_chunk() inserts a run of pages into the buddy queues. When
lazy initialization is enabled, only the first page of each run is

vm_phys: Make sure that vm_phys_enq_chunk() stays in bounds

vm_phys_enq_chunk() inserts a run of pages into the buddy queues. When
lazy initialization is enabled, only the first page of each run is
initialized; vm_phys_enq_chunk() thus initializes the page following the
just-inserted run.

This fails to account for the possibility that the page following the
run doesn't belong to the segment. Handle that in vm_phys_enq_chunk().

Reported by: KASAN
Reported by: syzbot+1097ef4cee8dfb240e31@syzkaller.appspotmail.com
Fixes: b16b4c22d2d1 ("vm_page: Implement lazy page initialization")

show more ...

vm_phys: Fix vm_phys_find_range() after commit 69cbb18746b6

vm_phys_seg_paddr_to_vm_page() expects a PA that's in bounds, but
vm_phys_find_range() purposefully returns a pointer to the end of the
la

vm_phys: Fix vm_phys_find_range() after commit 69cbb18746b6

vm_phys_seg_paddr_to_vm_page() expects a PA that's in bounds, but
vm_phys_find_range() purposefully returns a pointer to the end of the
last page in a segment.

Fixes: 69cbb18746b6 ("vm_phys: Add a vm_phys_seg_paddr_to_vm_page() helper")

show more ...

vm_page: Implement lazy page initialization

FreeBSD's boot times have decreased to the point where vm_page array
initialization represents a significant fraction of the total boot time.
For example,

vm_page: Implement lazy page initialization

FreeBSD's boot times have decreased to the point where vm_page array
initialization represents a significant fraction of the total boot time.
For example, when booting FreeBSD in Firecracker (a VMM designed to
support lightweight VMs) with 128MB and 1GB of RAM, vm_page
initialization consumes 9% (3ms) and 37% (21.5ms) of the kernel boot
time, respectively. This is generally relevant in cloud environments,
where one wants to be able to spin up VMs as quickly as possible.

This patch implements lazy initialization of (most) page structures,
following a suggestion from cperciva@. The idea is to introduce a new
free pool, VM_FREEPOOL_LAZYINIT, into which all vm_page structures are
initially placed. For this to work, we need only initialize the first
free page of each chunk placed into the buddy allocator. Then, early
page allocations draw from the lazy init pool and initialize vm_page
chunks (up to 16MB, 4096 pages) on demand. Once APs are started, an
idle-priority thread drains the lazy init pool in the background to
avoid introducing extra latency in the allocator. With this scheme,
almost all of the initialization work is moved out of the critical path.

A couple of vm_phys operations require the pool to be drained before
they can run: vm_phys_find_range() and vm_phys_unfree_page(). However,
these are rare operations. I believe that
vm_phys_find_freelist_contig() does not require any special treatment,
as it only ever accesses the first page in a power-of-2-sized free page
chunk, which is always initialized.

For now the new pool is only used on amd64 and arm64, since that's where
I can easily test and those platforms would get the most benefit.

Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D40403

show more ...

vm_phys: Add a vm_phys_seg_paddr_to_vm_page() helper

No functional change intended.

Suggested by: alc
Reviewed by: dougm, alc, kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.

vm_phys: Add a vm_phys_seg_paddr_to_vm_page() helper

No functional change intended.

Suggested by: alc
Reviewed by: dougm, alc, kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43636

show more ...

vm_phys: Factor out some calls to vm_freelist_add()

A subsequent patch will make this factoring more worthwhile.

No functional change intended.

Reviewed by: dougm, alc, kib, emaste
MFC after: 2 we

vm_phys: Factor out some calls to vm_freelist_add()

A subsequent patch will make this factoring more worthwhile.

No functional change intended.

Reviewed by: dougm, alc, kib, emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D40400

show more ...

vm_phys: use ilog2(x) instead of fls(x)-1

One of these changes saves two instructions on an amd64
GENERIC-NODEBUG build. The rest are entirely cosmetic, because the
compiler can deduce that x is non

vm_phys: use ilog2(x) instead of fls(x)-1

One of these changes saves two instructions on an amd64
GENERIC-NODEBUG build. The rest are entirely cosmetic, because the
compiler can deduce that x is nonzero, and avoid the needless test.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D45331

show more ...

Revert "subr_pctrie: use ilog2(x) instead of fls(x)-1"

This reverts commit 574ef650695088d56ea12df7da76155370286f9f.

subr_pctrie: use ilog2(x) instead of fls(x)-1

In three instances where fls(x)-1 is used, the compiler does not know
that x is nonzero and so adds needless zero checks. Using ilog(x)
instead saves,

subr_pctrie: use ilog2(x) instead of fls(x)-1

In three instances where fls(x)-1 is used, the compiler does not know
that x is nonzero and so adds needless zero checks. Using ilog(x)
instead saves, in each instance, about 4 instructions, including a
conditional, and 16 or so bytes, on an amd64 build.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D45330

show more ...

vm: add macro to mark arguments used when NUMA is defined

This fixes compiler warnings when -Wunused-arguments is enabled and
not quieted.

Reviewed by: kib, markj
Obtained from: Juniper Networks, I

vm: add macro to mark arguments used when NUMA is defined

This fixes compiler warnings when -Wunused-arguments is enabled and
not quieted.

Reviewed by: kib, markj
Obtained from: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D44623

show more ...

vm_phys; fix uncalled free_contig

Function vm_phys_free_contig does not always free memory properly when
the npages parameter is less than max block size. Change it so that it does.

Note that this

vm_phys; fix uncalled free_contig

Function vm_phys_free_contig does not always free memory properly when
the npages parameter is less than max block size. Change it so that it does.

Note that this function is not currently invoked, and this error was
not triggered in earlier versions of the code.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D42891

show more ...

vm_phys: fix freelist_contig

vm_phys_find_freelist_contig is called to search a list of max-sized
free page blocks and find one that, when joined with adjacent blocks
in memory, can satisfy a reques

vm_phys: fix freelist_contig

vm_phys_find_freelist_contig is called to search a list of max-sized
free page blocks and find one that, when joined with adjacent blocks
in memory, can satisfy a request for a memory allocation bigger than
any single max-sized free page block. In commit
fa8a6585c7522b7de6d29802967bd5eba2f2dcf1, I defined this function in
order to offer two improvements: 1) reduce the worst-case search time,
and 2) allow solutions that include less-than max-sized free page
blocks at the front or back of the giant allocation. However, it turns
out that this change introduced an error, reported in In Bug
274592. That error concerns failing to check segment boundaries. This
change fixes an error in vm_phys_find_freelist_config that resolves
that bug. It also abandons improvement 2), because the value of that
improvement is small and because preserving it would require more
testing than I am able to do.

PR: 274592
Reported by: shafaisal.us@gmail.com
Reviewed by: alc, markj
Tested by: shafaisal.us@gmail.com
Fixes: fa8a6585c752 vm_phys: avoid waste in multipage allocation
MFC after: 10 days
Differential Revision: https://reviews.freebsd.org/D42509

show more ...

vm_phys: Add corresponding sysctl knob for loader tunable

The loader tunable 'vm.numa.disabled' does not have corresponding sysctl
MIB entry. Add it so that it can be retrieved, and `sysctl -T` will

vm_phys: Add corresponding sysctl knob for loader tunable

The loader tunable 'vm.numa.disabled' does not have corresponding sysctl
MIB entry. Add it so that it can be retrieved, and `sysctl -T` will also
report it correctly.

Reviewed by: markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D42138

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

vm_phys: tune vm_phys_enqueue_contig loop

Rewrite the final loop in vm_phys_enqueue_contig as a new function,
vm_phys_enq_beg, to reduce amd64 code size.

Reviewed by: kib
Differential Revision: htt

vm_phys: tune vm_phys_enqueue_contig loop

Rewrite the final loop in vm_phys_enqueue_contig as a new function,
vm_phys_enq_beg, to reduce amd64 code size.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D41289

show more ...

vm_phys_enq_range: no alignment assert for npages==0

Do not assume that when vm_phys_enq_range is passed npages==0 that the
vm_page argument is valid in any way, much less that it has a
page-aligned

vm_phys_enq_range: no alignment assert for npages==0

Do not assume that when vm_phys_enq_range is passed npages==0 that the
vm_page argument is valid in any way, much less that it has a
page-aligned address. Just don't look at it. Assert nothing about it.

Reported by: karels
Differential Revision: https://reviews.freebsd.org/D41317

show more ...