vm_page: expose page_alloc_after

vm_page_alloc() just calls vm_page_alloc_after(), after it has found
the predecessor of a page parameter. Many callers of vm_page_alloc()
already know that predecess

vm_page: expose page_alloc_after

vm_page_alloc() just calls vm_page_alloc_after(), after it has found
the predecessor of a page parameter. Many callers of vm_page_alloc()
already know that predecessor. Letting them pass that to
vm_page_alloc_after() directly could save a little redundant
calculation.

Reviewed by: alc
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D49103

show more ...

vm_map: wrap map->system_map checks into wrapper

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47934

vm_fault: avoid vm_page_next()

Where vm_fault calls vm_page_next, replace it with a use of TAILQ_NEXT
and a KASSERT. This avoids needless computation in a NODEBUG kernel
and makes the error checkin

vm_fault: avoid vm_page_next()

Where vm_fault calls vm_page_next, replace it with a use of TAILQ_NEXT
and a KASSERT. This avoids needless computation in a NODEBUG kernel
and makes the error checking clearer in a GENERIC kernel.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46168

show more ...

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

pmap: Skip some superpage promotion attempts that will fail

Implement a simple heuristic to skip pointless promotion attempts by
pmap_enter_quick_locked() and moea64_enter(). Specifically, when
vm_

pmap: Skip some superpage promotion attempts that will fail

Implement a simple heuristic to skip pointless promotion attempts by
pmap_enter_quick_locked() and moea64_enter(). Specifically, when
vm_fault() calls pmap_enter_quick() to map neighboring pages at the end
of a copy-on-write fault, there is no point in attempting promotion in
pmap_enter_quick_locked() and moea64_enter(). Promotion will fail
because the base pages have differing protection.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45431
MFC after: 1 week

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

vm_fault: Revert commit 64087fd7f372

The underlying issue that originally triggered a kernel panic was
addressed and the fix was ported to all relevant pmaps, so the
safeguards placed in vm_fault.c

vm_fault: Revert commit 64087fd7f372

The underlying issue that originally triggered a kernel panic was
addressed and the fix was ported to all relevant pmaps, so the
safeguards placed in vm_fault.c can be removed now.

Reviewed by: alc, kib, markj
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D42517

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

vm_map: Add a macro to fetch a map entry's split boundary index

The resulting code is a bit more concise. No functional change
intended.

Reviewed by: alc, dougm, kib
MFC after: 1 week
Differential

vm_map: Add a macro to fetch a map entry's split boundary index

The resulting code is a bit more concise. No functional change
intended.

Reviewed by: alc, dougm, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D41249

show more ...

vm_fault: move FAULT_* return codes out of range for Mach errors

This way a possible clash between FAULT_* and KERN_* numbering is
avoided, and panics checks for fault_status confusion become more
e

vm_fault: move FAULT_* return codes out of range for Mach errors

This way a possible clash between FAULT_* and KERN_* numbering is
avoided, and panics checks for fault_status confusion become more
efficient.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D40771

show more ...

vm: add unlocked page lookup before trying vm_fault_soft_fast

Shaves a read lock + tryupgrade trip most of the time.

Stats from doing a kernel build (counters not present in the tree):
vm.fault_sof

vm: add unlocked page lookup before trying vm_fault_soft_fast

Shaves a read lock + tryupgrade trip most of the time.

Stats from doing a kernel build (counters not present in the tree):
vm.fault_soft_fast_ok: 262653
vm.fault_soft_fast_failed_other: 41
vm.fault_soft_fast_failed_no_page: 39595772
vm.fault_soft_fast_failed_page_busy: 1929
vm.fault_soft_fast_failed_page_invalid: 22183

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D39268

show more ...

vm: consistently prefix fault helpers with vm_fault_

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D39029

vm: avoid lock upgrade if possible in vm_fault_next

In my tests during buildkernel fs->m was always NULL at that stage.

Note the change has no impact on vm obj contention during said workload.

Rev

vm: avoid lock upgrade if possible in vm_fault_next

In my tests during buildkernel fs->m was always NULL at that stage.

Note the change has no impact on vm obj contention during said workload.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D39027

show more ...

vm: read-locked fault handling for backing objects

This is almost the simplest patch which manages to avoid write locking
for backing objects, as a result mostly fixing vm object contention
problems

vm: read-locked fault handling for backing objects

This is almost the simplest patch which manages to avoid write locking
for backing objects, as a result mostly fixing vm object contention
problems.

What is not fixed:
1. cacheline ping pong due to read-locks
2. cacheline ping pong due to pip
3. cacheling ping pong due to object busying
4. write locking on first object

On top of it the use of VM_OBJECT_UNLOCK instead of explicitly tracking
the state is slower multithreaded that it needs to be, done for
simplicity for the time being.

Sample lock profiling results doing -j 104 buildkernel on tmpfs:
before:
71446200 (rw:vmobject)
14689706 (sx:vm map (user))
4166251 (rw:pmap pv list)
2799924 (spin mutex:turnstile chain)

after:
19940411 (rw:vmobject)
8166012 (rw:pmap pv list)
6017608 (sx:vm map (user))
1151416 (sleep mutex:pipe mutex)

Reviewed by: kib
Reviewed by: markj
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D38964

show more ...

vm: move up object lock asserts in fault functions

No functional changes.

Reviewed by: kib
Reviewed by: markj
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D38964

vm_fault: Update a comment to reflect the removal of the default pager

Fixes: 5d32157d4e5c ("vm_object: Modify vm_object_allocate_anon() to return OBJT_SWAP objects")
Reviewed by: alc, kib
Different

vm_fault: Update a comment to reflect the removal of the default pager

Fixes: 5d32157d4e5c ("vm_object: Modify vm_object_allocate_anon() to return OBJT_SWAP objects")
Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D38985

show more ...

vm_fault: Fix a race in vm_fault_soft_fast()

When vm_fault_soft_fast() creates a mapping, it release the VM map lock
before unbusying the top-level object. Without the map lock, however,
nothing pr

vm_fault: Fix a race in vm_fault_soft_fast()

When vm_fault_soft_fast() creates a mapping, it release the VM map lock
before unbusying the top-level object. Without the map lock, however,
nothing prevents the VM object from being deallocated while still busy.

Fix the problem by unbusying the object before releasing the VM map
lock. If vm_fault_soft_fast() fails to create a mapping, the VM map
lock is not released, so those cases don't need to change.

Reported by: syzkaller
Reviewed by: kib (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D38527

show more ...

vm_pager: add method to veto page allocation

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D37097

vm_fault: Shoot down shared mappings in vm_fault_copy_entry()

As in vm_fault_cow(), it's possible, albeit rare, for multiple vm_maps
to share a shadow object. When copying a page from a backing obj

vm_fault: Shoot down shared mappings in vm_fault_copy_entry()

As in vm_fault_cow(), it's possible, albeit rare, for multiple vm_maps
to share a shadow object. When copying a page from a backing object
into the shadow, all mappings of the source page must therefore be
removed. Otherwise, future operations on the object tree may detect
that the source page is fully shadowed and thus can be freed.

Approved by: so
Security: FreeBSD-SA-22:11.vm
Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35635

show more ...

vm: Remove handling for OBJT_DEFAULT objects

Now that OBJT_DEFAULT objects can't be instantiated, we can simplify
checks of the form object->type == OBJT_DEFAULT || (object->flags &
OBJ_SWAP) != 0.

vm: Remove handling for OBJT_DEFAULT objects

Now that OBJT_DEFAULT objects can't be instantiated, we can simplify
checks of the form object->type == OBJT_DEFAULT || (object->flags &
OBJ_SWAP) != 0. No functional change intended.

Reviewed by: alc, kib
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35788

show more ...

vm_object: Modify vm_object_allocate_anon() to return OBJT_SWAP objects

With this change, OBJT_DEFAULT objects are no longer allocated.
Instead, anonymous objects are always of type OBJT_SWAP and al

vm_object: Modify vm_object_allocate_anon() to return OBJT_SWAP objects

With this change, OBJT_DEFAULT objects are no longer allocated.
Instead, anonymous objects are always of type OBJT_SWAP and always have
OBJ_SWAP set.

Modify the page fault handler to check the swap block radix tree in
places where it checked for objects of type OBJT_DEFAULT. In
particular, there's no need to invoke getpages for an OBJT_SWAP object
with no swap blocks assigned.

Reviewed by: alc, kib
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35785

show more ...

vm_fault: Fix some nits in vm_fault_copy_entry()

- Correct the description (vm_fault_copy_entry() does not create a
shadow object).
- Move some initialization and assertions out of the scope of th

vm_fault: Fix some nits in vm_fault_copy_entry()

- Correct the description (vm_fault_copy_entry() does not create a
shadow object).
- Move some initialization and assertions out of the scope of the object
locks, when doing so makes sense.
- Merge a pair of conditional blocks.
- Use __unused when appropriate.

No functional change intended.

Reviewed by: alc
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

show more ...

vm_fault: Avoid unnecessary object relocking in vm_fault_copy_entry()

Suggested by: alc
Reviewed by: alc, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://r

vm_fault: Avoid unnecessary object relocking in vm_fault_copy_entry()

Suggested by: alc
Reviewed by: alc, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35485

show more ...

vm_fault: Fix a racy copy of page valid bits

We do not hold the object lock or a page busy lock when copying src_m's
validity state. Prior to commit 45d72c7d7fca we marked dst_m as fully
valid.

Us

vm_fault: Fix a racy copy of page valid bits

We do not hold the object lock or a page busy lock when copying src_m's
validity state. Prior to commit 45d72c7d7fca we marked dst_m as fully
valid.

Use the source object's read lock to ensure that valid bits are not
concurrently cleared.

Reviewed by: alc, kib
Fixes: 45d72c7d7fca ("vm_fault_copy_entry: accept invalid source pages.")
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35471

show more ...

vm_fault_dontneed: Inline value of variable used once in an assertion.