vm: consistently prefix fault helpers with vm_fault_

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D39029

vm: avoid lock upgrade if possible in vm_fault_next

In my tests during buildkernel fs->m was always NULL at that stage.

Note the change has no impact on vm obj contention during said workload.

Rev

vm: avoid lock upgrade if possible in vm_fault_next

In my tests during buildkernel fs->m was always NULL at that stage.

Note the change has no impact on vm obj contention during said workload.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D39027

show more ...

vm: read-locked fault handling for backing objects

This is almost the simplest patch which manages to avoid write locking
for backing objects, as a result mostly fixing vm object contention
problems

vm: read-locked fault handling for backing objects

This is almost the simplest patch which manages to avoid write locking
for backing objects, as a result mostly fixing vm object contention
problems.

What is not fixed:
1. cacheline ping pong due to read-locks
2. cacheline ping pong due to pip
3. cacheling ping pong due to object busying
4. write locking on first object

On top of it the use of VM_OBJECT_UNLOCK instead of explicitly tracking
the state is slower multithreaded that it needs to be, done for
simplicity for the time being.

Sample lock profiling results doing -j 104 buildkernel on tmpfs:
before:
71446200 (rw:vmobject)
14689706 (sx:vm map (user))
4166251 (rw:pmap pv list)
2799924 (spin mutex:turnstile chain)

after:
19940411 (rw:vmobject)
8166012 (rw:pmap pv list)
6017608 (sx:vm map (user))
1151416 (sleep mutex:pipe mutex)

Reviewed by: kib
Reviewed by: markj
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D38964

show more ...

vm: move up object lock asserts in fault functions

No functional changes.

Reviewed by: kib
Reviewed by: markj
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D38964

vm_fault: Update a comment to reflect the removal of the default pager

Fixes: 5d32157d4e5c ("vm_object: Modify vm_object_allocate_anon() to return OBJT_SWAP objects")
Reviewed by: alc, kib
Different

vm_fault: Update a comment to reflect the removal of the default pager

Fixes: 5d32157d4e5c ("vm_object: Modify vm_object_allocate_anon() to return OBJT_SWAP objects")
Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D38985

show more ...

vm_fault: Fix a race in vm_fault_soft_fast()

When vm_fault_soft_fast() creates a mapping, it release the VM map lock
before unbusying the top-level object. Without the map lock, however,
nothing pr

vm_fault: Fix a race in vm_fault_soft_fast()

When vm_fault_soft_fast() creates a mapping, it release the VM map lock
before unbusying the top-level object. Without the map lock, however,
nothing prevents the VM object from being deallocated while still busy.

Fix the problem by unbusying the object before releasing the VM map
lock. If vm_fault_soft_fast() fails to create a mapping, the VM map
lock is not released, so those cases don't need to change.

Reported by: syzkaller
Reviewed by: kib (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D38527

show more ...

vm_pager: add method to veto page allocation

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D37097

vm_fault: Shoot down shared mappings in vm_fault_copy_entry()

As in vm_fault_cow(), it's possible, albeit rare, for multiple vm_maps
to share a shadow object. When copying a page from a backing obj

vm_fault: Shoot down shared mappings in vm_fault_copy_entry()

As in vm_fault_cow(), it's possible, albeit rare, for multiple vm_maps
to share a shadow object. When copying a page from a backing object
into the shadow, all mappings of the source page must therefore be
removed. Otherwise, future operations on the object tree may detect
that the source page is fully shadowed and thus can be freed.

Approved by: so
Security: FreeBSD-SA-22:11.vm
Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35635

show more ...

vm: Remove handling for OBJT_DEFAULT objects

Now that OBJT_DEFAULT objects can't be instantiated, we can simplify
checks of the form object->type == OBJT_DEFAULT || (object->flags &
OBJ_SWAP) != 0.

vm: Remove handling for OBJT_DEFAULT objects

Now that OBJT_DEFAULT objects can't be instantiated, we can simplify
checks of the form object->type == OBJT_DEFAULT || (object->flags &
OBJ_SWAP) != 0. No functional change intended.

Reviewed by: alc, kib
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35788

show more ...

vm_object: Modify vm_object_allocate_anon() to return OBJT_SWAP objects

With this change, OBJT_DEFAULT objects are no longer allocated.
Instead, anonymous objects are always of type OBJT_SWAP and al

vm_object: Modify vm_object_allocate_anon() to return OBJT_SWAP objects

With this change, OBJT_DEFAULT objects are no longer allocated.
Instead, anonymous objects are always of type OBJT_SWAP and always have
OBJ_SWAP set.

Modify the page fault handler to check the swap block radix tree in
places where it checked for objects of type OBJT_DEFAULT. In
particular, there's no need to invoke getpages for an OBJT_SWAP object
with no swap blocks assigned.

Reviewed by: alc, kib
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35785

show more ...

vm_fault: Fix some nits in vm_fault_copy_entry()

- Correct the description (vm_fault_copy_entry() does not create a
shadow object).
- Move some initialization and assertions out of the scope of th

vm_fault: Fix some nits in vm_fault_copy_entry()

- Correct the description (vm_fault_copy_entry() does not create a
shadow object).
- Move some initialization and assertions out of the scope of the object
locks, when doing so makes sense.
- Merge a pair of conditional blocks.
- Use __unused when appropriate.

No functional change intended.

Reviewed by: alc
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

show more ...

vm_fault: Avoid unnecessary object relocking in vm_fault_copy_entry()

Suggested by: alc
Reviewed by: alc, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://r

vm_fault: Avoid unnecessary object relocking in vm_fault_copy_entry()

Suggested by: alc
Reviewed by: alc, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35485

show more ...

vm_fault: Fix a racy copy of page valid bits

We do not hold the object lock or a page busy lock when copying src_m's
validity state. Prior to commit 45d72c7d7fca we marked dst_m as fully
valid.

Us

vm_fault: Fix a racy copy of page valid bits

We do not hold the object lock or a page busy lock when copying src_m's
validity state. Prior to commit 45d72c7d7fca we marked dst_m as fully
valid.

Use the source object's read lock to ensure that valid bits are not
concurrently cleared.

Reviewed by: alc, kib
Fixes: 45d72c7d7fca ("vm_fault_copy_entry: accept invalid source pages.")
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35471

show more ...

vm_fault_dontneed: Inline value of variable used once in an assertion.

kern: Fix typo in kassert message.

- s/unepxected/unexpected/
MFC after: 3 days

vm_fault: Fix vm_fault_populate()'s handling of VM_FAULT_WIRE

vm_map_wire() works by calling vm_fault(VM_FAULT_WIRE) on each page in
the rage. (For largepage mappings, it calls vm_fault() once per

vm_fault: Fix vm_fault_populate()'s handling of VM_FAULT_WIRE

vm_map_wire() works by calling vm_fault(VM_FAULT_WIRE) on each page in
the rage. (For largepage mappings, it calls vm_fault() once per large
page.)

A pager's populate method may return more than one page to be mapped.
If VM_FAULT_WIRE is also specified, we'd wire each page in the run, not
just the fault page. Consider an object with two pages mapped in a
vm_map_entry, and suppose vm_map_wire() is called on the entry. Then,
the first vm_fault() would allocate and wire both pages, and the second
would encounter a valid page upon lookup and wire it again in the
regular fault handler. So the second page is wired twice and will be
leaked when the object is destroyed.

Fix the problem by modify vm_fault_populate() to wire only the fault
page. Also modify the error handler for pmap_enter(psind=1) to not test
fs->wired, since it must be false.

PR: 260347
Reviewed by: alc, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33416

show more ...

vm_fault: Factor out per-object operations into vm_fault_object()

No functional change intended.

Obtained from: jeff (object_concurrency patches)
Reviewed by: kib
MFC after: 1 week
Differential Rev

vm_fault: Factor out per-object operations into vm_fault_object()

No functional change intended.

Obtained from: jeff (object_concurrency patches)
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33018

show more ...

vm_fault: Introduce a fault_status enum for internal return types

Rather than overloading the meanings of the Mach statuses, introduce a
new set for use internally in the fault code. This makes the

vm_fault: Introduce a fault_status enum for internal return types

Rather than overloading the meanings of the Mach statuses, introduce a
new set for use internally in the fault code. This makes the control
flow easier to follow and provides some extra error checking when a
fault status variable is used in a switch statement.

vm_fault_lookup() and vm_fault_relookup() continue to use Mach statuses
for now, as there isn't much benefit to converting them and they
effectively pass through a status from vm_map_lookup().

Obtained from: jeff (object_concurrency patches)
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D33017

show more ...

vm_fault: Move nera into faultstate

This makes it easier to factor out pieces of vm_fault(). No functional
change intended.

Obtained from: jeff (object_concurrency patches)
Reviewed by: kib
MFC af

vm_fault: Move nera into faultstate

This makes it easier to factor out pieces of vm_fault(). No functional
change intended.

Obtained from: jeff (object_concurrency patches)
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D33016

show more ...

vm_page: Consolidate page busy sleep mechanisms

- Modify vm_page_busy_sleep() and vm_page_busy_sleep_unlocked() to take
a VM_ALLOC_* flag indicating whether to sleep on shared-busy, and fix
up c

vm_page: Consolidate page busy sleep mechanisms

- Modify vm_page_busy_sleep() and vm_page_busy_sleep_unlocked() to take
a VM_ALLOC_* flag indicating whether to sleep on shared-busy, and fix
up callers.
- Modify vm_page_busy_sleep() to return a status indicating whether the
object lock was dropped, and fix up callers.
- Convert callers of vm_page_sleep_if_busy() to use vm_page_busy_sleep()
instead.
- Remove vm_page_sleep_if_(x)busy().

No functional change intended.

Obtained from: jeff (object_concurrency patches)
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D32947

show more ...

vm_fault: Stop specifying VM_ALLOC_ZERO

Now vm_page_alloc() and friends will unconditionally preserve PG_ZERO,
so there is no point in setting this flag.

Eliminate a local variable and add a commen

vm_fault: Stop specifying VM_ALLOC_ZERO

Now vm_page_alloc() and friends will unconditionally preserve PG_ZERO,
so there is no point in setting this flag.

Eliminate a local variable and add a comment explaining why we
prioritize the allocation when the process is doomed.

No functional change intended.

Reviewed by: kib, alc
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32036

show more ...

vm_fault: do not trigger OOM too early

Wakeup in vm_waitpfault() does not mean that the thread would get the
page on the next vm_page_alloc() call, other thread might steal the free
page we were wai

vm_fault: do not trigger OOM too early

Wakeup in vm_waitpfault() does not mean that the thread would get the
page on the next vm_page_alloc() call, other thread might steal the free
page we were waiting for. On the other hand, this wakeup might come much
earlier than just vm_pfault_oom_wait seconds, if the rate of the page
reclamation is high enough.

If wakeups come fast and we loose the allocation race enough times, OOM
could be undeservably triggered much earlier than vm_pfault_oom_attempts
x vm_pfault_oom_wait seconds. Fix it by not counting the number of sleeps,
but measuring the time to th first allocation failure, and triggering OOM
when it was older than oom_attempts x oom_wait seconds.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D32287

show more ...

Add OBJT_SWAP_TMPFS pager

This is OBJT_SWAP pager, specialized for tmpfs. Right now, both swap pager
and generic vm code have to explicitly handle swap objects which are tmpfs
vnode v_object, in th

Add OBJT_SWAP_TMPFS pager

This is OBJT_SWAP pager, specialized for tmpfs. Right now, both swap pager
and generic vm code have to explicitly handle swap objects which are tmpfs
vnode v_object, in the special ways. Replace (almost) all such places with
proper methods.

Since VM still needs a notion of the 'swap object', regardless of its
use, add yet another type-classification flag OBJ_SWAP. Set it in
vm_object_allocate() where other type-class flags are set.

This change almost completely eliminates the knowledge of tmpfs from VM,
and opens a way to make OBJT_SWAP_TMPFS loadable from tmpfs.ko.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D30070

show more ...

vm_fault: Shoot down multiply mapped COW source page mappings

Reviewed by: kib, rlibby
Discussed with: alc
Approved by: so
Security: CVE-2021-29626
Security: FreeBSD-SA-21:08.vm

Clean up a couple of MD warts in vm_fault_populate():

--Eliminate a big ifdef that encompassed all currently-supported
architectures except mips and powerpc32. This applied to the case
in which we'

Clean up a couple of MD warts in vm_fault_populate():

--Eliminate a big ifdef that encompassed all currently-supported
architectures except mips and powerpc32. This applied to the case
in which we've allocated a superpage but the pager-populated range
is insufficient for a superpage mapping. For platforms that don't
support superpages the check should be inexpensive as we shouldn't
get a superpage in the first place. Make the normal-page fallback
logic identical for all platforms and provide a simple implementation
of pmap_ps_enabled() for MIPS and Book-E/AIM32 powerpc.

--Apply the logic for handling pmap_enter() failure if a superpage
mapping can't be supported due to additional protection policy.
Use KERN_PROTECTION_FAILURE instead of KERN_FAILURE for this case,
and note Intel PKU on amd64 as the first example of such protection
policy.

Reviewed by: kib, markj, bdragon
Differential Revision: https://reviews.freebsd.org/D29439

show more ...