vm_fault: handle KERN_PROTECTION_FAILURE

pmap_enter(PMAP_ENTER_LARGEPAGE) may return KERN_PROTECTION_FAILURE due to
PKRU inconsistency. Handle it in the call place from vm_fault_populate(),
and in

vm_fault: handle KERN_PROTECTION_FAILURE

pmap_enter(PMAP_ENTER_LARGEPAGE) may return KERN_PROTECTION_FAILURE due to
PKRU inconsistency. Handle it in the call place from vm_fault_populate(),
and in places which decode errors from vm_fault_populate()/
vm_fault_allocate().

Reviewed by: jah, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D29442

show more ...

Make MAXPHYS tunable. Bump MAXPHYS to 1M.

Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.

Make b_pag

Make MAXPHYS tunable. Bump MAXPHYS to 1M.

Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.

Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.

Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.

Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.

Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225

show more ...

Implement superpages for PowerPC64 (HPT)

This change adds support for transparent superpages for PowerPC64
systems using Hashed Page Tables (HPT). All pmap operations are
supported.

The changes wer

Implement superpages for PowerPC64 (HPT)

This change adds support for transparent superpages for PowerPC64
systems using Hashed Page Tables (HPT). All pmap operations are
supported.

The changes were inspired by RISC-V implementation of superpages,
by @markj (r344106), but heavily adapted to fit PPC64 HPT architecture
and existing MMU OEA64 code.

While these changes are not better tested, superpages support is disabled by
default. To enable it, use vm.pmap.superpages_enabled=1.

In this initial implementation, when superpages are disabled, system
performance stays at the same level as without these changes. When
superpages are enabled, buildworld time increases a bit (~2%). However,
for workloads that put a heavy pressure on the TLB the performance boost
is much bigger (see HPC Challenge and pgbench on D25237).

Reviewed by: jhibbits
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D25237

show more ...

Implement sparse core dumps

Currently we allocate and map zero-filled anonymous pages when dumping
core. This can result in lots of needless disk I/O and page
allocations. This change tries to mak

Implement sparse core dumps

Currently we allocate and map zero-filled anonymous pages when dumping
core. This can result in lots of needless disk I/O and page
allocations. This change tries to make the core dumper more clever and
represent unbacked ranges of virtual memory by holes in the core dump
file.

Add a new page fault type, VM_FAULT_NOFILL, which causes vm_fault() to
clean up and return an error when it would otherwise map a zero-filled
page. Then, in the core dumper code, prefault all user pages and handle
errors by simply extending the size of the core file. This also fixes a
bug related to the fact that vn_io_fault1() does not attempt partial I/O
in the face of errors from vm_fault_quick_hold_pages(): if a truncated
file is mapped into a user process, an attempt to dump beyond the end of
the file results in an error, but this means that valid pages
immediately preceding the end of the file might not have been dumped
either.

The change reduces the core dump size of trivial programs by a factor of
ten simply by excluding unaccessed libc.so pages.

PR: 249067
Reviewed by: kib
Tested by: pho
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26590

show more ...

Add a vmparam.h constant indicating pmap support for large pages.

Enable SHM_LARGEPAGE support on arm64.

Reviewed by: alc, kib
Sponsored by: Juniper Networks, Inc., Klara, Inc.
Differential Revisio

Add a vmparam.h constant indicating pmap support for large pages.

Enable SHM_LARGEPAGE support on arm64.

Reviewed by: alc, kib
Sponsored by: Juniper Networks, Inc., Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D26467

show more ...

Support for userspace non-transparent superpages (largepages).

Created with shm_open2(SHM_LARGEPAGE) and then configured with
FIOSSHMLPGCNF ioctl, largepages posix shared memory objects guarantee
th

Support for userspace non-transparent superpages (largepages).

Created with shm_open2(SHM_LARGEPAGE) and then configured with
FIOSSHMLPGCNF ioctl, largepages posix shared memory objects guarantee
that all userspace mappings of it are served by superpage non-managed
mappings.

Only amd64 for now, both 2M and 1G superpages can be requested, the
later requires CPU feature.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D24652

show more ...

vm: clean up empty lines in .c and .h files

MFH

Sponsored by: Rubicon Communications, LLC (netgate.com)

Merge ^/head r364264 through r364278.

vfs: remove the thread argument from vget

It was already asserted to be curthread.

Semantic patch:

@@

expression arg1, arg2, arg3;

@@

- vget(arg1, arg2, arg3)
+ vget(arg1, arg2)

Add a helper function for validating VA ranges.

Functions which take untrusted user ranges must validate against the
bounds of the map, and also check for wraparound. Instead of having the
same log

Add a helper function for validating VA ranges.

Functions which take untrusted user ranges must validate against the
bounds of the map, and also check for wraparound. Instead of having the
same logic duplicated in a number of places, add a function to check.

Reviewed by: dougm, kib
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D25328

show more ...

Simplify the condition to enable superpage mappings in vm_fault_soft_fast().

The list of arches list there matches the list of arches where
default VM_NRESERVLEVEL > 0. Before sparc64 removal, that

Simplify the condition to enable superpage mappings in vm_fault_soft_fast().

The list of arches list there matches the list of arches where
default VM_NRESERVLEVEL > 0. Before sparc64 removal, that was the
only arch that defined VM_NRESERVLEVEL > 0 to help with cache coloring,
but did not implemented superpages. Now it can be simplified.

Submitted by: alc
Reviewed by: markj

show more ...

Properly sort ifdef archs in vm_fault_soft_fast superpage guards.

Sort broken in r360887.

powerpc64: Implement Radix MMU for POWER9 CPUs

Summary:
POWER9 supports two MMU formats: traditional hashed page tables, and Radix
page tables, similar to what's presesnt on most other architectures

powerpc64: Implement Radix MMU for POWER9 CPUs

Summary:
POWER9 supports two MMU formats: traditional hashed page tables, and Radix
page tables, similar to what's presesnt on most other architectures. The
PowerISA also specifies a process table -- a table of page table pointers--
which on the POWER9 is only available with the Radix MMU, so we can take
advantage of it with the Radix MMU driver.

Written by Matt Macy.

Differential Revision: https://reviews.freebsd.org/D19516

show more ...

Merge ^/head r358400 through r358465.

Add a blocking counter KPI.

refcount(9) was recently extended to support waiting on a refcount to
drop to zero, as this was needed for a lockless VM object
paging-in-progress counter. However, this

Add a blocking counter KPI.

refcount(9) was recently extended to support waiting on a refcount to
drop to zero, as this was needed for a lockless VM object
paging-in-progress counter. However, this adds overhead to all uses of
refcount(9) and doesn't really match traditional refcounting semantics:
once a counter has dropped to zero, the protected object may be freed at
any point and it is not safe to dereference the counter.

This change removes that extension and instead adds a new set of KPIs,
blockcount_*, for use by VM object PIP and busy.

Reviewed by: jeff, kib, mjg
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23723

show more ...

Merge ^/head r357179 through r357269.

Restore OOM logic on page fault after r357026.

Right now OOM is initiated unconditionally on the page allocation
failure, after the wait.

Reported by: Mark Millard <marklmi@yahoo.com>
Reviewed by:

Restore OOM logic on page fault after r357026.

Right now OOM is initiated unconditionally on the page allocation
failure, after the wait.

Reported by: Mark Millard <marklmi@yahoo.com>
Reviewed by: cy, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D23409

show more ...

Merge ^/head r356931 through r357118.

(fault 9/9) Move zero fill into a dedicated function to make the object lock
state more clear.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D23326

(fault 8/9) Restructure some code to reduce duplication and simplify flow
control.

Reviewed by: dougm, kib, markj
Differential Revision: https://reviews.freebsd.org/D23321

(fault 7/9) Move fault population and allocation into a dedicated function

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D23320

(fault 6/9) Move getpages and associated logic into a dedicated function.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D23311

(fault 5/9) Move the backing_object traversal into a dedicated function.

Reviewed by: dougm, kib, markj
Differential Revision: https://reviews.freebsd.org/D23310

(fault 4/9) Move copy-on-write into a dedicated function.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D23304