vm_map: convert several bool members into flags

Extend flags to u_int.
Move system_map and needs_wakeup bools into flags.

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision

vm_map: convert several bool members into flags

Extend flags to u_int.
Move system_map and needs_wakeup bools into flags.

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47934

show more ...

vm_map: wrap map->system_map checks into wrapper

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47934

vm_map: overlap system map mutex and user man sx

This saves 616-584 = 32 bytes per struct vmspace on amd64, which allows
to pack 7 vmspaces per page vs. 6 for non-overlapping layout.

I used anonymo

vm_map: overlap system map mutex and user man sx

This saves 616-584 = 32 bytes per struct vmspace on amd64, which allows
to pack 7 vmspaces per page vs. 6 for non-overlapping layout.

I used anonymous union member feature to avoid too much churn.

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D47934

show more ...

vm: rename MAP_STACK_GROWS_DOWN to MAP_STACK_AREA

Reviewed by: alc, dougm, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47892

vm_map: remove _GN suffix from MAP_ENTRY_STACK_GAP and MAP_CREATE_STACK_GAP symbols`

Reviewed by: alc, dougm, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebs

vm_map: remove _GN suffix from MAP_ENTRY_STACK_GAP and MAP_CREATE_STACK_GAP symbols`

Reviewed by: alc, dougm, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47892

show more ...

sys/vm: remove support for growing-up stacks

Reviewed by: alc, dougm, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47892

vm_map: add vm_map_find_locked(9)

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D46678

vm ASLR: Handle VM_NRESERV_LEVEL == 0

Some flavors of powerpc don't enable superpage reservations.

Fixes: 3e00c11a4f43 ("arm64: Support the L3 ATTR_CONTIGUOUS page ...")

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

vm: Reduce address space fragmentation

jemalloc performs two types of virtual memory allocations: (1) large
chunks of virtual memory, where the chunk size is a multiple of a
superpage and explicitly

vm: Reduce address space fragmentation

jemalloc performs two types of virtual memory allocations: (1) large
chunks of virtual memory, where the chunk size is a multiple of a
superpage and explicitly aligned, and (2) small allocations, mostly
128KB, where no alignment is requested. Typically, it starts with a
small allocation, and over time it makes both types of allocation.

With anon_loc being updated on every allocation, we wind up with a
repeating pattern of a small allocation, a large gap, and a large,
aligned allocation. (As an aside, we wind up allocating a reservation
for these small allocations, but it will never fill because the next
large, aligned allocation updates anon_loc, leaving a gap that will
never be filled with other small allocations.)

With this change, anon_loc isn't updated on every allocation. So, the
small allocations will be clustered together, the large allocations will
be clustered together, and there will be fewer gaps between the
anonymous memory allocations. In addition, I see a small reduction in
reservations allocated (e.g., 1.6% during buildworld), fewer partially
populated reservations, and a small increase in 64KB page promotions on
arm64.

Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39845

show more ...

uma: Deduplicate uma_small_alloc

This commit refactors the UMA small alloc code and
removes most UMA machine-dependent code.
The existing machine-dependent uma_small_alloc code is almost identical
a

uma: Deduplicate uma_small_alloc

This commit refactors the UMA small alloc code and
removes most UMA machine-dependent code.
The existing machine-dependent uma_small_alloc code is almost identical
across all architectures, except for powerpc where using the direct
map addresses involved extra steps in some cases.

The MI/MD split was replaced by a default uma_small_alloc
implementation that can be overridden by architecture-specific code by
defining the UMA_MD_SMALL_ALLOC symbol. Furthermore, UMA_USE_DMAP was
introduced to replace most UMA_MD_SMALL_ALLOC uses.

Reviewed by: markj, kib
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D45084

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

vm_map.c: fix syntax

Fixes: c718009884b3d65528deaff24712cbf98e3be656
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

vm_map.c: plug several more places which might modify entry->offset

for the GUARD entries protecting stacks gaps.

syzkaller: https://syzkaller.appspot.com/bug?extid=c325d6a75e4fd0a68714
Reviewed by

vm_map.c: plug several more places which might modify entry->offset

for the GUARD entries protecting stacks gaps.

syzkaller: https://syzkaller.appspot.com/bug?extid=c325d6a75e4fd0a68714
Reviewed by: dougm, markj (previous version)
Tested by: pho (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41475

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

vm: Fix address hints of 0 with MAP_32BIT

Also, rename min_addr to default_addr, which better reflects what it
represents. The min_addr is not a minimum address in the same way that
max_addr is act

vm: Fix address hints of 0 with MAP_32BIT

Also, rename min_addr to default_addr, which better reflects what it
represents. The min_addr is not a minimum address in the same way that
max_addr is actually a maximum address that can be allocated. For
example, a non-zero hint can be less than min_addr and be allocated.

Reported by: dchagin
Reviewed by: dchagin, kib, markj
Fixes: d8e6f4946cec0 "vm: Fix anonymous memory clustering under ASLR"
Differential Revision: https://reviews.freebsd.org/D41397

show more ...

vm_map_protect(): add VM_MAP_PROTECT_GROWSDOWN flag

which requests to propagate lowest stack segment protection to the grow gap.
This seems to be required for Linux emulation.

Reported by: dchagin

vm_map_protect(): add VM_MAP_PROTECT_GROWSDOWN flag

which requests to propagate lowest stack segment protection to the grow gap.
This seems to be required for Linux emulation.

Reported by: dchagin
Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...

vm_map_growstack(): restore stack gap data if gap entry was removed

and then restored.

Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential rev

vm_map_growstack(): restore stack gap data if gap entry was removed

and then restored.

Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...

vm_map: do not allow to merge stack gap entries

At least, offset handling is wrong for them.

Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differenti

vm_map: do not allow to merge stack gap entries

At least, offset handling is wrong for them.

Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...

vm_map_protect(): handle stack protection stored in the stack guard

mprotect(2) on the stack region needs to adjust guard stored protection,
so that e.g. enable executing on stack worked properly on

vm_map_protect(): handle stack protection stored in the stack guard

mprotect(2) on the stack region needs to adjust guard stored protection,
so that e.g. enable executing on stack worked properly on stack growth.

Reported by: dchagin
Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...

vm_map_protect(): move guard handling at the last phase into an empty dedicated helper

Restructure the first phase slightly, to facilitate further changes.

Reviewed by: alc, markj
Tested by: pho
Sp

vm_map_protect(): move guard handling at the last phase into an empty dedicated helper

Restructure the first phase slightly, to facilitate further changes.

Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...

vm_map_growstack(): handle max protection for stacks

Do not assume that protection is same as max_protection. Store both in
offset, packed in the same way as the prot syscall parameter.

Reviewed b

vm_map_growstack(): handle max protection for stacks

Do not assume that protection is same as max_protection. Store both in
offset, packed in the same way as the prot syscall parameter.

Reviewed by: alc, markj (previous version)
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...

vm_map.c: add CONTAINS_BITS macro

Suggested by: dougm
Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org

vm_map.c: add CONTAINS_BITS macro

Suggested by: dougm
Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...

Add vm_map_insert1(9)

The function returns the newly created entry.
Use vm_map_insert1() in stack grow code to avoid gap entry re-lookup.

The comment update for vm_map_try_merge_entries() was sugge

Add vm_map_insert1(9)

The function returns the newly created entry.
Use vm_map_insert1() in stack grow code to avoid gap entry re-lookup.

The comment update for vm_map_try_merge_entries() was suggested by dougm.

Suggested by: alc
Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...