arm64: Move DBM errata to the cpu feat framework

Move to the new cpu feature framework for enabling DBM and disabling
when there are errata affecting it.

Sponsored by: Arm Ltd
Differential Revision

arm64: Move DBM errata to the cpu feat framework

Move to the new cpu feature framework for enabling DBM and disabling
when there are errata affecting it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47815

show more ...

arm64: Support mapping a 52-bit physical adddress

When FEAT_LPA2 is enabled the physical address space increases from
48-bits to 52-bits. The top two address bits are moved to the now
unused shareab

arm64: Support mapping a 52-bit physical adddress

When FEAT_LPA2 is enabled the physical address space increases from
48-bits to 52-bits. The top two address bits are moved to the now
unused shareability field.

Update the kernel to support this new larger address space.

Reviewed by: alc, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46624

show more ...

arm64: Move setting TCR_HD to C code

To allow for it to be more selective when we enable it, e.g. if the
CPU has an erratum that prevents us from doing so, move the check for
setting TCR_HD to C.

R

arm64: Move setting TCR_HD to C code

To allow for it to be more selective when we enable it, e.g. if the
CPU has an erratum that prevents us from doing so, move the check for
setting TCR_HD to C.

Reviewed by: alc
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47808

show more ...

arm64: Remove VPIPT I-cache support from pmap

The VPIPT I-cache policy has been removed from the architecture.
Remove support for it from pmap. The vmm side was not imported so
calls to pmap_invalid

arm64: Remove VPIPT I-cache support from pmap

The VPIPT I-cache policy has been removed from the architecture.
Remove support for it from pmap. The vmm side was not imported so
calls to pmap_invalidate_vpipt_icache would cause the kernel to branch
to a NULL pointer.

Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47116

show more ...

arm64: Make shareability attributes dynamic

When LPA2 is enabled the shareability attribute in the page table are
replaces with output address bits. To support a larger physical address
space make t

arm64: Make shareability attributes dynamic

When LPA2 is enabled the shareability attribute in the page table are
replaces with output address bits. To support a larger physical address
space make this attribute dynamic so we only set it when appropriate.

Reviewed by: alc, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46394

show more ...

arm64, riscv: removed unused struct pv_addr

No functional change.

Reviewed by: markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45322

arm64: Add BTI support to pmap

Add a rangeset to the arm64 pmap to describe which address space needs
the Branch Target Identification (BTI) Guard Page flag set in the page
table.

On hardware that

arm64: Add BTI support to pmap

Add a rangeset to the arm64 pmap to describe which address space needs
the Branch Target Identification (BTI) Guard Page flag set in the page
table.

On hardware that supports BTI the Guard Page flag tells the hardware
to raise an exception if the target of a BR* and BLR* instruction is
not an appropriate landing pad instruction.

To support this in userspace we need to know which address space
should be guarded. For this add a rangeset to the arm64 pmap when the
hardware supports BTI. The kernel can then use pmap_bti_set and
pmap_bti_clear mark and unmark which address space is guarded.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42328

show more ...

arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directly

pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreove

arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directly

pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreover, the name is a
bit confusing given the existence of pmap_bootstrap_san(). Just remove
it and call kasan_init_early() directly like we do on amd64. It will
not be used by KMSAN in a forthcoming patch series.

No functional change intended.

MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43403

show more ...

arm64: Make kern_delta unneeded in the boot params

Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still

arm64: Make kern_delta unneeded in the boot params

Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still calculate it, we don't need to pass it to initarm or either
pmap bootstrap functions.

While here remove an unneeded printf that indirectly used it or was
related to the previous printf.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42567

show more ...

arm64: Use pmap_early_vtophys in pmap_bootstrap_san

Use pmap_early_vtophys to find the physical address of the kernel base
rather than using the calculated offset as it will be removed in a
latter c

arm64: Use pmap_early_vtophys in pmap_bootstrap_san

Use pmap_early_vtophys to find the physical address of the kernel base
rather than using the calculated offset as it will be removed in a
latter commit.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42566

show more ...

add pmap_active_cpus()

For amd64, i386, arm, and riscv, i.e. all architectures except arm64,
the custom implementation is provided since we maintain the bitmask of
active CPUs anyway.

Arm64 uses so

add pmap_active_cpus()

For amd64, i386, arm, and riscv, i.e. all architectures except arm64,
the custom implementation is provided since we maintain the bitmask of
active CPUs anyway.

Arm64 uses somewhat naive iteration over CPUs and match current vmspace'
pmap with the argument. It is not guaranteed that vmspace->pmap is the
same as the active pmap, but the inaccuracy should be toleratable.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32360

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 wh

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.

The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h

Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944

show more ...

pmap_{un}map_io_transient: Use bool instead of boolean_t.

Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D39920

Split out pmap_map_delete on arm64

This will be used when supporting some extensions, e.g. Branch Target
Identification (BTI).

Sponsored by: Arm Ltd

Add padding for future use on arm64

Allow new features to be supported without changing the size of
existing structures.

Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://review

Add padding for future use on arm64

Allow new features to be supported without changing the size of
existing structures.

Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39777

show more ...

amd64: fix PKRU and swapout interaction

When vm_map_remove() is called from vm_swapout_map_deactivate_pages()
due to swapout, PKRU attributes for the removed range must be kept
intact. Provide a va

amd64: fix PKRU and swapout interaction

When vm_map_remove() is called from vm_swapout_map_deactivate_pages()
due to swapout, PKRU attributes for the removed range must be kept
intact. Provide a variant of pmap_remove(), pmap_map_delete(), to
allow pmap to distinguish between real removes of the UVA mappings
and any other internal removes, e.g. swapout.

For non-amd64, pmap_map_delete() is stubbed by define to pmap_remove().

Reported by: andrew
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D39556

show more ...

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow m

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

show more ...

Support arm64 stage2 TLB invalidation

To invalidate stage 2 mappings on arm64 we may need to call into the
hypervisor so add a function pointer that bhyve can use to implement
this.

Sponsored by: T

Support arm64 stage2 TLB invalidation

To invalidate stage 2 mappings on arm64 we may need to call into the
hypervisor so add a function pointer that bhyve can use to implement
this.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37254

show more ...

sys: Consolidate common implementation details of PV entries.

Add a <sys/_pv_entry.h> intended for use in <machine/pmap.h> to
define struct pv_entry, pv_chunk, and related macros and inline
function

sys: Consolidate common implementation details of PV entries.

Add a <sys/_pv_entry.h> intended for use in <machine/pmap.h> to
define struct pv_entry, pv_chunk, and related macros and inline
functions.

Note that powerpc does not yet use this as while the mmu_radix pmap
in powerpc uses the new scheme (albeit with fewer PV entries in a
chunk than normal due to an used pv_pmap field in struct pv_entry),
the Book-E pmaps for powerpc use the older style PV entries without
chunks (and thus require the pv_pmap field).

Suggested by: kib
Reviewed by: kib
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36685

show more ...

Remove unneeded variables in the arm64 pmap bootstrap

These are now unneeded after cleaning up the pmap bootstrap process.
Remove them and the variables that set them.

Sponsored by: The FreeBSD Fou

Remove unneeded variables in the arm64 pmap bootstrap

These are now unneeded after cleaning up the pmap bootstrap process.
Remove them and the variables that set them.

Sponsored by: The FreeBSD Foundation

show more ...

pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.

This matches the return type of pmap_mapdev/bios.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.f

pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.

This matches the return type of pmap_mapdev/bios.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36548

show more ...

arm64 pmap: Simplify logic around pv_chunk sizes.

- Define PC_FREEL and _NPCM in terms of _NPCPV rather than via magic
numbers.

- Remove assertions about _NPC* values from pmap.c. This is less

arm64 pmap: Simplify logic around pv_chunk sizes.

- Define PC_FREEL and _NPCM in terms of _NPCPV rather than via magic
numbers.

- Remove assertions about _NPC* values from pmap.c. This is less
relevant now that PC_FREEL and _NPCM are derived from _NPCPV.

- Add a helper inline function pc_is_full() which uses a loop to check
if pc_map is all zeroes. Use this to replace three places that
check for a full mask assuming there are only 3 entries in pc_map.

Reviewed by: markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36217

show more ...

Disable promotion on pcpu memory on arm64

We need to be careful to not promote or demote the memory containing
the per-CPU structures as the exception handlers will dereference it
so any time it's i

Disable promotion on pcpu memory on arm64

We need to be careful to not promote or demote the memory containing
the per-CPU structures as the exception handlers will dereference it
so any time it's invalid may cause recursive exceptions.

Add a new pmap function to set a flag in the pte marking memory that
cannot be promoted or demoted and use it to mark pcpu memory.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35434

show more ...

pmap_mapdev: Consistently use vm_paddr_t for the first argument.

The devmap variants used vm_offset_t for some reason, and a few places
explicitly cast bus addresses to vm_offset_t. (Probably those

pmap_mapdev: Consistently use vm_paddr_t for the first argument.

The devmap variants used vm_offset_t for some reason, and a few places
explicitly cast bus addresses to vm_offset_t. (Probably those casts
along with similar casts for vm_size_t should just be removed and
instead permit the compiler to DTRT.)

Reviewed by: markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D35961

show more ...