arm64: Support mapping a 52-bit physical adddress

When FEAT_LPA2 is enabled the physical address space increases from
48-bits to 52-bits. The top two address bits are moved to the now
unused shareab

arm64: Support mapping a 52-bit physical adddress

When FEAT_LPA2 is enabled the physical address space increases from
48-bits to 52-bits. The top two address bits are moved to the now
unused shareability field.

Update the kernel to support this new larger address space.

Reviewed by: alc, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46624

show more ...

arm64: Remove ATTR_DEFAULT from pte.h

ATTR_SH(ATTR_SH_IS) will soon be dynamic as the field is moved out of
the page tables in FEAT_LPA2. When this happens ATTR_DEFAULT will just
be ATTR_AF.

Rather

arm64: Remove ATTR_DEFAULT from pte.h

ATTR_SH(ATTR_SH_IS) will soon be dynamic as the field is moved out of
the page tables in FEAT_LPA2. When this happens ATTR_DEFAULT will just
be ATTR_AF.

Rather than keeping ATTR_DEFAULT with one attribute remove it.

Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46466

show more ...

arm64: set ATTR_CONTIGUOUS on the DMAP's L2 blocks

On systems configured with 16KB pages, this change creates 1GB page
mappings in the direct map where possible. Previously, the largest page
size t

arm64: set ATTR_CONTIGUOUS on the DMAP's L2 blocks

On systems configured with 16KB pages, this change creates 1GB page
mappings in the direct map where possible. Previously, the largest page
size that was used to implement the direct map was 32MB. Similarly, on
systems configured with 4KB pages, this change creates 32MB page
mappings, instead of 2MB, in the direct map where 1GB is too large.

Implement demotion on L2C (32MB/1GB) page mappings within the DMAP.

Update sysctl vm.pmap.kernel_maps to report on L2C page mappings.

Reviewed by: markj
Tested by: gallatin, Eliot Solomon <ehs3@rice.edu>
Differential Revision: https://reviews.freebsd.org/D45224

show more ...

arm64 pmap: Add ATTR_CONTIGUOUS support [Part 1]

The ATTR_CONTIGUOUS bit within an L3 page table entry designates that
L3 page as being part of an aligned, physically contiguous collection
of L3 pag

arm64 pmap: Add ATTR_CONTIGUOUS support [Part 1]

The ATTR_CONTIGUOUS bit within an L3 page table entry designates that
L3 page as being part of an aligned, physically contiguous collection
of L3 pages. For example, 16 aligned, physically contiguous 4 KB pages
can form a 64 KB superpage, occupying a single TLB entry. While this
change only creates ATTR_CONTIGUOUS mappings in a few places,
specifically, the direct map and pmap_kenter{,_device}(), it adds all
of the necessary code for handling them once they exist, including
demotion, protection, and removal. Consequently, new ATTR_CONTIGUOUS
usage can be added (and tested) incrementally.

Modify the implementation of sysctl vm.pmap.kernel_maps so that it
correctly reports the number of ATTR_CONTIGUOUS mappings on machines
configured to use a 16 KB base page size, where an ATTR_CONTIGUOUS
mapping consists of 128 base pages.

Additionally, this change adds support for creating L2 superpage
mappings to pmap_kenter{,_device}().

Reviewed by: markj
Tested by: gallatin
Differential Revision: https://reviews.freebsd.org/D42737

show more ...

arm64: lop off another 24MB of KVA for early device mappings

This grows the block enough to fit a 4K 32-bit depth framebuffer; some
firmware would present smaller GOP modes to be able to boot with a

arm64: lop off another 24MB of KVA for early device mappings

This grows the block enough to fit a 4K 32-bit depth framebuffer; some
firmware would present smaller GOP modes to be able to boot with a
smaller framebuffer on these devices, but the Windows Devkit firmware
is simply not that nice. Instead, it offers exactly one GOP mode that
matches the current resolution of the attached display, so with limited
control over resolution on most of my displays it'd be nice if we could
Just Work(TM) at 4K.

andrew notes that he has some ideas for removing PMAP_MAPDEV_EARLY_SIZE
entirely, so this limitation could end up removed altogether in the
future.

Reviewed by: andrew, emaste
Differential Revision: https://reviews.freebsd.org/D42726

show more ...

arm64: Set the Guarded Page flag in the kernel

Now the kernel and modules are built with branch protection we can
enablethe Guarded Page flag in the page tables. This causes indirect
branches to a l

arm64: Set the Guarded Page flag in the kernel

Now the kernel and modules are built with branch protection we can
enablethe Guarded Page flag in the page tables. This causes indirect
branches to a location without a correct landing pad instruction to
raise an exception.

This should help mitigate some attacks where a function pointer is
changed to point somewhere other than the start of the function,
however it doesn't stop an attacker pointing it to an unintended
function.

Reviewed by: alc, scottph (both earlier version), markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42080

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 wh

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.

The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h

Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944

show more ...

arm64 pmap: introduce PHYS_TO_PTE macro

Introduce macro for PHYS_TO_PTE, setting the groundwork for future
support of various Arm VMSA extensions.

For extensions such as 52-bit VA/PA (FEAT_LPA2), t

arm64 pmap: introduce PHYS_TO_PTE macro

Introduce macro for PHYS_TO_PTE, setting the groundwork for future
support of various Arm VMSA extensions.

For extensions such as 52-bit VA/PA (FEAT_LPA2), the representation of
an address between a PTE and PA are not equivalent. This macro will
allow converting between the different representations.

Currently PHYS_TO_PTE is a NOP. Replace all instances where we go from
PA to PTE with new PHYS_TO_PTE macro.

Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39828

show more ...

arm64 pmap: introduce PTE_TO_PHYS macro

Introduce macro for PTE_TO_PHYS, setting the groundwork for future
support of various Arm VMSA extensions.

For extensions such as 52-bit VA/PA (FEAT_LPA2), t

arm64 pmap: introduce PTE_TO_PHYS macro

Introduce macro for PTE_TO_PHYS, setting the groundwork for future
support of various Arm VMSA extensions.

For extensions such as 52-bit VA/PA (FEAT_LPA2), the representation of
an address between a PTE and PA are not equivalent. This macro will
allow converting between the different representations.

Currently going from PTE to PA is achieved by masking off the upper and
lower attributes. Retain this behaviour but replace all instances with
the new macro instead.

Reviewed by: alc, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39827

show more ...

Disable promotion on pcpu memory on arm64

We need to be careful to not promote or demote the memory containing
the per-CPU structures as the exception handlers will dereference it
so any time it's i

Disable promotion on pcpu memory on arm64

We need to be careful to not promote or demote the memory containing
the per-CPU structures as the exception handlers will dereference it
so any time it's invalid may cause recursive exceptions.

Add a new pmap function to set a flag in the pte marking memory that
cannot be promoted or demoted and use it to mark pcpu memory.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35434

show more ...

Add experimental 16k page support on arm64

Add initial 16k page support on arm64. It is considered experimental,
with no guarantee of compatibility with a userspace or kernel modules
built with the

Add experimental 16k page support on arm64

Add initial 16k page support on arm64. It is considered experimental,
with no guarantee of compatibility with a userspace or kernel modules
built with the current a 4k page size as code will likely try to pass
in a too small size when working with APIs that take a multiple of a
page, e.g. mmap.

As this is experimental, and because userspace and the kernel need to
have the PAGE_SIZE macro kept in sync there is no kernel option to
enable this. To test a new image should be built with the
PAGE_{SIZE,SHIFT,MASK} macros changed to the 16k versions.

There are currently known issues with loading modules from an old
loader as it can misalign them to load on a non-16k boundary.

Testing has shown good results in kernel workloads that allocate and
free large amounts of memory as only a quarter of the number of calls
into the VM subsystem are needed in the best case.

Reviewed by: markj
Tested by: gallatin
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34793

show more ...

Remove L2_BLOCK_MASK from arm64

It's unneeded as it was just used to align KERNBASE to a level 2
block start address. KERNBASE was already aligned correctly.

Sponsored by: The FreeBSD Foundation

Add the arm64 table attributes and use them

Add the table page table attributes on arm64 and use them to add
restrictions to the block and page entries below them. This ensures
we are unable to incr

Add the arm64 table attributes and use them

Add the table page table attributes on arm64 and use them to add
restrictions to the block and page entries below them. This ensures
we are unable to increase the permissions in these last level entries
without also changing them in the upper levels.

Use the attributes to ensure the kernel can't execute from userspace
memory and vice versa, userspace has no access to read or write kernel
memory, and that the DMAP region is non-executable.

Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32081

show more ...

Fix the arm64 L2_BLOCK_MASK definition

It was missing the top 16 bits.

Sponsored by: The FreeBSD Foundation

arm64: eliminate a duplicated #define

Support creating and using arm64 pmap at stage 2

Add minimal support for creating stage 2 IPA -> PA mappings. For this we
need to:

- Create a new vmid set to allocate a vmid for each Virtual Machi

Support creating and using arm64 pmap at stage 2

Add minimal support for creating stage 2 IPA -> PA mappings. For this we
need to:

- Create a new vmid set to allocate a vmid for each Virtual Machine
- Add the missing stage 2 attributes
- Use these in pmap_enter to create a new mapping
- Handle stage 2 faults

The vmid set is based on the current asid set that was generalised in
r358328. It adds a function pointer for bhyve to use when the kernel needs
to reset the vmid set. This will need to call into EL2 and invalidate the
TLB.

The stage 2 attributes have been added. To simplify setting these fields
two new functions are added to get the memory type and protection fields.
These are slightly different on stage 1 and stage 2 tables. We then use
them in pmap_enter to set the new level 3 entry to be stored.

The D-cache on all entries is cleaned to the point of coherency. This is
to allow the data to be visible to the VM. To allow for userspace to load
code when creating a new executable entry an invalid entry is created. When
the VM tried to use it the I-cache is invalidated. As the D-cache has
already been cleaned this will ensure the I-cache is synchronised with the
D-cache.

When the hardware implements a VPIPT I-cache we need to either have the
correct VMID set or invalidate it from EL2. As the host kernel will have
the wrong VMID set we need to call into EL2 to clean it. For this a second
function pointer is added that is called when this invalidation is needed.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23875

show more ...

Merge ^/head r358269 through r358399.

Split out the stage 1 pte bits and add the stage 2 bits

In preperation for adding bhyve support to arm64 we need to split the
stage 1 and stage 2 pte fields to allow future changes to create stage 2

Split out the stage 1 pte bits and add the stage 2 bits

In preperation for adding bhyve support to arm64 we need to split the
stage 1 and stage 2 pte fields to allow future changes to create stage 2
page tables.

MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23669

show more ...

Ordinarily, during a superpage promotion or demotion within a pmap, the
pmap's lock ensures that other operations on the pmap don't observe the
old mapping being broken before the new mapping is esta

Ordinarily, during a superpage promotion or demotion within a pmap, the
pmap's lock ensures that other operations on the pmap don't observe the
old mapping being broken before the new mapping is established. However,
pmap_kextract() doesn't acquire the kernel pmap's lock, so it may observe
the broken mapping. And, if it does, it returns an incorrect result.

This revision implements a lock-free solution to this problem in
pmap_update_entry() and pmap_kextract() because pmap_kextract() can't
acquire the kernel pmap's lock.

Reported by: andrew, greg_unrelenting.technology
Reviewed by: andrew, markj
Tested by: greg_unrelenting.technology
X-MFC with: r350579
Differential Revision: https://reviews.freebsd.org/D21169

show more ...

Use ATTR_DBM even when hardware dirty bit management is not enabled.

The ARMv8 reference manual only states that the bit is reserved in
this case; following Linux's example, use it instead of a
soft

Use ATTR_DBM even when hardware dirty bit management is not enabled.

The ARMv8 reference manual only states that the bit is reserved in
this case; following Linux's example, use it instead of a
software-defined bit for the purpose of indicating that a managed
mapping is writable.

Reviewed by: alc, andrew
MFC after: r350004
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D21121

show more ...

MFHead @r350386

Sponsored by: The FreeBSD Foundation

Fix the arm64 page table entry attribute mask.

It did not include the DBM or contiguous bits.

Reported by: andrew
Reviewed by: andrew
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Implement software access and dirty bit management for arm64.

Previously the arm64 pmap did no reference or modification tracking;
all mappings were treated as referenced and all read-write mappings

Implement software access and dirty bit management for arm64.

Previously the arm64 pmap did no reference or modification tracking;
all mappings were treated as referenced and all read-write mappings
were treated as dirty. This change implements software management
of these attributes.

Dirty bit management is implemented to emulate ARMv8.1's optional
hardware dirty bit modifier management, following a suggestion from alc.
In particular, a mapping with ATTR_SW_DBM set is logically writeable and
is dirty if the ATTR_AP_RW_BIT bit is clear. Mappings with
ATTR_AP_RW_BIT set are write-protected, and a write access will trigger
a permission fault. pmap_fault() handles permission faults for such
mappings and marks the page dirty by clearing ATTR_AP_RW_BIT, thus
mapping the page read-write.

Reviewed by: alc
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D20907

show more ...

Merge ^/head r339015 through r339669.