dtrace/amd64: Make invop providers usable with KMSAN enabled

- Use a fresh context when entering dtrace_invop() via a breakpoint
exception.
- Mark the #BP trapframe as initialized.

MFC after: 2 w

dtrace/amd64: Make invop providers usable with KMSAN enabled

- Use a fresh context when entering dtrace_invop() via a breakpoint
exception.
- Mark the #BP trapframe as initialized.

MFC after: 2 weeks

show more ...

dtrace: remove x86 non-EARLY_AP_STARTUP support

After 792655abd64c EARLY_AP_STARTUP is mandatory for x86.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://revi

dtrace: remove x86 non-EARLY_AP_STARTUP support

After 792655abd64c EARLY_AP_STARTUP is mandatory for x86.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42139

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

dtrace: get rid of uchar_t types

Callers are specifying uint8_t anyway and this slightly reduces
dependencies on compatibility typedefs. No functional change intended.

Reviewed by: markj, mhorne
M

dtrace: get rid of uchar_t types

Callers are specifying uint8_t anyway and this slightly reduces
dependencies on compatibility typedefs. No functional change intended.

Reviewed by: markj, mhorne
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39490

show more ...

dtrace: Add a "regs" variable

This allows invop-based providers (i.e., fbt and kinst) to expose the
register file of the CPU at the point where the probe fired. It does
not work for SDT providers b

dtrace: Add a "regs" variable

This allows invop-based providers (i.e., fbt and kinst) to expose the
register file of the CPU at the point where the probe fired. It does
not work for SDT providers because their probes are implemented as plain
function calls and so don't save registers. It's not clear what
semantics "regs" should have for them anyway.

This is akin to "uregs", which nominally provides access to the
userspace registers. In fact, DIF already had a DIF_VAR_REGS variable
defined, it was simply unimplemented.

Usage example: print the contents of %rdi upon each call to
amd64_syscall():

fbt::amd64_syscall:entry {printf("%x", regs[R_RDI]);}

Note that the R_* constants are defined in /usr/lib/dtrace/regs_x86.d.
Currently there are no similar definitions for non-x86 platforms.

Reviewed by: christos
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D36799

show more ...

dtrace/amd64: Implement emulation of call instructions

Here, the provider is responsible for updating the trapframe to redirect
control flow and for computing the return address. Once software-save

dtrace/amd64: Implement emulation of call instructions

Here, the provider is responsible for updating the trapframe to redirect
control flow and for computing the return address. Once software-saved
registers are restored, the emulation shifts the remaining context down
on the stack to make space for the return address, then copies the
address provided by the invop handler. dtrace_invop() is modified to
allocate temporary storage space on the stack for use by the provider to
return the return address.

This is to support a new provider for amd64 which can instrument
arbitrary instructions, not just function entry and exit instructions as
FBT does.

In collaboration with: christos
Sponsored by: Google, Inc. (GSoC 2022)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

show more ...

MFH

Sponsored by: Rubicon Communications, LLC (netgate.com)

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would a

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872

show more ...

amd64 pmap: LA57 AKA 5-level paging

Since LA57 was moved to the main SDM document with revision 072, it
seems that we should have a support for it, and silicons are coming.

This patch makes pmap su

amd64 pmap: LA57 AKA 5-level paging

Since LA57 was moved to the main SDM document with revision 072, it
seems that we should have a support for it, and silicons are coming.

This patch makes pmap support both LA48 and LA57 hardware. The
selection of page table level is done at startup, kernel always
receives control from loader with 4-level paging. It is not clear how
UEFI spec would adapt LA57, for instance it could hand out control in
LA57 mode sometimes.

To switch from LA48 to LA57 requires turning off long mode, requesting
LA57 in CR4, then re-entering long mode. This is somewhat delicate
and done in pmap_bootstrap_la57(). AP startup in LA57 mode is much
easier, we only need to toggle a bit in CR4 and load right value in CR3.

I decided to not change kernel map for now. Single PML5 entry is
created that points to the existing kernel_pml4 (KML4Phys) page, and a
pml5 entry to create our recursive mapping for vtopte()/vtopde().
This decision is motivated by the fact that we cannot overcommit for
KVA, so large space there is unusable until machines start providing
wider physical memory addressing. Another reason is that I do not
want to break our fragile autotuning, so the KVA expansion is not
included into this first step. Nice side effect is that minidumps are
compatible.

On the other hand, (very) large address space is definitely
immediately useful for some userspace applications.

For userspace, numbering of pte entries (or page table pages) is
always done for 5-level structures even if we operate in 4-level mode.
The pmap_is_la57() function is added to report the mode of the
specified pmap, this is done not to allow simultaneous 4-/5-levels
(which is not allowed by hw), but to accomodate for EPT which has
separate level control and in principle might not allow 5-leve EPT
despite x86 paging supports it. Anyway, it does not seems critical to
have 5-level EPT support now.

Tested by: pho (LA48 hardware)
Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D25273

show more ...

Merge ^/head r352587 through r352763.

Implement x86 dtrace_invop_(un)init() in C.

There is no reason for these routines to be written in assembly. In
the ports of DTrace to other platforms, they are already written in C.
No functional

Implement x86 dtrace_invop_(un)init() in C.

There is no reason for these routines to be written in assembly. In
the ports of DTrace to other platforms, they are already written in C.
No functional change intended.

MFC after: 1 week
Sponsored by: Netflix

show more ...

MFHead@r345677

Use an explicit comparison with VM_GUEST_NO.

Reported by: jhb
MFC with: r345359
Sponsored by: The FreeBSD Foundation

Don't attempt to measure TSC skew when running as a VM guest.

It simply doesn't work in general since VCPUs may migrate between
physical cores. The approach used to measure skew also doesn't
make m

Don't attempt to measure TSC skew when running as a VM guest.

It simply doesn't work in general since VCPUs may migrate between
physical cores. The approach used to measure skew also doesn't
make much sense in a VM.

PR: 218452
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

show more ...

Remove very old and unused signal information codes.

These have been supplanted by the MI signal information codes in
<sys/signal.h> since 7.0. The FPE_*_TRAP ones were deprecated even
earlier in 1

Remove very old and unused signal information codes.

These have been supplanted by the MI signal information codes in
<sys/signal.h> since 7.0. The FPE_*_TRAP ones were deprecated even
earlier in 1999.

PR: 226579 (exp-run)
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D14637

show more ...

Avoid CPU migration in dtrace_gethrtime() on x86.

dtrace_gethrtime() may be called outside of probe context, and in
particular, from the DTRACEIOC_BUFSNAP handler.

Disable interrupts rather than us

Avoid CPU migration in dtrace_gethrtime() on x86.

dtrace_gethrtime() may be called outside of probe context, and in
particular, from the DTRACEIOC_BUFSNAP handler.

Disable interrupts rather than using sched_pin() to help ensure that
we don't call any external functions when in probe context.

PR: 218452
MFC after: 1 week

show more ...

Corrected misspelled versions of rendezvous.

The MFC will include a compat definition of smp_no_rendevous_barrier()
that calls smp_no_rendezvous_barrier().

Reviewed by: gnn, kib
MFC after: 1 week
D

Corrected misspelled versions of rendezvous.

The MFC will include a compat definition of smp_no_rendevous_barrier()
that calls smp_no_rendezvous_barrier().

Reviewed by: gnn, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D10313

show more ...

Merge ^/head r313644 through r313895.

Directly include needed headers rather than relying on pollution.

We get machine/cpu.h via kmem.h -> proc.h -> _vm_domain.h -> seq.h.

Reported by: Ryan Libby
Sponsored by: Dell EMC Isilon
X-MFC wit

Directly include needed headers rather than relying on pollution.

We get machine/cpu.h via kmem.h -> proc.h -> _vm_domain.h -> seq.h.

Reported by: Ryan Libby
Sponsored by: Dell EMC Isilon
X-MFC with: r313841

show more ...

Prevent CPU migration when checking the DTrace nofault flag on x86.

dtrace_trap() consumes page and protection faults triggered by code running
in DTrace probe context. Such faults occur with interr

Prevent CPU migration when checking the DTrace nofault flag on x86.

dtrace_trap() consumes page and protection faults triggered by code running
in DTrace probe context. Such faults occur with interrupts disabled and are
detected using a per-CPU flag. Regular faults cause dtrace_trap() to be
called with interrupts enabled, and nothing was ensuring that the flag was
read from the correct CPU. This may result in dtrace_trap() consuming
unrelated page and protection faults when DTrace is enabled, causing the
fault handler to return without actually having handled the fault.

Diagnosed by: Ryan Libby <rlibby@gmail.com>
MFC after: 3 days
Sponsored by: Dell EMC Isilon

show more ...

Add an EARLY_AP_STARTUP option to start APs earlier during boot.

Currently, Application Processors (non-boot CPUs) are started by
MD code at SI_SUB_CPU, but they are kept waiting in a "pen" until
SI

Add an EARLY_AP_STARTUP option to start APs earlier during boot.

Currently, Application Processors (non-boot CPUs) are started by
MD code at SI_SUB_CPU, but they are kept waiting in a "pen" until
SI_SUB_SMP at which point they are released to run kernel threads.
SI_SUB_SMP is one of the last SYSINIT levels, so APs don't enter
the scheduler and start running threads until fairly late in the
boot.

This change moves SI_SUB_SMP up to just before software interrupt
threads are created allowing the APs to start executing kernel
threads much sooner (before any devices are probed). This allows
several initialization routines that need to perform initialization
on all CPUs to now perform that initialization in one step rather
than having to defer the AP initialization to a second SYSINIT run
at SI_SUB_SMP. It also permits all CPUs to be available for
handling interrupts before any devices are probed.

This last feature fixes a problem on with interrupt vector exhaustion.
Specifically, in the old model all device interrupts were routed
onto the boot CPU during boot. Later after the APs were released at
SI_SUB_SMP, interrupts were redistributed across all CPUs.

However, several drivers for multiqueue hardware allocate N interrupts
per CPU in the system. In a system with many CPUs, just a few drivers
doing this could exhaust the available pool of interrupt vectors on
the boot CPU as each driver was allocating N * mp_ncpu vectors on the
boot CPU. Now, drivers will allocate interrupts on their desired CPUs
during boot meaning that only N interrupts are allocated from the boot
CPU instead of N * mp_ncpu.

Some other bits of code can also be simplified as smp_started is
now true much earlier and will now always be true for these bits of
code. This removes the need to treat the single-CPU boot environment
as a special case.

As a transition aid, the new behavior is available under a new kernel
option (EARLY_AP_STARTUP). This will allow the option to be turned off
if need be during initial testing. I plan to enable this on x86 by
default in a followup commit in the next few days and to have all
platforms moved over before 11.0. Once the transition is complete,
the option will be removed along with the !EARLY_AP_STARTUP code.

These changes have only been tested on x86. Other platform maintainers
are encouraged to port their architectures over as well. The main
things to check for are any uses of smp_started in MD code that can be
simplified and SI_SUB_SMP SYSINITs in MD code that can be removed in
the EARLY_AP_STARTUP case (e.g. the interrupt shuffling).

PR: kern/199321
Reviewed by: markj, gnn, kib
Sponsored by: Netflix

show more ...

Make the second argument of dtrace_invop() a trapframe pointer.

Currently this argument is a pointer into the stack which is used by FBT
to fetch the first five probe arguments. On all non-x86 archi

Make the second argument of dtrace_invop() a trapframe pointer.

Currently this argument is a pointer into the stack which is used by FBT
to fetch the first five probe arguments. On all non-x86 architectures it's
simply the trapframe address, so this change has no functional impact. On
amd64 it's a pointer into the trapframe such that stack[1 .. 5] gives the
first five argument registers, which are deliberately grouped together in
the amd64 trapframe definition.

A trapframe argument simplifies the invop handlers on !x86 and makes the
x86 FBT invop handler easier to understand. Moreover, it allows for invop
handlers that may want to modify the register set of the interrupted thread.

show more ...

MFH

Sponsored by: The FreeBSD Foundation

Initialize DTrace hrtimer frequency during SI_SUB_CPU on i386 and amd64.

This allows the hrtimer to be used earlier during boot. This is required
for boot-time DTrace: anonymous enablings are create

Initialize DTrace hrtimer frequency during SI_SUB_CPU on i386 and amd64.

This allows the hrtimer to be used earlier during boot. This is required
for boot-time DTrace: anonymous enablings are created during
SI_SUB_DTRACE_ANON, which runs before APs are started. In particular,
the DTrace deadman timer requires that the hrtimer be functional.

MFC after: 2 weeks

show more ...

MFH

Sponsored by: The FreeBSD Foundation