x86: Refactor kernel-mode NMI handling

This refactor aims to add the ability to share performance counter
interrupts by refactoring the kernel-mode NMI handler. The handler now
allows multiple drive

x86: Refactor kernel-mode NMI handling

This refactor aims to add the ability to share performance counter
interrupts by refactoring the kernel-mode NMI handler. The handler now
allows multiple drivers to service the same interrupt (e.g. hwpmc(4)
and hwt(4)'s Intel Processor Trace backend).

Reviewed by: kib, avg
Differential Revision: https://reviews.freebsd.org/D46421

show more ...

Use the correct idle routine on recent AMD EPYC servers

We have been incorrectly choosing the "hlt" idle method on modern AMD
EPYC servers for C1 idle. This is because AMD also uses the Functional
F

Use the correct idle routine on recent AMD EPYC servers

We have been incorrectly choosing the "hlt" idle method on modern AMD
EPYC servers for C1 idle. This is because AMD also uses the Functional
Fixed Hardware interface. Due to not parsing the table properly for
AMD, and due to a weird quirk where the mwait latency for C1 is
mis-interpreted as the latency for hlt, we wind up choosing hlt for
c1, which has a far higher wake up latency (similar to IO) of roughly
400us on my test system (AMD 7502P).

This patch fixes this by:

- Looking for AMD in addition to Intel in the FFH
(Note the vendor id of "2" for AMD is not publically documented, but
AMD has confirmed they are using "2" and has promised to document it.)

- Using mwait on AMD when specified in the table, and when CPUid says
its supported

- Fixing a weird issue where we copy the contents of cx_ptr for C1 and
when moving to C2, we do not reinitialize cx_ptr. This leads to
mwait being selected, and ignoring the specified i/o halt method
unless we clear mwait before looking at the table for C2.

Differential Revision: https://reviews.freebsd.org/D47444
Reviewed by: dab, kib, vangyzen
Sponsored by: Netflix

show more ...

AMD CPUs: update bits and data from CPUID 0x8000_0008

from AMD APM vol3 doc no 24594 Rev. 3.36 March 2024

Reviewed and tested by: emaste
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The

AMD CPUs: update bits and data from CPUID 0x8000_0008

from AMD APM vol3 doc no 24594 Rev. 3.36 March 2024

Reviewed and tested by: emaste
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D45188

show more ...

x86: Make cpu_model[] public

No functional change intended.

Reviewed by: emaste, imp, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43281

x86: AMD Zen2: Zenbleed chicken bit mitigation

Applies only to bare-metal Zen2 processors. The system currently
automatically applies it to all of them.

Tunable/sysctl 'machdep.mitigations.zenblee

x86: AMD Zen2: Zenbleed chicken bit mitigation

Applies only to bare-metal Zen2 processors. The system currently
automatically applies it to all of them.

Tunable/sysctl 'machdep.mitigations.zenbleed.enable' can be used to
forcibly enable or disable the mitigation at boot or run-time. Possible
values are:

0: Mitigation disabled
1: Mitigation enabled
2: Run the automatic determination.

Currently, value 2 is the default and has identical effect as value 1.
This might change in the future if we choose to take into account
microcode revisions in the automatic determination process.

The tunable/sysctl value is simply ignored on non-applicable CPU models,
which is useful to apply the same configuration on a set of machines
that do not all have Zen2 processors. Trying to set it to any integer
value not listed above is silently equivalent to setting it to value 2
(automatic determination).

The current mitigation state can be queried through sysctl
'machdep.mitigations.zenbleed.state', which returns "Not applicable",
"Mitigation enabled" or "Mitigation disabled". Note that this state is
not guaranteed to be accurate in case of intervening modifications of
the corresponding chicken bit directly via cpuctl(4) (this includes the
cpucontrol(8) utility). Resetting the desired policy through
'machdep.mitigations.zenbleed.enable' (possibly to its current value)
will reset the hardware state and ensure that the reported state is
again coherent with it.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41817

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

amd64: Eliminate write only cpu_fxsr.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D38289
MFC after: 1 week

i386: check that trap() and syscall() run on the thread kstack

and not on the trampoline stack. This is a useful way to ensure that
we did not enabled interrupts while on user %cr3 or trampoline st

i386: check that trap() and syscall() run on the thread kstack

and not on the trampoline stack. This is a useful way to ensure that
we did not enabled interrupts while on user %cr3 or trampoline stack.

Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

amd64: Reload CPU ext features after resume or cr4 changes

Reviewed by: kib
Differential revision: https://reviews.freebsd.org/D35555
MFC after: 2 weeks

Remove dead code.

is_physical_memory() dead since 235a54de.

Reviewed by: markj
Differential revision: https://reviews.freebsd.org/D35056
MFC after: 2 weeks

Simplify swi for bus_dma.

When a DMA request using bounce pages completes, a swi is triggered to
schedule pending DMA requests using the just-freed bounce pages. For
a long time this bus_dma swi ha

Simplify swi for bus_dma.

When a DMA request using bounce pages completes, a swi is triggered to
schedule pending DMA requests using the just-freed bounce pages. For
a long time this bus_dma swi has been tied to a "virtual memory" swi
(swi_vm). However, all of the swi_vm implementations are the same and
consist of checking a flag (busdma_swi_pending) which is always true
and if set calling busdma_swi. I suspect this dates back to the
pre-SMPng days and that the intention was for swi_vm to serve as a
mux. However, in the current scheme there's no need for the mux.

Instead, remove swi_vm and vm_ih. Each bus_dma implementation that
uses bounce pages is responsible for creating its own swi (busdma_ih)
which it now schedules directly. This swi invokes busdma_swi directly
removing the need for busdma_swi_pending.

One consequence is that the swi now works on RISC-V which had previously
failed to invoke busdma_swi from swi_vm.

Reviewed by: imp, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33447

show more ...

minidump: Parameterize minidumpsys()

The minidump code is written assuming that certain global state will not
change, and rightly so, since it executes from a kernel debugger
context. In order to su

minidump: Parameterize minidumpsys()

The minidump code is written assuming that certain global state will not
change, and rightly so, since it executes from a kernel debugger
context. In order to support taking minidumps of a live system, we
should allow copies of relevant global state that is likely to change to
be passed as parameters to the minidumpsys() function.

This patch does the work of parameterizing this function, by adding a
struct minidumpstate argument. For now, this struct allows for copies of
the kernel message buffer, and the bitset that tracks which pages should
be dumped (vm_page_dump). Follow-up changes will actually make use of
these arguments.

Notably, dump_avail[] does not need a snapshot, since it is not expected
to change after system initialization.

The existing minidumpsys() definitions are renamed, and a thin MI
wrapper is added to kern_dump.c, which handles the construction of
the state struct. Thus, calling minidumpsys() remains as simple as
before.

Reviewed by: kib, markj, jhb
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D31989

show more ...

x86: cpufunc: Add rdtsc_ordered()

Add a variant of 'rdtsc()' that performs the ordered version of 'rdtsc'
appropriate for the invoking x86 variant.

Also, expose the 'lfence'-ed and 'mfence'-ed 'rdt

x86: cpufunc: Add rdtsc_ordered()

Add a variant of 'rdtsc()' that performs the ordered version of 'rdtsc'
appropriate for the invoking x86 variant.

Also, expose the 'lfence'-ed and 'mfence'-ed 'rdtsc()' variants needed
by 'rdtsc_ordered()' for general use.

Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D31416

show more ...

fork: Allow ABI to specify fork return values for child.

At least Linux x86 ABI's does not use carry bit and expects that the dx register
is preserved. For this add a new sv_set_fork_retval hook and

fork: Allow ABI to specify fork return values for child.

At least Linux x86 ABI's does not use carry bit and expects that the dx register
is preserved. For this add a new sv_set_fork_retval hook and call it from cpu_fork().

Add a short comment about touching dx in x86_set_fork_retval(), for more details
see phab comments from kib@ and imp@.

Reviewed by: kib
Differential revision: https://reviews.freebsd.org/D31472
MFC after: 2 weeks

show more ...

x86_msr_op: extend the KPI to allow MSR read and single-CPU operations

Reivewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D3

x86_msr_op: extend the KPI to allow MSR read and single-CPU operations

Reivewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31386

show more ...

x86: add x86_clear_dbregs() helper

Move the code from exec_setregs() to reset debug registers state on exec,
to the x86_clear_dbregs() helper

Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
M

x86: add x86_clear_dbregs() helper

Move the code from exec_setregs() to reset debug registers state on exec,
to the x86_clear_dbregs() helper

Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D29687

show more ...

x86: implement kdb watchpoint functions

Add wrappers around the dbreg interface that can be consumed by MI
kernel debugger code. The dbreg functions themselves are updated to
return error codes, not

x86: implement kdb watchpoint functions

Add wrappers around the dbreg interface that can be consumed by MI
kernel debugger code. The dbreg functions themselves are updated to
return error codes, not just -1. dbreg_set_watchpoint() is extended to
accept access bits as an argument.

Reviewed by: jhb, kib, markj
MFC after: 3 weeks
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D29155

show more ...

x86: consolidate hw watchpoint logic into new file

This is a prerequisite to using these functions outside of ddb, but also
provides some cleanup and minor refactoring. This code is almost
entirely

x86: consolidate hw watchpoint logic into new file

This is a prerequisite to using these functions outside of ddb, but also
provides some cleanup and minor refactoring. This code is almost
entirely duplicated between the two implementations, the only
significant difference being the lack of dbreg synchronization on i386.

Cleanups are:
- demote some internal functions to static
- use the constant NDBREGS instead of a '4' literal
- remove K&R definitions
- some added comments

Reviewed by: kib, jhb
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D29153

show more ...

xen/boot: allow specifying boot method when booted from Xen

Allow setting the bootmethod variable from the Xen PVH entry point, in
order to be able to correctly set the underlying firmware mode when

xen/boot: allow specifying boot method when booted from Xen

Allow setting the bootmethod variable from the Xen PVH entry point, in
order to be able to correctly set the underlying firmware mode when
booted as a dom0.

Move the bootmethod variable to be defined in x86/cpu_machdep.c
instead so it can be shared by both i386 and amd64.

Sponsored by: Citrix Systems R&D
Reviewed by: kib
Differential revision: https://reviews.freebsd.org/D28619

show more ...

Limit workaround for errata E400 to appropriate AMD cpus.

From Linux sources and several datasheets I looked at, it seems that
the workaround is only needed on families 0xf and 0x10. For instance,

Limit workaround for errata E400 to appropriate AMD cpus.

From Linux sources and several datasheets I looked at, it seems that
the workaround is only needed on families 0xf and 0x10. For instance,
Ryzens do not implement the accessed MSR at all, it is documented as
reserved. Also, hypervisors should not allow guest to put CPU into
idle state, so activate workaround only when on bare hardware.

While there, style the code:
move MSR defines to specialreg.h
move identification to initcpu.c

Reported by: whu
Reviewed by: avg
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D26470

show more ...

Move ctx_switch_xsave declaration to amd64 md_var.h.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days

Move vm_page_dump bitset array definition to MI code

These definitions were repeated by all architectures, with small
variations. Consolidate the common definitons in machine
independent code and us

Move vm_page_dump bitset array definition to MI code

These definitions were repeated by all architectures, with small
variations. Consolidate the common definitons in machine
independent code and use bitset(9) macros for manipulation. Many
opportunities for deduplication remain in the machine dependent
minidump logic. The only intended functional change is increasing
the bit index type to vm_pindex_t, allowing the indexing of pages
with address of 8 TiB and greater.

Reviewed by: kib, markj
Approved by: scottl (implicit)
MFC after: 1 week
Sponsored by: Ampere Computing, Inc.
Differential Revision: https://reviews.freebsd.org/D26129

show more ...

MFH

Sponsored by: Rubicon Communications, LLC (netgate.com)

Export a routine to provide the TSC_AUX MSR value and use this in vmm.

Also, drop an unnecessary set of braces.

Requested by: kib
Reviewed by: kib
MFC after: 3 weeks

Control for Special Register Buffer Data Sampling mitigation.

New microcode update for Intel enables mitigation for SRBDS, which
slows down RDSEED and related instructions. The update also provides

Control for Special Register Buffer Data Sampling mitigation.

New microcode update for Intel enables mitigation for SRBDS, which
slows down RDSEED and related instructions. The update also provides
a control to limit the mitigation to SGX enclaves, which should
restore the speed of random generator by the cost of potential
cross-core bufer sampling.

See https://software.intel.com/security-software-guidance/insights/deep-dive-special-register-buffer-data-sampling

GIve the user control over it.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D25221

show more ...