riscv vmm: implement SBI RFNC extension.

The RISC-V SBI (Supervisor Binary Interface) RFNC (fence) extension is used
to provide a standardized mechanism for enforcing memory and I/O ordering
between

riscv vmm: implement SBI RFNC extension.

The RISC-V SBI (Supervisor Binary Interface) RFNC (fence) extension is used
to provide a standardized mechanism for enforcing memory and I/O ordering
between different execution contexts or cores in a system. Specifically, it
allows supervisors (such as an operating system kernel) to request certain
synchronization operations across CPUs or harts (hardware threads) via the
SBI.

Differential Revision: https://reviews.freebsd.org/D48441

show more ...

riscv: enable cpufreq_dt driver

Implement the small amount of MD code required; copied from arm/arm64.

One tweak is made to cpufreq_dt itself: if the opp-shared property is
missing, but there is on

riscv: enable cpufreq_dt driver

Implement the small amount of MD code required; copied from arm/arm64.

One tweak is made to cpufreq_dt itself: if the opp-shared property is
missing, but there is only one CPU, then we can still attach. This is
relevant for the single-core Allwinner D1.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48124

show more ...

riscv: enable Allwinner D1 USB drivers

Add the generic USB drivers and FDT glue to the build.

Make small tweaks to the aw_usbphy and aw_musb drivers for the Allwinner
D1.

Reviewed by: manu
Sponsor

riscv: enable Allwinner D1 USB drivers

Add the generic USB drivers and FDT glue to the build.

Make small tweaks to the aw_usbphy and aw_musb drivers for the Allwinner
D1.

Reviewed by: manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48126

show more ...

riscv vmm: virtual timer support.

Add a virtual timer implementation based on SBI Time extension.
This is needed for Eswin EIC7700 SoC which does not include the newer SSTC
extension.

Timer interru

riscv vmm: virtual timer support.

Add a virtual timer implementation based on SBI Time extension.
This is needed for Eswin EIC7700 SoC which does not include the newer SSTC
extension.

Timer interrupt pending bit (STIP) could not be cleared in the guest system,
so rework interrupts handling: add new "interrupts_pending" field. Use it
for timer interrupt only for now, but later we can extend to store all
pending interrupts (Timer, IPI and External).

With this I'm able to boot FreeBSD (SMP) guest on HiFive Premier P550,
which is the first real hardware with RISC-V 'H'-spec included.

Differential Revision: https://reviews.freebsd.org/D48133

show more ...

riscv: add custom T-HEAD dcache ops

This is the first major quirk we need to support in order to run on
current T-HEAD/XuanTie CPUs, e.g. the C906 or C910, found in several
existing RISC-V SBCs. Wit

riscv: add custom T-HEAD dcache ops

This is the first major quirk we need to support in order to run on
current T-HEAD/XuanTie CPUs, e.g. the C906 or C910, found in several
existing RISC-V SBCs. With these custom dcache routines installed,
busdma can reliably communicate with devices which are not coherent
w.r.t. the CPU's data caches.

This patch introduces the first quirk/errata handling functions to
identcpu.c, and thus is forced to make some decisions about how this
code is structured. It will be amended with the changes that follow in
the series, yet I feel the final result is (unavoidably) somewhat
clumsy. I expect the CPU identification code will continue to evolve as
more CPUs and their quirks are eventually supported.

Discussed with: jrtc27
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D47455

show more ...

riscv: dcache flush hooks

Cache management operations were, for a long time, unspecified by the
RISC-V ISA, and thus these functions have been no-ops. To cope, hardware
with non-coherent I/O has imp

riscv: dcache flush hooks

Cache management operations were, for a long time, unspecified by the
RISC-V ISA, and thus these functions have been no-ops. To cope, hardware
with non-coherent I/O has implemented custom cache flush mechanisms,
either in the form of custom instructions or special device registers.
Additionally, the RISC-V CMO extension is ratified and these official
instructions will start to show up in hardware eventually. Therefore, a
method is needed to select the dcache management routines at runtime.

Add a simple set of function hooks, as well as a routine to install them
and specify the minimum dcache line size. The first consumer will be the
non-standard cache management instructions for T-HEAD CPUs.

The unused I-cache variables and macros are removed.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D47454

show more ...

riscv: Add support for building vmm as a kernel module

- Update sys/modules/vmm/Makefile.
- Add some required symbol definitions.
- Hook up vmm in sys/modules/Makefile.

Reviewed by: br
Differential

riscv: Add support for building vmm as a kernel module

- Update sys/modules/vmm/Makefile.
- Add some required symbol definitions.
- Hook up vmm in sys/modules/Makefile.

Reviewed by: br
Differential Revision: https://reviews.freebsd.org/D47477

show more ...

riscv/vmm: Initial import.

Add kernel code for 'H' — Hypervisor Extension[1] to support
virtualization on RISC-V ISA.

This comes with a separate userspace patch allowing us to boot
unmodified freeb

riscv/vmm: Initial import.

Add kernel code for 'H' — Hypervisor Extension[1] to support
virtualization on RISC-V ISA.

This comes with a separate userspace patch allowing us to boot
unmodified freebsd/riscv guest. Other operating systems are untested.

This also comes with a U-Boot port that is configured to run in bhyve
guest environment — in RISC-V virtual supervisor mode.
The vmm SBI code emulates RISC-V machine-mode for the guest, handling
SBI calls partly in vmm kernel and partly in bhyve userspace.

Developed in Spike simulator during short period of time, the support
is considered experimental. The first real hardware with hypervisor
spec included should have just reached the market, so this was tested
in Spike and QEMU only. Note that this depends on Sstc extension
presence in the hardware (both Spike and QEMU have it).

Note that booting multiple guests at the same time is not tested and
may require additional work. Some TODOs are indicated within the
code, and some listed in the project's home page[2].

Many thanks to Jessica Clarke, Mitchell Horne and Mark Johnston
for help with parts, test and review.

1. https://riscv.org/technical/specifications/
2. https://wiki.freebsd.org/riscv/bhyve

Sponsored by: UK Research and Innovation
Differential Revision: https://reviews.freebsd.org/D45553

show more ...

dtrace: Avoid including dtrace_isa.c directly into dtrace.c

This was done in the original DTrace import, presumably because that
made it a bit easier to handle includes. However, this can cause
dtr

dtrace: Avoid including dtrace_isa.c directly into dtrace.c

This was done in the original DTrace import, presumably because that
made it a bit easier to handle includes. However, this can cause
dtrace_getpcstack() to be inlined into dtrace_probe(), resulting in a
missing frame in stack traces since dtrace_getpcstack() takes care to
bump "aframes" to account for its own stack frame.

To avoid this, compile dtrace_isa.c separately on all platforms. Add
requisite includes.

MFC after: 2 weeks
Sponsored by: Innovate UK

show more ...

sdt: Implement SDT probes using hot-patching

The idea here is to avoid a memory access and conditional branch per
probe site. Instead, the probe is represented by an "unreachable"
unconditional fun

sdt: Implement SDT probes using hot-patching

The idea here is to avoid a memory access and conditional branch per
probe site. Instead, the probe is represented by an "unreachable"
unconditional function call. asm goto is used to store the address of
the probe site (represented by a no-op sled) and the address of the
function call into a tracepoint record. Each SDT probe carries a list
of tracepoints.

When the probe is enabled, the no-op sled corresponding to each
tracepoint is overwritten with a jmp to the corresponding label. The
implementation uses smp_rendezvous() to park all other CPUs while the
instruction is being overwritten, as this can't be done atomically in
general. The compiler moves argument marshalling code and the
sdt_probe() function call out-of-line, i.e., to the end of the function.

Per gallatin@ in D43504, this approach has less overhead when probes are
disabled. To make the implementation a bit simpler, I removed support
for probes with 7 arguments; nothing makes use of this except a
regression test case. It could be re-added later if need be.

The approach taken in this patch enables some more improvements:
1. We can now automatically fill out the "function" field of SDT probe
names. The SDT macros let the programmer specify the function and
module names, but this is really a bug and shouldn't have been
allowed. The intent was to be able to have the same probe in
multiple functions and to let the user restrict which probes actually
get enabled by specifying a function name or glob.
2. We can avoid branching on SDT_PROBES_ENABLED() by adding the ability
to include blocks of code in the out-of-line path. For example:

if (SDT_PROBES_ENABLED()) {
int reason = CLD_EXITED;

if (WCOREDUMP(signo))
reason = CLD_DUMPED;
else if (WIFSIGNALED(signo))
reason = CLD_KILLED;
SDT_PROBE1(proc, , , exit, reason);
}

could be written

SDT_PROBE1_EXT(proc, , , exit, reason,
int reason;

reason = CLD_EXITED;
if (WCOREDUMP(signo))
reason = CLD_DUMPED;
else if (WIFSIGNALED(signo))
reason = CLD_KILLED;
);

In the future I would like to use this mechanism more generally, e.g.,
to remove branches and marshalling code used by hwpmc, and generally to
make it easier to add new tracepoint consumers without having to add
more conditional branches to hot code paths.

Reviewed by: Domagoj Stolfa, avg
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D44483

show more ...

riscv: Move sigcode out of locore.S

It really doesn't fit here anymore as locore is all about early startup
code. Thus, move it to its own file.

Reviewed by: br
MFC after: 1 week
Sponsored by: The

riscv: Move sigcode out of locore.S

It really doesn't fit here anymore as locore is all about early startup
code. Thus, move it to its own file.

Reviewed by: br
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45320

show more ...

uma: Deduplicate uma_small_alloc

This commit refactors the UMA small alloc code and
removes most UMA machine-dependent code.
The existing machine-dependent uma_small_alloc code is almost identical
a

uma: Deduplicate uma_small_alloc

This commit refactors the UMA small alloc code and
removes most UMA machine-dependent code.
The existing machine-dependent uma_small_alloc code is almost identical
across all architectures, except for powerpc where using the direct
map addresses involved extra steps in some cases.

The MI/MD split was replaced by a default uma_small_alloc
implementation that can be overridden by architecture-specific code by
defining the UMA_MD_SMALL_ALLOC symbol. Furthermore, UMA_USE_DMAP was
introduced to replace most UMA_MD_SMALL_ALLOC uses.

Reviewed by: markj, kib
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D45084

show more ...

riscv: Introduce support for APLIC interrupt controller

This patch introduces support for the RISC-V APLIC interrupt controller
[1]. Currently, it is only supports direct mode, i.e. without an IMSIC

riscv: Introduce support for APLIC interrupt controller

This patch introduces support for the RISC-V APLIC interrupt controller
[1]. Currently, it is only supports direct mode, i.e. without an IMSIC
and functionally replacing the legacy RISC-V PLIC. Work on IMSIC support
is in progress.

[1] https://github.com/riscv/riscv-aia/releases/tag/1.0

Reviewed by: mhorne
Discussed with: jrtc27
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D43293

show more ...

riscv: Convert local interrupt controller to a newbus PIC

Currently the local interrupt controller implementation is based on
pre-INTRNG arm/arm64 code, using hand-rolled event code rather than
INTR

riscv: Convert local interrupt controller to a newbus PIC

Currently the local interrupt controller implementation is based on
pre-INTRNG arm/arm64 code, using hand-rolled event code rather than
INTRNG. This then interacts weirdly with the PLIC, and other future
interrupt controllers like the APLIC and IMSICs in the upcoming AIA
specification, since they become the root PIC despite not being the
logical root. Instead, use a real newbus device for it and register
it as the root PIC.

This also adapts the IPI code to make use of the newly-added INTRNG
generic IPI handling framework, adding a new sbi_ipi as the PIC. In
future there will be alternative devices for sending IPIs that will
register with higher priorities, such as the proposed AIA IMSIC and
ACLINT SSWI.

Reviewed by: mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35901

show more ...

sys: Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Consistently provide ffs/fls using builtins

Use of compiler builtin ffs/ctz functions will result in optimized
instruction sequences when possible, and fall back to calling a function
provided by th

Consistently provide ffs/fls using builtins

Use of compiler builtin ffs/ctz functions will result in optimized
instruction sequences when possible, and fall back to calling a function
provided by the compiler run-time library. We have slowly shifted our
platforms to take advantage of these builtins in 60645781d613 (arm64),
1c76d3a9fbef (arm), 9e319462a03a (powerpc, partial).

Some platforms still rely on the libkern implementations of these
functions provided by libkern, namely riscv, powerpc (ffs*, flsll), and
i386 (ffsll and flsll). These routines are slow, as they perform a
linear search for the bit in question. Even on platforms lacking
dedicated bit-search instructions, such as riscv, the compiler library
will provide better-optimized routines, e.g. by using binary search.

Consolidate all definitions of these functions (whether currently using
builtins or not) to libkern.h. This should result in equivalent or
better performing routines in all cases.

One wart in all of this is the existing HAVE_INLINE_F*** macros, which
we use in a few places to conditionally avoid the slow libkern routines.
These aren't easily removed in one commit. For now, provide these
defines unconditionally, but marked for removal after subsequent
cleanup.

Removal of the now unused libkern routines will follow in the next
commit.

Reviewed by: dougm, imp (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40698

show more ...

riscv: Compile instr_size.c into the kernel when DTrace is configured

Reported by: Jenkins
Fixes: 080e56a6c98c ("dtrace: expose dtrace_instr_size() to userland and implement it for riscv")

Import an optimized str{n}cmp on arm64

These are from the Arm Optimized Routines and don't use the VFP so are
safe to use in the kernel.

Sponsored by: The FreeBSD Foundation

files: Make ext_resources non-optional

EXT_RESOURCES have been introduced in 12-CURRENT and all supported
releases have it enabled in their kernel config.

MFC after: 1 month
Differential Revision:

files: Make ext_resources non-optional

EXT_RESOURCES have been introduced in 12-CURRENT and all supported
releases have it enabled in their kernel config.

MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D33834

show more ...

riscv: gdb(4) support

Add the MD portion required for the gdb stub.

Reviewed by: jhb (earlier version)
Discussed with: jrtc27
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D3

riscv: gdb(4) support

Add the MD portion required for the gdb stub.

Reviewed by: jhb (earlier version)
Discussed with: jrtc27
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33734

show more ...

netinet: Deduplicate most in_cksum() implementations

in_cksum() and related routines are implemented separately for each
platform, but only i386 and arm have optimized versions. Other
platforms' co

netinet: Deduplicate most in_cksum() implementations

in_cksum() and related routines are implemented separately for each
platform, but only i386 and arm have optimized versions. Other
platforms' copies of in_cksum.c are identical except for style
differences and support for big-endian CPUs.

Deduplicate the implementations for the rest of the platforms. This
will make it easier to implement in_cksum() for unmapped mbufs. On arm
and i386, define HAVE_MD_IN_CKSUM to mean that the MI implementation is
not to be compiled.

No functional change intended.

Reviewed by: kp, glebius
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33095

show more ...

riscv: move signal delivery code to exec_machdep.c

Reviewed by: emaste, imp
Discussed with: jrtc27
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freeb

riscv: move signal delivery code to exec_machdep.c

Reviewed by: emaste, imp
Discussed with: jrtc27
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32310

show more ...

riscv: retire bcmp

Unused since ba96f37758412151 ("Use __builtin for various mem* and b* (e.g. bzero)
routines.")

Reviewed by: mhorne
Sponsored by: Rubicon Communications, LLC ("Netgate")

fu740_pci_dw: Add SiFive FU740 PCIe controller driver

Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31033

Rename ofwpci.c to ofw_pcib.c

It's a class0 driver that implements some pcib methods and creates
a pci bus as its children.
The "ofw_pci" name will be used by a new driver that will be a subclass
of

Rename ofwpci.c to ofw_pcib.c

It's a class0 driver that implements some pcib methods and creates
a pci bus as its children.
The "ofw_pci" name will be used by a new driver that will be a subclass
of the pci bus.
No functional changes intended.

Submitted by: Kornel Duleba <mindal@semihalf.com>
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: Alstom Group
Differential Revision: https://reviews.freebsd.org/D30226

show more ...