hwpmc/arm64: Hard code the initial pmcr state

We can't the fields in PMCR_EL0 are configured in the way we expect.
Rather than depending on prior configuration hard code the fields
we want set.

Spo

hwpmc/arm64: Hard code the initial pmcr state

We can't the fields in PMCR_EL0 are configured in the way we expect.
Rather than depending on prior configuration hard code the fields
we want set.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D50434

show more ...

hwpmc/arm64: Support 64-bit counters

When FEAT_PMUv3p5 is implemented the event counters become 64-bit. The
PMCR_EL0.LP field is set to move the overflow to be a 64-bit overflow.
As such we ignore i

hwpmc/arm64: Support 64-bit counters

When FEAT_PMUv3p5 is implemented the event counters become 64-bit. The
PMCR_EL0.LP field is set to move the overflow to be a 64-bit overflow.
As such we ignore it when getting the value for the user as there is
no space in a 64-bit variable to store is.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D50433

show more ...

hwpmc/arm64: The counter is 64-bit

Because we include the overflow count in the counter mark it as 64-bit.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D50432

hwpmc/arm64: The PMXEVCNTR_EL0 register is 64-bit

It may return data in the upper 32-bits when FEAT_PMUv3p5 is
implemented so mask those bits off.

Sponsored by: Arm Ltd
Differential Revision: https

hwpmc/arm64: The PMXEVCNTR_EL0 register is 64-bit

It may return data in the upper 32-bits when FEAT_PMUv3p5 is
implemented so mask those bits off.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D50431

show more ...

hwpmc/arm64: PMCR_EL0 is a 64-bit register

Use uint64_t to store PMCR_EL0 as it's a 64-bit register and fields may
be present in the upper 64-bits.

Sponsored by: Arm Ltd
Differential Revision: http

hwpmc/arm64: PMCR_EL0 is a 64-bit register

Use uint64_t to store PMCR_EL0 as it's a 64-bit register and fields may
be present in the upper 64-bits.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D50427

show more ...

hwpmc/arm64: Make the pmcr variable descriptive

Rename the generic reg to pmcr to make it easier to find places we
access the register.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.

hwpmc/arm64: Make the pmcr variable descriptive

Rename the generic reg to pmcr to make it easier to find places we
access the register.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D50426

show more ...

hwpmc/arm64: Support the kernel booting to VHE

When booting to VHE we need to handle using the PMU at either EL1 or
EL2. WE already support EL1 so add support for EL2.

Reviewed by: mhorne
Sponsored

hwpmc/arm64: Support the kernel booting to VHE

When booting to VHE we need to handle using the PMU at either EL1 or
EL2. WE already support EL1 so add support for EL2.

Reviewed by: mhorne
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45527

show more ...

hwpmc/arm64: fix the type for pm_arm64_evsel

It holds a 64-bit register, use a uint64_t.

Reviewed by: mhorne
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45526

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

hwpmc: tidy pcd_finalize methods

Make them a little bit more consistent with one another in terms of what
is done:
- Add/reduce assertions to what is most useful: a loop to ensure
pcpu_fini free

hwpmc: tidy pcd_finalize methods

Make them a little bit more consistent with one another in terms of what
is done:
- Add/reduce assertions to what is most useful: a loop to ensure
pcpu_fini freed everything
- Add PMCDBG trace entries

The exception is the dmc620/cmn600 classes, which behave a little
differently, so leave them untouched.

Reviewed by: jkoshy
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41270

show more ...

pmc: better distinguish pmu-events allocation path

Background:

The pm_ev field of struct pmc_op_pmcallocate and struct pmc
traditionally contains the index of the chosen event, corresponding to
the

pmc: better distinguish pmu-events allocation path

Background:

The pm_ev field of struct pmc_op_pmcallocate and struct pmc
traditionally contains the index of the chosen event, corresponding to
the __PMC_EVENTS array in pmc_events.h. This is a static list of events,
maintained by FreeBSD.

In the usual case, libpmc translates the user supplied event name
(string) into the pm_ev index, which is passed as an argument to the
allocation syscall. On the kernel side, the allocation method for the
relevant hwpmc class translates the given index into the event code that
will be written to an event selection register.

In 2018, a new source of performance event definitions was introduced:
the pmu-events json files, which are maintained by the Linux kernel. The
result was better coverage for newer Intel processors with a reduced
maintenance burden for libpmc/hwpmc. Intel and AMD CPUs were
unconditionally switched to allocate events from pmu-events instead of
the traditional scheme (959826ca1bb0a, 81eb4dcf9e0d).

Under the pmu-events scheme, the pm_ev field contains an index
corresponding to the selected event from the pmu-events table, something
which the kernel has no knowledge of. The configuration for the
performance counting registers is instead passed via class-dependent
fields (struct pmc_md_op_pmcallocate).

In 2021 I changed the allocation logic so that it would attempt to
pull from the pmu-events table first, and fall-back to the traditional
method (dfb4fb41166bc3). Later, pmu-events support for arm64 and power8
CPUs was added (28dd6730a5d6 and b48a2770d48b).

The problem that remains is that the pm_ev field is overloaded, without
a definitive way to determine whether the event allocation came from the
pmu-events table or FreeBSD's statically-defined PMC events. This
resulted in a recent fix, 21f7397a61f7.

Change:

To disambiguate these two supported but separate use-cases, add a new
flag, PMC_F_EV_PMU, to be set as part of the allocation, indicating that
the event index came from pmu-events.

This is useful in two ways:
1. On the kernel side, we can validate the syscall arguments better.
Some classes support only the traditional event scheme (e.g.
hwpmc_armv7), while others support only the pmu-events method (e.g.
hwpmc_core for Intel). We can now check for this. The hwpmc_arm64
class supports both methods, so the new flag supersedes the existing
MD flag, PM_MD_EVENT_RAW.

2. The flag will be tracked in struct pmc for the duration of its
lifetime, meaning it is communicated back to userspace. This allows
libpmc to perform the reverse index-to-event-name translation
without speculating about the meaning of the index value.

Adding the flag is a backwards-incompatible ABI change. We recently
bumped the major version of the hwpmc module, so this breakage is
acceptable.

Reviewed by: jkoshy
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40753

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

hwpmc_arm64: plug memory leak

Free allocated per-CPU structures in the pcpu_fini and finalize methods.

While here, add debug trace entries to these methods.

Reviewed by: jkoshy, andrew
MFC after:

hwpmc_arm64: plug memory leak

Free allocated per-CPU structures in the pcpu_fini and finalize methods.

While here, add debug trace entries to these methods.

Reviewed by: jkoshy, andrew
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41268

show more ...

hwpmc: pass pmc pointer to more class methods

In many cases this avoids an extra lookup, since the callers always have
pm at hand. We can also eliminate several assertions, mostly for pm !=
NULL. Th

hwpmc: pass pmc pointer to more class methods

In many cases this avoids an extra lookup, since the callers always have
pm at hand. We can also eliminate several assertions, mostly for pm !=
NULL. The class methods are an internal interface, and the callers
already handle such a scenario. No functional change intended.

Reviewed by: jkoshy
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39915

show more ...

hwpmc: remove stub pmd_switch_{in,out} methods

Most platforms (non-x86) don't require these methods and implement stub
versions. If we initialize the pmc_mdep structure to always point to the
generi

hwpmc: remove stub pmd_switch_{in,out} methods

Most platforms (non-x86) don't require these methods and implement stub
versions. If we initialize the pmc_mdep structure to always point to the
generic versions, then we can purge the duplicate stubs.

Reviewed by: jkoshy
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39913

show more ...

hwpmc: don't use deprecated copystr(9)

It is just wrapper around strlcpy(), but results in more complicated
code. Clean this up to use strlcpy() or snprintf() as appropriate.

Reviewed by: jkoshy
MF

hwpmc: don't use deprecated copystr(9)

It is just wrapper around strlcpy(), but results in more complicated
code. Clean this up to use strlcpy() or snprintf() as appropriate.

Reviewed by: jkoshy
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39910

show more ...

arm64: Fix hwpmc module for OPT_ACPI isn't selected

Fixes: 59191f3573f6 ("Add support of ARM CMN-600 controller ...")
Sponsored by: Beckhoff Automation GmbH & Co. KG

Adjust function definition in hwpmc to avoid clang 15 warning

With clang 15, the following -Werror warning is produced:

sys/dev/hwpmc/hwpmc_arm64.c:530:21: error: a function declaration without

Adjust function definition in hwpmc to avoid clang 15 warning

With clang 15, the following -Werror warning is produced:

sys/dev/hwpmc/hwpmc_arm64.c:530:21: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
pmc_arm64_initialize()
^
void

This is because many of the functions are declared with a (void)
argument list, but defined with an empty argument list. Make the
definition match the declaration.

Sponsored by: The FreeBSD Foundation

show more ...

Support of optional PMC classes.

Reviewed by: mhorne
Differential Revision: https://reviews.freebsd.org/D32316

hwpmc: Use __diagused for variables only used in KASSERT().

hwpmc_arm64: remove write only variables and the computations they do

Sponsored by: Netflix

arm64 hwpmc: Support restricting counters to user or kernel mode.

Support the "usr" and "os" qualifiers on arm64 events to restrict
event counting to either usermode or the kernel, respectively. If

arm64 hwpmc: Support restricting counters to user or kernel mode.

Support the "usr" and "os" qualifiers on arm64 events to restrict
event counting to either usermode or the kernel, respectively. If
neither qualifier is given, events are counted in both.

Reviewed by: emaste
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D34527

show more ...

hwpmc: Fix amd/arm64/armv7/uncore sampling overflow race

If a counter more than overflows just as we read it on switch out then,
if using sampling mode, we will negate this small value to give a hug

hwpmc: Fix amd/arm64/armv7/uncore sampling overflow race

If a counter more than overflows just as we read it on switch out then,
if using sampling mode, we will negate this small value to give a huge
reload count, and if we later switch back in that context we will
validate that value against pm_reloadcount and panic an INVARIANTS
kernel with:

panic: [pmc,1470] pmcval outside of expected range cpu=2 ri=16 pmcval=fffff292 pm_reloadcount=10000

or similar. Presumably in a non-INVARIANTS kernel we will instead just
use the provided value as the reload count, which would lead to the
overflow not happing for a very long time (e.g. 78 minutes for a 48-bit
counter incrementing at an averate rate of 1GHz).

Instead, clamp the reload count to 0 (which corresponds precisely to the
value we would compute if it had just overflowed and no more), which
will result in hwpmc using the full original reload count again. This is
the approach used by core for Intel (for both fixed and programmable
counters).

As part of this, armv7 and arm64 are made conceptually simpler; rather
than skipping modifying the overflow count for sampling mode counters so
it's always kept as ~0, those special cases are removed so it's always
applicable and the concatentation of it and the hardware counter can
always be viewed as a 64-bit counter, which also makes them look more
like other architectures.

Whilst here, fix an instance of UB (shifting a 1 into the sign bit) for
amd in its sign-extension code.

Reviewed by: andrew, mhorne, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33654

show more ...

hwpmc: initialize arm64 counter/interrupt state

Performance counters and overflow interrupts are assumed to be disabled
by default, but this is not guaranteed. Ensure we disable both during
per-cpu

hwpmc: initialize arm64 counter/interrupt state

Performance counters and overflow interrupts are assumed to be disabled
by default, but this is not guaranteed. Ensure we disable both during
per-cpu initialization, before enabling the PMU. Otherwise, some systems
(such as the Ampere eMAG) would experience an interrupt storm upon
loading the hwpmc module.

Reviewed by: br
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32854

show more ...

hwpmc: don't validate capabilities in allocation method

These checks were inconsistently applied across the various hwpmc
classes. The condition is already checked by the generic code in
hwpmc_mod.c

hwpmc: don't validate capabilities in allocation method

These checks were inconsistently applied across the various hwpmc
classes. The condition is already checked by the generic code in
hwpmc_mod.c, so remove them.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31388

show more ...