| 8020361d | 30-Apr-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla: Define _GNU_SOURCE in timerlat_bpf.c
Newer versions of glibc include a definition of struct sched_attr in
bits/sched.h (included through sched.h which is included by rtla).
Commit 0eecee340672
rtla: Define _GNU_SOURCE in timerlat_bpf.c
Newer versions of glibc include a definition of struct sched_attr in
bits/sched.h (included through sched.h which is included by rtla).
Commit 0eecee340672 ("tools/rtla: fix collision with glibc
sched_attr/sched_set_attr") has modified the definition of struct
sched_attr in utils.h, so that it is only applied with older versions of
glibc that do not define it, in order to prevent build failure.
The definition in bits/sched.h depends on _GNU_SOURCE.
timerlat_bpf.c does not define _GNU_SOURCE, making it fall back to the
definition in utils.h. The latter has two fields less, leading to
shifted offsets of struct timerlat_params in timerlat_bpf_init.
Because of the shift, timerlat_bpf_init incorrectly reads
params->entries as 0 for timerlat-hist and disables the creation of
histogram maps, causing breakage in BPF sample collection mode:
$ rtla timerlat hist -d 1s
Error pulling BPF data
Fix the issue by also defining _GNU_SOURCE in timerlat_bpf.c.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250430144651.621766-1-tglozar@redhat.com
Fixes: e34293ddcebd ("rtla/timerlat: Add BPF skeleton to collect samples")
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 6a38c51a | 22-Apr-2025 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
rtla: Define __NR_sched_setattr for LoongArch
When executing "make -C tools/tracing/rtla" on LoongArch, there exists
the following error:
src/utils.c:237:24: error: '__NR_sched_setattr' undeclare
rtla: Define __NR_sched_setattr for LoongArch
When executing "make -C tools/tracing/rtla" on LoongArch, there exists
the following error:
src/utils.c:237:24: error: '__NR_sched_setattr' undeclared
Just define __NR_sched_setattr for LoongArch if not exist.
Link: https://lore.kernel.org/20250422074917.25771-1-yangtiezhu@loongson.cn
Reported-by: Haiyong Sun <sunhaiyong@loongson.cn>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 73203269 | 21-Mar-2025 |
John Kacur <jkacur@redhat.com> |
rtla: Add the ability to create ctags and etags
- Add the ability to create and remove ctags and etags, using the following
make tags
make TAGS
make tags_clean
- fix a comment in Makefile.rtla with
rtla: Add the ability to create ctags and etags
- Add the ability to create and remove ctags and etags, using the following
make tags
make TAGS
make tags_clean
- fix a comment in Makefile.rtla with the correct spelling and don't
imply that the ability to create an rtla tarball will be removed
Cc: Tomas Glozar <tglozar@redhat.com>
Cc: "Luis Claudio R . Goncalves" <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250321175053.29048-1-jkacur@redhat.com
Signed-off-by: John Kacur <jkacur@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| a86150f3 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Test setting default options
Add function to test engine to test with pre-set osnoise options, and
use it to test whether osnoise period (as an example) is set correctly.
The test works
rtla/tests: Test setting default options
Add function to test engine to test with pre-set osnoise options, and
use it to test whether osnoise period (as an example) is set correctly.
The test works by pre-setting a high period of 10 minutes and stop on
threshold. Thus, it is easy to check whether rtla is properly resetting
the period to default: if it is, the test will complete on time, since
the first sample will overflow the threshold. If not, it will time out.
Cc: Luis Goncalves <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250320092500.101385-7-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Reviewed-by: John Kacur <jkacur@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 6c618272 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Reset osnoise options before check
Remove any dangling tracing instances from previous improperly exited
runs of rtla, and reset osnoise options to default before running a test
case.
T
rtla/tests: Reset osnoise options before check
Remove any dangling tracing instances from previous improperly exited
runs of rtla, and reset osnoise options to default before running a test
case.
This ensures that the test results are deterministic. Specific test
cases checked that rtla behaves correctly even when the tracer state is
not clean will be added later.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250320092500.101385-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 0122938a | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla: Always set all tracer options
rtla currently only sets tracer options that are explicitly set by the
user, with the exception of OSNOISE_WORKLOAD.
This leads to improper behavior in case rtla
rtla: Always set all tracer options
rtla currently only sets tracer options that are explicitly set by the
user, with the exception of OSNOISE_WORKLOAD.
This leads to improper behavior in case rtla is run with those options
not set to the default value. rtla does reset them to the original
value upon exiting, but that does not protect it from starting with
non-default values set either by an improperly exited rtla or by another
user of the tracers.
For example, after running this command:
$ echo 1 > /sys/kernel/tracing/osnoise/stop_tracing_us
all runs of rtla will stop at the 1us threshold, even if not requested
by the user:
$ rtla osnoise hist
Index CPU-000 CPU-001
1 8 5
2 5 9
3 1 2
4 6 1
5 2 1
6 0 1
8 1 1
12 0 1
14 1 0
15 1 0
over: 0 0
count: 25 21
min: 1 1
avg: 3.68 3.05
max: 15 12
rtla osnoise hit stop tracing
Fix the problem by setting the default value for all tracer options if
the user has not provided their own value.
For most of the options, it's enough to just drop the if clause checking
for the value being set. For cpus, "all" is used as the default value,
and for osnoise default period and runtime, default values of
the osnoise_data variable in trace_osnoise.c are used.
Cc: Luis Goncalves <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250320092500.101385-5-tglozar@redhat.com
Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode")
Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode")
Fixes: a828cd18bc4a ("rtla: Add timerlat tool and timelart top mode")
Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode")
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Reviewed-by: John Kacur <jkacur@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| a8122a63 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/osnoise: Set OSNOISE_WORKLOAD to true
If running rtla osnoise with NO_OSNOISE_WORKLOAD, it reports no samples:
$ echo NO_OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options
$ rtla osnoise h
rtla/osnoise: Set OSNOISE_WORKLOAD to true
If running rtla osnoise with NO_OSNOISE_WORKLOAD, it reports no samples:
$ echo NO_OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options
$ rtla osnoise hist -d 10s
Index
over: 0
count: 0
min: 0
avg: 0
max: 0
This situation can also happen when running rtla-osnoise after an
improperly exited rtla-timerlat run.
Set OSNOISE_WORKLOAD in rtla-osnoise, too, similarly to what we
already did for timerlat in commit 217f0b1e990e ("rtla/timerlat_top: Set
OSNOISE_WORKLOAD for kernel threads") and commit d8d866171a41
("rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threads").
Note that there is no user workload mode for rtla-osnoise yet, so
OSNOISE_WORKLOAD is always set to true.
Cc: Luis Goncalves <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250320092500.101385-4-tglozar@redhat.com
Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode")
Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode")
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Reviewed-by: John Kacur <jkacur@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 20d6b075 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla: Unify apply_config between top and hist
The functions osnoise_top_apply_config and osnoise_hist_apply_config, as
well as timerlat_top_apply_config and timerlat_hist_apply_config, are
mostly th
rtla: Unify apply_config between top and hist
The functions osnoise_top_apply_config and osnoise_hist_apply_config, as
well as timerlat_top_apply_config and timerlat_hist_apply_config, are
mostly the same.
Move common part from them into separate functions osnoise_apply_config
and timerlat_apply_config.
For rtla-timerlat, also unify params->user_hist and params->user_top
into one field called params->user_data, and move several fields used
only by timerlat-top into the top-only section of struct
timerlat_params.
Cc: Luis Goncalves <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250320092500.101385-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Reviewed-by: John Kacur <jkacur@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 025b2179 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/osnoise: Unify params struct
Instead of having separate structs osnoise_top_params and
osnoise_hist_params, use one struct osnoise_params for both.
This allows code using the structs to be sha
rtla/osnoise: Unify params struct
Instead of having separate structs osnoise_top_params and
osnoise_hist_params, use one struct osnoise_params for both.
This allows code using the structs to be shared between osnoise-top and
osnoise-hist.
Cc: Luis Goncalves <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250320092500.101385-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Reviewed-by: John Kacur <jkacur@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| c57c58a6 | 13-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla: Fix segfault in save_trace_to_file call
Running rtla with exit on threshold, but without saving trace leads to a
segmenetation fault:
$ rtla timerlat hist -T 10
...
Max timerlat IRQ latency f
rtla: Fix segfault in save_trace_to_file call
Running rtla with exit on threshold, but without saving trace leads to a
segmenetation fault:
$ rtla timerlat hist -T 10
...
Max timerlat IRQ latency from idle: 4.29 us in cpu 0
Segmentation fault
This is caused by null pointer deference in the call of
save_trace_to_file, which attempts to dereference an uninitialized
osnoise_tool variable:
save_trace_to_file(record->trace.inst, params->trace_output);
^ this is uninitialized if params->trace_output is
not set
Fix this by not attempting to dereference "record" if it is NULL and
passing NULL instead. As a safety measure, the first field is also
checked for NULL inside save_trace_to_file.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Cc: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/20250313141034.299117-1-tglozar@redhat.com
Fixes: dc4d4e7c72d1 ("rtla: Refactor save_trace_to_file")
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 814d051e | 26-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
tools/build: Use SYSTEM_BPFTOOL for system bpftool
The feature test for system bpftool uses BPFTOOL as the variable to set
its path, defaulting to just "bpftool" if not set by the user.
This confli
tools/build: Use SYSTEM_BPFTOOL for system bpftool
The feature test for system bpftool uses BPFTOOL as the variable to set
its path, defaulting to just "bpftool" if not set by the user.
This conflicts with selftests and a few other utilities, which expect
BPFTOOL to be set to the in-tree bpftool path by default. For example,
bpftool selftests fail to build:
$ make -C tools/testing/selftests/bpf/
make: Entering directory '/home/tglozar/dev/linux/tools/testing/selftests/bpf'
make: *** No rule to make target 'bpftool', needed by '/home/tglozar/dev/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h'. Stop.
make: Leaving directory '/home/tglozar/dev/linux/tools/testing/selftests/bpf'
Fix the problem by renaming the variable used for system bpftool from
BPFTOOL to SYSTEM_BPFTOOL, so that the new usage does not conflict with
the existing one of BPFTOOL.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Link: https://lore.kernel.org/20250326004018.248357-1-tglozar@redhat.com
Fixes: 8a635c3856dd ("tools/build: Add bpftool-skeletons feature test")
Closes: https://lore.kernel.org/linux-kernel/5df6968a-2e5f-468e-b457-fc201535dd4c@linux.ibm.com/
Reported-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Tested-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Suggested-by: Quentin Monnet <qmo@kernel.org>
Acked-by: Quentin Monnet <qmo@kernel.org>
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| dc4d4e7c | 19-Feb-2025 |
Costa Shulyupin <costa.shul@redhat.com> |
rtla: Refactor save_trace_to_file
The functions osnoise_hist_main(), osnoise_top_main(),
timerlat_hist_main(), and timerlat_top_main() are lengthy and contain
duplicated code.
Refactor by consolida
rtla: Refactor save_trace_to_file
The functions osnoise_hist_main(), osnoise_top_main(),
timerlat_hist_main(), and timerlat_top_main() are lengthy and contain
duplicated code.
Refactor by consolidating the duplicate lines into the
save_trace_to_file() function.
Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
Cc: John Kacur <jkacur@redhat.com>
Cc: "Luis Claudio R. Goncalves" <lgoncalv@redhat.com>
Cc: Eder Zulian <ezulian@redhat.com>
Cc: Dan Carpenter <dan.carpenter@linaro.org>
Cc: Gabriele Monaco <gmonaco@redhat.com>
Link: https://lore.kernel.org/20250219115138.406075-1-costa.shul@redhat.com
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Reviewed-by: Tomas Glozar <tglozar@redhat.com>
Tested-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 005682b4 | 18-Feb-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat: Test BPF mode
Using the RTLA_NO_BPF environmental variable, execute rtla-timerlat
tests both with and without BPF support to cover both paths.
If rtla is built without BPF or the osn
rtla/timerlat: Test BPF mode
Using the RTLA_NO_BPF environmental variable, execute rtla-timerlat
tests both with and without BPF support to cover both paths.
If rtla is built without BPF or the osnoise:timerlat_sample trace event
is not available, test only the non-BPF path.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Cc: Gabriele Monaco <gmonaco@redhat.com>
Cc: Clark Williams <williams@redhat.com>
Link: https://lore.kernel.org/20250218145859.27762-9-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 9a82a3fd | 18-Feb-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat_top: Use BPF to collect samples
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF
program fai
rtla/timerlat_top: Use BPF to collect samples
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF
program fails to load for whatever reason, rtla falls back to the old
implementation.
The collection of samples using the BPF program is fully self-contained
and requires no activity of the userspace part of rtla during the
measurement. Thus, rtla only pulls the summary from the BPF map and
displays it every second, improving the performance.
In --aa-only mode, the BPF program does not collect any data and only
signalizes the end of tracing to userspace. An optimization that re-used
the main trace instance for auto-analysis in aa-only mode was dropped, as
rtla no longer turns tracing on in the main trace instance, making it
useless for auto-analysis.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Cc: Gabriele Monaco <gmonaco@redhat.com>
Cc: Clark Williams <williams@redhat.com>
Link: https://lore.kernel.org/20250218145859.27762-8-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 18923806 | 18-Feb-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat_top: Move divisor to update
Unlike timerlat-hist, timerlat-top applies the output divisor used to
set ns/us mode when printing results instead of applying it when
collecting the sample
rtla/timerlat_top: Move divisor to update
Unlike timerlat-hist, timerlat-top applies the output divisor used to
set ns/us mode when printing results instead of applying it when
collecting the samples.
Move the application of the divisor from timerlat_top_print into
timerlat_top_update to make it consistent with timerlat-hist.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Cc: Gabriele Monaco <gmonaco@redhat.com>
Cc: Clark Williams <williams@redhat.com>
Link: https://lore.kernel.org/20250218145859.27762-7-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| fd7925cb | 18-Feb-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat_hist: Use BPF to collect samples
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF
program fa
rtla/timerlat_hist: Use BPF to collect samples
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF
program fails to load for whatever reason, rtla falls back to the old
implementation.
The collection of samples using the BPF program is fully self-contained
and requires no activity of the userspace part of rtla during the
measurement. Thus, instead of waking up every second to collect samples,
rtla simply sleeps until woken up by a signal or threshold overflow.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Cc: Gabriele Monaco <gmonaco@redhat.com>
Cc: Clark Williams <williams@redhat.com>
Link: https://lore.kernel.org/20250218145859.27762-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| e34293dd | 18-Feb-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat: Add BPF skeleton to collect samples
Add BPF program that attaches to the osnoise:timerlat_sample tracepoint
and collects both the summary and the histogram (if requested) into BPF
map
rtla/timerlat: Add BPF skeleton to collect samples
Add BPF program that attaches to the osnoise:timerlat_sample tracepoint
and collects both the summary and the histogram (if requested) into BPF
maps (one map of each kind per context).
The program is designed to be used for both timerlat-top and
timerlat-hist. If using with timerlat-top, the "entries" parameter is
set to zero, which prevents the BPF program from recording histogram
entries. In that case, the maps for histograms do not have to be
created, as the BPF verifier will identify the code using them as
unreachable.
An IRQ or thread latency threshold might be supplied to stop recording
if hit, similar to the timerlat tracer threshold, which stops ftrace
tracing if hit. A BPF ringbuffer is used to signal threshold overflow to
userspace. In aa-only mode, this is the only function of the BPF
program.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Cc: Gabriele Monaco <gmonaco@redhat.com>
Cc: Clark Williams <williams@redhat.com>
Link: https://lore.kernel.org/20250218145859.27762-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
| 9dc3766e | 18-Feb-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla: Add optional dependency on BPF tooling
If tooling required for building BPF CO-RE skeletons is present (that
is, libbpf, clang with BPF CO-RE support, and bpftool), turn on
HAVE_BPF_SKEL flag.
rtla: Add optional dependency on BPF tooling
If tooling required for building BPF CO-RE skeletons is present (that
is, libbpf, clang with BPF CO-RE support, and bpftool), turn on
HAVE_BPF_SKEL flag.
Those requirements are similar to what perf requires, with the
difference of using system libbpf and bpftool instead of in-tree
versions.
rtla can be forcefully built without BPF skeleton support by setting
BUILD_BPF_SKEL=0 manually; in that case, a warning is displayed.
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Cc: Gabriele Monaco <gmonaco@redhat.com>
Cc: Clark Williams <williams@redhat.com>
Link: https://lore.kernel.org/20250218145859.27762-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|