7b128f1d | 08-Aug-2025 |
Tao Chen <chen.dylane@linux.dev> |
rtla: Check pkg-config install
The tool pkg-config used to check libtraceevent and libtracefs, if not installed, it will report the libs not found, even though they have already been installed.
Bef
rtla: Check pkg-config install
The tool pkg-config used to check libtraceevent and libtracefs, if not installed, it will report the libs not found, even though they have already been installed.
Before: libtraceevent is missing. Please install libtraceevent-dev/libtraceevent-devel libtracefs is missing. Please install libtracefs-dev/libtracefs-devel
After: Makefile.config:10: *** Error: pkg-config needed by libtraceevent/libtracefs is missing on this system, please install it.
Link: https://lore.kernel.org/20250808040527.2036023-2-chen.dylane@linux.dev Fixes: 01474dc706ca ("tools/rtla: Use tools/build makefiles to build rtla") Signed-off-by: Tao Chen <chen.dylane@linux.dev> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
a80db1f8 | 25-Jul-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Test timerlat -P option using actions
The -P option is used to set priority of osnoise and timerlat threads.
Extend the test for -P with --on-threshold calling a script that looks for r
rtla/tests: Test timerlat -P option using actions
The -P option is used to set priority of osnoise and timerlat threads.
Extend the test for -P with --on-threshold calling a script that looks for running timerlat threads and checks if their priority is set correctly.
As --on-threshold is only supported by timerlat at the moment, this is only implemented there so far.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Link: https://lore.kernel.org/20250725133817.59237-3-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
892ae5f8 | 25-Jul-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Add grep checks for base test cases
Checking for patterns in rtla output with grep was added to test rtla actions. Add grep checks also for base tests where applicable.
Also fix trace e
rtla/tests: Add grep checks for base test cases
Checking for patterns in rtla output with grep was added to test rtla actions. Add grep checks also for base tests where applicable.
Also fix trace event histogram trigger check to use the correct syntax for the command-line option so that the test passes with the grep check.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Link: https://lore.kernel.org/20250725133817.59237-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
04f83716 | 26-Jun-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Limit duration to maximum of 10s
Many of the original rtla tests included durations of 1 minute and 30 seconds. Experience has shown this is unnecessary, since 10 seconds as waiting time
rtla/tests: Limit duration to maximum of 10s
Many of the original rtla tests included durations of 1 minute and 30 seconds. Experience has shown this is unnecessary, since 10 seconds as waiting time for samples to appear.
Change duration of all rtla tests to at most 10 seconds. This speeds up testing significantly.
Before: $ make check All tests successful. Files=3, Tests=54, 536 wallclock secs ( 0.03 usr 0.00 sys + 20.31 cusr 22.02 csys = 42.36 CPU) Result: PASS
After: $ make check ... All tests successful. Files=3, Tests=54, 196 wallclock secs ( 0.03 usr 0.01 sys + 20.28 cusr 20.68 csys = 41.00 CPU) Result: PASS
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Cc: Crystal Wood <crwood@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250626123405.1496931-9-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
4e26f84a | 26-Jun-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Add tests for actions
Add a bunch of tests covering most of both --on-threshold and --on-end. Parts sensitive to implementation of hist/top are tested for both.
Cc: John Kacur <jkacur@r
rtla/tests: Add tests for actions
Add a bunch of tests covering most of both --on-threshold and --on-end. Parts sensitive to implementation of hist/top are tested for both.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Cc: Crystal Wood <crwood@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250626123405.1496931-8-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
916a9c5b | 26-Jun-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Check rtla output with grep
Add argument to the check command in the test suite that takes a regular expression that the output of rtla command is checked against. This allows testing fo
rtla/tests: Check rtla output with grep
Add argument to the check command in the test suite that takes a regular expression that the output of rtla command is checked against. This allows testing for specific information in rtla output in addition to checking the return value.
Two minor improvements are included: running rtla with "eval" so that arguments with spaces can be passed to it via shell quotations, and the stdout of pushd and popd is suppressed to clean up the test output.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Cc: Crystal Wood <crwood@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250626123405.1496931-7-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
3aadb65d | 26-Jun-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat: Add action on end feature
Implement actions on end next to actions on threshold. A new option, --on-end is added, parallel to --on-threshold. Instead of being executed whenever a late
rtla/timerlat: Add action on end feature
Implement actions on end next to actions on threshold. A new option, --on-end is added, parallel to --on-threshold. Instead of being executed whenever a latency threshold is reached, it is executed at the end of the measurement.
For example:
$ rtla timerlat hist -d 5s --on-end trace
will save the trace output at the end.
All actions supported by --on-threshold are also supported by --on-end, except for continue, which does nothing with --on-end.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Cc: Crystal Wood <crwood@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250626123405.1496931-6-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
8d933d5c | 26-Jun-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat: Add continue action
Introduce option to resume tracing after a latency threshold overflow. The option is implemented as an action named "continue".
Example: $ rtla timerlat top -q -T
rtla/timerlat: Add continue action
Introduce option to resume tracing after a latency threshold overflow. The option is implemented as an action named "continue".
Example: $ rtla timerlat top -q -T 200 -d 1s --on-threshold \ exec,command="echo Threshold" --on-threshold continue Threshold Threshold Threshold Timer Latency ...
The feature is supported for both hist and top. After the continue action is executed, processing of the list of actions is stopped and tracing is resumed.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Cc: Crystal Wood <crwood@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250626123405.1496931-5-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
3b78670e | 26-Jun-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat_bpf: Allow resuming tracing
Currently, rtla-timerlat BPF program uses a global variable stored in a .bss section to store whether tracing has been stopped.
Move the information to a s
rtla/timerlat_bpf: Allow resuming tracing
Currently, rtla-timerlat BPF program uses a global variable stored in a .bss section to store whether tracing has been stopped.
Move the information to a separate map, so that it is easily writable from userspace, and add a function that clears the value, resuming tracing after it has been stopped.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Cc: Crystal Wood <crwood@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250626123405.1496931-4-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
6ea082b1 | 26-Jun-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/timerlat: Add action on threshold feature
Extend the functionality provided by the -t/--trace option, which triggers saving the contents of a tracefs buffer after tracing is stopped, to support
rtla/timerlat: Add action on threshold feature
Extend the functionality provided by the -t/--trace option, which triggers saving the contents of a tracefs buffer after tracing is stopped, to support implementing arbitrary actions.
A new option, --on-threshold, is added, taking an argument that further specifies the action. Actions added in this patch are:
- trace[,file=<filename>]: Saves tracefs buffer, optionally taking a filename. - signal,num=<sig>,pid=<pid>: Sends signal to process. "parent" might be specified instead of number to send signal to parent process. - shell,command=<command>: Execute shell command.
Multiple actions may be specified and will be executed in order, including multiple actions of the same type. Trace output requested via -t and -a now adds a trace action to the end of the list.
If an action fails, the following actions are not executed. For example, this command:
$ rtla timerlat -T 20 --on-threshold trace \ --on-threshold shell,command="grep ipi_send timerlat_trace.txt" \ --on-threshold signal,num=2,pid=parent
will send signal 2 (SIGINT) to parent process, but only if saved trace contains the text "ipi_send".
This way, the feature can be used for flexible reactions on latency spikes, and allows combining rtla with other tooling like perf.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Chang Yin <cyin@redhat.com> Cc: Costa Shulyupin <costa.shul@redhat.com> Cc: Crystal Wood <crwood@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250626123405.1496931-3-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
8020361d | 30-Apr-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla: Define _GNU_SOURCE in timerlat_bpf.c
Newer versions of glibc include a definition of struct sched_attr in bits/sched.h (included through sched.h which is included by rtla). Commit 0eecee340672
rtla: Define _GNU_SOURCE in timerlat_bpf.c
Newer versions of glibc include a definition of struct sched_attr in bits/sched.h (included through sched.h which is included by rtla). Commit 0eecee340672 ("tools/rtla: fix collision with glibc sched_attr/sched_set_attr") has modified the definition of struct sched_attr in utils.h, so that it is only applied with older versions of glibc that do not define it, in order to prevent build failure.
The definition in bits/sched.h depends on _GNU_SOURCE. timerlat_bpf.c does not define _GNU_SOURCE, making it fall back to the definition in utils.h. The latter has two fields less, leading to shifted offsets of struct timerlat_params in timerlat_bpf_init.
Because of the shift, timerlat_bpf_init incorrectly reads params->entries as 0 for timerlat-hist and disables the creation of histogram maps, causing breakage in BPF sample collection mode:
$ rtla timerlat hist -d 1s Error pulling BPF data
Fix the issue by also defining _GNU_SOURCE in timerlat_bpf.c.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250430144651.621766-1-tglozar@redhat.com Fixes: e34293ddcebd ("rtla/timerlat: Add BPF skeleton to collect samples") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
6a38c51a | 22-Apr-2025 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
rtla: Define __NR_sched_setattr for LoongArch
When executing "make -C tools/tracing/rtla" on LoongArch, there exists the following error:
src/utils.c:237:24: error: '__NR_sched_setattr' undeclare
rtla: Define __NR_sched_setattr for LoongArch
When executing "make -C tools/tracing/rtla" on LoongArch, there exists the following error:
src/utils.c:237:24: error: '__NR_sched_setattr' undeclared
Just define __NR_sched_setattr for LoongArch if not exist.
Link: https://lore.kernel.org/20250422074917.25771-1-yangtiezhu@loongson.cn Reported-by: Haiyong Sun <sunhaiyong@loongson.cn> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
73203269 | 21-Mar-2025 |
John Kacur <jkacur@redhat.com> |
rtla: Add the ability to create ctags and etags
- Add the ability to create and remove ctags and etags, using the following make tags make TAGS make tags_clean
- fix a comment in Makefile.rtla with
rtla: Add the ability to create ctags and etags
- Add the ability to create and remove ctags and etags, using the following make tags make TAGS make tags_clean
- fix a comment in Makefile.rtla with the correct spelling and don't imply that the ability to create an rtla tarball will be removed
Cc: Tomas Glozar <tglozar@redhat.com> Cc: "Luis Claudio R . Goncalves" <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250321175053.29048-1-jkacur@redhat.com Signed-off-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
a86150f3 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Test setting default options
Add function to test engine to test with pre-set osnoise options, and use it to test whether osnoise period (as an example) is set correctly.
The test works
rtla/tests: Test setting default options
Add function to test engine to test with pre-set osnoise options, and use it to test whether osnoise period (as an example) is set correctly.
The test works by pre-setting a high period of 10 minutes and stop on threshold. Thus, it is easy to check whether rtla is properly resetting the period to default: if it is, the test will complete on time, since the first sample will overflow the threshold. If not, it will time out.
Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-7-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
6c618272 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/tests: Reset osnoise options before check
Remove any dangling tracing instances from previous improperly exited runs of rtla, and reset osnoise options to default before running a test case.
T
rtla/tests: Reset osnoise options before check
Remove any dangling tracing instances from previous improperly exited runs of rtla, and reset osnoise options to default before running a test case.
This ensures that the test results are deterministic. Specific test cases checked that rtla behaves correctly even when the tracer state is not clean will be added later.
Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-6-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
0122938a | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla: Always set all tracer options
rtla currently only sets tracer options that are explicitly set by the user, with the exception of OSNOISE_WORKLOAD.
This leads to improper behavior in case rtla
rtla: Always set all tracer options
rtla currently only sets tracer options that are explicitly set by the user, with the exception of OSNOISE_WORKLOAD.
This leads to improper behavior in case rtla is run with those options not set to the default value. rtla does reset them to the original value upon exiting, but that does not protect it from starting with non-default values set either by an improperly exited rtla or by another user of the tracers.
For example, after running this command:
$ echo 1 > /sys/kernel/tracing/osnoise/stop_tracing_us
all runs of rtla will stop at the 1us threshold, even if not requested by the user:
$ rtla osnoise hist Index CPU-000 CPU-001 1 8 5 2 5 9 3 1 2 4 6 1 5 2 1 6 0 1 8 1 1 12 0 1 14 1 0 15 1 0 over: 0 0 count: 25 21 min: 1 1 avg: 3.68 3.05 max: 15 12 rtla osnoise hit stop tracing
Fix the problem by setting the default value for all tracer options if the user has not provided their own value.
For most of the options, it's enough to just drop the if clause checking for the value being set. For cpus, "all" is used as the default value, and for osnoise default period and runtime, default values of the osnoise_data variable in trace_osnoise.c are used.
Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-5-tglozar@redhat.com Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode") Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode") Fixes: a828cd18bc4a ("rtla: Add timerlat tool and timelart top mode") Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
a8122a63 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/osnoise: Set OSNOISE_WORKLOAD to true
If running rtla osnoise with NO_OSNOISE_WORKLOAD, it reports no samples:
$ echo NO_OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options $ rtla osnoise h
rtla/osnoise: Set OSNOISE_WORKLOAD to true
If running rtla osnoise with NO_OSNOISE_WORKLOAD, it reports no samples:
$ echo NO_OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options $ rtla osnoise hist -d 10s Index over: 0 count: 0 min: 0 avg: 0 max: 0
This situation can also happen when running rtla-osnoise after an improperly exited rtla-timerlat run.
Set OSNOISE_WORKLOAD in rtla-osnoise, too, similarly to what we already did for timerlat in commit 217f0b1e990e ("rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threads") and commit d8d866171a41 ("rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threads").
Note that there is no user workload mode for rtla-osnoise yet, so OSNOISE_WORKLOAD is always set to true.
Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-4-tglozar@redhat.com Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode") Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
20d6b075 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla: Unify apply_config between top and hist
The functions osnoise_top_apply_config and osnoise_hist_apply_config, as well as timerlat_top_apply_config and timerlat_hist_apply_config, are mostly th
rtla: Unify apply_config between top and hist
The functions osnoise_top_apply_config and osnoise_hist_apply_config, as well as timerlat_top_apply_config and timerlat_hist_apply_config, are mostly the same.
Move common part from them into separate functions osnoise_apply_config and timerlat_apply_config.
For rtla-timerlat, also unify params->user_hist and params->user_top into one field called params->user_data, and move several fields used only by timerlat-top into the top-only section of struct timerlat_params.
Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-3-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|
025b2179 | 20-Mar-2025 |
Tomas Glozar <tglozar@redhat.com> |
rtla/osnoise: Unify params struct
Instead of having separate structs osnoise_top_params and osnoise_hist_params, use one struct osnoise_params for both.
This allows code using the structs to be sha
rtla/osnoise: Unify params struct
Instead of having separate structs osnoise_top_params and osnoise_hist_params, use one struct osnoise_params for both.
This allows code using the structs to be shared between osnoise-top and osnoise-hist.
Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250320092500.101385-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
show more ...
|