587f05a8 | 12-Jun-2024 |
Luis Claudio R. Goncalves <lgoncalv@redhat.com> |
rtla/osnoise: Better report when histogram is empty
When osnoise hist does not observe any samples above the threshold, no entries are recorded and the final report shows empty entries for the usual
rtla/osnoise: Better report when histogram is empty
When osnoise hist does not observe any samples above the threshold, no entries are recorded and the final report shows empty entries for the usual statistics (count, min, max, avg):
[~]# osnoise hist -d 5s -T 500 # RTLA osnoise histogram # Time unit is microseconds (us) # Duration: 0 00:00:05 Index over: count: min: avg: max:
That could lead users to confusing interpretations of the results.
A simple solution is to report 0 for count and the statistics, making it clear that no noise (above the defined threshold) was observed:
[~]# osnoise hist -d 5s -T 500 # RTLA osnoise histogram # Time unit is microseconds (us) # Duration: 0 00:00:05 Index over: 0 count: 0 min: 0 avg: 0 max: 0
Link: https://lkml.kernel.org/r/Zml6JmH5cbS7-HfZ@uudg.org
Cc: Daniel Bristot de Oliveira <bristot@kernel.org> Cc: John Kacur <jkacur@redhat.com> Cc: Clark Williams <williams@redhat.com> Reviewed-by: John Kacur <jkacur@redhat.com> Signed-off-by: Luis Claudio R. Goncalves <lgoncalv@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
842fc5b8 | 15-May-2024 |
John Kacur <jkacur@redhat.com> |
rtla: Fix -t\--trace[=file]
The -t option has an optional argument. The usual case is for a short option to be specified without an '=' and for the long version to be specified with an '='
Various
rtla: Fix -t\--trace[=file]
The -t option has an optional argument. The usual case is for a short option to be specified without an '=' and for the long version to be specified with an '='
Various forms of this do not work as expected. For example: rtla timerlat hist -T50 -tfile.txt will result in a truncated file name of "ile.txt"
Another example is that the long form without the '=' will result in the default file name instead of the requested file name.
This patch properly parses the optional argument with and without '=' and with and without spaces for the short form.
This patch was also tested using -t and --trace without providing a file name both as the last requested option and with a following long and short option.
For example:
rtla timerlat hist -T50 -t -u rtla timerlat hist -T50 --trace -u
This fix is applied to both timerlat top and hist and to osnoise top and hist.
Here is the full testing for rtla timerlat hist. Before applying the patch
rtla timerlat hist -T50 -t=file.txt Works as expected, "file.txt"
rtla timerlat hist -T50 -tfile.txt Truncated file name "ile.txt"
rtla timerlat hist -T50 -t file.txt Default file name instead of file.txt
rtla timerlat hist -T50 --trace=file.txt Truncated file name "ile.txt"
rtla timerlat hist -T50 --trace file.txt Default file name "timerlat_trace.txt" instead of "file.txt"
After applying the patch:
rtla timerlat hist -T50 -t=file.txt Works as expected, "file.txt"
rtla timerlat hist -T50 -tfile.txt Works as expected, "file.txt"
rtla timerlat hist -T50 -t file.txt Works as expected, "file.txt"
rtla timerlat hist -T50 --trace=file.txt Works as expected, "file.txt"
rtla timerlat hist -T50 --trace file.txt Works as expected, "file.txt"
In addition the following tests were performed to make sure that the default file name worked as expected including with trailing options.
rtla timerlat hist -T50 -t Works as expected "timerlat_trace.txt"
rtla timerlat hist -T50 --trace Works as expected "timerlat_trace.txt"
rtla timerlat hist -T50 -t -u Works as expected "timerlat_trace.txt"
rtla timerlat hist -T50 --trace -u Works as expected "timerlat_trace.txt"
Link: https://lkml.kernel.org/r/20240515183024.59985-1-jkacur@redhat.com
Cc: Daniel Bristot de Oliveria <bristot@kernel.org> Signed-off-by: John Kacur <jkacur@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
01b05fc0 | 10-May-2024 |
John Kacur <jkacur@redhat.com> |
rtla/timerlat: Fix histogram report when a cpu count is 0
On short runs it is possible to get no samples on a cpu, like this:
# rtla timerlat hist -u -T50
Index IRQ-001 Thr-001 Usr-001
rtla/timerlat: Fix histogram report when a cpu count is 0
On short runs it is possible to get no samples on a cpu, like this:
# rtla timerlat hist -u -T50
Index IRQ-001 Thr-001 Usr-001 IRQ-002 Thr-002 Usr-002 2 1 0 0 0 0 0 33 0 1 0 0 0 0 36 0 0 1 0 0 0 49 0 0 0 1 0 0 52 0 0 0 0 1 0 over: 0 0 0 0 0 0 count: 1 1 1 1 1 0 min: 2 33 36 49 52 18446744073709551615 avg: 2 33 36 49 52 - max: 2 33 36 49 52 0 rtla timerlat hit stop tracing IRQ handler delay: (exit from idle) 48.21 us (91.09 %) IRQ latency: 49.11 us Timerlat IRQ duration: 2.17 us (4.09 %) Blocking thread: 1.01 us (1.90 %) swapper/2:0 1.01 us ------------------------------------------------------------------------ Thread latency: 52.93 us (100%)
Max timerlat IRQ latency from idle: 49.11 us in cpu 2
Note, the value 18446744073709551615 is the same as ~0.
Fix this by reporting no results for the min, avg and max if the count is 0.
Link: https://lkml.kernel.org/r/20240510190318.44295-1-jkacur@redhat.com
Cc: stable@vger.kernel.org Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Suggested-by: Daniel Bristot de Oliveria <bristot@kernel.org> Signed-off-by: John Kacur <jkacur@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
e9a4062e | 16-May-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla: Add --trace-buffer-size option
Add the option allow the users to set a different buffer size for the trace. For example, in large systems, the user might be interested on reducing the trace bu
rtla: Add --trace-buffer-size option
Add the option allow the users to set a different buffer size for the trace. For example, in large systems, the user might be interested on reducing the trace buffer to avoid large tracing files.
The buffer size is specified in kB, and it is only affecting the tracing instance.
The function trace_set_buffer_size() appears on libtracefs v1.6, so increase the minimum required version on Makefile.config.
Link: https://lkml.kernel.org/r/e7c9ca5b3865f28e131a49ec3b984fadf2d056c6.1715860611.git.bristot@kernel.org
Cc: Jonathan Corbet <corbet@lwn.net> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: John Kacur <jkacur@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
fb9e90a6 | 24-Apr-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla/timerlat: Make user-space threads the default
After ther -u addition, most of the known users are setting it. And it makes sense, as it adds more information, and inherits the default setup for
rtla/timerlat: Make user-space threads the default
After ther -u addition, most of the known users are setting it. And it makes sense, as it adds more information, and inherits the default setup for the threads - e.g., cgroups configs.
Thus, if the user-space interface is available, enable -u. Otherwise, use the in-kernel thread.
Add the -k option to allow the user to request kernel-threads.
Link: https://lkml.kernel.org/r/9241d3089de4091b124f780ed832a0e6646cadaa.1713968967.git.bristot@kernel.org
Cc: Jonathan Corbet <corbet@lwn.net> Cc: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
cdbf7196 | 24-Apr-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla: Add the --warm-up option
On many cases, the results right after the startup are different from the rest of the execution, biasing the results. For example, on osnoise, the scheduler might take
rtla: Add the --warm-up option
On many cases, the results right after the startup are different from the rest of the execution, biasing the results. For example, on osnoise, the scheduler might take some time to adapt to the new busy-loop workload.
Add the --warm-up <seconds> option, adding a warm-up phase (in seconds) where the workload is set, but the results are discarded.
Link: https://lkml.kernel.org/r/e682d5ce5af90f123bd13220f63d5c3d118a92be.1713968967.git.bristot@kernel.org
Cc: Jonathan Corbet <corbet@lwn.net> Cc: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
1462501c | 24-Apr-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla/timerlat: Add a summary for hist mode
Like on rtla timerlat top, add an overall summary at the bottom of timerlat hist. For instance:
# timerlat hist -c 0-1 -d 10s -E 20 # RTLA timerlat hi
rtla/timerlat: Add a summary for hist mode
Like on rtla timerlat top, add an overall summary at the bottom of timerlat hist. For instance:
# timerlat hist -c 0-1 -d 10s -E 20 # RTLA timerlat histogram # Time unit is microseconds (us) # Duration: 0 00:00:10 Index IRQ-000 Thr-000 IRQ-001 Thr-001 6 1 0 0 0 7 1 0 0 0 8 1 0 1 0 9 7 0 0 0 10 16 0 0 0 11 1 0 3 0 15 0 0 3 0 16 0 0 12 0 17 0 0 28 0 18 0 2 26 0 19 1 1 80 1 over: 9973 9998 9848 10000 count: 10001 10001 10001 10001 min: 6 18 8 19 avg: 185 204 95 113 max: 428 450 341 371 ALL: IRQ Thr count: 20002 20002 min: 6 18 avg: 140 159 max: 428 450
Link: https://lkml.kernel.org/r/a6bc06c798f72127edc57d1f99da8d57e1187cee.1713968967.git.bristot@kernel.org
Cc: Jonathan Corbet <corbet@lwn.net> Suggested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
285dcb76 | 24-Apr-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla/timerlat: Add a summary for top mode
While the per-cpu values are the results to take into consideration, the overall system values are also useful.
Add a summary at the bottom of rtla timerla
rtla/timerlat: Add a summary for top mode
While the per-cpu values are the results to take into consideration, the overall system values are also useful.
Add a summary at the bottom of rtla timerlat top showing the overall results. For instance:
Timer Latency 0 00:00:10 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max 0 #10003 | 113 19 150 441 | 134 35 170 459 1 #10003 | 63 8 99 462 | 84 15 119 481 2 #10003 | 3 2 89 396 | 21 8 108 414 3 #10002 | 206 11 210 394 | 223 21 228 415 ---------------|----------------------------------------|--------------------------------------- ALL #40011 e0 | 2 137 462 | 8 156 481
Link: https://lkml.kernel.org/r/5eb510d6faeb4ce745e09395196752df75a2dd1a.1713968967.git.bristot@kernel.org
Cc: Jonathan Corbet <corbet@lwn.net> Suggested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
f5c0cdad | 24-Apr-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla/timerlat: Use pretty formatting only on interactive tty
timerlat top does some background/font color formatting. While useful on terminal, it breaks the output on other formats. For example, wh
rtla/timerlat: Use pretty formatting only on interactive tty
timerlat top does some background/font color formatting. While useful on terminal, it breaks the output on other formats. For example, when piping the output for pastebin tools, the format strings are printed as characters. For instance:
[2;37;40m Timer Latency [0;0;0m 0 00:00:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) [2;30;47mCPU COUNT | cur min avg max | cur min avg max[0;0;0m 0 #1013 | 1 0 1 54 | 5 2 4 57 1 #1013 | 3 0 1 10 | 6 2 4 15
To avoid this problem, do the formatting only if running on a tty, and in !quiet mode.
Link: https://lkml.kernel.org/r/8288e1544ceab21557d5dda93a0f00339497c649.1713968967.git.bristot@kernel.org
Cc: Jonathan Corbet <corbet@lwn.net> Cc: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
a40e5e4d | 24-Apr-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla/auto-analysis: Replace \t with spaces
When copying timerlat auto-analysis from a terminal to some web pages or chats, the \t are being replaced with a single ' ' or ' ', breaking the output.
rtla/auto-analysis: Replace \t with spaces
When copying timerlat auto-analysis from a terminal to some web pages or chats, the \t are being replaced with a single ' ' or ' ', breaking the output.
For example: ## CPU 3 hit stop tracing, analyzing it ## IRQ handler delay: 1.30 us (0.11 %) IRQ latency: 1.90 us Timerlat IRQ duration: 3.00 us (0.24 %) Blocking thread: 1223.16 us (99.00 %) insync:4048 1223.16 us IRQ interference 4.93 us (0.40 %) local_timer:236 4.93 us ------------------------------------------------------------------------ Thread latency: 1235.47 us (100%)
Replace \t with spaces to avoid this problem.
Link: https://lkml.kernel.org/r/ec7ed2b2809c22ab0dfc8eb7c805ab9cddc4254a.1713968967.git.bristot@kernel.org
Cc: stable@vger.kernel.org Cc: Jonathan Corbet <corbet@lwn.net> Cc: Juri Lelli <juri.lelli@redhat.com> Fixes: 27e348b221f6 ("rtla/timerlat: Add auto-analysis core") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
a23c05fd | 06-Feb-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
tools/rtla: Add -U/--user-load option to timerlat
The timerlat tracer provides an interface for any application to wait for the timerlat's periodic wakeup. Currently, rtla timerlat uses it to dispat
tools/rtla: Add -U/--user-load option to timerlat
The timerlat tracer provides an interface for any application to wait for the timerlat's periodic wakeup. Currently, rtla timerlat uses it to dispatch its user-space workload (-u option).
But as the tracer interface is generic, rtla timerlat can also be used to monitor any workload that uses it. For example, a user might place their own workload to wait on the tracer interface, and monitor the results with rtla timerlat.
Add the -U option to rtla timerlat top and hist. With this option, rtla timerlat will not dispatch its workload but only setting up the system, waiting for a user to dispatch its workload.
The sample code in this patch is an example of python application that loops in the timerlat tracer fd.
To use it, dispatch:
# rtla timerlat -U
In a terminal, then run the python program on another terminal, specifying the CPU to run it. For example, setting on CPU 1:
#./timerlat_load.py 1
Then rtla timerlat will start printing the statistics of the ./timerlat_load.py app.
An interesting point is that the "Ret user Timer Latency" value is the overall response time of the load. The sample load does a memory copy to exemplify that.
The stop tracing options on rtla timerlat works in this setup as well, including auto analysis.
Link: https://lkml.kernel.org/r/36e6bcf18fe15c7601048fd4c65aeb193c502cc8.1707229706.git.bristot@kernel.org
Cc: Jonathan Corbet <corbet@lwn.net> Cc: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
b5f31936 | 03-Feb-2024 |
John Kacur <jkacur@redhat.com> |
tools/rtla: Exit with EXIT_SUCCESS when help is invoked
Fix rtla so that the following commands exit with 0 when help is invoked
rtla osnoise top -h rtla osnoise hist -h rtla timerlat top -h rtla t
tools/rtla: Exit with EXIT_SUCCESS when help is invoked
Fix rtla so that the following commands exit with 0 when help is invoked
rtla osnoise top -h rtla osnoise hist -h rtla timerlat top -h rtla timerlat hist -h
Link: https://lore.kernel.org/linux-trace-devel/20240203001607.69703-1-jkacur@redhat.com
Cc: stable@vger.kernel.org Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Signed-off-by: John Kacur <jkacur@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
14f08c97 | 07-Feb-2024 |
limingming3 <limingming890315@gmail.com> |
tools/rtla: Replace setting prio with nice for SCHED_OTHER
Since the sched_priority for SCHED_OTHER is always 0, it makes no sence to set it. Setting nice for SCHED_OTHER seems more meaningful.
Lin
tools/rtla: Replace setting prio with nice for SCHED_OTHER
Since the sched_priority for SCHED_OTHER is always 0, it makes no sence to set it. Setting nice for SCHED_OTHER seems more meaningful.
Link: https://lkml.kernel.org/r/20240207065142.1753909-1-limingming3@lixiang.com
Cc: stable@vger.kernel.org Fixes: b1696371d865 ("rtla: Helper functions for rtla") Signed-off-by: limingming3 <limingming3@lixiang.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
084ce16d | 06-Feb-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
tools/rtla: Remove unused sched_getattr() function
Clang is reporting:
$ make HOSTCC=clang CC=clang LLVM_IAS=1 [...] clang -O -g -DVERSION=\"6.8.0-rc3\" -flto=auto -fexceptions -fstack-protector-st
tools/rtla: Remove unused sched_getattr() function
Clang is reporting:
$ make HOSTCC=clang CC=clang LLVM_IAS=1 [...] clang -O -g -DVERSION=\"6.8.0-rc3\" -flto=auto -fexceptions -fstack-protector-strong -fasynchronous-unwind-tables -fstack-clash-protection -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS $(pkg-config --cflags libtracefs) -c -o src/utils.o src/utils.c src/utils.c:241:19: warning: unused function 'sched_getattr' [-Wunused-function] 241 | static inline int sched_getattr(pid_t pid, struct sched_attr *attr, | ^~~~~~~~~~~~~ 1 warning generated.
Which is correct, so remove the unused function.
Link: https://lkml.kernel.org/r/eaed7ba122c4ae88ce71277c824ef41cbf789385.1707217097.git.bristot@kernel.org
Cc: stable@vger.kernel.org Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Nathan Chancellor <nathan@kernel.org> Cc: Nick Desaulniers <ndesaulniers@google.com> Cc: Bill Wendling <morbo@google.com> Cc: Justin Stitt <justinstitt@google.com> Cc: Donald Zickus <dzickus@redhat.com> Fixes: b1696371d865 ("rtla: Helper functions for rtla") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
30369084 | 06-Feb-2024 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
tools/rtla: Fix clang warning about mount_point var size
clang is reporting this warning:
$ make HOSTCC=clang CC=clang LLVM_IAS=1 [...] clang -O -g -DVERSION=\"6.8.0-rc3\" -flto=auto -fexceptions
tools/rtla: Fix clang warning about mount_point var size
clang is reporting this warning:
$ make HOSTCC=clang CC=clang LLVM_IAS=1 [...] clang -O -g -DVERSION=\"6.8.0-rc3\" -flto=auto -fexceptions -fstack-protector-strong -fasynchronous-unwind-tables -fstack-clash-protection -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS $(pkg-config --cflags libtracefs) -c -o src/utils.o src/utils.c
src/utils.c:548:66: warning: 'fscanf' may overflow; destination buffer in argument 3 has size 1024, but the corresponding specifier may require size 1025 [-Wfortify-source] 548 | while (fscanf(fp, "%*s %" STR(MAX_PATH) "s %99s %*s %*d %*d\n", mount_point, type) == 2) { | ^
Increase mount_point variable size to MAX_PATH+1 to avoid the overflow.
Link: https://lkml.kernel.org/r/1b46712e93a2f4153909514a36016959dcc4021c.1707217097.git.bristot@kernel.org
Cc: stable@vger.kernel.org Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Nathan Chancellor <nathan@kernel.org> Cc: Nick Desaulniers <ndesaulniers@google.com> Cc: Bill Wendling <morbo@google.com> Cc: Justin Stitt <justinstitt@google.com> Cc: Donald Zickus <dzickus@redhat.com> Fixes: a957cbc02531 ("rtla: Add -C cgroup support") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
e8c44d3b | 15-Sep-2023 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla/timerlat: Do not stop user-space if a cpu is offline
If no CPU list is passed, timerlat in user-space will dispatch one thread per sysconf(_SC_NPROCESSORS_CONF). However, not all CPU might be a
rtla/timerlat: Do not stop user-space if a cpu is offline
If no CPU list is passed, timerlat in user-space will dispatch one thread per sysconf(_SC_NPROCESSORS_CONF). However, not all CPU might be available, for instance, if HT is disabled.
Currently, rtla timerlat is stopping the session if an user-space thread cannot set affinity to a CPU, or if a running user-space thread is killed. However, this is too restrictive.
So, reduce the error to a debug message, and rtla timerlat run as long as there is at least one user-space thread alive.
Link: https://lore.kernel.org/lkml/59cf2c882900ab7de91c6ee33b382ac7fa6b4ed0.1694781909.git.bristot@kernel.org
Fixes: cdca4f4e5e8e ("rtla/timerlat_top: Add timerlat user-space support") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|
301deca0 | 04-Aug-2023 |
Daniel Bristot de Oliveira <bristot@kernel.org> |
rtla/timerlat_aa: Fix previous IRQ delay for IRQs that happens after thread sample
timerlat auto-analysis takes note of all IRQs, before or after the execution of the timerlat thread.
Because we ca
rtla/timerlat_aa: Fix previous IRQ delay for IRQs that happens after thread sample
timerlat auto-analysis takes note of all IRQs, before or after the execution of the timerlat thread.
Because we cannot go backward in the trace (we will fix it when moving to trace-cmd lib?), timerlat aa take note of the last IRQ execution in the waiting for the IRQ state, and then print it if it is executed after the expected timer IRQ starting time.
After the thread sample, the timerlat starts recording the next IRQs as "previous" irq for the next occurrence.
However, if an IRQ happens after the thread measurement but before the tracing stops, it is classified as a previous IRQ. That is not wrong, as it can be "previous" for the subsequent activation. What is wrong is considering it as a potential source for the last activation.
Ignore the IRQ interference that happens after the IRQ starting time for now. A future improvement for timerlat can be either keeping a list of previous IRQ execution or using the trace-cmd library. Still, it requires further investigation - it is a new feature.
Link: https://lore.kernel.org/lkml/a44a3f5c801dcc697bacf7325b65d4a5b0460537.1691162043.git.bristot@kernel.org
Fixes: 27e348b221f6 ("rtla/timerlat: Add auto-analysis core") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
show more ...
|