1======================== 2ftrace - Function Tracer 3======================== 4 5Copyright 2008 Red Hat Inc. 6 7:Author: Steven Rostedt <srostedt@redhat.com> 8:License: The GNU Free Documentation License, Version 1.2 9 (dual licensed under the GPL v2) 10:Original Reviewers: Elias Oltmanns, Randy Dunlap, Andrew Morton, 11 John Kacur, and David Teigland. 12 13- Written for: 2.6.28-rc2 14- Updated for: 3.10 15- Updated for: 4.13 - Copyright 2017 VMware Inc. Steven Rostedt 16- Converted to rst format - Changbin Du <changbin.du@intel.com> 17 18Introduction 19------------ 20 21Ftrace is an internal tracer designed to help out developers and 22designers of systems to find what is going on inside the kernel. 23It can be used for debugging or analyzing latencies and 24performance issues that take place outside of user-space. 25 26Although ftrace is typically considered the function tracer, it 27is really a framework of several assorted tracing utilities. 28There's latency tracing to examine what occurs between interrupts 29disabled and enabled, as well as for preemption and from a time 30a task is woken to the task is actually scheduled in. 31 32One of the most common uses of ftrace is the event tracing. 33Throughout the kernel is hundreds of static event points that 34can be enabled via the tracefs file system to see what is 35going on in certain parts of the kernel. 36 37See events.rst for more information. 38 39 40Implementation Details 41---------------------- 42 43See Documentation/trace/ftrace-design.rst for details for arch porters and such. 44 45 46The File System 47--------------- 48 49Ftrace uses the tracefs file system to hold the control files as 50well as the files to display output. 51 52When tracefs is configured into the kernel (which selecting any ftrace 53option will do) the directory /sys/kernel/tracing will be created. To mount 54this directory, you can add to your /etc/fstab file:: 55 56 tracefs /sys/kernel/tracing tracefs defaults 0 0 57 58Or you can mount it at run time with:: 59 60 mount -t tracefs nodev /sys/kernel/tracing 61 62For quicker access to that directory you may want to make a soft link to 63it:: 64 65 ln -s /sys/kernel/tracing /tracing 66 67.. attention:: 68 69 Before 4.1, all ftrace tracing control files were within the debugfs 70 file system, which is typically located at /sys/kernel/debug/tracing. 71 For backward compatibility, when mounting the debugfs file system, 72 the tracefs file system will be automatically mounted at: 73 74 /sys/kernel/debug/tracing 75 76 All files located in the tracefs file system will be located in that 77 debugfs file system directory as well. 78 79.. attention:: 80 81 Any selected ftrace option will also create the tracefs file system. 82 The rest of the document will assume that you are in the ftrace directory 83 (cd /sys/kernel/tracing) and will only concentrate on the files within that 84 directory and not distract from the content with the extended 85 "/sys/kernel/tracing" path name. 86 87That's it! (assuming that you have ftrace configured into your kernel) 88 89After mounting tracefs you will have access to the control and output files 90of ftrace. Here is a list of some of the key files: 91 92 93 Note: all time values are in microseconds. 94 95 current_tracer: 96 97 This is used to set or display the current tracer 98 that is configured. Changing the current tracer clears 99 the ring buffer content as well as the "snapshot" buffer. 100 101 available_tracers: 102 103 This holds the different types of tracers that 104 have been compiled into the kernel. The 105 tracers listed here can be configured by 106 echoing their name into current_tracer. 107 108 tracing_on: 109 110 This sets or displays whether writing to the trace 111 ring buffer is enabled. Echo 0 into this file to disable 112 the tracer or 1 to enable it. Note, this only disables 113 writing to the ring buffer, the tracing overhead may 114 still be occurring. 115 116 The kernel function tracing_off() can be used within the 117 kernel to disable writing to the ring buffer, which will 118 set this file to "0". User space can re-enable tracing by 119 echoing "1" into the file. 120 121 Note, the function and event trigger "traceoff" will also 122 set this file to zero and stop tracing. Which can also 123 be re-enabled by user space using this file. 124 125 trace: 126 127 This file holds the output of the trace in a human 128 readable format (described below). Opening this file for 129 writing with the O_TRUNC flag clears the ring buffer content. 130 Note, this file is not a consumer. If tracing is off 131 (no tracer running, or tracing_on is zero), it will produce 132 the same output each time it is read. When tracing is on, 133 it may produce inconsistent results as it tries to read 134 the entire buffer without consuming it. 135 136 trace_pipe: 137 138 The output is the same as the "trace" file but this 139 file is meant to be streamed with live tracing. 140 Reads from this file will block until new data is 141 retrieved. Unlike the "trace" file, this file is a 142 consumer. This means reading from this file causes 143 sequential reads to display more current data. Once 144 data is read from this file, it is consumed, and 145 will not be read again with a sequential read. The 146 "trace" file is static, and if the tracer is not 147 adding more data, it will display the same 148 information every time it is read. 149 150 trace_options: 151 152 This file lets the user control the amount of data 153 that is displayed in one of the above output 154 files. Options also exist to modify how a tracer 155 or events work (stack traces, timestamps, etc). 156 157 options: 158 159 This is a directory that has a file for every available 160 trace option (also in trace_options). Options may also be set 161 or cleared by writing a "1" or "0" respectively into the 162 corresponding file with the option name. 163 164 tracing_max_latency: 165 166 Some of the tracers record the max latency. 167 For example, the maximum time that interrupts are disabled. 168 The maximum time is saved in this file. The max trace will also be 169 stored, and displayed by "trace". A new max trace will only be 170 recorded if the latency is greater than the value in this file 171 (in microseconds). 172 173 By echoing in a time into this file, no latency will be recorded 174 unless it is greater than the time in this file. 175 176 tracing_thresh: 177 178 Some latency tracers will record a trace whenever the 179 latency is greater than the number in this file. 180 Only active when the file contains a number greater than 0. 181 (in microseconds) 182 183 buffer_size_kb: 184 185 This sets or displays the number of kilobytes each CPU 186 buffer holds. By default, the trace buffers are the same size 187 for each CPU. The displayed number is the size of the 188 CPU buffer and not total size of all buffers. The 189 trace buffers are allocated in pages (blocks of memory 190 that the kernel uses for allocation, usually 4 KB in size). 191 A few extra pages may be allocated to accommodate buffer management 192 meta-data. If the last page allocated has room for more bytes 193 than requested, the rest of the page will be used, 194 making the actual allocation bigger than requested or shown. 195 ( Note, the size may not be a multiple of the page size 196 due to buffer management meta-data. ) 197 198 Buffer sizes for individual CPUs may vary 199 (see "per_cpu/cpu0/buffer_size_kb" below), and if they do 200 this file will show "X". 201 202 buffer_total_size_kb: 203 204 This displays the total combined size of all the trace buffers. 205 206 free_buffer: 207 208 If a process is performing tracing, and the ring buffer should be 209 shrunk "freed" when the process is finished, even if it were to be 210 killed by a signal, this file can be used for that purpose. On close 211 of this file, the ring buffer will be resized to its minimum size. 212 Having a process that is tracing also open this file, when the process 213 exits its file descriptor for this file will be closed, and in doing so, 214 the ring buffer will be "freed". 215 216 It may also stop tracing if disable_on_free option is set. 217 218 tracing_cpumask: 219 220 This is a mask that lets the user only trace on specified CPUs. 221 The format is a hex string representing the CPUs. 222 223 set_ftrace_filter: 224 225 When dynamic ftrace is configured in (see the 226 section below "dynamic ftrace"), the code is dynamically 227 modified (code text rewrite) to disable calling of the 228 function profiler (mcount). This lets tracing be configured 229 in with practically no overhead in performance. This also 230 has a side effect of enabling or disabling specific functions 231 to be traced. Echoing names of functions into this file 232 will limit the trace to only those functions. 233 This influences the tracers "function" and "function_graph" 234 and thus also function profiling (see "function_profile_enabled"). 235 236 The functions listed in "available_filter_functions" are what 237 can be written into this file. 238 239 This interface also allows for commands to be used. See the 240 "Filter commands" section for more details. 241 242 As a speed up, since processing strings can be quite expensive 243 and requires a check of all functions registered to tracing, instead 244 an index can be written into this file. A number (starting with "1") 245 written will instead select the same corresponding at the line position 246 of the "available_filter_functions" file. 247 248 set_ftrace_notrace: 249 250 This has an effect opposite to that of 251 set_ftrace_filter. Any function that is added here will not 252 be traced. If a function exists in both set_ftrace_filter 253 and set_ftrace_notrace, the function will _not_ be traced. 254 255 set_ftrace_pid: 256 257 Have the function tracer only trace the threads whose PID are 258 listed in this file. 259 260 If the "function-fork" option is set, then when a task whose 261 PID is listed in this file forks, the child's PID will 262 automatically be added to this file, and the child will be 263 traced by the function tracer as well. This option will also 264 cause PIDs of tasks that exit to be removed from the file. 265 266 set_ftrace_notrace_pid: 267 268 Have the function tracer ignore threads whose PID are listed in 269 this file. 270 271 If the "function-fork" option is set, then when a task whose 272 PID is listed in this file forks, the child's PID will 273 automatically be added to this file, and the child will not be 274 traced by the function tracer as well. This option will also 275 cause PIDs of tasks that exit to be removed from the file. 276 277 If a PID is in both this file and "set_ftrace_pid", then this 278 file takes precedence, and the thread will not be traced. 279 280 set_event_pid: 281 282 Have the events only trace a task with a PID listed in this file. 283 Note, sched_switch and sched_wake_up will also trace events 284 listed in this file. 285 286 To have the PIDs of children of tasks with their PID in this file 287 added on fork, enable the "event-fork" option. That option will also 288 cause the PIDs of tasks to be removed from this file when the task 289 exits. 290 291 set_event_notrace_pid: 292 293 Have the events not trace a task with a PID listed in this file. 294 Note, sched_switch and sched_wakeup will trace threads not listed 295 in this file, even if a thread's PID is in the file if the 296 sched_switch or sched_wakeup events also trace a thread that should 297 be traced. 298 299 To have the PIDs of children of tasks with their PID in this file 300 added on fork, enable the "event-fork" option. That option will also 301 cause the PIDs of tasks to be removed from this file when the task 302 exits. 303 304 set_graph_function: 305 306 Functions listed in this file will cause the function graph 307 tracer to only trace these functions and the functions that 308 they call. (See the section "dynamic ftrace" for more details). 309 Note, set_ftrace_filter and set_ftrace_notrace still affects 310 what functions are being traced. 311 312 set_graph_notrace: 313 314 Similar to set_graph_function, but will disable function graph 315 tracing when the function is hit until it exits the function. 316 This makes it possible to ignore tracing functions that are called 317 by a specific function. 318 319 available_filter_functions: 320 321 This lists the functions that ftrace has processed and can trace. 322 These are the function names that you can pass to 323 "set_ftrace_filter", "set_ftrace_notrace", 324 "set_graph_function", or "set_graph_notrace". 325 (See the section "dynamic ftrace" below for more details.) 326 327 dyn_ftrace_total_info: 328 329 This file is for debugging purposes. The number of functions that 330 have been converted to nops and are available to be traced. 331 332 enabled_functions: 333 334 This file is more for debugging ftrace, but can also be useful 335 in seeing if any function has a callback attached to it. 336 Not only does the trace infrastructure use ftrace function 337 trace utility, but other subsystems might too. This file 338 displays all functions that have a callback attached to them 339 as well as the number of callbacks that have been attached. 340 Note, a callback may also call multiple functions which will 341 not be listed in this count. 342 343 If the callback registered to be traced by a function with 344 the "save regs" attribute (thus even more overhead), a 'R' 345 will be displayed on the same line as the function that 346 is returning registers. 347 348 If the callback registered to be traced by a function with 349 the "ip modify" attribute (thus the regs->ip can be changed), 350 an 'I' will be displayed on the same line as the function that 351 can be overridden. 352 353 If the architecture supports it, it will also show what callback 354 is being directly called by the function. If the count is greater 355 than 1 it most likely will be ftrace_ops_list_func(). 356 357 If the callback of a function jumps to a trampoline that is 358 specific to the callback and which is not the standard trampoline, 359 its address will be printed as well as the function that the 360 trampoline calls. 361 362 function_profile_enabled: 363 364 When set it will enable all functions with either the function 365 tracer, or if configured, the function graph tracer. It will 366 keep a histogram of the number of functions that were called 367 and if the function graph tracer was configured, it will also keep 368 track of the time spent in those functions. The histogram 369 content can be displayed in the files: 370 371 trace_stat/function<cpu> ( function0, function1, etc). 372 373 trace_stat: 374 375 A directory that holds different tracing stats. 376 377 kprobe_events: 378 379 Enable dynamic trace points. See kprobetrace.rst. 380 381 kprobe_profile: 382 383 Dynamic trace points stats. See kprobetrace.rst. 384 385 max_graph_depth: 386 387 Used with the function graph tracer. This is the max depth 388 it will trace into a function. Setting this to a value of 389 one will show only the first kernel function that is called 390 from user space. 391 392 printk_formats: 393 394 This is for tools that read the raw format files. If an event in 395 the ring buffer references a string, only a pointer to the string 396 is recorded into the buffer and not the string itself. This prevents 397 tools from knowing what that string was. This file displays the string 398 and address for the string allowing tools to map the pointers to what 399 the strings were. 400 401 saved_cmdlines: 402 403 Only the pid of the task is recorded in a trace event unless 404 the event specifically saves the task comm as well. Ftrace 405 makes a cache of pid mappings to comms to try to display 406 comms for events. If a pid for a comm is not listed, then 407 "<...>" is displayed in the output. 408 409 If the option "record-cmd" is set to "0", then comms of tasks 410 will not be saved during recording. By default, it is enabled. 411 412 saved_cmdlines_size: 413 414 By default, 128 comms are saved (see "saved_cmdlines" above). To 415 increase or decrease the amount of comms that are cached, echo 416 the number of comms to cache into this file. 417 418 saved_tgids: 419 420 If the option "record-tgid" is set, on each scheduling context switch 421 the Task Group ID of a task is saved in a table mapping the PID of 422 the thread to its TGID. By default, the "record-tgid" option is 423 disabled. 424 425 snapshot: 426 427 This displays the "snapshot" buffer and also lets the user 428 take a snapshot of the current running trace. 429 See the "Snapshot" section below for more details. 430 431 stack_max_size: 432 433 When the stack tracer is activated, this will display the 434 maximum stack size it has encountered. 435 See the "Stack Trace" section below. 436 437 stack_trace: 438 439 This displays the stack back trace of the largest stack 440 that was encountered when the stack tracer is activated. 441 See the "Stack Trace" section below. 442 443 stack_trace_filter: 444 445 This is similar to "set_ftrace_filter" but it limits what 446 functions the stack tracer will check. 447 448 trace_clock: 449 450 Whenever an event is recorded into the ring buffer, a 451 "timestamp" is added. This stamp comes from a specified 452 clock. By default, ftrace uses the "local" clock. This 453 clock is very fast and strictly per cpu, but on some 454 systems it may not be monotonic with respect to other 455 CPUs. In other words, the local clocks may not be in sync 456 with local clocks on other CPUs. 457 458 Usual clocks for tracing:: 459 460 # cat trace_clock 461 [local] global counter x86-tsc 462 463 The clock with the square brackets around it is the one in effect. 464 465 local: 466 Default clock, but may not be in sync across CPUs 467 468 global: 469 This clock is in sync with all CPUs but may 470 be a bit slower than the local clock. 471 472 counter: 473 This is not a clock at all, but literally an atomic 474 counter. It counts up one by one, but is in sync 475 with all CPUs. This is useful when you need to 476 know exactly the order events occurred with respect to 477 each other on different CPUs. 478 479 uptime: 480 This uses the jiffies counter and the time stamp 481 is relative to the time since boot up. 482 483 perf: 484 This makes ftrace use the same clock that perf uses. 485 Eventually perf will be able to read ftrace buffers 486 and this will help out in interleaving the data. 487 488 x86-tsc: 489 Architectures may define their own clocks. For 490 example, x86 uses its own TSC cycle clock here. 491 492 ppc-tb: 493 This uses the powerpc timebase register value. 494 This is in sync across CPUs and can also be used 495 to correlate events across hypervisor/guest if 496 tb_offset is known. 497 498 mono: 499 This uses the fast monotonic clock (CLOCK_MONOTONIC) 500 which is monotonic and is subject to NTP rate adjustments. 501 502 mono_raw: 503 This is the raw monotonic clock (CLOCK_MONOTONIC_RAW) 504 which is monotonic but is not subject to any rate adjustments 505 and ticks at the same rate as the hardware clocksource. 506 507 boot: 508 This is the boot clock (CLOCK_BOOTTIME) and is based on the 509 fast monotonic clock, but also accounts for time spent in 510 suspend. Since the clock access is designed for use in 511 tracing in the suspend path, some side effects are possible 512 if clock is accessed after the suspend time is accounted before 513 the fast mono clock is updated. In this case, the clock update 514 appears to happen slightly sooner than it normally would have. 515 Also on 32-bit systems, it's possible that the 64-bit boot offset 516 sees a partial update. These effects are rare and post 517 processing should be able to handle them. See comments in the 518 ktime_get_boot_fast_ns() function for more information. 519 520 tai: 521 This is the tai clock (CLOCK_TAI) and is derived from the wall- 522 clock time. However, this clock does not experience 523 discontinuities and backwards jumps caused by NTP inserting leap 524 seconds. Since the clock access is designed for use in tracing, 525 side effects are possible. The clock access may yield wrong 526 readouts in case the internal TAI offset is updated e.g., caused 527 by setting the system time or using adjtimex() with an offset. 528 These effects are rare and post processing should be able to 529 handle them. See comments in the ktime_get_tai_fast_ns() 530 function for more information. 531 532 To set a clock, simply echo the clock name into this file:: 533 534 # echo global > trace_clock 535 536 Setting a clock clears the ring buffer content as well as the 537 "snapshot" buffer. 538 539 trace_marker: 540 541 This is a very useful file for synchronizing user space 542 with events happening in the kernel. Writing strings into 543 this file will be written into the ftrace buffer. 544 545 It is useful in applications to open this file at the start 546 of the application and just reference the file descriptor 547 for the file:: 548 549 void trace_write(const char *fmt, ...) 550 { 551 va_list ap; 552 char buf[256]; 553 int n; 554 555 if (trace_fd < 0) 556 return; 557 558 va_start(ap, fmt); 559 n = vsnprintf(buf, 256, fmt, ap); 560 va_end(ap); 561 562 write(trace_fd, buf, n); 563 } 564 565 start:: 566 567 trace_fd = open("trace_marker", O_WRONLY); 568 569 Note: Writing into the trace_marker file can also initiate triggers 570 that are written into /sys/kernel/tracing/events/ftrace/print/trigger 571 See "Event triggers" in Documentation/trace/events.rst and an 572 example in Documentation/trace/histogram.rst (Section 3.) 573 574 trace_marker_raw: 575 576 This is similar to trace_marker above, but is meant for binary data 577 to be written to it, where a tool can be used to parse the data 578 from trace_pipe_raw. 579 580 uprobe_events: 581 582 Add dynamic tracepoints in programs. 583 See uprobetracer.rst 584 585 uprobe_profile: 586 587 Uprobe statistics. See uprobetrace.txt 588 589 instances: 590 591 This is a way to make multiple trace buffers where different 592 events can be recorded in different buffers. 593 See "Instances" section below. 594 595 events: 596 597 This is the trace event directory. It holds event tracepoints 598 (also known as static tracepoints) that have been compiled 599 into the kernel. It shows what event tracepoints exist 600 and how they are grouped by system. There are "enable" 601 files at various levels that can enable the tracepoints 602 when a "1" is written to them. 603 604 See events.rst for more information. 605 606 set_event: 607 608 By echoing in the event into this file, will enable that event. 609 610 See events.rst for more information. 611 612 available_events: 613 614 A list of events that can be enabled in tracing. 615 616 See events.rst for more information. 617 618 timestamp_mode: 619 620 Certain tracers may change the timestamp mode used when 621 logging trace events into the event buffer. Events with 622 different modes can coexist within a buffer but the mode in 623 effect when an event is logged determines which timestamp mode 624 is used for that event. The default timestamp mode is 625 'delta'. 626 627 Usual timestamp modes for tracing: 628 629 # cat timestamp_mode 630 [delta] absolute 631 632 The timestamp mode with the square brackets around it is the 633 one in effect. 634 635 delta: Default timestamp mode - timestamp is a delta against 636 a per-buffer timestamp. 637 638 absolute: The timestamp is a full timestamp, not a delta 639 against some other value. As such it takes up more 640 space and is less efficient. 641 642 hwlat_detector: 643 644 Directory for the Hardware Latency Detector. 645 See "Hardware Latency Detector" section below. 646 647 per_cpu: 648 649 This is a directory that contains the trace per_cpu information. 650 651 per_cpu/cpu0/buffer_size_kb: 652 653 The ftrace buffer is defined per_cpu. That is, there's a separate 654 buffer for each CPU to allow writes to be done atomically, 655 and free from cache bouncing. These buffers may have different 656 size buffers. This file is similar to the buffer_size_kb 657 file, but it only displays or sets the buffer size for the 658 specific CPU. (here cpu0). 659 660 per_cpu/cpu0/trace: 661 662 This is similar to the "trace" file, but it will only display 663 the data specific for the CPU. If written to, it only clears 664 the specific CPU buffer. 665 666 per_cpu/cpu0/trace_pipe 667 668 This is similar to the "trace_pipe" file, and is a consuming 669 read, but it will only display (and consume) the data specific 670 for the CPU. 671 672 per_cpu/cpu0/trace_pipe_raw 673 674 For tools that can parse the ftrace ring buffer binary format, 675 the trace_pipe_raw file can be used to extract the data 676 from the ring buffer directly. With the use of the splice() 677 system call, the buffer data can be quickly transferred to 678 a file or to the network where a server is collecting the 679 data. 680 681 Like trace_pipe, this is a consuming reader, where multiple 682 reads will always produce different data. 683 684 per_cpu/cpu0/snapshot: 685 686 This is similar to the main "snapshot" file, but will only 687 snapshot the current CPU (if supported). It only displays 688 the content of the snapshot for a given CPU, and if 689 written to, only clears this CPU buffer. 690 691 per_cpu/cpu0/snapshot_raw: 692 693 Similar to the trace_pipe_raw, but will read the binary format 694 from the snapshot buffer for the given CPU. 695 696 per_cpu/cpu0/stats: 697 698 This displays certain stats about the ring buffer: 699 700 entries: 701 The number of events that are still in the buffer. 702 703 overrun: 704 The number of lost events due to overwriting when 705 the buffer was full. 706 707 commit overrun: 708 Should always be zero. 709 This gets set if so many events happened within a nested 710 event (ring buffer is re-entrant), that it fills the 711 buffer and starts dropping events. 712 713 bytes: 714 Bytes actually read (not overwritten). 715 716 oldest event ts: 717 The oldest timestamp in the buffer 718 719 now ts: 720 The current timestamp 721 722 dropped events: 723 Events lost due to overwrite option being off. 724 725 read events: 726 The number of events read. 727 728The Tracers 729----------- 730 731Here is the list of current tracers that may be configured. 732 733 "function" 734 735 Function call tracer to trace all kernel functions. 736 737 "function_graph" 738 739 Similar to the function tracer except that the 740 function tracer probes the functions on their entry 741 whereas the function graph tracer traces on both entry 742 and exit of the functions. It then provides the ability 743 to draw a graph of function calls similar to C code 744 source. 745 746 "blk" 747 748 The block tracer. The tracer used by the blktrace user 749 application. 750 751 "hwlat" 752 753 The Hardware Latency tracer is used to detect if the hardware 754 produces any latency. See "Hardware Latency Detector" section 755 below. 756 757 "irqsoff" 758 759 Traces the areas that disable interrupts and saves 760 the trace with the longest max latency. 761 See tracing_max_latency. When a new max is recorded, 762 it replaces the old trace. It is best to view this 763 trace with the latency-format option enabled, which 764 happens automatically when the tracer is selected. 765 766 "preemptoff" 767 768 Similar to irqsoff but traces and records the amount of 769 time for which preemption is disabled. 770 771 "preemptirqsoff" 772 773 Similar to irqsoff and preemptoff, but traces and 774 records the largest time for which irqs and/or preemption 775 is disabled. 776 777 "wakeup" 778 779 Traces and records the max latency that it takes for 780 the highest priority task to get scheduled after 781 it has been woken up. 782 Traces all tasks as an average developer would expect. 783 784 "wakeup_rt" 785 786 Traces and records the max latency that it takes for just 787 RT tasks (as the current "wakeup" does). This is useful 788 for those interested in wake up timings of RT tasks. 789 790 "wakeup_dl" 791 792 Traces and records the max latency that it takes for 793 a SCHED_DEADLINE task to be woken (as the "wakeup" and 794 "wakeup_rt" does). 795 796 "mmiotrace" 797 798 A special tracer that is used to trace binary module. 799 It will trace all the calls that a module makes to the 800 hardware. Everything it writes and reads from the I/O 801 as well. 802 803 "branch" 804 805 This tracer can be configured when tracing likely/unlikely 806 calls within the kernel. It will trace when a likely and 807 unlikely branch is hit and if it was correct in its prediction 808 of being correct. 809 810 "nop" 811 812 This is the "trace nothing" tracer. To remove all 813 tracers from tracing simply echo "nop" into 814 current_tracer. 815 816Error conditions 817---------------- 818 819 For most ftrace commands, failure modes are obvious and communicated 820 using standard return codes. 821 822 For other more involved commands, extended error information may be 823 available via the tracing/error_log file. For the commands that 824 support it, reading the tracing/error_log file after an error will 825 display more detailed information about what went wrong, if 826 information is available. The tracing/error_log file is a circular 827 error log displaying a small number (currently, 8) of ftrace errors 828 for the last (8) failed commands. 829 830 The extended error information and usage takes the form shown in 831 this example:: 832 833 # echo xxx > /sys/kernel/tracing/events/sched/sched_wakeup/trigger 834 echo: write error: Invalid argument 835 836 # cat /sys/kernel/tracing/error_log 837 [ 5348.887237] location: error: Couldn't yyy: zzz 838 Command: xxx 839 ^ 840 [ 7517.023364] location: error: Bad rrr: sss 841 Command: ppp qqq 842 ^ 843 844 To clear the error log, echo the empty string into it:: 845 846 # echo > /sys/kernel/tracing/error_log 847 848Examples of using the tracer 849---------------------------- 850 851Here are typical examples of using the tracers when controlling 852them only with the tracefs interface (without using any 853user-land utilities). 854 855Output format: 856-------------- 857 858Here is an example of the output format of the file "trace":: 859 860 # tracer: function 861 # 862 # entries-in-buffer/entries-written: 140080/250280 #P:4 863 # 864 # _-----=> irqs-off 865 # / _----=> need-resched 866 # | / _---=> hardirq/softirq 867 # || / _--=> preempt-depth 868 # ||| / delay 869 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 870 # | | | |||| | | 871 bash-1977 [000] .... 17284.993652: sys_close <-system_call_fastpath 872 bash-1977 [000] .... 17284.993653: __close_fd <-sys_close 873 bash-1977 [000] .... 17284.993653: _raw_spin_lock <-__close_fd 874 sshd-1974 [003] .... 17284.993653: __srcu_read_unlock <-fsnotify 875 bash-1977 [000] .... 17284.993654: add_preempt_count <-_raw_spin_lock 876 bash-1977 [000] ...1 17284.993655: _raw_spin_unlock <-__close_fd 877 bash-1977 [000] ...1 17284.993656: sub_preempt_count <-_raw_spin_unlock 878 bash-1977 [000] .... 17284.993657: filp_close <-__close_fd 879 bash-1977 [000] .... 17284.993657: dnotify_flush <-filp_close 880 sshd-1974 [003] .... 17284.993658: sys_select <-system_call_fastpath 881 .... 882 883A header is printed with the tracer name that is represented by 884the trace. In this case the tracer is "function". Then it shows the 885number of events in the buffer as well as the total number of entries 886that were written. The difference is the number of entries that were 887lost due to the buffer filling up (250280 - 140080 = 110200 events 888lost). 889 890The header explains the content of the events. Task name "bash", the task 891PID "1977", the CPU that it was running on "000", the latency format 892(explained below), the timestamp in <secs>.<usecs> format, the 893function name that was traced "sys_close" and the parent function that 894called this function "system_call_fastpath". The timestamp is the time 895at which the function was entered. 896 897Latency trace format 898-------------------- 899 900When the latency-format option is enabled or when one of the latency 901tracers is set, the trace file gives somewhat more information to see 902why a latency happened. Here is a typical trace:: 903 904 # tracer: irqsoff 905 # 906 # irqsoff latency trace v1.1.5 on 3.8.0-test+ 907 # -------------------------------------------------------------------- 908 # latency: 259 us, #4/4, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 909 # ----------------- 910 # | task: ps-6143 (uid:0 nice:0 policy:0 rt_prio:0) 911 # ----------------- 912 # => started at: __lock_task_sighand 913 # => ended at: _raw_spin_unlock_irqrestore 914 # 915 # 916 # _------=> CPU# 917 # / _-----=> irqs-off 918 # | / _----=> need-resched 919 # || / _---=> hardirq/softirq 920 # ||| / _--=> preempt-depth 921 # |||| / delay 922 # cmd pid ||||| time | caller 923 # \ / ||||| \ | / 924 ps-6143 2d... 0us!: trace_hardirqs_off <-__lock_task_sighand 925 ps-6143 2d..1 259us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore 926 ps-6143 2d..1 263us+: time_hardirqs_on <-_raw_spin_unlock_irqrestore 927 ps-6143 2d..1 306us : <stack trace> 928 => trace_hardirqs_on_caller 929 => trace_hardirqs_on 930 => _raw_spin_unlock_irqrestore 931 => do_task_stat 932 => proc_tgid_stat 933 => proc_single_show 934 => seq_read 935 => vfs_read 936 => sys_read 937 => system_call_fastpath 938 939 940This shows that the current tracer is "irqsoff" tracing the time 941for which interrupts were disabled. It gives the trace version (which 942never changes) and the version of the kernel upon which this was executed on 943(3.8). Then it displays the max latency in microseconds (259 us). The number 944of trace entries displayed and the total number (both are four: #4/4). 945VP, KP, SP, and HP are always zero and are reserved for later use. 946#P is the number of online CPUs (#P:4). 947 948The task is the process that was running when the latency 949occurred. (ps pid: 6143). 950 951The start and stop (the functions in which the interrupts were 952disabled and enabled respectively) that caused the latencies: 953 954 - __lock_task_sighand is where the interrupts were disabled. 955 - _raw_spin_unlock_irqrestore is where they were enabled again. 956 957The next lines after the header are the trace itself. The header 958explains which is which. 959 960 cmd: The name of the process in the trace. 961 962 pid: The PID of that process. 963 964 CPU#: The CPU which the process was running on. 965 966 irqs-off: 'd' interrupts are disabled. '.' otherwise. 967 .. caution:: If the architecture does not support a way to 968 read the irq flags variable, an 'X' will always 969 be printed here. 970 971 need-resched: 972 - 'N' both TIF_NEED_RESCHED and PREEMPT_NEED_RESCHED is set, 973 - 'n' only TIF_NEED_RESCHED is set, 974 - 'p' only PREEMPT_NEED_RESCHED is set, 975 - '.' otherwise. 976 977 hardirq/softirq: 978 - 'Z' - NMI occurred inside a hardirq 979 - 'z' - NMI is running 980 - 'H' - hard irq occurred inside a softirq. 981 - 'h' - hard irq is running 982 - 's' - soft irq is running 983 - '.' - normal context. 984 985 preempt-depth: The level of preempt_disabled 986 987The above is mostly meaningful for kernel developers. 988 989 time: 990 When the latency-format option is enabled, the trace file 991 output includes a timestamp relative to the start of the 992 trace. This differs from the output when latency-format 993 is disabled, which includes an absolute timestamp. 994 995 delay: 996 This is just to help catch your eye a bit better. And 997 needs to be fixed to be only relative to the same CPU. 998 The marks are determined by the difference between this 999 current trace and the next trace. 1000 1001 - '$' - greater than 1 second 1002 - '@' - greater than 100 millisecond 1003 - '*' - greater than 10 millisecond 1004 - '#' - greater than 1000 microsecond 1005 - '!' - greater than 100 microsecond 1006 - '+' - greater than 10 microsecond 1007 - ' ' - less than or equal to 10 microsecond. 1008 1009 The rest is the same as the 'trace' file. 1010 1011 Note, the latency tracers will usually end with a back trace 1012 to easily find where the latency occurred. 1013 1014trace_options 1015------------- 1016 1017The trace_options file (or the options directory) is used to control 1018what gets printed in the trace output, or manipulate the tracers. 1019To see what is available, simply cat the file:: 1020 1021 cat trace_options 1022 print-parent 1023 nosym-offset 1024 nosym-addr 1025 noverbose 1026 noraw 1027 nohex 1028 nobin 1029 noblock 1030 trace_printk 1031 annotate 1032 nouserstacktrace 1033 nosym-userobj 1034 noprintk-msg-only 1035 context-info 1036 nolatency-format 1037 record-cmd 1038 norecord-tgid 1039 overwrite 1040 nodisable_on_free 1041 irq-info 1042 markers 1043 noevent-fork 1044 function-trace 1045 nofunction-fork 1046 nodisplay-graph 1047 nostacktrace 1048 nobranch 1049 1050To disable one of the options, echo in the option prepended with 1051"no":: 1052 1053 echo noprint-parent > trace_options 1054 1055To enable an option, leave off the "no":: 1056 1057 echo sym-offset > trace_options 1058 1059Here are the available options: 1060 1061 print-parent 1062 On function traces, display the calling (parent) 1063 function as well as the function being traced. 1064 :: 1065 1066 print-parent: 1067 bash-4000 [01] 1477.606694: simple_strtoul <-kstrtoul 1068 1069 noprint-parent: 1070 bash-4000 [01] 1477.606694: simple_strtoul 1071 1072 1073 sym-offset 1074 Display not only the function name, but also the 1075 offset in the function. For example, instead of 1076 seeing just "ktime_get", you will see 1077 "ktime_get+0xb/0x20". 1078 :: 1079 1080 sym-offset: 1081 bash-4000 [01] 1477.606694: simple_strtoul+0x6/0xa0 1082 1083 sym-addr 1084 This will also display the function address as well 1085 as the function name. 1086 :: 1087 1088 sym-addr: 1089 bash-4000 [01] 1477.606694: simple_strtoul <c0339346> 1090 1091 verbose 1092 This deals with the trace file when the 1093 latency-format option is enabled. 1094 :: 1095 1096 bash 4000 1 0 00000000 00010a95 [58127d26] 1720.415ms \ 1097 (+0.000ms): simple_strtoul (kstrtoul) 1098 1099 raw 1100 This will display raw numbers. This option is best for 1101 use with user applications that can translate the raw 1102 numbers better than having it done in the kernel. 1103 1104 hex 1105 Similar to raw, but the numbers will be in a hexadecimal format. 1106 1107 bin 1108 This will print out the formats in raw binary. 1109 1110 block 1111 When set, reading trace_pipe will not block when polled. 1112 1113 trace_printk 1114 Can disable trace_printk() from writing into the buffer. 1115 1116 annotate 1117 It is sometimes confusing when the CPU buffers are full 1118 and one CPU buffer had a lot of events recently, thus 1119 a shorter time frame, were another CPU may have only had 1120 a few events, which lets it have older events. When 1121 the trace is reported, it shows the oldest events first, 1122 and it may look like only one CPU ran (the one with the 1123 oldest events). When the annotate option is set, it will 1124 display when a new CPU buffer started:: 1125 1126 <idle>-0 [001] dNs4 21169.031481: wake_up_idle_cpu <-add_timer_on 1127 <idle>-0 [001] dNs4 21169.031482: _raw_spin_unlock_irqrestore <-add_timer_on 1128 <idle>-0 [001] .Ns4 21169.031484: sub_preempt_count <-_raw_spin_unlock_irqrestore 1129 ##### CPU 2 buffer started #### 1130 <idle>-0 [002] .N.1 21169.031484: rcu_idle_exit <-cpu_idle 1131 <idle>-0 [001] .Ns3 21169.031484: _raw_spin_unlock <-clocksource_watchdog 1132 <idle>-0 [001] .Ns3 21169.031485: sub_preempt_count <-_raw_spin_unlock 1133 1134 userstacktrace 1135 This option changes the trace. It records a 1136 stacktrace of the current user space thread after 1137 each trace event. 1138 1139 sym-userobj 1140 when user stacktrace are enabled, look up which 1141 object the address belongs to, and print a 1142 relative address. This is especially useful when 1143 ASLR is on, otherwise you don't get a chance to 1144 resolve the address to object/file/line after 1145 the app is no longer running 1146 1147 The lookup is performed when you read 1148 trace,trace_pipe. Example:: 1149 1150 a.out-1623 [000] 40874.465068: /root/a.out[+0x480] <-/root/a.out[+0 1151 x494] <- /root/a.out[+0x4a8] <- /lib/libc-2.7.so[+0x1e1a6] 1152 1153 1154 printk-msg-only 1155 When set, trace_printk()s will only show the format 1156 and not their parameters (if trace_bprintk() or 1157 trace_bputs() was used to save the trace_printk()). 1158 1159 context-info 1160 Show only the event data. Hides the comm, PID, 1161 timestamp, CPU, and other useful data. 1162 1163 latency-format 1164 This option changes the trace output. When it is enabled, 1165 the trace displays additional information about the 1166 latency, as described in "Latency trace format". 1167 1168 pause-on-trace 1169 When set, opening the trace file for read, will pause 1170 writing to the ring buffer (as if tracing_on was set to zero). 1171 This simulates the original behavior of the trace file. 1172 When the file is closed, tracing will be enabled again. 1173 1174 hash-ptr 1175 When set, "%p" in the event printk format displays the 1176 hashed pointer value instead of real address. 1177 This will be useful if you want to find out which hashed 1178 value is corresponding to the real value in trace log. 1179 1180 record-cmd 1181 When any event or tracer is enabled, a hook is enabled 1182 in the sched_switch trace point to fill comm cache 1183 with mapped pids and comms. But this may cause some 1184 overhead, and if you only care about pids, and not the 1185 name of the task, disabling this option can lower the 1186 impact of tracing. See "saved_cmdlines". 1187 1188 record-tgid 1189 When any event or tracer is enabled, a hook is enabled 1190 in the sched_switch trace point to fill the cache of 1191 mapped Thread Group IDs (TGID) mapping to pids. See 1192 "saved_tgids". 1193 1194 overwrite 1195 This controls what happens when the trace buffer is 1196 full. If "1" (default), the oldest events are 1197 discarded and overwritten. If "0", then the newest 1198 events are discarded. 1199 (see per_cpu/cpu0/stats for overrun and dropped) 1200 1201 disable_on_free 1202 When the free_buffer is closed, tracing will 1203 stop (tracing_on set to 0). 1204 1205 irq-info 1206 Shows the interrupt, preempt count, need resched data. 1207 When disabled, the trace looks like:: 1208 1209 # tracer: function 1210 # 1211 # entries-in-buffer/entries-written: 144405/9452052 #P:4 1212 # 1213 # TASK-PID CPU# TIMESTAMP FUNCTION 1214 # | | | | | 1215 <idle>-0 [002] 23636.756054: ttwu_do_activate.constprop.89 <-try_to_wake_up 1216 <idle>-0 [002] 23636.756054: activate_task <-ttwu_do_activate.constprop.89 1217 <idle>-0 [002] 23636.756055: enqueue_task <-activate_task 1218 1219 1220 markers 1221 When set, the trace_marker is writable (only by root). 1222 When disabled, the trace_marker will error with EINVAL 1223 on write. 1224 1225 event-fork 1226 When set, tasks with PIDs listed in set_event_pid will have 1227 the PIDs of their children added to set_event_pid when those 1228 tasks fork. Also, when tasks with PIDs in set_event_pid exit, 1229 their PIDs will be removed from the file. 1230 1231 This affects PIDs listed in set_event_notrace_pid as well. 1232 1233 function-trace 1234 The latency tracers will enable function tracing 1235 if this option is enabled (default it is). When 1236 it is disabled, the latency tracers do not trace 1237 functions. This keeps the overhead of the tracer down 1238 when performing latency tests. 1239 1240 function-fork 1241 When set, tasks with PIDs listed in set_ftrace_pid will 1242 have the PIDs of their children added to set_ftrace_pid 1243 when those tasks fork. Also, when tasks with PIDs in 1244 set_ftrace_pid exit, their PIDs will be removed from the 1245 file. 1246 1247 This affects PIDs in set_ftrace_notrace_pid as well. 1248 1249 display-graph 1250 When set, the latency tracers (irqsoff, wakeup, etc) will 1251 use function graph tracing instead of function tracing. 1252 1253 stacktrace 1254 When set, a stack trace is recorded after any trace event 1255 is recorded. 1256 1257 branch 1258 Enable branch tracing with the tracer. This enables branch 1259 tracer along with the currently set tracer. Enabling this 1260 with the "nop" tracer is the same as just enabling the 1261 "branch" tracer. 1262 1263.. tip:: Some tracers have their own options. They only appear in this 1264 file when the tracer is active. They always appear in the 1265 options directory. 1266 1267 1268Here are the per tracer options: 1269 1270Options for function tracer: 1271 1272 func_stack_trace 1273 When set, a stack trace is recorded after every 1274 function that is recorded. NOTE! Limit the functions 1275 that are recorded before enabling this, with 1276 "set_ftrace_filter" otherwise the system performance 1277 will be critically degraded. Remember to disable 1278 this option before clearing the function filter. 1279 1280Options for function_graph tracer: 1281 1282 Since the function_graph tracer has a slightly different output 1283 it has its own options to control what is displayed. 1284 1285 funcgraph-overrun 1286 When set, the "overrun" of the graph stack is 1287 displayed after each function traced. The 1288 overrun, is when the stack depth of the calls 1289 is greater than what is reserved for each task. 1290 Each task has a fixed array of functions to 1291 trace in the call graph. If the depth of the 1292 calls exceeds that, the function is not traced. 1293 The overrun is the number of functions missed 1294 due to exceeding this array. 1295 1296 funcgraph-cpu 1297 When set, the CPU number of the CPU where the trace 1298 occurred is displayed. 1299 1300 funcgraph-overhead 1301 When set, if the function takes longer than 1302 A certain amount, then a delay marker is 1303 displayed. See "delay" above, under the 1304 header description. 1305 1306 funcgraph-proc 1307 Unlike other tracers, the process' command line 1308 is not displayed by default, but instead only 1309 when a task is traced in and out during a context 1310 switch. Enabling this options has the command 1311 of each process displayed at every line. 1312 1313 funcgraph-duration 1314 At the end of each function (the return) 1315 the duration of the amount of time in the 1316 function is displayed in microseconds. 1317 1318 funcgraph-abstime 1319 When set, the timestamp is displayed at each line. 1320 1321 funcgraph-irqs 1322 When disabled, functions that happen inside an 1323 interrupt will not be traced. 1324 1325 funcgraph-tail 1326 When set, the return event will include the function 1327 that it represents. By default this is off, and 1328 only a closing curly bracket "}" is displayed for 1329 the return of a function. 1330 1331 sleep-time 1332 When running function graph tracer, to include 1333 the time a task schedules out in its function. 1334 When enabled, it will account time the task has been 1335 scheduled out as part of the function call. 1336 1337 graph-time 1338 When running function profiler with function graph tracer, 1339 to include the time to call nested functions. When this is 1340 not set, the time reported for the function will only 1341 include the time the function itself executed for, not the 1342 time for functions that it called. 1343 1344Options for blk tracer: 1345 1346 blk_classic 1347 Shows a more minimalistic output. 1348 1349 1350irqsoff 1351------- 1352 1353When interrupts are disabled, the CPU can not react to any other 1354external event (besides NMIs and SMIs). This prevents the timer 1355interrupt from triggering or the mouse interrupt from letting 1356the kernel know of a new mouse event. The result is a latency 1357with the reaction time. 1358 1359The irqsoff tracer tracks the time for which interrupts are 1360disabled. When a new maximum latency is hit, the tracer saves 1361the trace leading up to that latency point so that every time a 1362new maximum is reached, the old saved trace is discarded and the 1363new trace is saved. 1364 1365To reset the maximum, echo 0 into tracing_max_latency. Here is 1366an example:: 1367 1368 # echo 0 > options/function-trace 1369 # echo irqsoff > current_tracer 1370 # echo 1 > tracing_on 1371 # echo 0 > tracing_max_latency 1372 # ls -ltr 1373 [...] 1374 # echo 0 > tracing_on 1375 # cat trace 1376 # tracer: irqsoff 1377 # 1378 # irqsoff latency trace v1.1.5 on 3.8.0-test+ 1379 # -------------------------------------------------------------------- 1380 # latency: 16 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 1381 # ----------------- 1382 # | task: swapper/0-0 (uid:0 nice:0 policy:0 rt_prio:0) 1383 # ----------------- 1384 # => started at: run_timer_softirq 1385 # => ended at: run_timer_softirq 1386 # 1387 # 1388 # _------=> CPU# 1389 # / _-----=> irqs-off 1390 # | / _----=> need-resched 1391 # || / _---=> hardirq/softirq 1392 # ||| / _--=> preempt-depth 1393 # |||| / delay 1394 # cmd pid ||||| time | caller 1395 # \ / ||||| \ | / 1396 <idle>-0 0d.s2 0us+: _raw_spin_lock_irq <-run_timer_softirq 1397 <idle>-0 0dNs3 17us : _raw_spin_unlock_irq <-run_timer_softirq 1398 <idle>-0 0dNs3 17us+: trace_hardirqs_on <-run_timer_softirq 1399 <idle>-0 0dNs3 25us : <stack trace> 1400 => _raw_spin_unlock_irq 1401 => run_timer_softirq 1402 => __do_softirq 1403 => call_softirq 1404 => do_softirq 1405 => irq_exit 1406 => smp_apic_timer_interrupt 1407 => apic_timer_interrupt 1408 => rcu_idle_exit 1409 => cpu_idle 1410 => rest_init 1411 => start_kernel 1412 => x86_64_start_reservations 1413 => x86_64_start_kernel 1414 1415Here we see that we had a latency of 16 microseconds (which is 1416very good). The _raw_spin_lock_irq in run_timer_softirq disabled 1417interrupts. The difference between the 16 and the displayed 1418timestamp 25us occurred because the clock was incremented 1419between the time of recording the max latency and the time of 1420recording the function that had that latency. 1421 1422Note the above example had function-trace not set. If we set 1423function-trace, we get a much larger output:: 1424 1425 with echo 1 > options/function-trace 1426 1427 # tracer: irqsoff 1428 # 1429 # irqsoff latency trace v1.1.5 on 3.8.0-test+ 1430 # -------------------------------------------------------------------- 1431 # latency: 71 us, #168/168, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 1432 # ----------------- 1433 # | task: bash-2042 (uid:0 nice:0 policy:0 rt_prio:0) 1434 # ----------------- 1435 # => started at: ata_scsi_queuecmd 1436 # => ended at: ata_scsi_queuecmd 1437 # 1438 # 1439 # _------=> CPU# 1440 # / _-----=> irqs-off 1441 # | / _----=> need-resched 1442 # || / _---=> hardirq/softirq 1443 # ||| / _--=> preempt-depth 1444 # |||| / delay 1445 # cmd pid ||||| time | caller 1446 # \ / ||||| \ | / 1447 bash-2042 3d... 0us : _raw_spin_lock_irqsave <-ata_scsi_queuecmd 1448 bash-2042 3d... 0us : add_preempt_count <-_raw_spin_lock_irqsave 1449 bash-2042 3d..1 1us : ata_scsi_find_dev <-ata_scsi_queuecmd 1450 bash-2042 3d..1 1us : __ata_scsi_find_dev <-ata_scsi_find_dev 1451 bash-2042 3d..1 2us : ata_find_dev.part.14 <-__ata_scsi_find_dev 1452 bash-2042 3d..1 2us : ata_qc_new_init <-__ata_scsi_queuecmd 1453 bash-2042 3d..1 3us : ata_sg_init <-__ata_scsi_queuecmd 1454 bash-2042 3d..1 4us : ata_scsi_rw_xlat <-__ata_scsi_queuecmd 1455 bash-2042 3d..1 4us : ata_build_rw_tf <-ata_scsi_rw_xlat 1456 [...] 1457 bash-2042 3d..1 67us : delay_tsc <-__delay 1458 bash-2042 3d..1 67us : add_preempt_count <-delay_tsc 1459 bash-2042 3d..2 67us : sub_preempt_count <-delay_tsc 1460 bash-2042 3d..1 67us : add_preempt_count <-delay_tsc 1461 bash-2042 3d..2 68us : sub_preempt_count <-delay_tsc 1462 bash-2042 3d..1 68us+: ata_bmdma_start <-ata_bmdma_qc_issue 1463 bash-2042 3d..1 71us : _raw_spin_unlock_irqrestore <-ata_scsi_queuecmd 1464 bash-2042 3d..1 71us : _raw_spin_unlock_irqrestore <-ata_scsi_queuecmd 1465 bash-2042 3d..1 72us+: trace_hardirqs_on <-ata_scsi_queuecmd 1466 bash-2042 3d..1 120us : <stack trace> 1467 => _raw_spin_unlock_irqrestore 1468 => ata_scsi_queuecmd 1469 => scsi_dispatch_cmd 1470 => scsi_request_fn 1471 => __blk_run_queue_uncond 1472 => __blk_run_queue 1473 => blk_queue_bio 1474 => submit_bio_noacct 1475 => submit_bio 1476 => submit_bh 1477 => __ext3_get_inode_loc 1478 => ext3_iget 1479 => ext3_lookup 1480 => lookup_real 1481 => __lookup_hash 1482 => walk_component 1483 => lookup_last 1484 => path_lookupat 1485 => filename_lookup 1486 => user_path_at_empty 1487 => user_path_at 1488 => vfs_fstatat 1489 => vfs_stat 1490 => sys_newstat 1491 => system_call_fastpath 1492 1493 1494Here we traced a 71 microsecond latency. But we also see all the 1495functions that were called during that time. Note that by 1496enabling function tracing, we incur an added overhead. This 1497overhead may extend the latency times. But nevertheless, this 1498trace has provided some very helpful debugging information. 1499 1500If we prefer function graph output instead of function, we can set 1501display-graph option:: 1502 1503 with echo 1 > options/display-graph 1504 1505 # tracer: irqsoff 1506 # 1507 # irqsoff latency trace v1.1.5 on 4.20.0-rc6+ 1508 # -------------------------------------------------------------------- 1509 # latency: 3751 us, #274/274, CPU#0 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:4) 1510 # ----------------- 1511 # | task: bash-1507 (uid:0 nice:0 policy:0 rt_prio:0) 1512 # ----------------- 1513 # => started at: free_debug_processing 1514 # => ended at: return_to_handler 1515 # 1516 # 1517 # _-----=> irqs-off 1518 # / _----=> need-resched 1519 # | / _---=> hardirq/softirq 1520 # || / _--=> preempt-depth 1521 # ||| / 1522 # REL TIME CPU TASK/PID |||| DURATION FUNCTION CALLS 1523 # | | | | |||| | | | | | | 1524 0 us | 0) bash-1507 | d... | 0.000 us | _raw_spin_lock_irqsave(); 1525 0 us | 0) bash-1507 | d..1 | 0.378 us | do_raw_spin_trylock(); 1526 1 us | 0) bash-1507 | d..2 | | set_track() { 1527 2 us | 0) bash-1507 | d..2 | | save_stack_trace() { 1528 2 us | 0) bash-1507 | d..2 | | __save_stack_trace() { 1529 3 us | 0) bash-1507 | d..2 | | __unwind_start() { 1530 3 us | 0) bash-1507 | d..2 | | get_stack_info() { 1531 3 us | 0) bash-1507 | d..2 | 0.351 us | in_task_stack(); 1532 4 us | 0) bash-1507 | d..2 | 1.107 us | } 1533 [...] 1534 3750 us | 0) bash-1507 | d..1 | 0.516 us | do_raw_spin_unlock(); 1535 3750 us | 0) bash-1507 | d..1 | 0.000 us | _raw_spin_unlock_irqrestore(); 1536 3764 us | 0) bash-1507 | d..1 | 0.000 us | tracer_hardirqs_on(); 1537 bash-1507 0d..1 3792us : <stack trace> 1538 => free_debug_processing 1539 => __slab_free 1540 => kmem_cache_free 1541 => vm_area_free 1542 => remove_vma 1543 => exit_mmap 1544 => mmput 1545 => begin_new_exec 1546 => load_elf_binary 1547 => search_binary_handler 1548 => __do_execve_file.isra.32 1549 => __x64_sys_execve 1550 => do_syscall_64 1551 => entry_SYSCALL_64_after_hwframe 1552 1553preemptoff 1554---------- 1555 1556When preemption is disabled, we may be able to receive 1557interrupts but the task cannot be preempted and a higher 1558priority task must wait for preemption to be enabled again 1559before it can preempt a lower priority task. 1560 1561The preemptoff tracer traces the places that disable preemption. 1562Like the irqsoff tracer, it records the maximum latency for 1563which preemption was disabled. The control of preemptoff tracer 1564is much like the irqsoff tracer. 1565:: 1566 1567 # echo 0 > options/function-trace 1568 # echo preemptoff > current_tracer 1569 # echo 1 > tracing_on 1570 # echo 0 > tracing_max_latency 1571 # ls -ltr 1572 [...] 1573 # echo 0 > tracing_on 1574 # cat trace 1575 # tracer: preemptoff 1576 # 1577 # preemptoff latency trace v1.1.5 on 3.8.0-test+ 1578 # -------------------------------------------------------------------- 1579 # latency: 46 us, #4/4, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 1580 # ----------------- 1581 # | task: sshd-1991 (uid:0 nice:0 policy:0 rt_prio:0) 1582 # ----------------- 1583 # => started at: do_IRQ 1584 # => ended at: do_IRQ 1585 # 1586 # 1587 # _------=> CPU# 1588 # / _-----=> irqs-off 1589 # | / _----=> need-resched 1590 # || / _---=> hardirq/softirq 1591 # ||| / _--=> preempt-depth 1592 # |||| / delay 1593 # cmd pid ||||| time | caller 1594 # \ / ||||| \ | / 1595 sshd-1991 1d.h. 0us+: irq_enter <-do_IRQ 1596 sshd-1991 1d..1 46us : irq_exit <-do_IRQ 1597 sshd-1991 1d..1 47us+: trace_preempt_on <-do_IRQ 1598 sshd-1991 1d..1 52us : <stack trace> 1599 => sub_preempt_count 1600 => irq_exit 1601 => do_IRQ 1602 => ret_from_intr 1603 1604 1605This has some more changes. Preemption was disabled when an 1606interrupt came in (notice the 'h'), and was enabled on exit. 1607But we also see that interrupts have been disabled when entering 1608the preempt off section and leaving it (the 'd'). We do not know if 1609interrupts were enabled in the mean time or shortly after this 1610was over. 1611:: 1612 1613 # tracer: preemptoff 1614 # 1615 # preemptoff latency trace v1.1.5 on 3.8.0-test+ 1616 # -------------------------------------------------------------------- 1617 # latency: 83 us, #241/241, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 1618 # ----------------- 1619 # | task: bash-1994 (uid:0 nice:0 policy:0 rt_prio:0) 1620 # ----------------- 1621 # => started at: wake_up_new_task 1622 # => ended at: task_rq_unlock 1623 # 1624 # 1625 # _------=> CPU# 1626 # / _-----=> irqs-off 1627 # | / _----=> need-resched 1628 # || / _---=> hardirq/softirq 1629 # ||| / _--=> preempt-depth 1630 # |||| / delay 1631 # cmd pid ||||| time | caller 1632 # \ / ||||| \ | / 1633 bash-1994 1d..1 0us : _raw_spin_lock_irqsave <-wake_up_new_task 1634 bash-1994 1d..1 0us : select_task_rq_fair <-select_task_rq 1635 bash-1994 1d..1 1us : __rcu_read_lock <-select_task_rq_fair 1636 bash-1994 1d..1 1us : source_load <-select_task_rq_fair 1637 bash-1994 1d..1 1us : source_load <-select_task_rq_fair 1638 [...] 1639 bash-1994 1d..1 12us : irq_enter <-smp_apic_timer_interrupt 1640 bash-1994 1d..1 12us : rcu_irq_enter <-irq_enter 1641 bash-1994 1d..1 13us : add_preempt_count <-irq_enter 1642 bash-1994 1d.h1 13us : exit_idle <-smp_apic_timer_interrupt 1643 bash-1994 1d.h1 13us : hrtimer_interrupt <-smp_apic_timer_interrupt 1644 bash-1994 1d.h1 13us : _raw_spin_lock <-hrtimer_interrupt 1645 bash-1994 1d.h1 14us : add_preempt_count <-_raw_spin_lock 1646 bash-1994 1d.h2 14us : ktime_get_update_offsets <-hrtimer_interrupt 1647 [...] 1648 bash-1994 1d.h1 35us : lapic_next_event <-clockevents_program_event 1649 bash-1994 1d.h1 35us : irq_exit <-smp_apic_timer_interrupt 1650 bash-1994 1d.h1 36us : sub_preempt_count <-irq_exit 1651 bash-1994 1d..2 36us : do_softirq <-irq_exit 1652 bash-1994 1d..2 36us : __do_softirq <-call_softirq 1653 bash-1994 1d..2 36us : __local_bh_disable <-__do_softirq 1654 bash-1994 1d.s2 37us : add_preempt_count <-_raw_spin_lock_irq 1655 bash-1994 1d.s3 38us : _raw_spin_unlock <-run_timer_softirq 1656 bash-1994 1d.s3 39us : sub_preempt_count <-_raw_spin_unlock 1657 bash-1994 1d.s2 39us : call_timer_fn <-run_timer_softirq 1658 [...] 1659 bash-1994 1dNs2 81us : cpu_needs_another_gp <-rcu_process_callbacks 1660 bash-1994 1dNs2 82us : __local_bh_enable <-__do_softirq 1661 bash-1994 1dNs2 82us : sub_preempt_count <-__local_bh_enable 1662 bash-1994 1dN.2 82us : idle_cpu <-irq_exit 1663 bash-1994 1dN.2 83us : rcu_irq_exit <-irq_exit 1664 bash-1994 1dN.2 83us : sub_preempt_count <-irq_exit 1665 bash-1994 1.N.1 84us : _raw_spin_unlock_irqrestore <-task_rq_unlock 1666 bash-1994 1.N.1 84us+: trace_preempt_on <-task_rq_unlock 1667 bash-1994 1.N.1 104us : <stack trace> 1668 => sub_preempt_count 1669 => _raw_spin_unlock_irqrestore 1670 => task_rq_unlock 1671 => wake_up_new_task 1672 => do_fork 1673 => sys_clone 1674 => stub_clone 1675 1676 1677The above is an example of the preemptoff trace with 1678function-trace set. Here we see that interrupts were not disabled 1679the entire time. The irq_enter code lets us know that we entered 1680an interrupt 'h'. Before that, the functions being traced still 1681show that it is not in an interrupt, but we can see from the 1682functions themselves that this is not the case. 1683 1684preemptirqsoff 1685-------------- 1686 1687Knowing the locations that have interrupts disabled or 1688preemption disabled for the longest times is helpful. But 1689sometimes we would like to know when either preemption and/or 1690interrupts are disabled. 1691 1692Consider the following code:: 1693 1694 local_irq_disable(); 1695 call_function_with_irqs_off(); 1696 preempt_disable(); 1697 call_function_with_irqs_and_preemption_off(); 1698 local_irq_enable(); 1699 call_function_with_preemption_off(); 1700 preempt_enable(); 1701 1702The irqsoff tracer will record the total length of 1703call_function_with_irqs_off() and 1704call_function_with_irqs_and_preemption_off(). 1705 1706The preemptoff tracer will record the total length of 1707call_function_with_irqs_and_preemption_off() and 1708call_function_with_preemption_off(). 1709 1710But neither will trace the time that interrupts and/or 1711preemption is disabled. This total time is the time that we can 1712not schedule. To record this time, use the preemptirqsoff 1713tracer. 1714 1715Again, using this trace is much like the irqsoff and preemptoff 1716tracers. 1717:: 1718 1719 # echo 0 > options/function-trace 1720 # echo preemptirqsoff > current_tracer 1721 # echo 1 > tracing_on 1722 # echo 0 > tracing_max_latency 1723 # ls -ltr 1724 [...] 1725 # echo 0 > tracing_on 1726 # cat trace 1727 # tracer: preemptirqsoff 1728 # 1729 # preemptirqsoff latency trace v1.1.5 on 3.8.0-test+ 1730 # -------------------------------------------------------------------- 1731 # latency: 100 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 1732 # ----------------- 1733 # | task: ls-2230 (uid:0 nice:0 policy:0 rt_prio:0) 1734 # ----------------- 1735 # => started at: ata_scsi_queuecmd 1736 # => ended at: ata_scsi_queuecmd 1737 # 1738 # 1739 # _------=> CPU# 1740 # / _-----=> irqs-off 1741 # | / _----=> need-resched 1742 # || / _---=> hardirq/softirq 1743 # ||| / _--=> preempt-depth 1744 # |||| / delay 1745 # cmd pid ||||| time | caller 1746 # \ / ||||| \ | / 1747 ls-2230 3d... 0us+: _raw_spin_lock_irqsave <-ata_scsi_queuecmd 1748 ls-2230 3...1 100us : _raw_spin_unlock_irqrestore <-ata_scsi_queuecmd 1749 ls-2230 3...1 101us+: trace_preempt_on <-ata_scsi_queuecmd 1750 ls-2230 3...1 111us : <stack trace> 1751 => sub_preempt_count 1752 => _raw_spin_unlock_irqrestore 1753 => ata_scsi_queuecmd 1754 => scsi_dispatch_cmd 1755 => scsi_request_fn 1756 => __blk_run_queue_uncond 1757 => __blk_run_queue 1758 => blk_queue_bio 1759 => submit_bio_noacct 1760 => submit_bio 1761 => submit_bh 1762 => ext3_bread 1763 => ext3_dir_bread 1764 => htree_dirblock_to_tree 1765 => ext3_htree_fill_tree 1766 => ext3_readdir 1767 => vfs_readdir 1768 => sys_getdents 1769 => system_call_fastpath 1770 1771 1772The trace_hardirqs_off_thunk is called from assembly on x86 when 1773interrupts are disabled in the assembly code. Without the 1774function tracing, we do not know if interrupts were enabled 1775within the preemption points. We do see that it started with 1776preemption enabled. 1777 1778Here is a trace with function-trace set:: 1779 1780 # tracer: preemptirqsoff 1781 # 1782 # preemptirqsoff latency trace v1.1.5 on 3.8.0-test+ 1783 # -------------------------------------------------------------------- 1784 # latency: 161 us, #339/339, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 1785 # ----------------- 1786 # | task: ls-2269 (uid:0 nice:0 policy:0 rt_prio:0) 1787 # ----------------- 1788 # => started at: schedule 1789 # => ended at: mutex_unlock 1790 # 1791 # 1792 # _------=> CPU# 1793 # / _-----=> irqs-off 1794 # | / _----=> need-resched 1795 # || / _---=> hardirq/softirq 1796 # ||| / _--=> preempt-depth 1797 # |||| / delay 1798 # cmd pid ||||| time | caller 1799 # \ / ||||| \ | / 1800 kworker/-59 3...1 0us : __schedule <-schedule 1801 kworker/-59 3d..1 0us : rcu_preempt_qs <-rcu_note_context_switch 1802 kworker/-59 3d..1 1us : add_preempt_count <-_raw_spin_lock_irq 1803 kworker/-59 3d..2 1us : deactivate_task <-__schedule 1804 kworker/-59 3d..2 1us : dequeue_task <-deactivate_task 1805 kworker/-59 3d..2 2us : update_rq_clock <-dequeue_task 1806 kworker/-59 3d..2 2us : dequeue_task_fair <-dequeue_task 1807 kworker/-59 3d..2 2us : update_curr <-dequeue_task_fair 1808 kworker/-59 3d..2 2us : update_min_vruntime <-update_curr 1809 kworker/-59 3d..2 3us : cpuacct_charge <-update_curr 1810 kworker/-59 3d..2 3us : __rcu_read_lock <-cpuacct_charge 1811 kworker/-59 3d..2 3us : __rcu_read_unlock <-cpuacct_charge 1812 kworker/-59 3d..2 3us : update_cfs_rq_blocked_load <-dequeue_task_fair 1813 kworker/-59 3d..2 4us : clear_buddies <-dequeue_task_fair 1814 kworker/-59 3d..2 4us : account_entity_dequeue <-dequeue_task_fair 1815 kworker/-59 3d..2 4us : update_min_vruntime <-dequeue_task_fair 1816 kworker/-59 3d..2 4us : update_cfs_shares <-dequeue_task_fair 1817 kworker/-59 3d..2 5us : hrtick_update <-dequeue_task_fair 1818 kworker/-59 3d..2 5us : wq_worker_sleeping <-__schedule 1819 kworker/-59 3d..2 5us : kthread_data <-wq_worker_sleeping 1820 kworker/-59 3d..2 5us : put_prev_task_fair <-__schedule 1821 kworker/-59 3d..2 6us : pick_next_task_fair <-pick_next_task 1822 kworker/-59 3d..2 6us : clear_buddies <-pick_next_task_fair 1823 kworker/-59 3d..2 6us : set_next_entity <-pick_next_task_fair 1824 kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity 1825 ls-2269 3d..2 7us : finish_task_switch <-__schedule 1826 ls-2269 3d..2 7us : _raw_spin_unlock_irq <-finish_task_switch 1827 ls-2269 3d..2 8us : do_IRQ <-ret_from_intr 1828 ls-2269 3d..2 8us : irq_enter <-do_IRQ 1829 ls-2269 3d..2 8us : rcu_irq_enter <-irq_enter 1830 ls-2269 3d..2 9us : add_preempt_count <-irq_enter 1831 ls-2269 3d.h2 9us : exit_idle <-do_IRQ 1832 [...] 1833 ls-2269 3d.h3 20us : sub_preempt_count <-_raw_spin_unlock 1834 ls-2269 3d.h2 20us : irq_exit <-do_IRQ 1835 ls-2269 3d.h2 21us : sub_preempt_count <-irq_exit 1836 ls-2269 3d..3 21us : do_softirq <-irq_exit 1837 ls-2269 3d..3 21us : __do_softirq <-call_softirq 1838 ls-2269 3d..3 21us+: __local_bh_disable <-__do_softirq 1839 ls-2269 3d.s4 29us : sub_preempt_count <-_local_bh_enable_ip 1840 ls-2269 3d.s5 29us : sub_preempt_count <-_local_bh_enable_ip 1841 ls-2269 3d.s5 31us : do_IRQ <-ret_from_intr 1842 ls-2269 3d.s5 31us : irq_enter <-do_IRQ 1843 ls-2269 3d.s5 31us : rcu_irq_enter <-irq_enter 1844 [...] 1845 ls-2269 3d.s5 31us : rcu_irq_enter <-irq_enter 1846 ls-2269 3d.s5 32us : add_preempt_count <-irq_enter 1847 ls-2269 3d.H5 32us : exit_idle <-do_IRQ 1848 ls-2269 3d.H5 32us : handle_irq <-do_IRQ 1849 ls-2269 3d.H5 32us : irq_to_desc <-handle_irq 1850 ls-2269 3d.H5 33us : handle_fasteoi_irq <-handle_irq 1851 [...] 1852 ls-2269 3d.s5 158us : _raw_spin_unlock_irqrestore <-rtl8139_poll 1853 ls-2269 3d.s3 158us : net_rps_action_and_irq_enable.isra.65 <-net_rx_action 1854 ls-2269 3d.s3 159us : __local_bh_enable <-__do_softirq 1855 ls-2269 3d.s3 159us : sub_preempt_count <-__local_bh_enable 1856 ls-2269 3d..3 159us : idle_cpu <-irq_exit 1857 ls-2269 3d..3 159us : rcu_irq_exit <-irq_exit 1858 ls-2269 3d..3 160us : sub_preempt_count <-irq_exit 1859 ls-2269 3d... 161us : __mutex_unlock_slowpath <-mutex_unlock 1860 ls-2269 3d... 162us+: trace_hardirqs_on <-mutex_unlock 1861 ls-2269 3d... 186us : <stack trace> 1862 => __mutex_unlock_slowpath 1863 => mutex_unlock 1864 => process_output 1865 => n_tty_write 1866 => tty_write 1867 => vfs_write 1868 => sys_write 1869 => system_call_fastpath 1870 1871This is an interesting trace. It started with kworker running and 1872scheduling out and ls taking over. But as soon as ls released the 1873rq lock and enabled interrupts (but not preemption) an interrupt 1874triggered. When the interrupt finished, it started running softirqs. 1875But while the softirq was running, another interrupt triggered. 1876When an interrupt is running inside a softirq, the annotation is 'H'. 1877 1878 1879wakeup 1880------ 1881 1882One common case that people are interested in tracing is the 1883time it takes for a task that is woken to actually wake up. 1884Now for non Real-Time tasks, this can be arbitrary. But tracing 1885it none the less can be interesting. 1886 1887Without function tracing:: 1888 1889 # echo 0 > options/function-trace 1890 # echo wakeup > current_tracer 1891 # echo 1 > tracing_on 1892 # echo 0 > tracing_max_latency 1893 # chrt -f 5 sleep 1 1894 # echo 0 > tracing_on 1895 # cat trace 1896 # tracer: wakeup 1897 # 1898 # wakeup latency trace v1.1.5 on 3.8.0-test+ 1899 # -------------------------------------------------------------------- 1900 # latency: 15 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 1901 # ----------------- 1902 # | task: kworker/3:1H-312 (uid:0 nice:-20 policy:0 rt_prio:0) 1903 # ----------------- 1904 # 1905 # _------=> CPU# 1906 # / _-----=> irqs-off 1907 # | / _----=> need-resched 1908 # || / _---=> hardirq/softirq 1909 # ||| / _--=> preempt-depth 1910 # |||| / delay 1911 # cmd pid ||||| time | caller 1912 # \ / ||||| \ | / 1913 <idle>-0 3dNs7 0us : 0:120:R + [003] 312:100:R kworker/3:1H 1914 <idle>-0 3dNs7 1us+: ttwu_do_activate.constprop.87 <-try_to_wake_up 1915 <idle>-0 3d..3 15us : __schedule <-schedule 1916 <idle>-0 3d..3 15us : 0:120:R ==> [003] 312:100:R kworker/3:1H 1917 1918The tracer only traces the highest priority task in the system 1919to avoid tracing the normal circumstances. Here we see that 1920the kworker with a nice priority of -20 (not very nice), took 1921just 15 microseconds from the time it woke up, to the time it 1922ran. 1923 1924Non Real-Time tasks are not that interesting. A more interesting 1925trace is to concentrate only on Real-Time tasks. 1926 1927wakeup_rt 1928--------- 1929 1930In a Real-Time environment it is very important to know the 1931wakeup time it takes for the highest priority task that is woken 1932up to the time that it executes. This is also known as "schedule 1933latency". I stress the point that this is about RT tasks. It is 1934also important to know the scheduling latency of non-RT tasks, 1935but the average schedule latency is better for non-RT tasks. 1936Tools like LatencyTop are more appropriate for such 1937measurements. 1938 1939Real-Time environments are interested in the worst case latency. 1940That is the longest latency it takes for something to happen, 1941and not the average. We can have a very fast scheduler that may 1942only have a large latency once in a while, but that would not 1943work well with Real-Time tasks. The wakeup_rt tracer was designed 1944to record the worst case wakeups of RT tasks. Non-RT tasks are 1945not recorded because the tracer only records one worst case and 1946tracing non-RT tasks that are unpredictable will overwrite the 1947worst case latency of RT tasks (just run the normal wakeup 1948tracer for a while to see that effect). 1949 1950Since this tracer only deals with RT tasks, we will run this 1951slightly differently than we did with the previous tracers. 1952Instead of performing an 'ls', we will run 'sleep 1' under 1953'chrt' which changes the priority of the task. 1954:: 1955 1956 # echo 0 > options/function-trace 1957 # echo wakeup_rt > current_tracer 1958 # echo 1 > tracing_on 1959 # echo 0 > tracing_max_latency 1960 # chrt -f 5 sleep 1 1961 # echo 0 > tracing_on 1962 # cat trace 1963 # tracer: wakeup 1964 # 1965 # tracer: wakeup_rt 1966 # 1967 # wakeup_rt latency trace v1.1.5 on 3.8.0-test+ 1968 # -------------------------------------------------------------------- 1969 # latency: 5 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 1970 # ----------------- 1971 # | task: sleep-2389 (uid:0 nice:0 policy:1 rt_prio:5) 1972 # ----------------- 1973 # 1974 # _------=> CPU# 1975 # / _-----=> irqs-off 1976 # | / _----=> need-resched 1977 # || / _---=> hardirq/softirq 1978 # ||| / _--=> preempt-depth 1979 # |||| / delay 1980 # cmd pid ||||| time | caller 1981 # \ / ||||| \ | / 1982 <idle>-0 3d.h4 0us : 0:120:R + [003] 2389: 94:R sleep 1983 <idle>-0 3d.h4 1us+: ttwu_do_activate.constprop.87 <-try_to_wake_up 1984 <idle>-0 3d..3 5us : __schedule <-schedule 1985 <idle>-0 3d..3 5us : 0:120:R ==> [003] 2389: 94:R sleep 1986 1987 1988Running this on an idle system, we see that it only took 5 microseconds 1989to perform the task switch. Note, since the trace point in the schedule 1990is before the actual "switch", we stop the tracing when the recorded task 1991is about to schedule in. This may change if we add a new marker at the 1992end of the scheduler. 1993 1994Notice that the recorded task is 'sleep' with the PID of 2389 1995and it has an rt_prio of 5. This priority is user-space priority 1996and not the internal kernel priority. The policy is 1 for 1997SCHED_FIFO and 2 for SCHED_RR. 1998 1999Note, that the trace data shows the internal priority (99 - rtprio). 2000:: 2001 2002 <idle>-0 3d..3 5us : 0:120:R ==> [003] 2389: 94:R sleep 2003 2004The 0:120:R means idle was running with a nice priority of 0 (120 - 120) 2005and in the running state 'R'. The sleep task was scheduled in with 20062389: 94:R. That is the priority is the kernel rtprio (99 - 5 = 94) 2007and it too is in the running state. 2008 2009Doing the same with chrt -r 5 and function-trace set. 2010:: 2011 2012 echo 1 > options/function-trace 2013 2014 # tracer: wakeup_rt 2015 # 2016 # wakeup_rt latency trace v1.1.5 on 3.8.0-test+ 2017 # -------------------------------------------------------------------- 2018 # latency: 29 us, #85/85, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 2019 # ----------------- 2020 # | task: sleep-2448 (uid:0 nice:0 policy:1 rt_prio:5) 2021 # ----------------- 2022 # 2023 # _------=> CPU# 2024 # / _-----=> irqs-off 2025 # | / _----=> need-resched 2026 # || / _---=> hardirq/softirq 2027 # ||| / _--=> preempt-depth 2028 # |||| / delay 2029 # cmd pid ||||| time | caller 2030 # \ / ||||| \ | / 2031 <idle>-0 3d.h4 1us+: 0:120:R + [003] 2448: 94:R sleep 2032 <idle>-0 3d.h4 2us : ttwu_do_activate.constprop.87 <-try_to_wake_up 2033 <idle>-0 3d.h3 3us : check_preempt_curr <-ttwu_do_wakeup 2034 <idle>-0 3d.h3 3us : resched_curr <-check_preempt_curr 2035 <idle>-0 3dNh3 4us : task_woken_rt <-ttwu_do_wakeup 2036 <idle>-0 3dNh3 4us : _raw_spin_unlock <-try_to_wake_up 2037 <idle>-0 3dNh3 4us : sub_preempt_count <-_raw_spin_unlock 2038 <idle>-0 3dNh2 5us : ttwu_stat <-try_to_wake_up 2039 <idle>-0 3dNh2 5us : _raw_spin_unlock_irqrestore <-try_to_wake_up 2040 <idle>-0 3dNh2 6us : sub_preempt_count <-_raw_spin_unlock_irqrestore 2041 <idle>-0 3dNh1 6us : _raw_spin_lock <-__run_hrtimer 2042 <idle>-0 3dNh1 6us : add_preempt_count <-_raw_spin_lock 2043 <idle>-0 3dNh2 7us : _raw_spin_unlock <-hrtimer_interrupt 2044 <idle>-0 3dNh2 7us : sub_preempt_count <-_raw_spin_unlock 2045 <idle>-0 3dNh1 7us : tick_program_event <-hrtimer_interrupt 2046 <idle>-0 3dNh1 7us : clockevents_program_event <-tick_program_event 2047 <idle>-0 3dNh1 8us : ktime_get <-clockevents_program_event 2048 <idle>-0 3dNh1 8us : lapic_next_event <-clockevents_program_event 2049 <idle>-0 3dNh1 8us : irq_exit <-smp_apic_timer_interrupt 2050 <idle>-0 3dNh1 9us : sub_preempt_count <-irq_exit 2051 <idle>-0 3dN.2 9us : idle_cpu <-irq_exit 2052 <idle>-0 3dN.2 9us : rcu_irq_exit <-irq_exit 2053 <idle>-0 3dN.2 10us : rcu_eqs_enter_common.isra.45 <-rcu_irq_exit 2054 <idle>-0 3dN.2 10us : sub_preempt_count <-irq_exit 2055 <idle>-0 3.N.1 11us : rcu_idle_exit <-cpu_idle 2056 <idle>-0 3dN.1 11us : rcu_eqs_exit_common.isra.43 <-rcu_idle_exit 2057 <idle>-0 3.N.1 11us : tick_nohz_idle_exit <-cpu_idle 2058 <idle>-0 3dN.1 12us : menu_hrtimer_cancel <-tick_nohz_idle_exit 2059 <idle>-0 3dN.1 12us : ktime_get <-tick_nohz_idle_exit 2060 <idle>-0 3dN.1 12us : tick_do_update_jiffies64 <-tick_nohz_idle_exit 2061 <idle>-0 3dN.1 13us : cpu_load_update_nohz <-tick_nohz_idle_exit 2062 <idle>-0 3dN.1 13us : _raw_spin_lock <-cpu_load_update_nohz 2063 <idle>-0 3dN.1 13us : add_preempt_count <-_raw_spin_lock 2064 <idle>-0 3dN.2 13us : __cpu_load_update <-cpu_load_update_nohz 2065 <idle>-0 3dN.2 14us : sched_avg_update <-__cpu_load_update 2066 <idle>-0 3dN.2 14us : _raw_spin_unlock <-cpu_load_update_nohz 2067 <idle>-0 3dN.2 14us : sub_preempt_count <-_raw_spin_unlock 2068 <idle>-0 3dN.1 15us : calc_load_nohz_stop <-tick_nohz_idle_exit 2069 <idle>-0 3dN.1 15us : touch_softlockup_watchdog <-tick_nohz_idle_exit 2070 <idle>-0 3dN.1 15us : hrtimer_cancel <-tick_nohz_idle_exit 2071 <idle>-0 3dN.1 15us : hrtimer_try_to_cancel <-hrtimer_cancel 2072 <idle>-0 3dN.1 16us : lock_hrtimer_base.isra.18 <-hrtimer_try_to_cancel 2073 <idle>-0 3dN.1 16us : _raw_spin_lock_irqsave <-lock_hrtimer_base.isra.18 2074 <idle>-0 3dN.1 16us : add_preempt_count <-_raw_spin_lock_irqsave 2075 <idle>-0 3dN.2 17us : __remove_hrtimer <-remove_hrtimer.part.16 2076 <idle>-0 3dN.2 17us : hrtimer_force_reprogram <-__remove_hrtimer 2077 <idle>-0 3dN.2 17us : tick_program_event <-hrtimer_force_reprogram 2078 <idle>-0 3dN.2 18us : clockevents_program_event <-tick_program_event 2079 <idle>-0 3dN.2 18us : ktime_get <-clockevents_program_event 2080 <idle>-0 3dN.2 18us : lapic_next_event <-clockevents_program_event 2081 <idle>-0 3dN.2 19us : _raw_spin_unlock_irqrestore <-hrtimer_try_to_cancel 2082 <idle>-0 3dN.2 19us : sub_preempt_count <-_raw_spin_unlock_irqrestore 2083 <idle>-0 3dN.1 19us : hrtimer_forward <-tick_nohz_idle_exit 2084 <idle>-0 3dN.1 20us : ktime_add_safe <-hrtimer_forward 2085 <idle>-0 3dN.1 20us : ktime_add_safe <-hrtimer_forward 2086 <idle>-0 3dN.1 20us : hrtimer_start_range_ns <-hrtimer_start_expires.constprop.11 2087 <idle>-0 3dN.1 20us : __hrtimer_start_range_ns <-hrtimer_start_range_ns 2088 <idle>-0 3dN.1 21us : lock_hrtimer_base.isra.18 <-__hrtimer_start_range_ns 2089 <idle>-0 3dN.1 21us : _raw_spin_lock_irqsave <-lock_hrtimer_base.isra.18 2090 <idle>-0 3dN.1 21us : add_preempt_count <-_raw_spin_lock_irqsave 2091 <idle>-0 3dN.2 22us : ktime_add_safe <-__hrtimer_start_range_ns 2092 <idle>-0 3dN.2 22us : enqueue_hrtimer <-__hrtimer_start_range_ns 2093 <idle>-0 3dN.2 22us : tick_program_event <-__hrtimer_start_range_ns 2094 <idle>-0 3dN.2 23us : clockevents_program_event <-tick_program_event 2095 <idle>-0 3dN.2 23us : ktime_get <-clockevents_program_event 2096 <idle>-0 3dN.2 23us : lapic_next_event <-clockevents_program_event 2097 <idle>-0 3dN.2 24us : _raw_spin_unlock_irqrestore <-__hrtimer_start_range_ns 2098 <idle>-0 3dN.2 24us : sub_preempt_count <-_raw_spin_unlock_irqrestore 2099 <idle>-0 3dN.1 24us : account_idle_ticks <-tick_nohz_idle_exit 2100 <idle>-0 3dN.1 24us : account_idle_time <-account_idle_ticks 2101 <idle>-0 3.N.1 25us : sub_preempt_count <-cpu_idle 2102 <idle>-0 3.N.. 25us : schedule <-cpu_idle 2103 <idle>-0 3.N.. 25us : __schedule <-preempt_schedule 2104 <idle>-0 3.N.. 26us : add_preempt_count <-__schedule 2105 <idle>-0 3.N.1 26us : rcu_note_context_switch <-__schedule 2106 <idle>-0 3.N.1 26us : rcu_sched_qs <-rcu_note_context_switch 2107 <idle>-0 3dN.1 27us : rcu_preempt_qs <-rcu_note_context_switch 2108 <idle>-0 3.N.1 27us : _raw_spin_lock_irq <-__schedule 2109 <idle>-0 3dN.1 27us : add_preempt_count <-_raw_spin_lock_irq 2110 <idle>-0 3dN.2 28us : put_prev_task_idle <-__schedule 2111 <idle>-0 3dN.2 28us : pick_next_task_stop <-pick_next_task 2112 <idle>-0 3dN.2 28us : pick_next_task_rt <-pick_next_task 2113 <idle>-0 3dN.2 29us : dequeue_pushable_task <-pick_next_task_rt 2114 <idle>-0 3d..3 29us : __schedule <-preempt_schedule 2115 <idle>-0 3d..3 30us : 0:120:R ==> [003] 2448: 94:R sleep 2116 2117This isn't that big of a trace, even with function tracing enabled, 2118so I included the entire trace. 2119 2120The interrupt went off while when the system was idle. Somewhere 2121before task_woken_rt() was called, the NEED_RESCHED flag was set, 2122this is indicated by the first occurrence of the 'N' flag. 2123 2124Latency tracing and events 2125-------------------------- 2126As function tracing can induce a much larger latency, but without 2127seeing what happens within the latency it is hard to know what 2128caused it. There is a middle ground, and that is with enabling 2129events. 2130:: 2131 2132 # echo 0 > options/function-trace 2133 # echo wakeup_rt > current_tracer 2134 # echo 1 > events/enable 2135 # echo 1 > tracing_on 2136 # echo 0 > tracing_max_latency 2137 # chrt -f 5 sleep 1 2138 # echo 0 > tracing_on 2139 # cat trace 2140 # tracer: wakeup_rt 2141 # 2142 # wakeup_rt latency trace v1.1.5 on 3.8.0-test+ 2143 # -------------------------------------------------------------------- 2144 # latency: 6 us, #12/12, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) 2145 # ----------------- 2146 # | task: sleep-5882 (uid:0 nice:0 policy:1 rt_prio:5) 2147 # ----------------- 2148 # 2149 # _------=> CPU# 2150 # / _-----=> irqs-off 2151 # | / _----=> need-resched 2152 # || / _---=> hardirq/softirq 2153 # ||| / _--=> preempt-depth 2154 # |||| / delay 2155 # cmd pid ||||| time | caller 2156 # \ / ||||| \ | / 2157 <idle>-0 2d.h4 0us : 0:120:R + [002] 5882: 94:R sleep 2158 <idle>-0 2d.h4 0us : ttwu_do_activate.constprop.87 <-try_to_wake_up 2159 <idle>-0 2d.h4 1us : sched_wakeup: comm=sleep pid=5882 prio=94 success=1 target_cpu=002 2160 <idle>-0 2dNh2 1us : hrtimer_expire_exit: hrtimer=ffff88007796feb8 2161 <idle>-0 2.N.2 2us : power_end: cpu_id=2 2162 <idle>-0 2.N.2 3us : cpu_idle: state=4294967295 cpu_id=2 2163 <idle>-0 2dN.3 4us : hrtimer_cancel: hrtimer=ffff88007d50d5e0 2164 <idle>-0 2dN.3 4us : hrtimer_start: hrtimer=ffff88007d50d5e0 function=tick_sched_timer expires=34311211000000 softexpires=34311211000000 2165 <idle>-0 2.N.2 5us : rcu_utilization: Start context switch 2166 <idle>-0 2.N.2 5us : rcu_utilization: End context switch 2167 <idle>-0 2d..3 6us : __schedule <-schedule 2168 <idle>-0 2d..3 6us : 0:120:R ==> [002] 5882: 94:R sleep 2169 2170 2171Hardware Latency Detector 2172------------------------- 2173 2174The hardware latency detector is executed by enabling the "hwlat" tracer. 2175 2176NOTE, this tracer will affect the performance of the system as it will 2177periodically make a CPU constantly busy with interrupts disabled. 2178:: 2179 2180 # echo hwlat > current_tracer 2181 # sleep 100 2182 # cat trace 2183 # tracer: hwlat 2184 # 2185 # entries-in-buffer/entries-written: 13/13 #P:8 2186 # 2187 # _-----=> irqs-off 2188 # / _----=> need-resched 2189 # | / _---=> hardirq/softirq 2190 # || / _--=> preempt-depth 2191 # ||| / delay 2192 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 2193 # | | | |||| | | 2194 <...>-1729 [001] d... 678.473449: #1 inner/outer(us): 11/12 ts:1581527483.343962693 count:6 2195 <...>-1729 [004] d... 689.556542: #2 inner/outer(us): 16/9 ts:1581527494.889008092 count:1 2196 <...>-1729 [005] d... 714.756290: #3 inner/outer(us): 16/16 ts:1581527519.678961629 count:5 2197 <...>-1729 [001] d... 718.788247: #4 inner/outer(us): 9/17 ts:1581527523.889012713 count:1 2198 <...>-1729 [002] d... 719.796341: #5 inner/outer(us): 13/9 ts:1581527524.912872606 count:1 2199 <...>-1729 [006] d... 844.787091: #6 inner/outer(us): 9/12 ts:1581527649.889048502 count:2 2200 <...>-1729 [003] d... 849.827033: #7 inner/outer(us): 18/9 ts:1581527654.889013793 count:1 2201 <...>-1729 [007] d... 853.859002: #8 inner/outer(us): 9/12 ts:1581527658.889065736 count:1 2202 <...>-1729 [001] d... 855.874978: #9 inner/outer(us): 9/11 ts:1581527660.861991877 count:1 2203 <...>-1729 [001] d... 863.938932: #10 inner/outer(us): 9/11 ts:1581527668.970010500 count:1 nmi-total:7 nmi-count:1 2204 <...>-1729 [007] d... 878.050780: #11 inner/outer(us): 9/12 ts:1581527683.385002600 count:1 nmi-total:5 nmi-count:1 2205 <...>-1729 [007] d... 886.114702: #12 inner/outer(us): 9/12 ts:1581527691.385001600 count:1 2206 2207 2208The above output is somewhat the same in the header. All events will have 2209interrupts disabled 'd'. Under the FUNCTION title there is: 2210 2211 #1 2212 This is the count of events recorded that were greater than the 2213 tracing_threshold (See below). 2214 2215 inner/outer(us): 11/11 2216 2217 This shows two numbers as "inner latency" and "outer latency". The test 2218 runs in a loop checking a timestamp twice. The latency detected within 2219 the two timestamps is the "inner latency" and the latency detected 2220 after the previous timestamp and the next timestamp in the loop is 2221 the "outer latency". 2222 2223 ts:1581527483.343962693 2224 2225 The absolute timestamp that the first latency was recorded in the window. 2226 2227 count:6 2228 2229 The number of times a latency was detected during the window. 2230 2231 nmi-total:7 nmi-count:1 2232 2233 On architectures that support it, if an NMI comes in during the 2234 test, the time spent in NMI is reported in "nmi-total" (in 2235 microseconds). 2236 2237 All architectures that have NMIs will show the "nmi-count" if an 2238 NMI comes in during the test. 2239 2240hwlat files: 2241 2242 tracing_threshold 2243 This gets automatically set to "10" to represent 10 2244 microseconds. This is the threshold of latency that 2245 needs to be detected before the trace will be recorded. 2246 2247 Note, when hwlat tracer is finished (another tracer is 2248 written into "current_tracer"), the original value for 2249 tracing_threshold is placed back into this file. 2250 2251 hwlat_detector/width 2252 The length of time the test runs with interrupts disabled. 2253 2254 hwlat_detector/window 2255 The length of time of the window which the test 2256 runs. That is, the test will run for "width" 2257 microseconds per "window" microseconds 2258 2259 tracing_cpumask 2260 When the test is started. A kernel thread is created that 2261 runs the test. This thread will alternate between CPUs 2262 listed in the tracing_cpumask between each period 2263 (one "window"). To limit the test to specific CPUs 2264 set the mask in this file to only the CPUs that the test 2265 should run on. 2266 2267function 2268-------- 2269 2270This tracer is the function tracer. Enabling the function tracer 2271can be done from the debug file system. Make sure the 2272ftrace_enabled is set; otherwise this tracer is a nop. 2273See the "ftrace_enabled" section below. 2274:: 2275 2276 # sysctl kernel.ftrace_enabled=1 2277 # echo function > current_tracer 2278 # echo 1 > tracing_on 2279 # usleep 1 2280 # echo 0 > tracing_on 2281 # cat trace 2282 # tracer: function 2283 # 2284 # entries-in-buffer/entries-written: 24799/24799 #P:4 2285 # 2286 # _-----=> irqs-off 2287 # / _----=> need-resched 2288 # | / _---=> hardirq/softirq 2289 # || / _--=> preempt-depth 2290 # ||| / delay 2291 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 2292 # | | | |||| | | 2293 bash-1994 [002] .... 3082.063030: mutex_unlock <-rb_simple_write 2294 bash-1994 [002] .... 3082.063031: __mutex_unlock_slowpath <-mutex_unlock 2295 bash-1994 [002] .... 3082.063031: __fsnotify_parent <-fsnotify_modify 2296 bash-1994 [002] .... 3082.063032: fsnotify <-fsnotify_modify 2297 bash-1994 [002] .... 3082.063032: __srcu_read_lock <-fsnotify 2298 bash-1994 [002] .... 3082.063032: add_preempt_count <-__srcu_read_lock 2299 bash-1994 [002] ...1 3082.063032: sub_preempt_count <-__srcu_read_lock 2300 bash-1994 [002] .... 3082.063033: __srcu_read_unlock <-fsnotify 2301 [...] 2302 2303 2304Note: function tracer uses ring buffers to store the above 2305entries. The newest data may overwrite the oldest data. 2306Sometimes using echo to stop the trace is not sufficient because 2307the tracing could have overwritten the data that you wanted to 2308record. For this reason, it is sometimes better to disable 2309tracing directly from a program. This allows you to stop the 2310tracing at the point that you hit the part that you are 2311interested in. To disable the tracing directly from a C program, 2312something like following code snippet can be used:: 2313 2314 int trace_fd; 2315 [...] 2316 int main(int argc, char *argv[]) { 2317 [...] 2318 trace_fd = open(tracing_file("tracing_on"), O_WRONLY); 2319 [...] 2320 if (condition_hit()) { 2321 write(trace_fd, "0", 1); 2322 } 2323 [...] 2324 } 2325 2326 2327Single thread tracing 2328--------------------- 2329 2330By writing into set_ftrace_pid you can trace a 2331single thread. For example:: 2332 2333 # cat set_ftrace_pid 2334 no pid 2335 # echo 3111 > set_ftrace_pid 2336 # cat set_ftrace_pid 2337 3111 2338 # echo function > current_tracer 2339 # cat trace | head 2340 # tracer: function 2341 # 2342 # TASK-PID CPU# TIMESTAMP FUNCTION 2343 # | | | | | 2344 yum-updatesd-3111 [003] 1637.254676: finish_task_switch <-thread_return 2345 yum-updatesd-3111 [003] 1637.254681: hrtimer_cancel <-schedule_hrtimeout_range 2346 yum-updatesd-3111 [003] 1637.254682: hrtimer_try_to_cancel <-hrtimer_cancel 2347 yum-updatesd-3111 [003] 1637.254683: lock_hrtimer_base <-hrtimer_try_to_cancel 2348 yum-updatesd-3111 [003] 1637.254685: fget_light <-do_sys_poll 2349 yum-updatesd-3111 [003] 1637.254686: pipe_poll <-do_sys_poll 2350 # echo > set_ftrace_pid 2351 # cat trace |head 2352 # tracer: function 2353 # 2354 # TASK-PID CPU# TIMESTAMP FUNCTION 2355 # | | | | | 2356 ##### CPU 3 buffer started #### 2357 yum-updatesd-3111 [003] 1701.957688: free_poll_entry <-poll_freewait 2358 yum-updatesd-3111 [003] 1701.957689: remove_wait_queue <-free_poll_entry 2359 yum-updatesd-3111 [003] 1701.957691: fput <-free_poll_entry 2360 yum-updatesd-3111 [003] 1701.957692: audit_syscall_exit <-sysret_audit 2361 yum-updatesd-3111 [003] 1701.957693: path_put <-audit_syscall_exit 2362 2363If you want to trace a function when executing, you could use 2364something like this simple program. 2365:: 2366 2367 #include <stdio.h> 2368 #include <stdlib.h> 2369 #include <sys/types.h> 2370 #include <sys/stat.h> 2371 #include <fcntl.h> 2372 #include <unistd.h> 2373 #include <string.h> 2374 2375 #define _STR(x) #x 2376 #define STR(x) _STR(x) 2377 #define MAX_PATH 256 2378 2379 const char *find_tracefs(void) 2380 { 2381 static char tracefs[MAX_PATH+1]; 2382 static int tracefs_found; 2383 char type[100]; 2384 FILE *fp; 2385 2386 if (tracefs_found) 2387 return tracefs; 2388 2389 if ((fp = fopen("/proc/mounts","r")) == NULL) { 2390 perror("/proc/mounts"); 2391 return NULL; 2392 } 2393 2394 while (fscanf(fp, "%*s %" 2395 STR(MAX_PATH) 2396 "s %99s %*s %*d %*d\n", 2397 tracefs, type) == 2) { 2398 if (strcmp(type, "tracefs") == 0) 2399 break; 2400 } 2401 fclose(fp); 2402 2403 if (strcmp(type, "tracefs") != 0) { 2404 fprintf(stderr, "tracefs not mounted"); 2405 return NULL; 2406 } 2407 2408 strcat(tracefs, "/tracing/"); 2409 tracefs_found = 1; 2410 2411 return tracefs; 2412 } 2413 2414 const char *tracing_file(const char *file_name) 2415 { 2416 static char trace_file[MAX_PATH+1]; 2417 snprintf(trace_file, MAX_PATH, "%s/%s", find_tracefs(), file_name); 2418 return trace_file; 2419 } 2420 2421 int main (int argc, char **argv) 2422 { 2423 if (argc < 1) 2424 exit(-1); 2425 2426 if (fork() > 0) { 2427 int fd, ffd; 2428 char line[64]; 2429 int s; 2430 2431 ffd = open(tracing_file("current_tracer"), O_WRONLY); 2432 if (ffd < 0) 2433 exit(-1); 2434 write(ffd, "nop", 3); 2435 2436 fd = open(tracing_file("set_ftrace_pid"), O_WRONLY); 2437 s = sprintf(line, "%d\n", getpid()); 2438 write(fd, line, s); 2439 2440 write(ffd, "function", 8); 2441 2442 close(fd); 2443 close(ffd); 2444 2445 execvp(argv[1], argv+1); 2446 } 2447 2448 return 0; 2449 } 2450 2451Or this simple script! 2452:: 2453 2454 #!/bin/bash 2455 2456 tracefs=`sed -ne 's/^tracefs \(.*\) tracefs.*/\1/p' /proc/mounts` 2457 echo 0 > $tracefs/tracing_on 2458 echo $$ > $tracefs/set_ftrace_pid 2459 echo function > $tracefs/current_tracer 2460 echo 1 > $tracefs/tracing_on 2461 exec "$@" 2462 2463 2464function graph tracer 2465--------------------------- 2466 2467This tracer is similar to the function tracer except that it 2468probes a function on its entry and its exit. This is done by 2469using a dynamically allocated stack of return addresses in each 2470task_struct. On function entry the tracer overwrites the return 2471address of each function traced to set a custom probe. Thus the 2472original return address is stored on the stack of return address 2473in the task_struct. 2474 2475Probing on both ends of a function leads to special features 2476such as: 2477 2478- measure of a function's time execution 2479- having a reliable call stack to draw function calls graph 2480 2481This tracer is useful in several situations: 2482 2483- you want to find the reason of a strange kernel behavior and 2484 need to see what happens in detail on any areas (or specific 2485 ones). 2486 2487- you are experiencing weird latencies but it's difficult to 2488 find its origin. 2489 2490- you want to find quickly which path is taken by a specific 2491 function 2492 2493- you just want to peek inside a working kernel and want to see 2494 what happens there. 2495 2496:: 2497 2498 # tracer: function_graph 2499 # 2500 # CPU DURATION FUNCTION CALLS 2501 # | | | | | | | 2502 2503 0) | sys_open() { 2504 0) | do_sys_open() { 2505 0) | getname() { 2506 0) | kmem_cache_alloc() { 2507 0) 1.382 us | __might_sleep(); 2508 0) 2.478 us | } 2509 0) | strncpy_from_user() { 2510 0) | might_fault() { 2511 0) 1.389 us | __might_sleep(); 2512 0) 2.553 us | } 2513 0) 3.807 us | } 2514 0) 7.876 us | } 2515 0) | alloc_fd() { 2516 0) 0.668 us | _spin_lock(); 2517 0) 0.570 us | expand_files(); 2518 0) 0.586 us | _spin_unlock(); 2519 2520 2521There are several columns that can be dynamically 2522enabled/disabled. You can use every combination of options you 2523want, depending on your needs. 2524 2525- The cpu number on which the function executed is default 2526 enabled. It is sometimes better to only trace one cpu (see 2527 tracing_cpu_mask file) or you might sometimes see unordered 2528 function calls while cpu tracing switch. 2529 2530 - hide: echo nofuncgraph-cpu > trace_options 2531 - show: echo funcgraph-cpu > trace_options 2532 2533- The duration (function's time of execution) is displayed on 2534 the closing bracket line of a function or on the same line 2535 than the current function in case of a leaf one. It is default 2536 enabled. 2537 2538 - hide: echo nofuncgraph-duration > trace_options 2539 - show: echo funcgraph-duration > trace_options 2540 2541- The overhead field precedes the duration field in case of 2542 reached duration thresholds. 2543 2544 - hide: echo nofuncgraph-overhead > trace_options 2545 - show: echo funcgraph-overhead > trace_options 2546 - depends on: funcgraph-duration 2547 2548 ie:: 2549 2550 3) # 1837.709 us | } /* __switch_to */ 2551 3) | finish_task_switch() { 2552 3) 0.313 us | _raw_spin_unlock_irq(); 2553 3) 3.177 us | } 2554 3) # 1889.063 us | } /* __schedule */ 2555 3) ! 140.417 us | } /* __schedule */ 2556 3) # 2034.948 us | } /* schedule */ 2557 3) * 33998.59 us | } /* schedule_preempt_disabled */ 2558 2559 [...] 2560 2561 1) 0.260 us | msecs_to_jiffies(); 2562 1) 0.313 us | __rcu_read_unlock(); 2563 1) + 61.770 us | } 2564 1) + 64.479 us | } 2565 1) 0.313 us | rcu_bh_qs(); 2566 1) 0.313 us | __local_bh_enable(); 2567 1) ! 217.240 us | } 2568 1) 0.365 us | idle_cpu(); 2569 1) | rcu_irq_exit() { 2570 1) 0.417 us | rcu_eqs_enter_common.isra.47(); 2571 1) 3.125 us | } 2572 1) ! 227.812 us | } 2573 1) ! 457.395 us | } 2574 1) @ 119760.2 us | } 2575 2576 [...] 2577 2578 2) | handle_IPI() { 2579 1) 6.979 us | } 2580 2) 0.417 us | scheduler_ipi(); 2581 1) 9.791 us | } 2582 1) + 12.917 us | } 2583 2) 3.490 us | } 2584 1) + 15.729 us | } 2585 1) + 18.542 us | } 2586 2) $ 3594274 us | } 2587 2588Flags:: 2589 2590 + means that the function exceeded 10 usecs. 2591 ! means that the function exceeded 100 usecs. 2592 # means that the function exceeded 1000 usecs. 2593 * means that the function exceeded 10 msecs. 2594 @ means that the function exceeded 100 msecs. 2595 $ means that the function exceeded 1 sec. 2596 2597 2598- The task/pid field displays the thread cmdline and pid which 2599 executed the function. It is default disabled. 2600 2601 - hide: echo nofuncgraph-proc > trace_options 2602 - show: echo funcgraph-proc > trace_options 2603 2604 ie:: 2605 2606 # tracer: function_graph 2607 # 2608 # CPU TASK/PID DURATION FUNCTION CALLS 2609 # | | | | | | | | | 2610 0) sh-4802 | | d_free() { 2611 0) sh-4802 | | call_rcu() { 2612 0) sh-4802 | | __call_rcu() { 2613 0) sh-4802 | 0.616 us | rcu_process_gp_end(); 2614 0) sh-4802 | 0.586 us | check_for_new_grace_period(); 2615 0) sh-4802 | 2.899 us | } 2616 0) sh-4802 | 4.040 us | } 2617 0) sh-4802 | 5.151 us | } 2618 0) sh-4802 | + 49.370 us | } 2619 2620 2621- The absolute time field is an absolute timestamp given by the 2622 system clock since it started. A snapshot of this time is 2623 given on each entry/exit of functions 2624 2625 - hide: echo nofuncgraph-abstime > trace_options 2626 - show: echo funcgraph-abstime > trace_options 2627 2628 ie:: 2629 2630 # 2631 # TIME CPU DURATION FUNCTION CALLS 2632 # | | | | | | | | 2633 360.774522 | 1) 0.541 us | } 2634 360.774522 | 1) 4.663 us | } 2635 360.774523 | 1) 0.541 us | __wake_up_bit(); 2636 360.774524 | 1) 6.796 us | } 2637 360.774524 | 1) 7.952 us | } 2638 360.774525 | 1) 9.063 us | } 2639 360.774525 | 1) 0.615 us | journal_mark_dirty(); 2640 360.774527 | 1) 0.578 us | __brelse(); 2641 360.774528 | 1) | reiserfs_prepare_for_journal() { 2642 360.774528 | 1) | unlock_buffer() { 2643 360.774529 | 1) | wake_up_bit() { 2644 360.774529 | 1) | bit_waitqueue() { 2645 360.774530 | 1) 0.594 us | __phys_addr(); 2646 2647 2648The function name is always displayed after the closing bracket 2649for a function if the start of that function is not in the 2650trace buffer. 2651 2652Display of the function name after the closing bracket may be 2653enabled for functions whose start is in the trace buffer, 2654allowing easier searching with grep for function durations. 2655It is default disabled. 2656 2657 - hide: echo nofuncgraph-tail > trace_options 2658 - show: echo funcgraph-tail > trace_options 2659 2660 Example with nofuncgraph-tail (default):: 2661 2662 0) | putname() { 2663 0) | kmem_cache_free() { 2664 0) 0.518 us | __phys_addr(); 2665 0) 1.757 us | } 2666 0) 2.861 us | } 2667 2668 Example with funcgraph-tail:: 2669 2670 0) | putname() { 2671 0) | kmem_cache_free() { 2672 0) 0.518 us | __phys_addr(); 2673 0) 1.757 us | } /* kmem_cache_free() */ 2674 0) 2.861 us | } /* putname() */ 2675 2676You can put some comments on specific functions by using 2677trace_printk() For example, if you want to put a comment inside 2678the __might_sleep() function, you just have to include 2679<linux/ftrace.h> and call trace_printk() inside __might_sleep():: 2680 2681 trace_printk("I'm a comment!\n") 2682 2683will produce:: 2684 2685 1) | __might_sleep() { 2686 1) | /* I'm a comment! */ 2687 1) 1.449 us | } 2688 2689 2690You might find other useful features for this tracer in the 2691following "dynamic ftrace" section such as tracing only specific 2692functions or tasks. 2693 2694dynamic ftrace 2695-------------- 2696 2697If CONFIG_DYNAMIC_FTRACE is set, the system will run with 2698virtually no overhead when function tracing is disabled. The way 2699this works is the mcount function call (placed at the start of 2700every kernel function, produced by the -pg switch in gcc), 2701starts of pointing to a simple return. (Enabling FTRACE will 2702include the -pg switch in the compiling of the kernel.) 2703 2704At compile time every C file object is run through the 2705recordmcount program (located in the scripts directory). This 2706program will parse the ELF headers in the C object to find all 2707the locations in the .text section that call mcount. Starting 2708with gcc version 4.6, the -mfentry has been added for x86, which 2709calls "__fentry__" instead of "mcount". Which is called before 2710the creation of the stack frame. 2711 2712Note, not all sections are traced. They may be prevented by either 2713a notrace, or blocked another way and all inline functions are not 2714traced. Check the "available_filter_functions" file to see what functions 2715can be traced. 2716 2717A section called "__mcount_loc" is created that holds 2718references to all the mcount/fentry call sites in the .text section. 2719The recordmcount program re-links this section back into the 2720original object. The final linking stage of the kernel will add all these 2721references into a single table. 2722 2723On boot up, before SMP is initialized, the dynamic ftrace code 2724scans this table and updates all the locations into nops. It 2725also records the locations, which are added to the 2726available_filter_functions list. Modules are processed as they 2727are loaded and before they are executed. When a module is 2728unloaded, it also removes its functions from the ftrace function 2729list. This is automatic in the module unload code, and the 2730module author does not need to worry about it. 2731 2732When tracing is enabled, the process of modifying the function 2733tracepoints is dependent on architecture. The old method is to use 2734kstop_machine to prevent races with the CPUs executing code being 2735modified (which can cause the CPU to do undesirable things, especially 2736if the modified code crosses cache (or page) boundaries), and the nops are 2737patched back to calls. But this time, they do not call mcount 2738(which is just a function stub). They now call into the ftrace 2739infrastructure. 2740 2741The new method of modifying the function tracepoints is to place 2742a breakpoint at the location to be modified, sync all CPUs, modify 2743the rest of the instruction not covered by the breakpoint. Sync 2744all CPUs again, and then remove the breakpoint with the finished 2745version to the ftrace call site. 2746 2747Some archs do not even need to monkey around with the synchronization, 2748and can just slap the new code on top of the old without any 2749problems with other CPUs executing it at the same time. 2750 2751One special side-effect to the recording of the functions being 2752traced is that we can now selectively choose which functions we 2753wish to trace and which ones we want the mcount calls to remain 2754as nops. 2755 2756Two files are used, one for enabling and one for disabling the 2757tracing of specified functions. They are: 2758 2759 set_ftrace_filter 2760 2761and 2762 2763 set_ftrace_notrace 2764 2765A list of available functions that you can add to these files is 2766listed in: 2767 2768 available_filter_functions 2769 2770:: 2771 2772 # cat available_filter_functions 2773 put_prev_task_idle 2774 kmem_cache_create 2775 pick_next_task_rt 2776 cpus_read_lock 2777 pick_next_task_fair 2778 mutex_lock 2779 [...] 2780 2781If I am only interested in sys_nanosleep and hrtimer_interrupt:: 2782 2783 # echo sys_nanosleep hrtimer_interrupt > set_ftrace_filter 2784 # echo function > current_tracer 2785 # echo 1 > tracing_on 2786 # usleep 1 2787 # echo 0 > tracing_on 2788 # cat trace 2789 # tracer: function 2790 # 2791 # entries-in-buffer/entries-written: 5/5 #P:4 2792 # 2793 # _-----=> irqs-off 2794 # / _----=> need-resched 2795 # | / _---=> hardirq/softirq 2796 # || / _--=> preempt-depth 2797 # ||| / delay 2798 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 2799 # | | | |||| | | 2800 usleep-2665 [001] .... 4186.475355: sys_nanosleep <-system_call_fastpath 2801 <idle>-0 [001] d.h1 4186.475409: hrtimer_interrupt <-smp_apic_timer_interrupt 2802 usleep-2665 [001] d.h1 4186.475426: hrtimer_interrupt <-smp_apic_timer_interrupt 2803 <idle>-0 [003] d.h1 4186.475426: hrtimer_interrupt <-smp_apic_timer_interrupt 2804 <idle>-0 [002] d.h1 4186.475427: hrtimer_interrupt <-smp_apic_timer_interrupt 2805 2806To see which functions are being traced, you can cat the file: 2807:: 2808 2809 # cat set_ftrace_filter 2810 hrtimer_interrupt 2811 sys_nanosleep 2812 2813 2814Perhaps this is not enough. The filters also allow glob(7) matching. 2815 2816 ``<match>*`` 2817 will match functions that begin with <match> 2818 ``*<match>`` 2819 will match functions that end with <match> 2820 ``*<match>*`` 2821 will match functions that have <match> in it 2822 ``<match1>*<match2>`` 2823 will match functions that begin with <match1> and end with <match2> 2824 2825.. note:: 2826 It is better to use quotes to enclose the wild cards, 2827 otherwise the shell may expand the parameters into names 2828 of files in the local directory. 2829 2830:: 2831 2832 # echo 'hrtimer_*' > set_ftrace_filter 2833 2834Produces:: 2835 2836 # tracer: function 2837 # 2838 # entries-in-buffer/entries-written: 897/897 #P:4 2839 # 2840 # _-----=> irqs-off 2841 # / _----=> need-resched 2842 # | / _---=> hardirq/softirq 2843 # || / _--=> preempt-depth 2844 # ||| / delay 2845 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 2846 # | | | |||| | | 2847 <idle>-0 [003] dN.1 4228.547803: hrtimer_cancel <-tick_nohz_idle_exit 2848 <idle>-0 [003] dN.1 4228.547804: hrtimer_try_to_cancel <-hrtimer_cancel 2849 <idle>-0 [003] dN.2 4228.547805: hrtimer_force_reprogram <-__remove_hrtimer 2850 <idle>-0 [003] dN.1 4228.547805: hrtimer_forward <-tick_nohz_idle_exit 2851 <idle>-0 [003] dN.1 4228.547805: hrtimer_start_range_ns <-hrtimer_start_expires.constprop.11 2852 <idle>-0 [003] d..1 4228.547858: hrtimer_get_next_event <-get_next_timer_interrupt 2853 <idle>-0 [003] d..1 4228.547859: hrtimer_start <-__tick_nohz_idle_enter 2854 <idle>-0 [003] d..2 4228.547860: hrtimer_force_reprogram <-__rem 2855 2856Notice that we lost the sys_nanosleep. 2857:: 2858 2859 # cat set_ftrace_filter 2860 hrtimer_run_queues 2861 hrtimer_run_pending 2862 hrtimer_init 2863 hrtimer_cancel 2864 hrtimer_try_to_cancel 2865 hrtimer_forward 2866 hrtimer_start 2867 hrtimer_reprogram 2868 hrtimer_force_reprogram 2869 hrtimer_get_next_event 2870 hrtimer_interrupt 2871 hrtimer_nanosleep 2872 hrtimer_wakeup 2873 hrtimer_get_remaining 2874 hrtimer_get_res 2875 hrtimer_init_sleeper 2876 2877 2878This is because the '>' and '>>' act just like they do in bash. 2879To rewrite the filters, use '>' 2880To append to the filters, use '>>' 2881 2882To clear out a filter so that all functions will be recorded 2883again:: 2884 2885 # echo > set_ftrace_filter 2886 # cat set_ftrace_filter 2887 # 2888 2889Again, now we want to append. 2890 2891:: 2892 2893 # echo sys_nanosleep > set_ftrace_filter 2894 # cat set_ftrace_filter 2895 sys_nanosleep 2896 # echo 'hrtimer_*' >> set_ftrace_filter 2897 # cat set_ftrace_filter 2898 hrtimer_run_queues 2899 hrtimer_run_pending 2900 hrtimer_init 2901 hrtimer_cancel 2902 hrtimer_try_to_cancel 2903 hrtimer_forward 2904 hrtimer_start 2905 hrtimer_reprogram 2906 hrtimer_force_reprogram 2907 hrtimer_get_next_event 2908 hrtimer_interrupt 2909 sys_nanosleep 2910 hrtimer_nanosleep 2911 hrtimer_wakeup 2912 hrtimer_get_remaining 2913 hrtimer_get_res 2914 hrtimer_init_sleeper 2915 2916 2917The set_ftrace_notrace prevents those functions from being 2918traced. 2919:: 2920 2921 # echo '*preempt*' '*lock*' > set_ftrace_notrace 2922 2923Produces:: 2924 2925 # tracer: function 2926 # 2927 # entries-in-buffer/entries-written: 39608/39608 #P:4 2928 # 2929 # _-----=> irqs-off 2930 # / _----=> need-resched 2931 # | / _---=> hardirq/softirq 2932 # || / _--=> preempt-depth 2933 # ||| / delay 2934 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 2935 # | | | |||| | | 2936 bash-1994 [000] .... 4342.324896: file_ra_state_init <-do_dentry_open 2937 bash-1994 [000] .... 4342.324897: open_check_o_direct <-do_last 2938 bash-1994 [000] .... 4342.324897: ima_file_check <-do_last 2939 bash-1994 [000] .... 4342.324898: process_measurement <-ima_file_check 2940 bash-1994 [000] .... 4342.324898: ima_get_action <-process_measurement 2941 bash-1994 [000] .... 4342.324898: ima_match_policy <-ima_get_action 2942 bash-1994 [000] .... 4342.324899: do_truncate <-do_last 2943 bash-1994 [000] .... 4342.324899: setattr_should_drop_suidgid <-do_truncate 2944 bash-1994 [000] .... 4342.324899: notify_change <-do_truncate 2945 bash-1994 [000] .... 4342.324900: current_fs_time <-notify_change 2946 bash-1994 [000] .... 4342.324900: current_kernel_time <-current_fs_time 2947 bash-1994 [000] .... 4342.324900: timespec_trunc <-current_fs_time 2948 2949We can see that there's no more lock or preempt tracing. 2950 2951Selecting function filters via index 2952------------------------------------ 2953 2954Because processing of strings is expensive (the address of the function 2955needs to be looked up before comparing to the string being passed in), 2956an index can be used as well to enable functions. This is useful in the 2957case of setting thousands of specific functions at a time. By passing 2958in a list of numbers, no string processing will occur. Instead, the function 2959at the specific location in the internal array (which corresponds to the 2960functions in the "available_filter_functions" file), is selected. 2961 2962:: 2963 2964 # echo 1 > set_ftrace_filter 2965 2966Will select the first function listed in "available_filter_functions" 2967 2968:: 2969 2970 # head -1 available_filter_functions 2971 trace_initcall_finish_cb 2972 2973 # cat set_ftrace_filter 2974 trace_initcall_finish_cb 2975 2976 # head -50 available_filter_functions | tail -1 2977 x86_pmu_commit_txn 2978 2979 # echo 1 50 > set_ftrace_filter 2980 # cat set_ftrace_filter 2981 trace_initcall_finish_cb 2982 x86_pmu_commit_txn 2983 2984Dynamic ftrace with the function graph tracer 2985--------------------------------------------- 2986 2987Although what has been explained above concerns both the 2988function tracer and the function-graph-tracer, there are some 2989special features only available in the function-graph tracer. 2990 2991If you want to trace only one function and all of its children, 2992you just have to echo its name into set_graph_function:: 2993 2994 echo __do_fault > set_graph_function 2995 2996will produce the following "expanded" trace of the __do_fault() 2997function:: 2998 2999 0) | __do_fault() { 3000 0) | filemap_fault() { 3001 0) | find_lock_page() { 3002 0) 0.804 us | find_get_page(); 3003 0) | __might_sleep() { 3004 0) 1.329 us | } 3005 0) 3.904 us | } 3006 0) 4.979 us | } 3007 0) 0.653 us | _spin_lock(); 3008 0) 0.578 us | page_add_file_rmap(); 3009 0) 0.525 us | native_set_pte_at(); 3010 0) 0.585 us | _spin_unlock(); 3011 0) | unlock_page() { 3012 0) 0.541 us | page_waitqueue(); 3013 0) 0.639 us | __wake_up_bit(); 3014 0) 2.786 us | } 3015 0) + 14.237 us | } 3016 0) | __do_fault() { 3017 0) | filemap_fault() { 3018 0) | find_lock_page() { 3019 0) 0.698 us | find_get_page(); 3020 0) | __might_sleep() { 3021 0) 1.412 us | } 3022 0) 3.950 us | } 3023 0) 5.098 us | } 3024 0) 0.631 us | _spin_lock(); 3025 0) 0.571 us | page_add_file_rmap(); 3026 0) 0.526 us | native_set_pte_at(); 3027 0) 0.586 us | _spin_unlock(); 3028 0) | unlock_page() { 3029 0) 0.533 us | page_waitqueue(); 3030 0) 0.638 us | __wake_up_bit(); 3031 0) 2.793 us | } 3032 0) + 14.012 us | } 3033 3034You can also expand several functions at once:: 3035 3036 echo sys_open > set_graph_function 3037 echo sys_close >> set_graph_function 3038 3039Now if you want to go back to trace all functions you can clear 3040this special filter via:: 3041 3042 echo > set_graph_function 3043 3044 3045ftrace_enabled 3046-------------- 3047 3048Note, the proc sysctl ftrace_enable is a big on/off switch for the 3049function tracer. By default it is enabled (when function tracing is 3050enabled in the kernel). If it is disabled, all function tracing is 3051disabled. This includes not only the function tracers for ftrace, but 3052also for any other uses (perf, kprobes, stack tracing, profiling, etc). It 3053cannot be disabled if there is a callback with FTRACE_OPS_FL_PERMANENT set 3054registered. 3055 3056Please disable this with care. 3057 3058This can be disable (and enabled) with:: 3059 3060 sysctl kernel.ftrace_enabled=0 3061 sysctl kernel.ftrace_enabled=1 3062 3063 or 3064 3065 echo 0 > /proc/sys/kernel/ftrace_enabled 3066 echo 1 > /proc/sys/kernel/ftrace_enabled 3067 3068 3069Filter commands 3070--------------- 3071 3072A few commands are supported by the set_ftrace_filter interface. 3073Trace commands have the following format:: 3074 3075 <function>:<command>:<parameter> 3076 3077The following commands are supported: 3078 3079- mod: 3080 This command enables function filtering per module. The 3081 parameter defines the module. For example, if only the write* 3082 functions in the ext3 module are desired, run: 3083 3084 echo 'write*:mod:ext3' > set_ftrace_filter 3085 3086 This command interacts with the filter in the same way as 3087 filtering based on function names. Thus, adding more functions 3088 in a different module is accomplished by appending (>>) to the 3089 filter file. Remove specific module functions by prepending 3090 '!':: 3091 3092 echo '!writeback*:mod:ext3' >> set_ftrace_filter 3093 3094 Mod command supports module globbing. Disable tracing for all 3095 functions except a specific module:: 3096 3097 echo '!*:mod:!ext3' >> set_ftrace_filter 3098 3099 Disable tracing for all modules, but still trace kernel:: 3100 3101 echo '!*:mod:*' >> set_ftrace_filter 3102 3103 Enable filter only for kernel:: 3104 3105 echo '*write*:mod:!*' >> set_ftrace_filter 3106 3107 Enable filter for module globbing:: 3108 3109 echo '*write*:mod:*snd*' >> set_ftrace_filter 3110 3111- traceon/traceoff: 3112 These commands turn tracing on and off when the specified 3113 functions are hit. The parameter determines how many times the 3114 tracing system is turned on and off. If unspecified, there is 3115 no limit. For example, to disable tracing when a schedule bug 3116 is hit the first 5 times, run:: 3117 3118 echo '__schedule_bug:traceoff:5' > set_ftrace_filter 3119 3120 To always disable tracing when __schedule_bug is hit:: 3121 3122 echo '__schedule_bug:traceoff' > set_ftrace_filter 3123 3124 These commands are cumulative whether or not they are appended 3125 to set_ftrace_filter. To remove a command, prepend it by '!' 3126 and drop the parameter:: 3127 3128 echo '!__schedule_bug:traceoff:0' > set_ftrace_filter 3129 3130 The above removes the traceoff command for __schedule_bug 3131 that have a counter. To remove commands without counters:: 3132 3133 echo '!__schedule_bug:traceoff' > set_ftrace_filter 3134 3135- snapshot: 3136 Will cause a snapshot to be triggered when the function is hit. 3137 :: 3138 3139 echo 'native_flush_tlb_others:snapshot' > set_ftrace_filter 3140 3141 To only snapshot once: 3142 :: 3143 3144 echo 'native_flush_tlb_others:snapshot:1' > set_ftrace_filter 3145 3146 To remove the above commands:: 3147 3148 echo '!native_flush_tlb_others:snapshot' > set_ftrace_filter 3149 echo '!native_flush_tlb_others:snapshot:0' > set_ftrace_filter 3150 3151- enable_event/disable_event: 3152 These commands can enable or disable a trace event. Note, because 3153 function tracing callbacks are very sensitive, when these commands 3154 are registered, the trace point is activated, but disabled in 3155 a "soft" mode. That is, the tracepoint will be called, but 3156 just will not be traced. The event tracepoint stays in this mode 3157 as long as there's a command that triggers it. 3158 :: 3159 3160 echo 'try_to_wake_up:enable_event:sched:sched_switch:2' > \ 3161 set_ftrace_filter 3162 3163 The format is:: 3164 3165 <function>:enable_event:<system>:<event>[:count] 3166 <function>:disable_event:<system>:<event>[:count] 3167 3168 To remove the events commands:: 3169 3170 echo '!try_to_wake_up:enable_event:sched:sched_switch:0' > \ 3171 set_ftrace_filter 3172 echo '!schedule:disable_event:sched:sched_switch' > \ 3173 set_ftrace_filter 3174 3175- dump: 3176 When the function is hit, it will dump the contents of the ftrace 3177 ring buffer to the console. This is useful if you need to debug 3178 something, and want to dump the trace when a certain function 3179 is hit. Perhaps it's a function that is called before a triple 3180 fault happens and does not allow you to get a regular dump. 3181 3182- cpudump: 3183 When the function is hit, it will dump the contents of the ftrace 3184 ring buffer for the current CPU to the console. Unlike the "dump" 3185 command, it only prints out the contents of the ring buffer for the 3186 CPU that executed the function that triggered the dump. 3187 3188- stacktrace: 3189 When the function is hit, a stack trace is recorded. 3190 3191trace_pipe 3192---------- 3193 3194The trace_pipe outputs the same content as the trace file, but 3195the effect on the tracing is different. Every read from 3196trace_pipe is consumed. This means that subsequent reads will be 3197different. The trace is live. 3198:: 3199 3200 # echo function > current_tracer 3201 # cat trace_pipe > /tmp/trace.out & 3202 [1] 4153 3203 # echo 1 > tracing_on 3204 # usleep 1 3205 # echo 0 > tracing_on 3206 # cat trace 3207 # tracer: function 3208 # 3209 # entries-in-buffer/entries-written: 0/0 #P:4 3210 # 3211 # _-----=> irqs-off 3212 # / _----=> need-resched 3213 # | / _---=> hardirq/softirq 3214 # || / _--=> preempt-depth 3215 # ||| / delay 3216 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 3217 # | | | |||| | | 3218 3219 # 3220 # cat /tmp/trace.out 3221 bash-1994 [000] .... 5281.568961: mutex_unlock <-rb_simple_write 3222 bash-1994 [000] .... 5281.568963: __mutex_unlock_slowpath <-mutex_unlock 3223 bash-1994 [000] .... 5281.568963: __fsnotify_parent <-fsnotify_modify 3224 bash-1994 [000] .... 5281.568964: fsnotify <-fsnotify_modify 3225 bash-1994 [000] .... 5281.568964: __srcu_read_lock <-fsnotify 3226 bash-1994 [000] .... 5281.568964: add_preempt_count <-__srcu_read_lock 3227 bash-1994 [000] ...1 5281.568965: sub_preempt_count <-__srcu_read_lock 3228 bash-1994 [000] .... 5281.568965: __srcu_read_unlock <-fsnotify 3229 bash-1994 [000] .... 5281.568967: sys_dup2 <-system_call_fastpath 3230 3231 3232Note, reading the trace_pipe file will block until more input is 3233added. This is contrary to the trace file. If any process opened 3234the trace file for reading, it will actually disable tracing and 3235prevent new entries from being added. The trace_pipe file does 3236not have this limitation. 3237 3238trace entries 3239------------- 3240 3241Having too much or not enough data can be troublesome in 3242diagnosing an issue in the kernel. The file buffer_size_kb is 3243used to modify the size of the internal trace buffers. The 3244number listed is the number of entries that can be recorded per 3245CPU. To know the full size, multiply the number of possible CPUs 3246with the number of entries. 3247:: 3248 3249 # cat buffer_size_kb 3250 1408 (units kilobytes) 3251 3252Or simply read buffer_total_size_kb 3253:: 3254 3255 # cat buffer_total_size_kb 3256 5632 3257 3258To modify the buffer, simple echo in a number (in 1024 byte segments). 3259:: 3260 3261 # echo 10000 > buffer_size_kb 3262 # cat buffer_size_kb 3263 10000 (units kilobytes) 3264 3265It will try to allocate as much as possible. If you allocate too 3266much, it can cause Out-Of-Memory to trigger. 3267:: 3268 3269 # echo 1000000000000 > buffer_size_kb 3270 -bash: echo: write error: Cannot allocate memory 3271 # cat buffer_size_kb 3272 85 3273 3274The per_cpu buffers can be changed individually as well: 3275:: 3276 3277 # echo 10000 > per_cpu/cpu0/buffer_size_kb 3278 # echo 100 > per_cpu/cpu1/buffer_size_kb 3279 3280When the per_cpu buffers are not the same, the buffer_size_kb 3281at the top level will just show an X 3282:: 3283 3284 # cat buffer_size_kb 3285 X 3286 3287This is where the buffer_total_size_kb is useful: 3288:: 3289 3290 # cat buffer_total_size_kb 3291 12916 3292 3293Writing to the top level buffer_size_kb will reset all the buffers 3294to be the same again. 3295 3296Snapshot 3297-------- 3298CONFIG_TRACER_SNAPSHOT makes a generic snapshot feature 3299available to all non latency tracers. (Latency tracers which 3300record max latency, such as "irqsoff" or "wakeup", can't use 3301this feature, since those are already using the snapshot 3302mechanism internally.) 3303 3304Snapshot preserves a current trace buffer at a particular point 3305in time without stopping tracing. Ftrace swaps the current 3306buffer with a spare buffer, and tracing continues in the new 3307current (=previous spare) buffer. 3308 3309The following tracefs files in "tracing" are related to this 3310feature: 3311 3312 snapshot: 3313 3314 This is used to take a snapshot and to read the output 3315 of the snapshot. Echo 1 into this file to allocate a 3316 spare buffer and to take a snapshot (swap), then read 3317 the snapshot from this file in the same format as 3318 "trace" (described above in the section "The File 3319 System"). Both reads snapshot and tracing are executable 3320 in parallel. When the spare buffer is allocated, echoing 3321 0 frees it, and echoing else (positive) values clear the 3322 snapshot contents. 3323 More details are shown in the table below. 3324 3325 +--------------+------------+------------+------------+ 3326 |status\\input | 0 | 1 | else | 3327 +==============+============+============+============+ 3328 |not allocated |(do nothing)| alloc+swap |(do nothing)| 3329 +--------------+------------+------------+------------+ 3330 |allocated | free | swap | clear | 3331 +--------------+------------+------------+------------+ 3332 3333Here is an example of using the snapshot feature. 3334:: 3335 3336 # echo 1 > events/sched/enable 3337 # echo 1 > snapshot 3338 # cat snapshot 3339 # tracer: nop 3340 # 3341 # entries-in-buffer/entries-written: 71/71 #P:8 3342 # 3343 # _-----=> irqs-off 3344 # / _----=> need-resched 3345 # | / _---=> hardirq/softirq 3346 # || / _--=> preempt-depth 3347 # ||| / delay 3348 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 3349 # | | | |||| | | 3350 <idle>-0 [005] d... 2440.603828: sched_switch: prev_comm=swapper/5 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2242 next_prio=120 3351 sleep-2242 [005] d... 2440.603846: sched_switch: prev_comm=snapshot-test-2 prev_pid=2242 prev_prio=120 prev_state=R ==> next_comm=kworker/5:1 next_pid=60 next_prio=120 3352 [...] 3353 <idle>-0 [002] d... 2440.707230: sched_switch: prev_comm=swapper/2 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2229 next_prio=120 3354 3355 # cat trace 3356 # tracer: nop 3357 # 3358 # entries-in-buffer/entries-written: 77/77 #P:8 3359 # 3360 # _-----=> irqs-off 3361 # / _----=> need-resched 3362 # | / _---=> hardirq/softirq 3363 # || / _--=> preempt-depth 3364 # ||| / delay 3365 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 3366 # | | | |||| | | 3367 <idle>-0 [007] d... 2440.707395: sched_switch: prev_comm=swapper/7 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2243 next_prio=120 3368 snapshot-test-2-2229 [002] d... 2440.707438: sched_switch: prev_comm=snapshot-test-2 prev_pid=2229 prev_prio=120 prev_state=S ==> next_comm=swapper/2 next_pid=0 next_prio=120 3369 [...] 3370 3371 3372If you try to use this snapshot feature when current tracer is 3373one of the latency tracers, you will get the following results. 3374:: 3375 3376 # echo wakeup > current_tracer 3377 # echo 1 > snapshot 3378 bash: echo: write error: Device or resource busy 3379 # cat snapshot 3380 cat: snapshot: Device or resource busy 3381 3382 3383Instances 3384--------- 3385In the tracefs tracing directory, there is a directory called "instances". 3386This directory can have new directories created inside of it using 3387mkdir, and removing directories with rmdir. The directory created 3388with mkdir in this directory will already contain files and other 3389directories after it is created. 3390:: 3391 3392 # mkdir instances/foo 3393 # ls instances/foo 3394 buffer_size_kb buffer_total_size_kb events free_buffer per_cpu 3395 set_event snapshot trace trace_clock trace_marker trace_options 3396 trace_pipe tracing_on 3397 3398As you can see, the new directory looks similar to the tracing directory 3399itself. In fact, it is very similar, except that the buffer and 3400events are agnostic from the main directory, or from any other 3401instances that are created. 3402 3403The files in the new directory work just like the files with the 3404same name in the tracing directory except the buffer that is used 3405is a separate and new buffer. The files affect that buffer but do not 3406affect the main buffer with the exception of trace_options. Currently, 3407the trace_options affect all instances and the top level buffer 3408the same, but this may change in future releases. That is, options 3409may become specific to the instance they reside in. 3410 3411Notice that none of the function tracer files are there, nor is 3412current_tracer and available_tracers. This is because the buffers 3413can currently only have events enabled for them. 3414:: 3415 3416 # mkdir instances/foo 3417 # mkdir instances/bar 3418 # mkdir instances/zoot 3419 # echo 100000 > buffer_size_kb 3420 # echo 1000 > instances/foo/buffer_size_kb 3421 # echo 5000 > instances/bar/per_cpu/cpu1/buffer_size_kb 3422 # echo function > current_trace 3423 # echo 1 > instances/foo/events/sched/sched_wakeup/enable 3424 # echo 1 > instances/foo/events/sched/sched_wakeup_new/enable 3425 # echo 1 > instances/foo/events/sched/sched_switch/enable 3426 # echo 1 > instances/bar/events/irq/enable 3427 # echo 1 > instances/zoot/events/syscalls/enable 3428 # cat trace_pipe 3429 CPU:2 [LOST 11745 EVENTS] 3430 bash-2044 [002] .... 10594.481032: _raw_spin_lock_irqsave <-get_page_from_freelist 3431 bash-2044 [002] d... 10594.481032: add_preempt_count <-_raw_spin_lock_irqsave 3432 bash-2044 [002] d..1 10594.481032: __rmqueue <-get_page_from_freelist 3433 bash-2044 [002] d..1 10594.481033: _raw_spin_unlock <-get_page_from_freelist 3434 bash-2044 [002] d..1 10594.481033: sub_preempt_count <-_raw_spin_unlock 3435 bash-2044 [002] d... 10594.481033: get_pageblock_flags_group <-get_pageblock_migratetype 3436 bash-2044 [002] d... 10594.481034: __mod_zone_page_state <-get_page_from_freelist 3437 bash-2044 [002] d... 10594.481034: zone_statistics <-get_page_from_freelist 3438 bash-2044 [002] d... 10594.481034: __inc_zone_state <-zone_statistics 3439 bash-2044 [002] d... 10594.481034: __inc_zone_state <-zone_statistics 3440 bash-2044 [002] .... 10594.481035: arch_dup_task_struct <-copy_process 3441 [...] 3442 3443 # cat instances/foo/trace_pipe 3444 bash-1998 [000] d..4 136.676759: sched_wakeup: comm=kworker/0:1 pid=59 prio=120 success=1 target_cpu=000 3445 bash-1998 [000] dN.4 136.676760: sched_wakeup: comm=bash pid=1998 prio=120 success=1 target_cpu=000 3446 <idle>-0 [003] d.h3 136.676906: sched_wakeup: comm=rcu_preempt pid=9 prio=120 success=1 target_cpu=003 3447 <idle>-0 [003] d..3 136.676909: sched_switch: prev_comm=swapper/3 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=rcu_preempt next_pid=9 next_prio=120 3448 rcu_preempt-9 [003] d..3 136.676916: sched_switch: prev_comm=rcu_preempt prev_pid=9 prev_prio=120 prev_state=S ==> next_comm=swapper/3 next_pid=0 next_prio=120 3449 bash-1998 [000] d..4 136.677014: sched_wakeup: comm=kworker/0:1 pid=59 prio=120 success=1 target_cpu=000 3450 bash-1998 [000] dN.4 136.677016: sched_wakeup: comm=bash pid=1998 prio=120 success=1 target_cpu=000 3451 bash-1998 [000] d..3 136.677018: sched_switch: prev_comm=bash prev_pid=1998 prev_prio=120 prev_state=R+ ==> next_comm=kworker/0:1 next_pid=59 next_prio=120 3452 kworker/0:1-59 [000] d..4 136.677022: sched_wakeup: comm=sshd pid=1995 prio=120 success=1 target_cpu=001 3453 kworker/0:1-59 [000] d..3 136.677025: sched_switch: prev_comm=kworker/0:1 prev_pid=59 prev_prio=120 prev_state=S ==> next_comm=bash next_pid=1998 next_prio=120 3454 [...] 3455 3456 # cat instances/bar/trace_pipe 3457 migration/1-14 [001] d.h3 138.732674: softirq_raise: vec=3 [action=NET_RX] 3458 <idle>-0 [001] dNh3 138.732725: softirq_raise: vec=3 [action=NET_RX] 3459 bash-1998 [000] d.h1 138.733101: softirq_raise: vec=1 [action=TIMER] 3460 bash-1998 [000] d.h1 138.733102: softirq_raise: vec=9 [action=RCU] 3461 bash-1998 [000] ..s2 138.733105: softirq_entry: vec=1 [action=TIMER] 3462 bash-1998 [000] ..s2 138.733106: softirq_exit: vec=1 [action=TIMER] 3463 bash-1998 [000] ..s2 138.733106: softirq_entry: vec=9 [action=RCU] 3464 bash-1998 [000] ..s2 138.733109: softirq_exit: vec=9 [action=RCU] 3465 sshd-1995 [001] d.h1 138.733278: irq_handler_entry: irq=21 name=uhci_hcd:usb4 3466 sshd-1995 [001] d.h1 138.733280: irq_handler_exit: irq=21 ret=unhandled 3467 sshd-1995 [001] d.h1 138.733281: irq_handler_entry: irq=21 name=eth0 3468 sshd-1995 [001] d.h1 138.733283: irq_handler_exit: irq=21 ret=handled 3469 [...] 3470 3471 # cat instances/zoot/trace 3472 # tracer: nop 3473 # 3474 # entries-in-buffer/entries-written: 18996/18996 #P:4 3475 # 3476 # _-----=> irqs-off 3477 # / _----=> need-resched 3478 # | / _---=> hardirq/softirq 3479 # || / _--=> preempt-depth 3480 # ||| / delay 3481 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 3482 # | | | |||| | | 3483 bash-1998 [000] d... 140.733501: sys_write -> 0x2 3484 bash-1998 [000] d... 140.733504: sys_dup2(oldfd: a, newfd: 1) 3485 bash-1998 [000] d... 140.733506: sys_dup2 -> 0x1 3486 bash-1998 [000] d... 140.733508: sys_fcntl(fd: a, cmd: 1, arg: 0) 3487 bash-1998 [000] d... 140.733509: sys_fcntl -> 0x1 3488 bash-1998 [000] d... 140.733510: sys_close(fd: a) 3489 bash-1998 [000] d... 140.733510: sys_close -> 0x0 3490 bash-1998 [000] d... 140.733514: sys_rt_sigprocmask(how: 0, nset: 0, oset: 6e2768, sigsetsize: 8) 3491 bash-1998 [000] d... 140.733515: sys_rt_sigprocmask -> 0x0 3492 bash-1998 [000] d... 140.733516: sys_rt_sigaction(sig: 2, act: 7fff718846f0, oact: 7fff71884650, sigsetsize: 8) 3493 bash-1998 [000] d... 140.733516: sys_rt_sigaction -> 0x0 3494 3495You can see that the trace of the top most trace buffer shows only 3496the function tracing. The foo instance displays wakeups and task 3497switches. 3498 3499To remove the instances, simply delete their directories: 3500:: 3501 3502 # rmdir instances/foo 3503 # rmdir instances/bar 3504 # rmdir instances/zoot 3505 3506Note, if a process has a trace file open in one of the instance 3507directories, the rmdir will fail with EBUSY. 3508 3509 3510Stack trace 3511----------- 3512Since the kernel has a fixed sized stack, it is important not to 3513waste it in functions. A kernel developer must be conscience of 3514what they allocate on the stack. If they add too much, the system 3515can be in danger of a stack overflow, and corruption will occur, 3516usually leading to a system panic. 3517 3518There are some tools that check this, usually with interrupts 3519periodically checking usage. But if you can perform a check 3520at every function call that will become very useful. As ftrace provides 3521a function tracer, it makes it convenient to check the stack size 3522at every function call. This is enabled via the stack tracer. 3523 3524CONFIG_STACK_TRACER enables the ftrace stack tracing functionality. 3525To enable it, write a '1' into /proc/sys/kernel/stack_tracer_enabled. 3526:: 3527 3528 # echo 1 > /proc/sys/kernel/stack_tracer_enabled 3529 3530You can also enable it from the kernel command line to trace 3531the stack size of the kernel during boot up, by adding "stacktrace" 3532to the kernel command line parameter. 3533 3534After running it for a few minutes, the output looks like: 3535:: 3536 3537 # cat stack_max_size 3538 2928 3539 3540 # cat stack_trace 3541 Depth Size Location (18 entries) 3542 ----- ---- -------- 3543 0) 2928 224 update_sd_lb_stats+0xbc/0x4ac 3544 1) 2704 160 find_busiest_group+0x31/0x1f1 3545 2) 2544 256 load_balance+0xd9/0x662 3546 3) 2288 80 idle_balance+0xbb/0x130 3547 4) 2208 128 __schedule+0x26e/0x5b9 3548 5) 2080 16 schedule+0x64/0x66 3549 6) 2064 128 schedule_timeout+0x34/0xe0 3550 7) 1936 112 wait_for_common+0x97/0xf1 3551 8) 1824 16 wait_for_completion+0x1d/0x1f 3552 9) 1808 128 flush_work+0xfe/0x119 3553 10) 1680 16 tty_flush_to_ldisc+0x1e/0x20 3554 11) 1664 48 input_available_p+0x1d/0x5c 3555 12) 1616 48 n_tty_poll+0x6d/0x134 3556 13) 1568 64 tty_poll+0x64/0x7f 3557 14) 1504 880 do_select+0x31e/0x511 3558 15) 624 400 core_sys_select+0x177/0x216 3559 16) 224 96 sys_select+0x91/0xb9 3560 17) 128 128 system_call_fastpath+0x16/0x1b 3561 3562Note, if -mfentry is being used by gcc, functions get traced before 3563they set up the stack frame. This means that leaf level functions 3564are not tested by the stack tracer when -mfentry is used. 3565 3566Currently, -mfentry is used by gcc 4.6.0 and above on x86 only. 3567 3568More 3569---- 3570More details can be found in the source code, in the `kernel/trace/*.c` files. 3571