1.. SPDX-License-Identifier: GPL-2.0 2 3================= 4Boot-time tracing 5================= 6 7:Author: Masami Hiramatsu <mhiramat@kernel.org> 8 9Overview 10======== 11 12Boot-time tracing allows users to trace boot-time process including 13device initialization with full features of ftrace including per-event 14filter and actions, histograms, kprobe-events and synthetic-events, 15and trace instances. 16Since kernel command line is not enough to control these complex features, 17this uses bootconfig file to describe tracing feature programming. 18 19Options in the Boot Config 20========================== 21 22Here is the list of available options list for boot time tracing in 23boot config file [1]_. All options are under "ftrace." or "kernel." 24prefix. See kernel parameters for the options which starts 25with "kernel." prefix [2]_. 26 27.. [1] See :ref:`Documentation/admin-guide/bootconfig.rst <bootconfig>` 28.. [2] See :ref:`Documentation/admin-guide/kernel-parameters.rst <kernelparameters>` 29 30Ftrace Global Options 31--------------------- 32 33Ftrace global options have "kernel." prefix in boot config, which means 34these options are passed as a part of kernel legacy command line. 35 36kernel.tp_printk 37 Output trace-event data on printk buffer too. 38 39kernel.dump_on_oops [= MODE] 40 Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer 41 on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops. 42 43kernel.traceoff_on_warning 44 Stop tracing if WARN_ON() occurs. 45 46kernel.fgraph_max_depth = MAX_DEPTH 47 Set MAX_DEPTH to maximum depth of fgraph tracer. 48 49kernel.fgraph_filters = FILTER[, FILTER2...] 50 Add fgraph tracing function filters. 51 52kernel.fgraph_notraces = FILTER[, FILTER2...] 53 Add fgraph non-tracing function filters. 54 55 56Ftrace Per-instance Options 57--------------------------- 58 59These options can be used for each instance including global ftrace node. 60 61ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]] 62 Enable given ftrace options. 63 64ftrace.[instance.INSTANCE.]tracing_on = 0|1 65 Enable/Disable tracing on this instance when starting boot-time tracing. 66 (you can enable it by the "traceon" event trigger action) 67 68ftrace.[instance.INSTANCE.]trace_clock = CLOCK 69 Set given CLOCK to ftrace's trace_clock. 70 71ftrace.[instance.INSTANCE.]buffer_size = SIZE 72 Configure ftrace buffer size to SIZE. You can use "KB" or "MB" 73 for that SIZE. 74 75ftrace.[instance.INSTANCE.]alloc_snapshot 76 Allocate snapshot buffer. 77 78ftrace.[instance.INSTANCE.]cpumask = CPUMASK 79 Set CPUMASK as trace cpu-mask. 80 81ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2[...]] 82 Enable given events on boot. You can use a wild card in EVENT. 83 84ftrace.[instance.INSTANCE.]tracer = TRACER 85 Set TRACER to current tracer on boot. (e.g. function) 86 87ftrace.[instance.INSTANCE.]ftrace.filters 88 This will take an array of tracing function filter rules. 89 90ftrace.[instance.INSTANCE.]ftrace.notraces 91 This will take an array of NON-tracing function filter rules. 92 93 94Ftrace Per-Event Options 95------------------------ 96 97These options are setting per-event options. 98 99ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable 100 Enable GROUP:EVENT tracing. 101 102ftrace.[instance.INSTANCE.]event.GROUP.enable 103 Enable all event tracing within GROUP. 104 105ftrace.[instance.INSTANCE.]event.enable 106 Enable all event tracing. 107 108ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER 109 Set FILTER rule to the GROUP:EVENT. 110 111ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2[...]] 112 Set ACTIONs to the GROUP:EVENT. 113 114ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2[...]] 115 Defines new kprobe event based on PROBEs. It is able to define 116 multiple probes on one event, but those must have same type of 117 arguments. This option is available only for the event which 118 group name is "kprobes". 119 120ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]] 121 Defines new synthetic event with FIELDs. Each field should be 122 "type varname". 123 124Note that kprobe and synthetic event definitions can be written under 125instance node, but those are also visible from other instances. So please 126take care for event name conflict. 127 128Ftrace Histogram Options 129------------------------ 130 131Since it is too long to write a histogram action as a string for per-event 132action option, there are tree-style options under per-event 'hist' subkey 133for the histogram actions. For the detail of the each parameter, 134please read the event histogram document (Documentation/trace/histogram.rst) 135 136ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]keys = KEY1[, KEY2[...]] 137 Set histogram key parameters. (Mandatory) 138 The 'N' is a digit string for the multiple histogram. You can omit it 139 if there is one histogram on the event. 140 141ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]values = VAL1[, VAL2[...]] 142 Set histogram value parameters. 143 144ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]sort = SORT1[, SORT2[...]] 145 Set histogram sort parameter options. 146 147ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]size = NR_ENTRIES 148 Set histogram size (number of entries). 149 150ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]name = NAME 151 Set histogram name. 152 153ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]var.VARIABLE = EXPR 154 Define a new VARIABLE by EXPR expression. 155 156ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<pause|continue|clear> 157 Set histogram control parameter. You can set one of them. 158 159ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]event = GROUP.EVENT 160 Set histogram 'onmatch' handler matching event parameter. 161 The 'M' is a digit string for the multiple 'onmatch' handler. You can omit it 162 if there is one 'onmatch' handler on this histogram. 163 164ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]trace = EVENT[, ARG1[...]] 165 Set histogram 'trace' action for 'onmatch'. 166 EVENT must be a synthetic event name, and ARG1... are parameters 167 for that event. Mandatory if 'onmatch.event' option is set. 168 169ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmax.[M.]var = VAR 170 Set histogram 'onmax' handler variable parameter. 171 172ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onchange.[M.]var = VAR 173 Set histogram 'onchange' handler variable parameter. 174 175ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]save = ARG1[, ARG2[...]] 176 Set histogram 'save' action parameters for 'onmax' or 'onchange' handler. 177 This option or below 'snapshot' option is mandatory if 'onmax.var' or 178 'onchange.var' option is set. 179 180ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]snapshot 181 Set histogram 'snapshot' action for 'onmax' or 'onchange' handler. 182 This option or above 'save' option is mandatory if 'onmax.var' or 183 'onchange.var' option is set. 184 185ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.filter = FILTER_EXPR 186 Set histogram filter expression. You don't need 'if' in the FILTER_EXPR. 187 188Note that this 'hist' option can conflict with the per-event 'actions' 189option if the 'actions' option has a histogram action. 190 191 192When to Start 193============= 194 195All boot-time tracing options starting with ``ftrace`` will be enabled at the 196end of core_initcall. This means you can trace the events from postcore_initcall. 197Most of the subsystems and architecture dependent drivers will be initialized 198after that (arch_initcall or subsys_initcall). Thus, you can trace those with 199boot-time tracing. 200If you want to trace events before core_initcall, you can use the options 201starting with ``kernel``. Some of them will be enabled eariler than the initcall 202processing (for example,. ``kernel.ftrace=function`` and ``kernel.trace_event`` 203will start before the initcall.) 204 205 206Examples 207======== 208 209For example, to add filter and actions for each event, define kprobe 210events, and synthetic events with histogram, write a boot config like 211below:: 212 213 ftrace.event { 214 task.task_newtask { 215 filter = "pid < 128" 216 enable 217 } 218 kprobes.vfs_read { 219 probes = "vfs_read $arg1 $arg2" 220 filter = "common_pid < 200" 221 enable 222 } 223 synthetic.initcall_latency { 224 fields = "unsigned long func", "u64 lat" 225 hist { 226 keys = func.sym, lat 227 values = lat 228 sort = lat 229 } 230 } 231 initcall.initcall_start.hist { 232 keys = func 233 var.ts0 = common_timestamp.usecs 234 } 235 initcall.initcall_finish.hist { 236 keys = func 237 var.lat = common_timestamp.usecs - $ts0 238 onmatch { 239 event = initcall.initcall_start 240 trace = initcall_latency, func, $lat 241 } 242 } 243 } 244 245Also, boot-time tracing supports "instance" node, which allows us to run 246several tracers for different purpose at once. For example, one tracer 247is for tracing functions starting with "user\_", and others tracing 248"kernel\_" functions, you can write boot config as below:: 249 250 ftrace.instance { 251 foo { 252 tracer = "function" 253 ftrace.filters = "user_*" 254 } 255 bar { 256 tracer = "function" 257 ftrace.filters = "kernel_*" 258 } 259 } 260 261The instance node also accepts event nodes so that each instance 262can customize its event tracing. 263 264With the trigger action and kprobes, you can trace function-graph while 265a function is called. For example, this will trace all function calls in 266the pci_proc_init():: 267 268 ftrace { 269 tracing_on = 0 270 tracer = function_graph 271 event.kprobes { 272 start_event { 273 probes = "pci_proc_init" 274 actions = "traceon" 275 } 276 end_event { 277 probes = "pci_proc_init%return" 278 actions = "traceoff" 279 } 280 } 281 } 282 283 284This boot-time tracing also supports ftrace kernel parameters via boot 285config. 286For example, following kernel parameters:: 287 288 trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*" 289 290This can be written in boot config like below:: 291 292 kernel { 293 trace_options = sym-addr 294 trace_event = "initcall:*" 295 tp_printk 296 trace_buf_size = 1M 297 ftrace = function 298 ftrace_filter = "vfs*" 299 } 300 301Note that parameters start with "kernel" prefix instead of "ftrace". 302