1============= 2Event Tracing 3============= 4 5:Author: Theodore Ts'o 6:Updated: Li Zefan and Tom Zanussi 7 81. Introduction 9=============== 10 11Tracepoints (see Documentation/trace/tracepoints.rst) can be used 12without creating custom kernel modules to register probe functions 13using the event tracing infrastructure. 14 15Not all tracepoints can be traced using the event tracing system; 16the kernel developer must provide code snippets which define how the 17tracing information is saved into the tracing buffer, and how the 18tracing information should be printed. 19 202. Using Event Tracing 21====================== 22 232.1 Via the 'set_event' interface 24--------------------------------- 25 26The events which are available for tracing can be found in the file 27/sys/kernel/debug/tracing/available_events. 28 29To enable a particular event, such as 'sched_wakeup', simply echo it 30to /sys/kernel/debug/tracing/set_event. For example:: 31 32 # echo sched_wakeup >> /sys/kernel/debug/tracing/set_event 33 34.. Note:: '>>' is necessary, otherwise it will firstly disable all the events. 35 36To disable an event, echo the event name to the set_event file prefixed 37with an exclamation point:: 38 39 # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event 40 41To disable all events, echo an empty line to the set_event file:: 42 43 # echo > /sys/kernel/debug/tracing/set_event 44 45To enable all events, echo ``*:*`` or ``*:`` to the set_event file:: 46 47 # echo *:* > /sys/kernel/debug/tracing/set_event 48 49The events are organized into subsystems, such as ext4, irq, sched, 50etc., and a full event name looks like this: <subsystem>:<event>. The 51subsystem name is optional, but it is displayed in the available_events 52file. All of the events in a subsystem can be specified via the syntax 53``<subsystem>:*``; for example, to enable all irq events, you can use the 54command:: 55 56 # echo 'irq:*' > /sys/kernel/debug/tracing/set_event 57 582.2 Via the 'enable' toggle 59--------------------------- 60 61The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy 62of directories. 63 64To enable event 'sched_wakeup':: 65 66 # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable 67 68To disable it:: 69 70 # echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable 71 72To enable all events in sched subsystem:: 73 74 # echo 1 > /sys/kernel/debug/tracing/events/sched/enable 75 76To enable all events:: 77 78 # echo 1 > /sys/kernel/debug/tracing/events/enable 79 80When reading one of these enable files, there are four results: 81 82 - 0 - all events this file affects are disabled 83 - 1 - all events this file affects are enabled 84 - X - there is a mixture of events enabled and disabled 85 - ? - this file does not affect any event 86 872.3 Boot option 88--------------- 89 90In order to facilitate early boot debugging, use boot option:: 91 92 trace_event=[event-list] 93 94event-list is a comma separated list of events. See section 2.1 for event 95format. 96 973. Defining an event-enabled tracepoint 98======================================= 99 100See The example provided in samples/trace_events 101 1024. Event formats 103================ 104 105Each trace event has a 'format' file associated with it that contains 106a description of each field in a logged event. This information can 107be used to parse the binary trace stream, and is also the place to 108find the field names that can be used in event filters (see section 5). 109 110It also displays the format string that will be used to print the 111event in text mode, along with the event name and ID used for 112profiling. 113 114Every event has a set of ``common`` fields associated with it; these are 115the fields prefixed with ``common_``. The other fields vary between 116events and correspond to the fields defined in the TRACE_EVENT 117definition for that event. 118 119Each field in the format has the form:: 120 121 field:field-type field-name; offset:N; size:N; 122 123where offset is the offset of the field in the trace record and size 124is the size of the data item, in bytes. 125 126For example, here's the information displayed for the 'sched_wakeup' 127event:: 128 129 # cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format 130 131 name: sched_wakeup 132 ID: 60 133 format: 134 field:unsigned short common_type; offset:0; size:2; 135 field:unsigned char common_flags; offset:2; size:1; 136 field:unsigned char common_preempt_count; offset:3; size:1; 137 field:int common_pid; offset:4; size:4; 138 field:int common_tgid; offset:8; size:4; 139 140 field:char comm[TASK_COMM_LEN]; offset:12; size:16; 141 field:pid_t pid; offset:28; size:4; 142 field:int prio; offset:32; size:4; 143 field:int success; offset:36; size:4; 144 field:int cpu; offset:40; size:4; 145 146 print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid, 147 REC->prio, REC->success, REC->cpu 148 149This event contains 10 fields, the first 5 common and the remaining 5 150event-specific. All the fields for this event are numeric, except for 151'comm' which is a string, a distinction important for event filtering. 152 1535. Event filtering 154================== 155 156Trace events can be filtered in the kernel by associating boolean 157'filter expressions' with them. As soon as an event is logged into 158the trace buffer, its fields are checked against the filter expression 159associated with that event type. An event with field values that 160'match' the filter will appear in the trace output, and an event whose 161values don't match will be discarded. An event with no filter 162associated with it matches everything, and is the default when no 163filter has been set for an event. 164 1655.1 Expression syntax 166--------------------- 167 168A filter expression consists of one or more 'predicates' that can be 169combined using the logical operators '&&' and '||'. A predicate is 170simply a clause that compares the value of a field contained within a 171logged event with a constant value and returns either 0 or 1 depending 172on whether the field value matched (1) or didn't match (0):: 173 174 field-name relational-operator value 175 176Parentheses can be used to provide arbitrary logical groupings and 177double-quotes can be used to prevent the shell from interpreting 178operators as shell metacharacters. 179 180The field-names available for use in filters can be found in the 181'format' files for trace events (see section 4). 182 183The relational-operators depend on the type of the field being tested: 184 185The operators available for numeric fields are: 186 187==, !=, <, <=, >, >=, & 188 189And for string fields they are: 190 191==, !=, ~ 192 193The glob (~) accepts a wild card character (\*,?) and character classes 194([). For example:: 195 196 prev_comm ~ "*sh" 197 prev_comm ~ "sh*" 198 prev_comm ~ "*sh*" 199 prev_comm ~ "ba*sh" 200 2015.2 Setting filters 202------------------- 203 204A filter for an individual event is set by writing a filter expression 205to the 'filter' file for the given event. 206 207For example:: 208 209 # cd /sys/kernel/debug/tracing/events/sched/sched_wakeup 210 # echo "common_preempt_count > 4" > filter 211 212A slightly more involved example:: 213 214 # cd /sys/kernel/debug/tracing/events/signal/signal_generate 215 # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter 216 217If there is an error in the expression, you'll get an 'Invalid 218argument' error when setting it, and the erroneous string along with 219an error message can be seen by looking at the filter e.g.:: 220 221 # cd /sys/kernel/debug/tracing/events/signal/signal_generate 222 # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter 223 -bash: echo: write error: Invalid argument 224 # cat filter 225 ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash 226 ^ 227 parse_error: Field not found 228 229Currently the caret ('^') for an error always appears at the beginning of 230the filter string; the error message should still be useful though 231even without more accurate position info. 232 2335.3 Clearing filters 234-------------------- 235 236To clear the filter for an event, write a '0' to the event's filter 237file. 238 239To clear the filters for all events in a subsystem, write a '0' to the 240subsystem's filter file. 241 2425.3 Subsystem filters 243--------------------- 244 245For convenience, filters for every event in a subsystem can be set or 246cleared as a group by writing a filter expression into the filter file 247at the root of the subsystem. Note however, that if a filter for any 248event within the subsystem lacks a field specified in the subsystem 249filter, or if the filter can't be applied for any other reason, the 250filter for that event will retain its previous setting. This can 251result in an unintended mixture of filters which could lead to 252confusing (to the user who might think different filters are in 253effect) trace output. Only filters that reference just the common 254fields can be guaranteed to propagate successfully to all events. 255 256Here are a few subsystem filter examples that also illustrate the 257above points: 258 259Clear the filters on all events in the sched subsystem:: 260 261 # cd /sys/kernel/debug/tracing/events/sched 262 # echo 0 > filter 263 # cat sched_switch/filter 264 none 265 # cat sched_wakeup/filter 266 none 267 268Set a filter using only common fields for all events in the sched 269subsystem (all events end up with the same filter):: 270 271 # cd /sys/kernel/debug/tracing/events/sched 272 # echo common_pid == 0 > filter 273 # cat sched_switch/filter 274 common_pid == 0 275 # cat sched_wakeup/filter 276 common_pid == 0 277 278Attempt to set a filter using a non-common field for all events in the 279sched subsystem (all events but those that have a prev_pid field retain 280their old filters):: 281 282 # cd /sys/kernel/debug/tracing/events/sched 283 # echo prev_pid == 0 > filter 284 # cat sched_switch/filter 285 prev_pid == 0 286 # cat sched_wakeup/filter 287 common_pid == 0 288 2895.4 PID filtering 290----------------- 291 292The set_event_pid file in the same directory as the top events directory 293exists, will filter all events from tracing any task that does not have the 294PID listed in the set_event_pid file. 295:: 296 297 # cd /sys/kernel/debug/tracing 298 # echo $$ > set_event_pid 299 # echo 1 > events/enable 300 301Will only trace events for the current task. 302 303To add more PIDs without losing the PIDs already included, use '>>'. 304:: 305 306 # echo 123 244 1 >> set_event_pid 307 308 3096. Event triggers 310================= 311 312Trace events can be made to conditionally invoke trigger 'commands' 313which can take various forms and are described in detail below; 314examples would be enabling or disabling other trace events or invoking 315a stack trace whenever the trace event is hit. Whenever a trace event 316with attached triggers is invoked, the set of trigger commands 317associated with that event is invoked. Any given trigger can 318additionally have an event filter of the same form as described in 319section 5 (Event filtering) associated with it - the command will only 320be invoked if the event being invoked passes the associated filter. 321If no filter is associated with the trigger, it always passes. 322 323Triggers are added to and removed from a particular event by writing 324trigger expressions to the 'trigger' file for the given event. 325 326A given event can have any number of triggers associated with it, 327subject to any restrictions that individual commands may have in that 328regard. 329 330Event triggers are implemented on top of "soft" mode, which means that 331whenever a trace event has one or more triggers associated with it, 332the event is activated even if it isn't actually enabled, but is 333disabled in a "soft" mode. That is, the tracepoint will be called, 334but just will not be traced, unless of course it's actually enabled. 335This scheme allows triggers to be invoked even for events that aren't 336enabled, and also allows the current event filter implementation to be 337used for conditionally invoking triggers. 338 339The syntax for event triggers is roughly based on the syntax for 340set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands' 341section of Documentation/trace/ftrace.rst), but there are major 342differences and the implementation isn't currently tied to it in any 343way, so beware about making generalizations between the two. 344 345.. Note:: 346 Writing into trace_marker (See Documentation/trace/ftrace.rst) 347 can also enable triggers that are written into 348 /sys/kernel/tracing/events/ftrace/print/trigger 349 3506.1 Expression syntax 351--------------------- 352 353Triggers are added by echoing the command to the 'trigger' file:: 354 355 # echo 'command[:count] [if filter]' > trigger 356 357Triggers are removed by echoing the same command but starting with '!' 358to the 'trigger' file:: 359 360 # echo '!command[:count] [if filter]' > trigger 361 362The [if filter] part isn't used in matching commands when removing, so 363leaving that off in a '!' command will accomplish the same thing as 364having it in. 365 366The filter syntax is the same as that described in the 'Event 367filtering' section above. 368 369For ease of use, writing to the trigger file using '>' currently just 370adds or removes a single trigger and there's no explicit '>>' support 371('>' actually behaves like '>>') or truncation support to remove all 372triggers (you have to use '!' for each one added.) 373 3746.2 Supported trigger commands 375------------------------------ 376 377The following commands are supported: 378 379- enable_event/disable_event 380 381 These commands can enable or disable another trace event whenever 382 the triggering event is hit. When these commands are registered, 383 the other trace event is activated, but disabled in a "soft" mode. 384 That is, the tracepoint will be called, but just will not be traced. 385 The event tracepoint stays in this mode as long as there's a trigger 386 in effect that can trigger it. 387 388 For example, the following trigger causes kmalloc events to be 389 traced when a read system call is entered, and the :1 at the end 390 specifies that this enablement happens only once:: 391 392 # echo 'enable_event:kmem:kmalloc:1' > \ 393 /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger 394 395 The following trigger causes kmalloc events to stop being traced 396 when a read system call exits. This disablement happens on every 397 read system call exit:: 398 399 # echo 'disable_event:kmem:kmalloc' > \ 400 /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger 401 402 The format is:: 403 404 enable_event:<system>:<event>[:count] 405 disable_event:<system>:<event>[:count] 406 407 To remove the above commands:: 408 409 # echo '!enable_event:kmem:kmalloc:1' > \ 410 /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger 411 412 # echo '!disable_event:kmem:kmalloc' > \ 413 /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger 414 415 Note that there can be any number of enable/disable_event triggers 416 per triggering event, but there can only be one trigger per 417 triggered event. e.g. sys_enter_read can have triggers enabling both 418 kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc 419 versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if 420 bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they 421 could be combined into a single filter on kmem:kmalloc though). 422 423- stacktrace 424 425 This command dumps a stacktrace in the trace buffer whenever the 426 triggering event occurs. 427 428 For example, the following trigger dumps a stacktrace every time the 429 kmalloc tracepoint is hit:: 430 431 # echo 'stacktrace' > \ 432 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 433 434 The following trigger dumps a stacktrace the first 5 times a kmalloc 435 request happens with a size >= 64K:: 436 437 # echo 'stacktrace:5 if bytes_req >= 65536' > \ 438 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 439 440 The format is:: 441 442 stacktrace[:count] 443 444 To remove the above commands:: 445 446 # echo '!stacktrace' > \ 447 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 448 449 # echo '!stacktrace:5 if bytes_req >= 65536' > \ 450 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 451 452 The latter can also be removed more simply by the following (without 453 the filter):: 454 455 # echo '!stacktrace:5' > \ 456 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger 457 458 Note that there can be only one stacktrace trigger per triggering 459 event. 460 461- snapshot 462 463 This command causes a snapshot to be triggered whenever the 464 triggering event occurs. 465 466 The following command creates a snapshot every time a block request 467 queue is unplugged with a depth > 1. If you were tracing a set of 468 events or functions at the time, the snapshot trace buffer would 469 capture those events when the trigger event occurred:: 470 471 # echo 'snapshot if nr_rq > 1' > \ 472 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 473 474 To only snapshot once:: 475 476 # echo 'snapshot:1 if nr_rq > 1' > \ 477 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 478 479 To remove the above commands:: 480 481 # echo '!snapshot if nr_rq > 1' > \ 482 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 483 484 # echo '!snapshot:1 if nr_rq > 1' > \ 485 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 486 487 Note that there can be only one snapshot trigger per triggering 488 event. 489 490- traceon/traceoff 491 492 These commands turn tracing on and off when the specified events are 493 hit. The parameter determines how many times the tracing system is 494 turned on and off. If unspecified, there is no limit. 495 496 The following command turns tracing off the first time a block 497 request queue is unplugged with a depth > 1. If you were tracing a 498 set of events or functions at the time, you could then examine the 499 trace buffer to see the sequence of events that led up to the 500 trigger event:: 501 502 # echo 'traceoff:1 if nr_rq > 1' > \ 503 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 504 505 To always disable tracing when nr_rq > 1:: 506 507 # echo 'traceoff if nr_rq > 1' > \ 508 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 509 510 To remove the above commands:: 511 512 # echo '!traceoff:1 if nr_rq > 1' > \ 513 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 514 515 # echo '!traceoff if nr_rq > 1' > \ 516 /sys/kernel/debug/tracing/events/block/block_unplug/trigger 517 518 Note that there can be only one traceon or traceoff trigger per 519 triggering event. 520 521- hist 522 523 This command aggregates event hits into a hash table keyed on one or 524 more trace event format fields (or stacktrace) and a set of running 525 totals derived from one or more trace event format fields and/or 526 event counts (hitcount). 527 528 See Documentation/trace/histogram.rst for details and examples. 529 5307. In-kernel trace event API 531============================ 532 533In most cases, the command-line interface to trace events is more than 534sufficient. Sometimes, however, applications might find the need for 535more complex relationships than can be expressed through a simple 536series of linked command-line expressions, or putting together sets of 537commands may be simply too cumbersome. An example might be an 538application that needs to 'listen' to the trace stream in order to 539maintain an in-kernel state machine detecting, for instance, when an 540illegal kernel state occurs in the scheduler. 541 542The trace event subsystem provides an in-kernel API allowing modules 543or other kernel code to generate user-defined 'synthetic' events at 544will, which can be used to either augment the existing trace stream 545and/or signal that a particular important state has occurred. 546 547A similar in-kernel API is also available for creating kprobe and 548kretprobe events. 549 550Both the synthetic event and k/ret/probe event APIs are built on top 551of a lower-level "dynevent_cmd" event command API, which is also 552available for more specialized applications, or as the basis of other 553higher-level trace event APIs. 554 555The API provided for these purposes is describe below and allows the 556following: 557 558 - dynamically creating synthetic event definitions 559 - dynamically creating kprobe and kretprobe event definitions 560 - tracing synthetic events from in-kernel code 561 - the low-level "dynevent_cmd" API 562 5637.1 Dyamically creating synthetic event definitions 564--------------------------------------------------- 565 566There are a couple ways to create a new synthetic event from a kernel 567module or other kernel code. 568 569The first creates the event in one step, using synth_event_create(). 570In this method, the name of the event to create and an array defining 571the fields is supplied to synth_event_create(). If successful, a 572synthetic event with that name and fields will exist following that 573call. For example, to create a new "schedtest" synthetic event:: 574 575 ret = synth_event_create("schedtest", sched_fields, 576 ARRAY_SIZE(sched_fields), THIS_MODULE); 577 578The sched_fields param in this example points to an array of struct 579synth_field_desc, each of which describes an event field by type and 580name:: 581 582 static struct synth_field_desc sched_fields[] = { 583 { .type = "pid_t", .name = "next_pid_field" }, 584 { .type = "char[16]", .name = "next_comm_field" }, 585 { .type = "u64", .name = "ts_ns" }, 586 { .type = "u64", .name = "ts_ms" }, 587 { .type = "unsigned int", .name = "cpu" }, 588 { .type = "char[64]", .name = "my_string_field" }, 589 { .type = "int", .name = "my_int_field" }, 590 }; 591 592See synth_field_size() for available types. 593 594If field_name contains [n], the field is considered to be a static array. 595 596If field_names contains[] (no subscript), the field is considered to 597be a dynamic array, which will only take as much space in the event as 598is required to hold the array. 599 600Because space for an event is reserved before assigning field values 601to the event, using dynamic arrays implies that the piecewise 602in-kernel API described below can't be used with dynamic arrays. The 603other non-piecewise in-kernel APIs can, however, be used with dynamic 604arrays. 605 606If the event is created from within a module, a pointer to the module 607must be passed to synth_event_create(). This will ensure that the 608trace buffer won't contain unreadable events when the module is 609removed. 610 611At this point, the event object is ready to be used for generating new 612events. 613 614In the second method, the event is created in several steps. This 615allows events to be created dynamically and without the need to create 616and populate an array of fields beforehand. 617 618To use this method, an empty or partially empty synthetic event should 619first be created using synth_event_gen_cmd_start() or 620synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(), 621the name of the event along with one or more pairs of args each pair 622representing a 'type field_name;' field specification should be 623supplied. For synth_event_gen_cmd_array_start(), the name of the 624event along with an array of struct synth_field_desc should be 625supplied. Before calling synth_event_gen_cmd_start() or 626synth_event_gen_cmd_array_start(), the user should create and 627initialize a dynevent_cmd object using synth_event_cmd_init(). 628 629For example, to create a new "schedtest" synthetic event with two 630fields:: 631 632 struct dynevent_cmd cmd; 633 char *buf; 634 635 /* Create a buffer to hold the generated command */ 636 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); 637 638 /* Before generating the command, initialize the cmd object */ 639 synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN); 640 641 ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE, 642 "pid_t", "next_pid_field", 643 "u64", "ts_ns"); 644 645Alternatively, using an array of struct synth_field_desc fields 646containing the same information:: 647 648 ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE, 649 fields, n_fields); 650 651Once the synthetic event object has been created, it can then be 652populated with more fields. Fields are added one by one using 653synth_event_add_field(), supplying the dynevent_cmd object, a field 654type, and a field name. For example, to add a new int field named 655"intfield", the following call should be made:: 656 657 ret = synth_event_add_field(&cmd, "int", "intfield"); 658 659See synth_field_size() for available types. If field_name contains [n] 660the field is considered to be an array. 661 662A group of fields can also be added all at once using an array of 663synth_field_desc with add_synth_fields(). For example, this would add 664just the first four sched_fields:: 665 666 ret = synth_event_add_fields(&cmd, sched_fields, 4); 667 668If you already have a string of the form 'type field_name', 669synth_event_add_field_str() can be used to add it as-is; it will 670also automatically append a ';' to the string. 671 672Once all the fields have been added, the event should be finalized and 673registered by calling the synth_event_gen_cmd_end() function:: 674 675 ret = synth_event_gen_cmd_end(&cmd); 676 677At this point, the event object is ready to be used for tracing new 678events. 679 6807.2 Tracing synthetic events from in-kernel code 681------------------------------------------------ 682 683To trace a synthetic event, there are several options. The first 684option is to trace the event in one call, using synth_event_trace() 685with a variable number of values, or synth_event_trace_array() with an 686array of values to be set. A second option can be used to avoid the 687need for a pre-formed array of values or list of arguments, via 688synth_event_trace_start() and synth_event_trace_end() along with 689synth_event_add_next_val() or synth_event_add_val() to add the values 690piecewise. 691 6927.2.1 Tracing a synthetic event all at once 693------------------------------------------- 694 695To trace a synthetic event all at once, the synth_event_trace() or 696synth_event_trace_array() functions can be used. 697 698The synth_event_trace() function is passed the trace_event_file 699representing the synthetic event (which can be retrieved using 700trace_get_event_file() using the synthetic event name, "synthetic" as 701the system name, and the trace instance name (NULL if using the global 702trace array)), along with an variable number of u64 args, one for each 703synthetic event field, and the number of values being passed. 704 705So, to trace an event corresponding to the synthetic event definition 706above, code like the following could be used:: 707 708 ret = synth_event_trace(create_synth_test, 7, /* number of values */ 709 444, /* next_pid_field */ 710 (u64)"clackers", /* next_comm_field */ 711 1000000, /* ts_ns */ 712 1000, /* ts_ms */ 713 smp_processor_id(),/* cpu */ 714 (u64)"Thneed", /* my_string_field */ 715 999); /* my_int_field */ 716 717All vals should be cast to u64, and string vals are just pointers to 718strings, cast to u64. Strings will be copied into space reserved in 719the event for the string, using these pointers. 720 721Alternatively, the synth_event_trace_array() function can be used to 722accomplish the same thing. It is passed the trace_event_file 723representing the synthetic event (which can be retrieved using 724trace_get_event_file() using the synthetic event name, "synthetic" as 725the system name, and the trace instance name (NULL if using the global 726trace array)), along with an array of u64, one for each synthetic 727event field. 728 729To trace an event corresponding to the synthetic event definition 730above, code like the following could be used:: 731 732 u64 vals[7]; 733 734 vals[0] = 777; /* next_pid_field */ 735 vals[1] = (u64)"tiddlywinks"; /* next_comm_field */ 736 vals[2] = 1000000; /* ts_ns */ 737 vals[3] = 1000; /* ts_ms */ 738 vals[4] = smp_processor_id(); /* cpu */ 739 vals[5] = (u64)"thneed"; /* my_string_field */ 740 vals[6] = 398; /* my_int_field */ 741 742The 'vals' array is just an array of u64, the number of which must 743match the number of field in the synthetic event, and which must be in 744the same order as the synthetic event fields. 745 746All vals should be cast to u64, and string vals are just pointers to 747strings, cast to u64. Strings will be copied into space reserved in 748the event for the string, using these pointers. 749 750In order to trace a synthetic event, a pointer to the trace event file 751is needed. The trace_get_event_file() function can be used to get 752it - it will find the file in the given trace instance (in this case 753NULL since the top trace array is being used) while at the same time 754preventing the instance containing it from going away:: 755 756 schedtest_event_file = trace_get_event_file(NULL, "synthetic", 757 "schedtest"); 758 759Before tracing the event, it should be enabled in some way, otherwise 760the synthetic event won't actually show up in the trace buffer. 761 762To enable a synthetic event from the kernel, trace_array_set_clr_event() 763can be used (which is not specific to synthetic events, so does need 764the "synthetic" system name to be specified explicitly). 765 766To enable the event, pass 'true' to it:: 767 768 trace_array_set_clr_event(schedtest_event_file->tr, 769 "synthetic", "schedtest", true); 770 771To disable it pass false:: 772 773 trace_array_set_clr_event(schedtest_event_file->tr, 774 "synthetic", "schedtest", false); 775 776Finally, synth_event_trace_array() can be used to actually trace the 777event, which should be visible in the trace buffer afterwards:: 778 779 ret = synth_event_trace_array(schedtest_event_file, vals, 780 ARRAY_SIZE(vals)); 781 782To remove the synthetic event, the event should be disabled, and the 783trace instance should be 'put' back using trace_put_event_file():: 784 785 trace_array_set_clr_event(schedtest_event_file->tr, 786 "synthetic", "schedtest", false); 787 trace_put_event_file(schedtest_event_file); 788 789If those have been successful, synth_event_delete() can be called to 790remove the event:: 791 792 ret = synth_event_delete("schedtest"); 793 7947.2.2 Tracing a synthetic event piecewise 795----------------------------------------- 796 797To trace a synthetic using the piecewise method described above, the 798synth_event_trace_start() function is used to 'open' the synthetic 799event trace:: 800 801 struct synth_event_trace_state trace_state; 802 803 ret = synth_event_trace_start(schedtest_event_file, &trace_state); 804 805It's passed the trace_event_file representing the synthetic event 806using the same methods as described above, along with a pointer to a 807struct synth_event_trace_state object, which will be zeroed before use and 808used to maintain state between this and following calls. 809 810Once the event has been opened, which means space for it has been 811reserved in the trace buffer, the individual fields can be set. There 812are two ways to do that, either one after another for each field in 813the event, which requires no lookups, or by name, which does. The 814tradeoff is flexibility in doing the assignments vs the cost of a 815lookup per field. 816 817To assign the values one after the other without lookups, 818synth_event_add_next_val() should be used. Each call is passed the 819same synth_event_trace_state object used in the synth_event_trace_start(), 820along with the value to set the next field in the event. After each 821field is set, the 'cursor' points to the next field, which will be set 822by the subsequent call, continuing until all the fields have been set 823in order. The same sequence of calls as in the above examples using 824this method would be (without error-handling code):: 825 826 /* next_pid_field */ 827 ret = synth_event_add_next_val(777, &trace_state); 828 829 /* next_comm_field */ 830 ret = synth_event_add_next_val((u64)"slinky", &trace_state); 831 832 /* ts_ns */ 833 ret = synth_event_add_next_val(1000000, &trace_state); 834 835 /* ts_ms */ 836 ret = synth_event_add_next_val(1000, &trace_state); 837 838 /* cpu */ 839 ret = synth_event_add_next_val(smp_processor_id(), &trace_state); 840 841 /* my_string_field */ 842 ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state); 843 844 /* my_int_field */ 845 ret = synth_event_add_next_val(395, &trace_state); 846 847To assign the values in any order, synth_event_add_val() should be 848used. Each call is passed the same synth_event_trace_state object used in 849the synth_event_trace_start(), along with the field name of the field 850to set and the value to set it to. The same sequence of calls as in 851the above examples using this method would be (without error-handling 852code):: 853 854 ret = synth_event_add_val("next_pid_field", 777, &trace_state); 855 ret = synth_event_add_val("next_comm_field", (u64)"silly putty", 856 &trace_state); 857 ret = synth_event_add_val("ts_ns", 1000000, &trace_state); 858 ret = synth_event_add_val("ts_ms", 1000, &trace_state); 859 ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state); 860 ret = synth_event_add_val("my_string_field", (u64)"thneed_9", 861 &trace_state); 862 ret = synth_event_add_val("my_int_field", 3999, &trace_state); 863 864Note that synth_event_add_next_val() and synth_event_add_val() are 865incompatible if used within the same trace of an event - either one 866can be used but not both at the same time. 867 868Finally, the event won't be actually traced until it's 'closed', 869which is done using synth_event_trace_end(), which takes only the 870struct synth_event_trace_state object used in the previous calls:: 871 872 ret = synth_event_trace_end(&trace_state); 873 874Note that synth_event_trace_end() must be called at the end regardless 875of whether any of the add calls failed (say due to a bad field name 876being passed in). 877 8787.3 Dyamically creating kprobe and kretprobe event definitions 879-------------------------------------------------------------- 880 881To create a kprobe or kretprobe trace event from kernel code, the 882kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start() 883functions can be used. 884 885To create a kprobe event, an empty or partially empty kprobe event 886should first be created using kprobe_event_gen_cmd_start(). The name 887of the event and the probe location should be specfied along with one 888or args each representing a probe field should be supplied to this 889function. Before calling kprobe_event_gen_cmd_start(), the user 890should create and initialize a dynevent_cmd object using 891kprobe_event_cmd_init(). 892 893For example, to create a new "schedtest" kprobe event with two fields:: 894 895 struct dynevent_cmd cmd; 896 char *buf; 897 898 /* Create a buffer to hold the generated command */ 899 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); 900 901 /* Before generating the command, initialize the cmd object */ 902 kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN); 903 904 /* 905 * Define the gen_kprobe_test event with the first 2 kprobe 906 * fields. 907 */ 908 ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open", 909 "dfd=%ax", "filename=%dx"); 910 911Once the kprobe event object has been created, it can then be 912populated with more fields. Fields can be added using 913kprobe_event_add_fields(), supplying the dynevent_cmd object along 914with a variable arg list of probe fields. For example, to add a 915couple additional fields, the following call could be made:: 916 917 ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)"); 918 919Once all the fields have been added, the event should be finalized and 920registered by calling the kprobe_event_gen_cmd_end() or 921kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe 922or kretprobe command was started:: 923 924 ret = kprobe_event_gen_cmd_end(&cmd); 925 926or:: 927 928 ret = kretprobe_event_gen_cmd_end(&cmd); 929 930At this point, the event object is ready to be used for tracing new 931events. 932 933Similarly, a kretprobe event can be created using 934kretprobe_event_gen_cmd_start() with a probe name and location and 935additional params such as $retval:: 936 937 ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test", 938 "do_sys_open", "$retval"); 939 940Similar to the synthetic event case, code like the following can be 941used to enable the newly created kprobe event:: 942 943 gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test"); 944 945 ret = trace_array_set_clr_event(gen_kprobe_test->tr, 946 "kprobes", "gen_kprobe_test", true); 947 948Finally, also similar to synthetic events, the following code can be 949used to give the kprobe event file back and delete the event:: 950 951 trace_put_event_file(gen_kprobe_test); 952 953 ret = kprobe_event_delete("gen_kprobe_test"); 954 9557.4 The "dynevent_cmd" low-level API 956------------------------------------ 957 958Both the in-kernel synthetic event and kprobe interfaces are built on 959top of a lower-level "dynevent_cmd" interface. This interface is 960meant to provide the basis for higher-level interfaces such as the 961synthetic and kprobe interfaces, which can be used as examples. 962 963The basic idea is simple and amounts to providing a general-purpose 964layer that can be used to generate trace event commands. The 965generated command strings can then be passed to the command-parsing 966and event creation code that already exists in the trace event 967subystem for creating the corresponding trace events. 968 969In a nutshell, the way it works is that the higher-level interface 970code creates a struct dynevent_cmd object, then uses a couple 971functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up 972a command string, which finally causes the command to be executed 973using the dynevent_create() function. The details of the interface 974are described below. 975 976The first step in building a new command string is to create and 977initialize an instance of a dynevent_cmd. Here, for instance, we 978create a dynevent_cmd on the stack and initialize it:: 979 980 struct dynevent_cmd cmd; 981 char *buf; 982 int ret; 983 984 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); 985 986 dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO, 987 foo_event_run_command); 988 989The dynevent_cmd initialization needs to be given a user-specified 990buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used 991for this purpose - at 2k it's generally too big to be comfortably put 992on the stack, so is dynamically allocated), a dynevent type id, which 993is meant to be used to check that further API calls are for the 994correct command type, and a pointer to an event-specific run_command() 995callback that will be called to actually execute the event-specific 996command function. 997 998Once that's done, the command string can by built up by successive 999calls to argument-adding functions. 1000 1001To add a single argument, define and initialize a struct dynevent_arg 1002or struct dynevent_arg_pair object. Here's an example of the simplest 1003possible arg addition, which is simply to append the given string as 1004a whitespace-separated argument to the command:: 1005 1006 struct dynevent_arg arg; 1007 1008 dynevent_arg_init(&arg, NULL, 0); 1009 1010 arg.str = name; 1011 1012 ret = dynevent_arg_add(cmd, &arg); 1013 1014The arg object is first initialized using dynevent_arg_init() and in 1015this case the parameters are NULL or 0, which means there's no 1016optional sanity-checking function or separator appended to the end of 1017the arg. 1018 1019Here's another more complicated example using an 'arg pair', which is 1020used to create an argument that consists of a couple components added 1021together as a unit, for example, a 'type field_name;' arg or a simple 1022expression arg e.g. 'flags=%cx':: 1023 1024 struct dynevent_arg_pair arg_pair; 1025 1026 dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';'); 1027 1028 arg_pair.lhs = type; 1029 arg_pair.rhs = name; 1030 1031 ret = dynevent_arg_pair_add(cmd, &arg_pair); 1032 1033Again, the arg_pair is first initialized, in this case with a callback 1034function used to check the sanity of the args (for example, that 1035neither part of the pair is NULL), along with a character to be used 1036to add an operator between the pair (here none) and a separator to be 1037appended onto the end of the arg pair (here ';'). 1038 1039There's also a dynevent_str_add() function that can be used to simply 1040add a string as-is, with no spaces, delimeters, or arg check. 1041 1042Any number of dynevent_*_add() calls can be made to build up the string 1043(until its length surpasses cmd->maxlen). When all the arguments have 1044been added and the command string is complete, the only thing left to 1045do is run the command, which happens by simply calling 1046dynevent_create():: 1047 1048 ret = dynevent_create(&cmd); 1049 1050At that point, if the return value is 0, the dynamic event has been 1051created and is ready to use. 1052 1053See the dynevent_cmd function definitions themselves for the details 1054of the API. 1055