1514f1c67SArnaldo Carvalho de Meloperf-trace(1) 2514f1c67SArnaldo Carvalho de Melo============= 3514f1c67SArnaldo Carvalho de Melo 4514f1c67SArnaldo Carvalho de MeloNAME 5514f1c67SArnaldo Carvalho de Melo---- 6514f1c67SArnaldo Carvalho de Meloperf-trace - strace inspired tool 7514f1c67SArnaldo Carvalho de Melo 8514f1c67SArnaldo Carvalho de MeloSYNOPSIS 9514f1c67SArnaldo Carvalho de Melo-------- 10514f1c67SArnaldo Carvalho de Melo[verse] 11514f1c67SArnaldo Carvalho de Melo'perf trace' 125e2485b1SDavid Ahern'perf trace record' 13514f1c67SArnaldo Carvalho de Melo 14514f1c67SArnaldo Carvalho de MeloDESCRIPTION 15514f1c67SArnaldo Carvalho de Melo----------- 16514f1c67SArnaldo Carvalho de MeloThis command will show the events associated with the target, initially 17514f1c67SArnaldo Carvalho de Melosyscalls, but other system events like pagefaults, task lifetime events, 18514f1c67SArnaldo Carvalho de Meloscheduling events, etc. 19514f1c67SArnaldo Carvalho de Melo 205e2485b1SDavid AhernThis is a live mode tool in addition to working with perf.data files like 215e2485b1SDavid Ahernthe other perf tools. Files can be generated using the 'perf record' command 225e2485b1SDavid Ahernbut the session needs to include the raw_syscalls events (-e 'raw_syscalls:*'). 2396355f2cSMasanari IidaAlternatively, 'perf trace record' can be used as a shortcut to 245e2485b1SDavid Ahernautomatically include the raw_syscalls events when writing events to a file. 255e2485b1SDavid Ahern 265e2485b1SDavid AhernThe following options apply to perf trace; options to perf trace record are 275e2485b1SDavid Ahernfound in the perf record man page. 28514f1c67SArnaldo Carvalho de Melo 29514f1c67SArnaldo Carvalho de MeloOPTIONS 30514f1c67SArnaldo Carvalho de Melo------- 31514f1c67SArnaldo Carvalho de Melo 32ac9be8eeSDavid Ahern-a:: 33514f1c67SArnaldo Carvalho de Melo--all-cpus:: 34514f1c67SArnaldo Carvalho de Melo System-wide collection from all CPUs. 35514f1c67SArnaldo Carvalho de Melo 362ae3a312SArnaldo Carvalho de Melo-e:: 372ae3a312SArnaldo Carvalho de Melo--expr:: 38d303e85aSArnaldo Carvalho de Melo List of syscalls to show, currently only syscall names. 39b059efdfSArnaldo Carvalho de Melo Prefixing with ! shows all syscalls but the ones specified. You may 40b059efdfSArnaldo Carvalho de Melo need to escape it. 412ae3a312SArnaldo Carvalho de Melo 42*e36b7821SAlexis Berlemont-D msecs:: 43*e36b7821SAlexis Berlemont--delay msecs:: 44*e36b7821SAlexis BerlemontAfter starting the program, wait msecs before measuring. This is useful to 45*e36b7821SAlexis Berlemontfilter out the startup phase of the program, which is often very different. 46*e36b7821SAlexis Berlemont 47c24ff998SArnaldo Carvalho de Melo-o:: 48c24ff998SArnaldo Carvalho de Melo--output=:: 49c24ff998SArnaldo Carvalho de Melo Output file name. 50c24ff998SArnaldo Carvalho de Melo 51514f1c67SArnaldo Carvalho de Melo-p:: 52514f1c67SArnaldo Carvalho de Melo--pid=:: 53514f1c67SArnaldo Carvalho de Melo Record events on existing process ID (comma separated list). 54514f1c67SArnaldo Carvalho de Melo 55ac9be8eeSDavid Ahern-t:: 56514f1c67SArnaldo Carvalho de Melo--tid=:: 57514f1c67SArnaldo Carvalho de Melo Record events on existing thread ID (comma separated list). 58514f1c67SArnaldo Carvalho de Melo 59ac9be8eeSDavid Ahern-u:: 60514f1c67SArnaldo Carvalho de Melo--uid=:: 61514f1c67SArnaldo Carvalho de Melo Record events in threads owned by uid. Name or number. 62514f1c67SArnaldo Carvalho de Melo 63f078c385SArnaldo Carvalho de Melo--filter-pids=:: 64f078c385SArnaldo Carvalho de Melo Filter out events for these pids and for 'trace' itself (comma separated list). 65f078c385SArnaldo Carvalho de Melo 667c304ee0SArnaldo Carvalho de Melo-v:: 677c304ee0SArnaldo Carvalho de Melo--verbose=:: 687c304ee0SArnaldo Carvalho de Melo Verbosity level. 697c304ee0SArnaldo Carvalho de Melo 70514f1c67SArnaldo Carvalho de Melo--no-inherit:: 71514f1c67SArnaldo Carvalho de Melo Child tasks do not inherit counters. 72514f1c67SArnaldo Carvalho de Melo 73ac9be8eeSDavid Ahern-m:: 74514f1c67SArnaldo Carvalho de Melo--mmap-pages=:: 7527050f53SJiri Olsa Number of mmap data pages (must be a power of two) or size 7627050f53SJiri Olsa specification with appended unit character - B/K/M/G. The 7727050f53SJiri Olsa size is rounded up to have nearest pages power of two value. 78514f1c67SArnaldo Carvalho de Melo 79ac9be8eeSDavid Ahern-C:: 80514f1c67SArnaldo Carvalho de Melo--cpu:: 81514f1c67SArnaldo Carvalho de MeloCollect samples only on the list of CPUs provided. Multiple CPUs can be provided as a 82514f1c67SArnaldo Carvalho de Melocomma-separated list with no space: 0,1. Ranges of CPUs are specified with -: 0-2. 83514f1c67SArnaldo Carvalho de MeloIn per-thread mode with inheritance mode on (default), Events are captured only when 84514f1c67SArnaldo Carvalho de Melothe thread executes on the designated CPUs. Default is to monitor all CPUs. 85514f1c67SArnaldo Carvalho de Melo 86ae9ed035SArnaldo Carvalho de Melo--duration: 87ae9ed035SArnaldo Carvalho de Melo Show only events that had a duration greater than N.M ms. 88ae9ed035SArnaldo Carvalho de Melo 891302d88eSArnaldo Carvalho de Melo--sched: 901302d88eSArnaldo Carvalho de Melo Accrue thread runtime and provide a summary at the end of the session. 911302d88eSArnaldo Carvalho de Melo 926810fc91SDavid Ahern-i 936810fc91SDavid Ahern--input 946810fc91SDavid Ahern Process events from a given perf data file. 956810fc91SDavid Ahern 964bb09192SDavid Ahern-T 974bb09192SDavid Ahern--time 984bb09192SDavid Ahern Print full timestamp rather time relative to first sample. 994bb09192SDavid Ahern 10050c95cbdSArnaldo Carvalho de Melo--comm:: 10150c95cbdSArnaldo Carvalho de Melo Show process COMM right beside its ID, on by default, disable with --no-comm. 10250c95cbdSArnaldo Carvalho de Melo 103fd2eabafSDavid Ahern-s:: 104bf2575c1SDavid Ahern--summary:: 105fd2eabafSDavid Ahern Show only a summary of syscalls by thread with min, max, and average times 106fd2eabafSDavid Ahern (in msec) and relative stddev. 107fd2eabafSDavid Ahern 108fd2eabafSDavid Ahern-S:: 109fd2eabafSDavid Ahern--with-summary:: 110fd2eabafSDavid Ahern Show all syscalls followed by a summary by thread with min, max, and 111fd2eabafSDavid Ahern average times (in msec) and relative stddev. 112bf2575c1SDavid Ahern 113c522739dSArnaldo Carvalho de Melo--tool_stats:: 114c522739dSArnaldo Carvalho de Melo Show tool stats such as number of times fd->pathname was discovered thru 115c522739dSArnaldo Carvalho de Melo hooking the open syscall return + vfs_getname or via reading /proc/pid/fd, etc. 116c522739dSArnaldo Carvalho de Melo 117598d02c5SStanislav Fomichev-F=[all|min|maj]:: 118598d02c5SStanislav Fomichev--pf=[all|min|maj]:: 119598d02c5SStanislav Fomichev Trace pagefaults. Optionally, you can specify whether you want minor, 120598d02c5SStanislav Fomichev major or all pagefaults. Default value is maj. 121598d02c5SStanislav Fomichev 122e281a960SStanislav Fomichev--syscalls:: 123e281a960SStanislav Fomichev Trace system calls. This options is enabled by default. 124e281a960SStanislav Fomichev 125566a0885SMilian Wolff--call-graph [mode,type,min[,limit],order[,key][,branch]]:: 126566a0885SMilian Wolff Setup and enable call-graph (stack chain/backtrace) recording. 127566a0885SMilian Wolff See `--call-graph` section in perf-record and perf-report 128566a0885SMilian Wolff man pages for details. The ones that are most useful in 'perf trace' 129566a0885SMilian Wolff are 'dwarf' and 'lbr', where available, try: 'perf trace --call-graph dwarf'. 130566a0885SMilian Wolff 131f3e459d1SArnaldo Carvalho de Melo Using this will, for the root user, bump the value of --mmap-pages to 4 132f3e459d1SArnaldo Carvalho de Melo times the maximum for non-root users, based on the kernel.perf_event_mlock_kb 133f3e459d1SArnaldo Carvalho de Melo sysctl. This is done only if the user doesn't specify a --mmap-pages value. 134f3e459d1SArnaldo Carvalho de Melo 13544621819SArnaldo Carvalho de Melo--kernel-syscall-graph:: 13644621819SArnaldo Carvalho de Melo Show the kernel callchains on the syscall exit path. 13744621819SArnaldo Carvalho de Melo 13877c92582SArnaldo Carvalho de Melo--event:: 13977c92582SArnaldo Carvalho de Melo Trace other events, see 'perf list' for a complete list. 14077c92582SArnaldo Carvalho de Melo 141c6d4a494SArnaldo Carvalho de Melo--max-stack:: 142c6d4a494SArnaldo Carvalho de Melo Set the stack depth limit when parsing the callchain, anything 143c6d4a494SArnaldo Carvalho de Melo beyond the specified depth will be ignored. Note that at this point 144c6d4a494SArnaldo Carvalho de Melo this is just about the presentation part, i.e. the kernel is still 145c6d4a494SArnaldo Carvalho de Melo not limiting, the overhead of callchains needs to be set via the 146c6d4a494SArnaldo Carvalho de Melo knobs in --call-graph dwarf. 147c6d4a494SArnaldo Carvalho de Melo 14805614993SArnaldo Carvalho de Melo Implies '--call-graph dwarf' when --call-graph not present on the 14905614993SArnaldo Carvalho de Melo command line, on systems where DWARF unwinding was built in. 15005614993SArnaldo Carvalho de Melo 151fe176085SArnaldo Carvalho de Melo Default: /proc/sys/kernel/perf_event_max_stack when present for 152fe176085SArnaldo Carvalho de Melo live sessions (without --input/-i), 127 otherwise. 153c6d4a494SArnaldo Carvalho de Melo 1545cf9c84eSArnaldo Carvalho de Melo--min-stack:: 1555cf9c84eSArnaldo Carvalho de Melo Set the stack depth limit when parsing the callchain, anything 1565cf9c84eSArnaldo Carvalho de Melo below the specified depth will be ignored. Disabled by default. 1575cf9c84eSArnaldo Carvalho de Melo 15805614993SArnaldo Carvalho de Melo Implies '--call-graph dwarf' when --call-graph not present on the 15905614993SArnaldo Carvalho de Melo command line, on systems where DWARF unwinding was built in. 16005614993SArnaldo Carvalho de Melo 1619d9cad76SKan Liang--proc-map-timeout:: 1629d9cad76SKan Liang When processing pre-existing threads /proc/XXX/mmap, it may take a long time, 1639d9cad76SKan Liang because the file may be huge. A time out is needed in such cases. 1649d9cad76SKan Liang This option sets the time out limit. The default value is 500 ms. 1659d9cad76SKan Liang 166598d02c5SStanislav FomichevPAGEFAULTS 167598d02c5SStanislav Fomichev---------- 168598d02c5SStanislav Fomichev 169598d02c5SStanislav FomichevWhen tracing pagefaults, the format of the trace is as follows: 170598d02c5SStanislav Fomichev 171598d02c5SStanislav Fomichev<min|maj>fault [<ip.symbol>+<ip.offset>] => <addr.dso@addr.offset> (<map type><addr level>). 172598d02c5SStanislav Fomichev 173598d02c5SStanislav Fomichev- min/maj indicates whether fault event is minor or major; 174598d02c5SStanislav Fomichev- ip.symbol shows symbol for instruction pointer (the code that generated the 175598d02c5SStanislav Fomichev fault); if no debug symbols available, perf trace will print raw IP; 176598d02c5SStanislav Fomichev- addr.dso shows DSO for the faulted address; 177598d02c5SStanislav Fomichev- map type is either 'd' for non-executable maps or 'x' for executable maps; 178598d02c5SStanislav Fomichev- addr level is either 'k' for kernel dso or '.' for user dso. 179598d02c5SStanislav Fomichev 180598d02c5SStanislav FomichevFor symbols resolution you may need to install debugging symbols. 181598d02c5SStanislav Fomichev 182598d02c5SStanislav FomichevPlease be aware that duration is currently always 0 and doesn't reflect actual 183598d02c5SStanislav Fomichevtime it took for fault to be handled! 184598d02c5SStanislav Fomichev 185598d02c5SStanislav FomichevWhen --verbose specified, perf trace tries to print all available information 186598d02c5SStanislav Fomichevfor both IP and fault address in the form of dso@symbol+offset. 187598d02c5SStanislav Fomichev 188598d02c5SStanislav FomichevEXAMPLES 189598d02c5SStanislav Fomichev-------- 190598d02c5SStanislav Fomichev 191e281a960SStanislav FomichevTrace only major pagefaults: 192e281a960SStanislav Fomichev 193e281a960SStanislav Fomichev $ perf trace --no-syscalls -F 194e281a960SStanislav Fomichev 195598d02c5SStanislav FomichevTrace syscalls, major and minor pagefaults: 196598d02c5SStanislav Fomichev 197598d02c5SStanislav Fomichev $ perf trace -F all 198598d02c5SStanislav Fomichev 199598d02c5SStanislav Fomichev 1416.547 ( 0.000 ms): python/20235 majfault [CRYPTO_push_info_+0x0] => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0@0x61be0 (x.) 200598d02c5SStanislav Fomichev 201598d02c5SStanislav Fomichev As you can see, there was major pagefault in python process, from 202598d02c5SStanislav Fomichev CRYPTO_push_info_ routine which faulted somewhere in libcrypto.so. 203598d02c5SStanislav Fomichev 204514f1c67SArnaldo Carvalho de MeloSEE ALSO 205514f1c67SArnaldo Carvalho de Melo-------- 206514f1c67SArnaldo Carvalho de Melolinkperf:perf-record[1], linkperf:perf-script[1] 207