1514f1c67SArnaldo Carvalho de Meloperf-trace(1) 2514f1c67SArnaldo Carvalho de Melo============= 3514f1c67SArnaldo Carvalho de Melo 4514f1c67SArnaldo Carvalho de MeloNAME 5514f1c67SArnaldo Carvalho de Melo---- 6514f1c67SArnaldo Carvalho de Meloperf-trace - strace inspired tool 7514f1c67SArnaldo Carvalho de Melo 8514f1c67SArnaldo Carvalho de MeloSYNOPSIS 9514f1c67SArnaldo Carvalho de Melo-------- 10514f1c67SArnaldo Carvalho de Melo[verse] 11514f1c67SArnaldo Carvalho de Melo'perf trace' 125e2485b1SDavid Ahern'perf trace record' 13514f1c67SArnaldo Carvalho de Melo 14514f1c67SArnaldo Carvalho de MeloDESCRIPTION 15514f1c67SArnaldo Carvalho de Melo----------- 16514f1c67SArnaldo Carvalho de MeloThis command will show the events associated with the target, initially 17514f1c67SArnaldo Carvalho de Melosyscalls, but other system events like pagefaults, task lifetime events, 18514f1c67SArnaldo Carvalho de Meloscheduling events, etc. 19514f1c67SArnaldo Carvalho de Melo 205e2485b1SDavid AhernThis is a live mode tool in addition to working with perf.data files like 215e2485b1SDavid Ahernthe other perf tools. Files can be generated using the 'perf record' command 225e2485b1SDavid Ahernbut the session needs to include the raw_syscalls events (-e 'raw_syscalls:*'). 2396355f2cSMasanari IidaAlternatively, 'perf trace record' can be used as a shortcut to 245e2485b1SDavid Ahernautomatically include the raw_syscalls events when writing events to a file. 255e2485b1SDavid Ahern 265e2485b1SDavid AhernThe following options apply to perf trace; options to perf trace record are 275e2485b1SDavid Ahernfound in the perf record man page. 28514f1c67SArnaldo Carvalho de Melo 29514f1c67SArnaldo Carvalho de MeloOPTIONS 30514f1c67SArnaldo Carvalho de Melo------- 31514f1c67SArnaldo Carvalho de Melo 32ac9be8eeSDavid Ahern-a:: 33514f1c67SArnaldo Carvalho de Melo--all-cpus:: 34514f1c67SArnaldo Carvalho de Melo System-wide collection from all CPUs. 35514f1c67SArnaldo Carvalho de Melo 362ae3a312SArnaldo Carvalho de Melo-e:: 372ae3a312SArnaldo Carvalho de Melo--expr:: 38d303e85aSArnaldo Carvalho de Melo List of syscalls to show, currently only syscall names. 39b059efdfSArnaldo Carvalho de Melo Prefixing with ! shows all syscalls but the ones specified. You may 40b059efdfSArnaldo Carvalho de Melo need to escape it. 412ae3a312SArnaldo Carvalho de Melo 42c24ff998SArnaldo Carvalho de Melo-o:: 43c24ff998SArnaldo Carvalho de Melo--output=:: 44c24ff998SArnaldo Carvalho de Melo Output file name. 45c24ff998SArnaldo Carvalho de Melo 46514f1c67SArnaldo Carvalho de Melo-p:: 47514f1c67SArnaldo Carvalho de Melo--pid=:: 48514f1c67SArnaldo Carvalho de Melo Record events on existing process ID (comma separated list). 49514f1c67SArnaldo Carvalho de Melo 50ac9be8eeSDavid Ahern-t:: 51514f1c67SArnaldo Carvalho de Melo--tid=:: 52514f1c67SArnaldo Carvalho de Melo Record events on existing thread ID (comma separated list). 53514f1c67SArnaldo Carvalho de Melo 54ac9be8eeSDavid Ahern-u:: 55514f1c67SArnaldo Carvalho de Melo--uid=:: 56514f1c67SArnaldo Carvalho de Melo Record events in threads owned by uid. Name or number. 57514f1c67SArnaldo Carvalho de Melo 58f078c385SArnaldo Carvalho de Melo--filter-pids=:: 59f078c385SArnaldo Carvalho de Melo Filter out events for these pids and for 'trace' itself (comma separated list). 60f078c385SArnaldo Carvalho de Melo 617c304ee0SArnaldo Carvalho de Melo-v:: 627c304ee0SArnaldo Carvalho de Melo--verbose=:: 637c304ee0SArnaldo Carvalho de Melo Verbosity level. 647c304ee0SArnaldo Carvalho de Melo 65514f1c67SArnaldo Carvalho de Melo--no-inherit:: 66514f1c67SArnaldo Carvalho de Melo Child tasks do not inherit counters. 67514f1c67SArnaldo Carvalho de Melo 68ac9be8eeSDavid Ahern-m:: 69514f1c67SArnaldo Carvalho de Melo--mmap-pages=:: 7027050f53SJiri Olsa Number of mmap data pages (must be a power of two) or size 7127050f53SJiri Olsa specification with appended unit character - B/K/M/G. The 7227050f53SJiri Olsa size is rounded up to have nearest pages power of two value. 73514f1c67SArnaldo Carvalho de Melo 74ac9be8eeSDavid Ahern-C:: 75514f1c67SArnaldo Carvalho de Melo--cpu:: 76514f1c67SArnaldo Carvalho de MeloCollect samples only on the list of CPUs provided. Multiple CPUs can be provided as a 77514f1c67SArnaldo Carvalho de Melocomma-separated list with no space: 0,1. Ranges of CPUs are specified with -: 0-2. 78514f1c67SArnaldo Carvalho de MeloIn per-thread mode with inheritance mode on (default), Events are captured only when 79514f1c67SArnaldo Carvalho de Melothe thread executes on the designated CPUs. Default is to monitor all CPUs. 80514f1c67SArnaldo Carvalho de Melo 81ae9ed035SArnaldo Carvalho de Melo--duration: 82ae9ed035SArnaldo Carvalho de Melo Show only events that had a duration greater than N.M ms. 83ae9ed035SArnaldo Carvalho de Melo 841302d88eSArnaldo Carvalho de Melo--sched: 851302d88eSArnaldo Carvalho de Melo Accrue thread runtime and provide a summary at the end of the session. 861302d88eSArnaldo Carvalho de Melo 876810fc91SDavid Ahern-i 886810fc91SDavid Ahern--input 896810fc91SDavid Ahern Process events from a given perf data file. 906810fc91SDavid Ahern 914bb09192SDavid Ahern-T 924bb09192SDavid Ahern--time 934bb09192SDavid Ahern Print full timestamp rather time relative to first sample. 944bb09192SDavid Ahern 9550c95cbdSArnaldo Carvalho de Melo--comm:: 9650c95cbdSArnaldo Carvalho de Melo Show process COMM right beside its ID, on by default, disable with --no-comm. 9750c95cbdSArnaldo Carvalho de Melo 98fd2eabafSDavid Ahern-s:: 99bf2575c1SDavid Ahern--summary:: 100fd2eabafSDavid Ahern Show only a summary of syscalls by thread with min, max, and average times 101fd2eabafSDavid Ahern (in msec) and relative stddev. 102fd2eabafSDavid Ahern 103fd2eabafSDavid Ahern-S:: 104fd2eabafSDavid Ahern--with-summary:: 105fd2eabafSDavid Ahern Show all syscalls followed by a summary by thread with min, max, and 106fd2eabafSDavid Ahern average times (in msec) and relative stddev. 107bf2575c1SDavid Ahern 108c522739dSArnaldo Carvalho de Melo--tool_stats:: 109c522739dSArnaldo Carvalho de Melo Show tool stats such as number of times fd->pathname was discovered thru 110c522739dSArnaldo Carvalho de Melo hooking the open syscall return + vfs_getname or via reading /proc/pid/fd, etc. 111c522739dSArnaldo Carvalho de Melo 112598d02c5SStanislav Fomichev-F=[all|min|maj]:: 113598d02c5SStanislav Fomichev--pf=[all|min|maj]:: 114598d02c5SStanislav Fomichev Trace pagefaults. Optionally, you can specify whether you want minor, 115598d02c5SStanislav Fomichev major or all pagefaults. Default value is maj. 116598d02c5SStanislav Fomichev 117e281a960SStanislav Fomichev--syscalls:: 118e281a960SStanislav Fomichev Trace system calls. This options is enabled by default. 119e281a960SStanislav Fomichev 120566a0885SMilian Wolff--call-graph [mode,type,min[,limit],order[,key][,branch]]:: 121566a0885SMilian Wolff Setup and enable call-graph (stack chain/backtrace) recording. 122566a0885SMilian Wolff See `--call-graph` section in perf-record and perf-report 123566a0885SMilian Wolff man pages for details. The ones that are most useful in 'perf trace' 124566a0885SMilian Wolff are 'dwarf' and 'lbr', where available, try: 'perf trace --call-graph dwarf'. 125566a0885SMilian Wolff 12644621819SArnaldo Carvalho de Melo--kernel-syscall-graph:: 12744621819SArnaldo Carvalho de Melo Show the kernel callchains on the syscall exit path. 12844621819SArnaldo Carvalho de Melo 12977c92582SArnaldo Carvalho de Melo--event:: 13077c92582SArnaldo Carvalho de Melo Trace other events, see 'perf list' for a complete list. 13177c92582SArnaldo Carvalho de Melo 132c6d4a494SArnaldo Carvalho de Melo--max-stack:: 133c6d4a494SArnaldo Carvalho de Melo Set the stack depth limit when parsing the callchain, anything 134c6d4a494SArnaldo Carvalho de Melo beyond the specified depth will be ignored. Note that at this point 135c6d4a494SArnaldo Carvalho de Melo this is just about the presentation part, i.e. the kernel is still 136c6d4a494SArnaldo Carvalho de Melo not limiting, the overhead of callchains needs to be set via the 137c6d4a494SArnaldo Carvalho de Melo knobs in --call-graph dwarf. 138c6d4a494SArnaldo Carvalho de Melo 139*05614993SArnaldo Carvalho de Melo Implies '--call-graph dwarf' when --call-graph not present on the 140*05614993SArnaldo Carvalho de Melo command line, on systems where DWARF unwinding was built in. 141*05614993SArnaldo Carvalho de Melo 142c6d4a494SArnaldo Carvalho de Melo Default: 127 143c6d4a494SArnaldo Carvalho de Melo 1445cf9c84eSArnaldo Carvalho de Melo--min-stack:: 1455cf9c84eSArnaldo Carvalho de Melo Set the stack depth limit when parsing the callchain, anything 1465cf9c84eSArnaldo Carvalho de Melo below the specified depth will be ignored. Disabled by default. 1475cf9c84eSArnaldo Carvalho de Melo 148*05614993SArnaldo Carvalho de Melo Implies '--call-graph dwarf' when --call-graph not present on the 149*05614993SArnaldo Carvalho de Melo command line, on systems where DWARF unwinding was built in. 150*05614993SArnaldo Carvalho de Melo 1519d9cad76SKan Liang--proc-map-timeout:: 1529d9cad76SKan Liang When processing pre-existing threads /proc/XXX/mmap, it may take a long time, 1539d9cad76SKan Liang because the file may be huge. A time out is needed in such cases. 1549d9cad76SKan Liang This option sets the time out limit. The default value is 500 ms. 1559d9cad76SKan Liang 156598d02c5SStanislav FomichevPAGEFAULTS 157598d02c5SStanislav Fomichev---------- 158598d02c5SStanislav Fomichev 159598d02c5SStanislav FomichevWhen tracing pagefaults, the format of the trace is as follows: 160598d02c5SStanislav Fomichev 161598d02c5SStanislav Fomichev<min|maj>fault [<ip.symbol>+<ip.offset>] => <addr.dso@addr.offset> (<map type><addr level>). 162598d02c5SStanislav Fomichev 163598d02c5SStanislav Fomichev- min/maj indicates whether fault event is minor or major; 164598d02c5SStanislav Fomichev- ip.symbol shows symbol for instruction pointer (the code that generated the 165598d02c5SStanislav Fomichev fault); if no debug symbols available, perf trace will print raw IP; 166598d02c5SStanislav Fomichev- addr.dso shows DSO for the faulted address; 167598d02c5SStanislav Fomichev- map type is either 'd' for non-executable maps or 'x' for executable maps; 168598d02c5SStanislav Fomichev- addr level is either 'k' for kernel dso or '.' for user dso. 169598d02c5SStanislav Fomichev 170598d02c5SStanislav FomichevFor symbols resolution you may need to install debugging symbols. 171598d02c5SStanislav Fomichev 172598d02c5SStanislav FomichevPlease be aware that duration is currently always 0 and doesn't reflect actual 173598d02c5SStanislav Fomichevtime it took for fault to be handled! 174598d02c5SStanislav Fomichev 175598d02c5SStanislav FomichevWhen --verbose specified, perf trace tries to print all available information 176598d02c5SStanislav Fomichevfor both IP and fault address in the form of dso@symbol+offset. 177598d02c5SStanislav Fomichev 178598d02c5SStanislav FomichevEXAMPLES 179598d02c5SStanislav Fomichev-------- 180598d02c5SStanislav Fomichev 181e281a960SStanislav FomichevTrace only major pagefaults: 182e281a960SStanislav Fomichev 183e281a960SStanislav Fomichev $ perf trace --no-syscalls -F 184e281a960SStanislav Fomichev 185598d02c5SStanislav FomichevTrace syscalls, major and minor pagefaults: 186598d02c5SStanislav Fomichev 187598d02c5SStanislav Fomichev $ perf trace -F all 188598d02c5SStanislav Fomichev 189598d02c5SStanislav Fomichev 1416.547 ( 0.000 ms): python/20235 majfault [CRYPTO_push_info_+0x0] => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0@0x61be0 (x.) 190598d02c5SStanislav Fomichev 191598d02c5SStanislav Fomichev As you can see, there was major pagefault in python process, from 192598d02c5SStanislav Fomichev CRYPTO_push_info_ routine which faulted somewhere in libcrypto.so. 193598d02c5SStanislav Fomichev 194514f1c67SArnaldo Carvalho de MeloSEE ALSO 195514f1c67SArnaldo Carvalho de Melo-------- 196514f1c67SArnaldo Carvalho de Melolinkperf:perf-record[1], linkperf:perf-script[1] 197