1514f1c67SArnaldo Carvalho de Meloperf-trace(1) 2514f1c67SArnaldo Carvalho de Melo============= 3514f1c67SArnaldo Carvalho de Melo 4514f1c67SArnaldo Carvalho de MeloNAME 5514f1c67SArnaldo Carvalho de Melo---- 6514f1c67SArnaldo Carvalho de Meloperf-trace - strace inspired tool 7514f1c67SArnaldo Carvalho de Melo 8514f1c67SArnaldo Carvalho de MeloSYNOPSIS 9514f1c67SArnaldo Carvalho de Melo-------- 10514f1c67SArnaldo Carvalho de Melo[verse] 11514f1c67SArnaldo Carvalho de Melo'perf trace' 125e2485b1SDavid Ahern'perf trace record' 13514f1c67SArnaldo Carvalho de Melo 14514f1c67SArnaldo Carvalho de MeloDESCRIPTION 15514f1c67SArnaldo Carvalho de Melo----------- 16514f1c67SArnaldo Carvalho de MeloThis command will show the events associated with the target, initially 17514f1c67SArnaldo Carvalho de Melosyscalls, but other system events like pagefaults, task lifetime events, 18514f1c67SArnaldo Carvalho de Meloscheduling events, etc. 19514f1c67SArnaldo Carvalho de Melo 205e2485b1SDavid AhernThis is a live mode tool in addition to working with perf.data files like 215e2485b1SDavid Ahernthe other perf tools. Files can be generated using the 'perf record' command 225e2485b1SDavid Ahernbut the session needs to include the raw_syscalls events (-e 'raw_syscalls:*'). 2396355f2cSMasanari IidaAlternatively, 'perf trace record' can be used as a shortcut to 245e2485b1SDavid Ahernautomatically include the raw_syscalls events when writing events to a file. 255e2485b1SDavid Ahern 265e2485b1SDavid AhernThe following options apply to perf trace; options to perf trace record are 275e2485b1SDavid Ahernfound in the perf record man page. 28514f1c67SArnaldo Carvalho de Melo 29514f1c67SArnaldo Carvalho de MeloOPTIONS 30514f1c67SArnaldo Carvalho de Melo------- 31514f1c67SArnaldo Carvalho de Melo 32ac9be8eeSDavid Ahern-a:: 33514f1c67SArnaldo Carvalho de Melo--all-cpus:: 34514f1c67SArnaldo Carvalho de Melo System-wide collection from all CPUs. 35514f1c67SArnaldo Carvalho de Melo 362ae3a312SArnaldo Carvalho de Melo-e:: 372ae3a312SArnaldo Carvalho de Melo--expr:: 38*017037ffSArnaldo Carvalho de Melo--event:: 39*017037ffSArnaldo Carvalho de Melo List of syscalls and other perf events (tracepoints, HW cache events, 40*017037ffSArnaldo Carvalho de Melo etc) to show. 41*017037ffSArnaldo Carvalho de Melo See 'perf list' for a complete list of events. 42b059efdfSArnaldo Carvalho de Melo Prefixing with ! shows all syscalls but the ones specified. You may 43b059efdfSArnaldo Carvalho de Melo need to escape it. 442ae3a312SArnaldo Carvalho de Melo 45e36b7821SAlexis Berlemont-D msecs:: 46e36b7821SAlexis Berlemont--delay msecs:: 47e36b7821SAlexis BerlemontAfter starting the program, wait msecs before measuring. This is useful to 48e36b7821SAlexis Berlemontfilter out the startup phase of the program, which is often very different. 49e36b7821SAlexis Berlemont 50c24ff998SArnaldo Carvalho de Melo-o:: 51c24ff998SArnaldo Carvalho de Melo--output=:: 52c24ff998SArnaldo Carvalho de Melo Output file name. 53c24ff998SArnaldo Carvalho de Melo 54514f1c67SArnaldo Carvalho de Melo-p:: 55514f1c67SArnaldo Carvalho de Melo--pid=:: 56514f1c67SArnaldo Carvalho de Melo Record events on existing process ID (comma separated list). 57514f1c67SArnaldo Carvalho de Melo 58ac9be8eeSDavid Ahern-t:: 59514f1c67SArnaldo Carvalho de Melo--tid=:: 60514f1c67SArnaldo Carvalho de Melo Record events on existing thread ID (comma separated list). 61514f1c67SArnaldo Carvalho de Melo 62ac9be8eeSDavid Ahern-u:: 63514f1c67SArnaldo Carvalho de Melo--uid=:: 64514f1c67SArnaldo Carvalho de Melo Record events in threads owned by uid. Name or number. 65514f1c67SArnaldo Carvalho de Melo 66f078c385SArnaldo Carvalho de Melo--filter-pids=:: 67f078c385SArnaldo Carvalho de Melo Filter out events for these pids and for 'trace' itself (comma separated list). 68f078c385SArnaldo Carvalho de Melo 697c304ee0SArnaldo Carvalho de Melo-v:: 707c304ee0SArnaldo Carvalho de Melo--verbose=:: 717c304ee0SArnaldo Carvalho de Melo Verbosity level. 727c304ee0SArnaldo Carvalho de Melo 73514f1c67SArnaldo Carvalho de Melo--no-inherit:: 74514f1c67SArnaldo Carvalho de Melo Child tasks do not inherit counters. 75514f1c67SArnaldo Carvalho de Melo 76ac9be8eeSDavid Ahern-m:: 77514f1c67SArnaldo Carvalho de Melo--mmap-pages=:: 7827050f53SJiri Olsa Number of mmap data pages (must be a power of two) or size 7927050f53SJiri Olsa specification with appended unit character - B/K/M/G. The 8027050f53SJiri Olsa size is rounded up to have nearest pages power of two value. 81514f1c67SArnaldo Carvalho de Melo 82ac9be8eeSDavid Ahern-C:: 83514f1c67SArnaldo Carvalho de Melo--cpu:: 84514f1c67SArnaldo Carvalho de MeloCollect samples only on the list of CPUs provided. Multiple CPUs can be provided as a 85514f1c67SArnaldo Carvalho de Melocomma-separated list with no space: 0,1. Ranges of CPUs are specified with -: 0-2. 86514f1c67SArnaldo Carvalho de MeloIn per-thread mode with inheritance mode on (default), Events are captured only when 87514f1c67SArnaldo Carvalho de Melothe thread executes on the designated CPUs. Default is to monitor all CPUs. 88514f1c67SArnaldo Carvalho de Melo 89ae9ed035SArnaldo Carvalho de Melo--duration: 90ae9ed035SArnaldo Carvalho de Melo Show only events that had a duration greater than N.M ms. 91ae9ed035SArnaldo Carvalho de Melo 921302d88eSArnaldo Carvalho de Melo--sched: 931302d88eSArnaldo Carvalho de Melo Accrue thread runtime and provide a summary at the end of the session. 941302d88eSArnaldo Carvalho de Melo 956810fc91SDavid Ahern-i 966810fc91SDavid Ahern--input 976810fc91SDavid Ahern Process events from a given perf data file. 986810fc91SDavid Ahern 994bb09192SDavid Ahern-T 1004bb09192SDavid Ahern--time 1014bb09192SDavid Ahern Print full timestamp rather time relative to first sample. 1024bb09192SDavid Ahern 10350c95cbdSArnaldo Carvalho de Melo--comm:: 10450c95cbdSArnaldo Carvalho de Melo Show process COMM right beside its ID, on by default, disable with --no-comm. 10550c95cbdSArnaldo Carvalho de Melo 106fd2eabafSDavid Ahern-s:: 107bf2575c1SDavid Ahern--summary:: 108fd2eabafSDavid Ahern Show only a summary of syscalls by thread with min, max, and average times 109fd2eabafSDavid Ahern (in msec) and relative stddev. 110fd2eabafSDavid Ahern 111fd2eabafSDavid Ahern-S:: 112fd2eabafSDavid Ahern--with-summary:: 113fd2eabafSDavid Ahern Show all syscalls followed by a summary by thread with min, max, and 114fd2eabafSDavid Ahern average times (in msec) and relative stddev. 115bf2575c1SDavid Ahern 116c522739dSArnaldo Carvalho de Melo--tool_stats:: 117c522739dSArnaldo Carvalho de Melo Show tool stats such as number of times fd->pathname was discovered thru 118c522739dSArnaldo Carvalho de Melo hooking the open syscall return + vfs_getname or via reading /proc/pid/fd, etc. 119c522739dSArnaldo Carvalho de Melo 120598d02c5SStanislav Fomichev-F=[all|min|maj]:: 121598d02c5SStanislav Fomichev--pf=[all|min|maj]:: 122598d02c5SStanislav Fomichev Trace pagefaults. Optionally, you can specify whether you want minor, 123598d02c5SStanislav Fomichev major or all pagefaults. Default value is maj. 124598d02c5SStanislav Fomichev 125e281a960SStanislav Fomichev--syscalls:: 126e281a960SStanislav Fomichev Trace system calls. This options is enabled by default. 127e281a960SStanislav Fomichev 128566a0885SMilian Wolff--call-graph [mode,type,min[,limit],order[,key][,branch]]:: 129566a0885SMilian Wolff Setup and enable call-graph (stack chain/backtrace) recording. 130566a0885SMilian Wolff See `--call-graph` section in perf-record and perf-report 131566a0885SMilian Wolff man pages for details. The ones that are most useful in 'perf trace' 132566a0885SMilian Wolff are 'dwarf' and 'lbr', where available, try: 'perf trace --call-graph dwarf'. 133566a0885SMilian Wolff 134f3e459d1SArnaldo Carvalho de Melo Using this will, for the root user, bump the value of --mmap-pages to 4 135f3e459d1SArnaldo Carvalho de Melo times the maximum for non-root users, based on the kernel.perf_event_mlock_kb 136f3e459d1SArnaldo Carvalho de Melo sysctl. This is done only if the user doesn't specify a --mmap-pages value. 137f3e459d1SArnaldo Carvalho de Melo 13844621819SArnaldo Carvalho de Melo--kernel-syscall-graph:: 13944621819SArnaldo Carvalho de Melo Show the kernel callchains on the syscall exit path. 14044621819SArnaldo Carvalho de Melo 141c6d4a494SArnaldo Carvalho de Melo--max-stack:: 142c6d4a494SArnaldo Carvalho de Melo Set the stack depth limit when parsing the callchain, anything 143c6d4a494SArnaldo Carvalho de Melo beyond the specified depth will be ignored. Note that at this point 144c6d4a494SArnaldo Carvalho de Melo this is just about the presentation part, i.e. the kernel is still 145c6d4a494SArnaldo Carvalho de Melo not limiting, the overhead of callchains needs to be set via the 146c6d4a494SArnaldo Carvalho de Melo knobs in --call-graph dwarf. 147c6d4a494SArnaldo Carvalho de Melo 14805614993SArnaldo Carvalho de Melo Implies '--call-graph dwarf' when --call-graph not present on the 14905614993SArnaldo Carvalho de Melo command line, on systems where DWARF unwinding was built in. 15005614993SArnaldo Carvalho de Melo 151fe176085SArnaldo Carvalho de Melo Default: /proc/sys/kernel/perf_event_max_stack when present for 152fe176085SArnaldo Carvalho de Melo live sessions (without --input/-i), 127 otherwise. 153c6d4a494SArnaldo Carvalho de Melo 1545cf9c84eSArnaldo Carvalho de Melo--min-stack:: 1555cf9c84eSArnaldo Carvalho de Melo Set the stack depth limit when parsing the callchain, anything 1565cf9c84eSArnaldo Carvalho de Melo below the specified depth will be ignored. Disabled by default. 1575cf9c84eSArnaldo Carvalho de Melo 15805614993SArnaldo Carvalho de Melo Implies '--call-graph dwarf' when --call-graph not present on the 15905614993SArnaldo Carvalho de Melo command line, on systems where DWARF unwinding was built in. 16005614993SArnaldo Carvalho de Melo 1619d9cad76SKan Liang--proc-map-timeout:: 1629d9cad76SKan Liang When processing pre-existing threads /proc/XXX/mmap, it may take a long time, 1639d9cad76SKan Liang because the file may be huge. A time out is needed in such cases. 1649d9cad76SKan Liang This option sets the time out limit. The default value is 500 ms. 1659d9cad76SKan Liang 166598d02c5SStanislav FomichevPAGEFAULTS 167598d02c5SStanislav Fomichev---------- 168598d02c5SStanislav Fomichev 169598d02c5SStanislav FomichevWhen tracing pagefaults, the format of the trace is as follows: 170598d02c5SStanislav Fomichev 171598d02c5SStanislav Fomichev<min|maj>fault [<ip.symbol>+<ip.offset>] => <addr.dso@addr.offset> (<map type><addr level>). 172598d02c5SStanislav Fomichev 173598d02c5SStanislav Fomichev- min/maj indicates whether fault event is minor or major; 174598d02c5SStanislav Fomichev- ip.symbol shows symbol for instruction pointer (the code that generated the 175598d02c5SStanislav Fomichev fault); if no debug symbols available, perf trace will print raw IP; 176598d02c5SStanislav Fomichev- addr.dso shows DSO for the faulted address; 177598d02c5SStanislav Fomichev- map type is either 'd' for non-executable maps or 'x' for executable maps; 178598d02c5SStanislav Fomichev- addr level is either 'k' for kernel dso or '.' for user dso. 179598d02c5SStanislav Fomichev 180598d02c5SStanislav FomichevFor symbols resolution you may need to install debugging symbols. 181598d02c5SStanislav Fomichev 182598d02c5SStanislav FomichevPlease be aware that duration is currently always 0 and doesn't reflect actual 183598d02c5SStanislav Fomichevtime it took for fault to be handled! 184598d02c5SStanislav Fomichev 185598d02c5SStanislav FomichevWhen --verbose specified, perf trace tries to print all available information 186598d02c5SStanislav Fomichevfor both IP and fault address in the form of dso@symbol+offset. 187598d02c5SStanislav Fomichev 188598d02c5SStanislav FomichevEXAMPLES 189598d02c5SStanislav Fomichev-------- 190598d02c5SStanislav Fomichev 191e281a960SStanislav FomichevTrace only major pagefaults: 192e281a960SStanislav Fomichev 193e281a960SStanislav Fomichev $ perf trace --no-syscalls -F 194e281a960SStanislav Fomichev 195598d02c5SStanislav FomichevTrace syscalls, major and minor pagefaults: 196598d02c5SStanislav Fomichev 197598d02c5SStanislav Fomichev $ perf trace -F all 198598d02c5SStanislav Fomichev 199598d02c5SStanislav Fomichev 1416.547 ( 0.000 ms): python/20235 majfault [CRYPTO_push_info_+0x0] => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0@0x61be0 (x.) 200598d02c5SStanislav Fomichev 201598d02c5SStanislav Fomichev As you can see, there was major pagefault in python process, from 202598d02c5SStanislav Fomichev CRYPTO_push_info_ routine which faulted somewhere in libcrypto.so. 203598d02c5SStanislav Fomichev 204514f1c67SArnaldo Carvalho de MeloSEE ALSO 205514f1c67SArnaldo Carvalho de Melo-------- 206514f1c67SArnaldo Carvalho de Melolinkperf:perf-record[1], linkperf:perf-script[1] 207