12c9a583bSTodd Brandt _ 22c9a583bSTodd Brandt _ __ _ __ ___ __ _ _ __ __ _ _ __ | |__ 32c9a583bSTodd Brandt | '_ \| '_ ` _ \ _____ / _` | '__/ _` | '_ \| '_ \ 42c9a583bSTodd Brandt | |_) | | | | | |_____| (_| | | | (_| | |_) | | | | 52c9a583bSTodd Brandt | .__/|_| |_| |_| \__, |_| \__,_| .__/|_| |_| 62c9a583bSTodd Brandt |_| |___/ |_| 7d5a5e4ecSTodd Brandt 8d5a5e4ecSTodd Brandt pm-graph: suspend/resume/boot timing analysis tools 9*04175527STodd Brandt Version: 5.11 10d5a5e4ecSTodd Brandt Author: Todd Brandt <todd.e.brandt@intel.com> 119bfb0977STodd Brandt Home Page: https://www.intel.com/content/www/us/en/developer/topic-technology/open/pm-graph/overview.html 12d5a5e4ecSTodd Brandt 13d5a5e4ecSTodd Brandt Report bugs/issues at bugzilla.kernel.org Tools/pm-graph 14d5a5e4ecSTodd Brandt - https://bugzilla.kernel.org/buglist.cgi?component=pm-graph&product=Tools 15d5a5e4ecSTodd Brandt 16d5a5e4ecSTodd Brandt Full documentation available online & in man pages 17d5a5e4ecSTodd Brandt - Getting Started: 189bfb0977STodd Brandt https://www.intel.com/content/www/us/en/developer/articles/technical/usage.html 19d5a5e4ecSTodd Brandt 209bfb0977STodd Brandt - Feature Summary: 219bfb0977STodd Brandt https://www.intel.com/content/www/us/en/developer/topic-technology/open/pm-graph/features.html 22d5a5e4ecSTodd Brandt 23d5a5e4ecSTodd Brandt - upstream version in git: 249bfb0977STodd Brandt git clone https://github.com/intel/pm-graph/ 25d5a5e4ecSTodd Brandt 26d5a5e4ecSTodd Brandt Table of Contents 27d5a5e4ecSTodd Brandt - Overview 28d5a5e4ecSTodd Brandt - Setup 29d5a5e4ecSTodd Brandt - Usage 30d5a5e4ecSTodd Brandt - Basic Usage 31d5a5e4ecSTodd Brandt - Dev Mode Usage 32d5a5e4ecSTodd Brandt - Proc Mode Usage 332c9a583bSTodd Brandt - Endurance Testing 342c9a583bSTodd Brandt - Usage Examples 35d5a5e4ecSTodd Brandt - Configuration Files 36d5a5e4ecSTodd Brandt - Usage Examples 37d5a5e4ecSTodd Brandt - Config File Options 38d5a5e4ecSTodd Brandt - Custom Timeline Entries 39d5a5e4ecSTodd Brandt - Adding/Editing Timeline Functions 40d5a5e4ecSTodd Brandt - Adding/Editing Dev Timeline Source Functions 41d5a5e4ecSTodd Brandt - Verifying your Custom Functions 42d5a5e4ecSTodd Brandt - Testing on consumer linux Operating Systems 43d5a5e4ecSTodd Brandt - Android 44d5a5e4ecSTodd Brandt 45d5a5e4ecSTodd Brandt------------------------------------------------------------------ 46d5a5e4ecSTodd Brandt| OVERVIEW | 47d5a5e4ecSTodd Brandt------------------------------------------------------------------ 48d5a5e4ecSTodd Brandt 49d5a5e4ecSTodd Brandt This tool suite is designed to assist kernel and OS developers in optimizing 50d5a5e4ecSTodd Brandt their linux stack's suspend/resume & boot time. Using a kernel image built 51d5a5e4ecSTodd Brandt with a few extra options enabled, the tools will execute a suspend or boot, 52d5a5e4ecSTodd Brandt and will capture dmesg and ftrace data. This data is transformed into a set of 53d5a5e4ecSTodd Brandt timelines and a callgraph to give a quick and detailed view of which devices 54d5a5e4ecSTodd Brandt and kernel processes are taking the most time in suspend/resume & boot. 55d5a5e4ecSTodd Brandt 56d5a5e4ecSTodd Brandt------------------------------------------------------------------ 57d5a5e4ecSTodd Brandt| SETUP | 58d5a5e4ecSTodd Brandt------------------------------------------------------------------ 59d5a5e4ecSTodd Brandt 602c9a583bSTodd Brandt Package Requirements 612c9a583bSTodd Brandt - runs with python2 or python3, choice is made by /usr/bin/python link 62d5a5e4ecSTodd Brandt - python 632c9a583bSTodd Brandt - python-configparser (for python2 sleepgraph) 64d23e95c0STodd Brandt - python-requests (for stresstester.py) 652c9a583bSTodd Brandt - linux-tools-common (for turbostat usage in sleepgraph) 66d5a5e4ecSTodd Brandt 67d5a5e4ecSTodd Brandt Ubuntu: 682c9a583bSTodd Brandt sudo apt-get install python python-configparser python-requests linux-tools-common 69d5a5e4ecSTodd Brandt 70d5a5e4ecSTodd Brandt Fedora: 712c9a583bSTodd Brandt sudo dnf install python python-configparser python-requests linux-tools-common 72d5a5e4ecSTodd Brandt 73d5a5e4ecSTodd Brandt The tools can most easily be installed via git clone and make install 74d5a5e4ecSTodd Brandt 75d5a5e4ecSTodd Brandt $> git clone http://github.com/intel/pm-graph.git 76d5a5e4ecSTodd Brandt $> cd pm-graph 77d5a5e4ecSTodd Brandt $> sudo make install 78d5a5e4ecSTodd Brandt $> man sleepgraph ; man bootgraph 79d5a5e4ecSTodd Brandt 80d5a5e4ecSTodd Brandt Setup involves some minor kernel configuration 81d5a5e4ecSTodd Brandt 82d5a5e4ecSTodd Brandt The following kernel build options are required for all kernels: 83d5a5e4ecSTodd Brandt CONFIG_DEVMEM=y 84d5a5e4ecSTodd Brandt CONFIG_PM_DEBUG=y 85d5a5e4ecSTodd Brandt CONFIG_PM_SLEEP_DEBUG=y 86d5a5e4ecSTodd Brandt CONFIG_FTRACE=y 87d5a5e4ecSTodd Brandt CONFIG_FUNCTION_TRACER=y 88d5a5e4ecSTodd Brandt CONFIG_FUNCTION_GRAPH_TRACER=y 89d5a5e4ecSTodd Brandt CONFIG_KPROBES=y 90d5a5e4ecSTodd Brandt CONFIG_KPROBES_ON_FTRACE=y 91d5a5e4ecSTodd Brandt 92d5a5e4ecSTodd Brandt In kernel 3.15.0, two patches were upstreamed which enable the 93d5a5e4ecSTodd Brandt v3.0 behavior. These patches allow the tool to read all the 94d5a5e4ecSTodd Brandt data from trace events instead of from dmesg. You can enable 95d5a5e4ecSTodd Brandt this behavior on earlier kernels with these patches: 96d5a5e4ecSTodd Brandt 97d5a5e4ecSTodd Brandt (kernel/pre-3.15/enable_trace_events_suspend_resume.patch) 98d5a5e4ecSTodd Brandt (kernel/pre-3.15/enable_trace_events_device_pm_callback.patch) 99d5a5e4ecSTodd Brandt 100b3f6c43dSTodd Brandt If you're using bootgraph, or sleepgraph with a kernel older than 3.15.0, 101b3f6c43dSTodd Brandt the following additional kernel parameters are required: 102d5a5e4ecSTodd Brandt (e.g. in file /etc/default/grub) 103d5a5e4ecSTodd Brandt GRUB_CMDLINE_LINUX_DEFAULT="... initcall_debug log_buf_len=32M ..." 104d5a5e4ecSTodd Brandt 105d5a5e4ecSTodd Brandt If you're using a kernel older than 3.11-rc2, the following simple 106d5a5e4ecSTodd Brandt patch must be applied to enable ftrace data: 107d5a5e4ecSTodd Brandt in file: kernel/power/suspend.c 108d5a5e4ecSTodd Brandt in function: int suspend_devices_and_enter(suspend_state_t state) 109d5a5e4ecSTodd Brandt remove call to "ftrace_stop();" 110d5a5e4ecSTodd Brandt remove call to "ftrace_start();" 111d5a5e4ecSTodd Brandt 112d5a5e4ecSTodd Brandt There is a patch which does this for kernel v3.8.0: 113d5a5e4ecSTodd Brandt (kernel/pre-3.11-rc2/enable_ftrace_in_suspendresume.patch) 114d5a5e4ecSTodd Brandt 115d5a5e4ecSTodd Brandt 116d5a5e4ecSTodd Brandt 117d5a5e4ecSTodd Brandt------------------------------------------------------------------ 118d5a5e4ecSTodd Brandt| USAGE | 119d5a5e4ecSTodd Brandt------------------------------------------------------------------ 120d5a5e4ecSTodd Brandt 121d5a5e4ecSTodd BrandtBasic Usage 122d5a5e4ecSTodd Brandt___________ 123d5a5e4ecSTodd Brandt 124d5a5e4ecSTodd Brandt 1) First configure a kernel using the instructions from the previous sections. 125d5a5e4ecSTodd Brandt Then build, install, and boot with it. 126d5a5e4ecSTodd Brandt 2) Open up a terminal window and execute the mode list command: 127d5a5e4ecSTodd Brandt 128d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -modes 129d5a5e4ecSTodd Brandt ['freeze', 'mem', 'disk'] 130d5a5e4ecSTodd Brandt 131d5a5e4ecSTodd Brandt Execute a test using one of the available power modes, e.g. mem (S3): 132d5a5e4ecSTodd Brandt 133d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -m mem -rtcwake 15 134d5a5e4ecSTodd Brandt 135d5a5e4ecSTodd Brandt or with a config file 136d5a5e4ecSTodd Brandt 137d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -config config/suspend.cfg 138d5a5e4ecSTodd Brandt 139d5a5e4ecSTodd Brandt When the system comes back you'll see the script finishing up and 140d5a5e4ecSTodd Brandt creating the output files in the test subdir. It generates output 141d5a5e4ecSTodd Brandt files in subdirectory: suspend-mmddyy-HHMMSS. The ftrace file can 142d5a5e4ecSTodd Brandt be used to regenerate the html timeline with different options 143d5a5e4ecSTodd Brandt 144d5a5e4ecSTodd Brandt HTML output: <hostname>_<mode>.html 145d5a5e4ecSTodd Brandt raw dmesg output: <hostname>_<mode>_dmesg.txt 146d5a5e4ecSTodd Brandt raw ftrace output: <hostname>_<mode>_ftrace.txt 147d5a5e4ecSTodd Brandt 148d5a5e4ecSTodd Brandt View the html in firefox or chrome. 149d5a5e4ecSTodd Brandt 150d5a5e4ecSTodd Brandt 151d5a5e4ecSTodd BrandtDev Mode Usage 152d5a5e4ecSTodd Brandt______________ 153d5a5e4ecSTodd Brandt 154d5a5e4ecSTodd Brandt Developer mode adds information on low level source calls to the timeline. 155d5a5e4ecSTodd Brandt The tool sets kprobes on all delay and mutex calls to see which devices 156d5a5e4ecSTodd Brandt are waiting for something and when. It also sets a suite of kprobes on 157d5a5e4ecSTodd Brandt subsystem dependent calls to better fill out the timeline. 158d5a5e4ecSTodd Brandt 159d5a5e4ecSTodd Brandt The tool will also expose kernel threads that don't normally show up in the 160d5a5e4ecSTodd Brandt timeline. This is useful in discovering dependent threads to get a better 161d5a5e4ecSTodd Brandt idea of what each device is waiting for. For instance, the scsi_eh thread, 162d5a5e4ecSTodd Brandt a.k.a. scsi resume error handler, is what each SATA disk device waits for 163d5a5e4ecSTodd Brandt before it can continue resume. 164d5a5e4ecSTodd Brandt 165d5a5e4ecSTodd Brandt The timeline will be much larger if run with dev mode, so it can be useful 166d5a5e4ecSTodd Brandt to set the -mindev option to clip out any device blocks that are too small 167d5a5e4ecSTodd Brandt to see easily. The following command will give a nice dev mode run: 168d5a5e4ecSTodd Brandt 169d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -m mem -rtcwake 15 -mindev 1 -dev 170d5a5e4ecSTodd Brandt 171d5a5e4ecSTodd Brandt or with a config file 172d5a5e4ecSTodd Brandt 173d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -config config/suspend-dev.cfg 174d5a5e4ecSTodd Brandt 175d5a5e4ecSTodd Brandt 176d5a5e4ecSTodd BrandtProc Mode Usage 177d5a5e4ecSTodd Brandt_______________ 178d5a5e4ecSTodd Brandt 179d5a5e4ecSTodd Brandt Proc mode adds user process info to the timeline. This is done in a manner 180d5a5e4ecSTodd Brandt similar to the bootchart utility, which graphs init processes and their 181d5a5e4ecSTodd Brandt execution as the system boots. This tool option does the same thing but for 182d5a5e4ecSTodd Brandt the period before and after suspend/resume. 183d5a5e4ecSTodd Brandt 184d5a5e4ecSTodd Brandt In order to see any process info, there needs to be some delay before or 185d5a5e4ecSTodd Brandt after resume since processes are frozen in suspend_prepare and thawed in 186d5a5e4ecSTodd Brandt resume_complete. The predelay and postdelay args allow you to do this. It 187d5a5e4ecSTodd Brandt can also be useful to run in x2 mode with an x2 delay, this way you can 188d5a5e4ecSTodd Brandt see process activity before and after resume, and in between two 189d5a5e4ecSTodd Brandt successive suspend/resumes. 190d5a5e4ecSTodd Brandt 191d5a5e4ecSTodd Brandt The command can be run like this: 192d5a5e4ecSTodd Brandt 193d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -m mem -rtcwake 15 -x2 -x2delay 1000 -predelay 1000 -postdelay 1000 -proc 194d5a5e4ecSTodd Brandt 195d5a5e4ecSTodd Brandt or with a config file 196d5a5e4ecSTodd Brandt 197d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -config config/suspend-proc.cfg 198d5a5e4ecSTodd Brandt 1992c9a583bSTodd Brandt------------------------------------------------------------------ 2002c9a583bSTodd Brandt| ENDURANCE TESTING | 2012c9a583bSTodd Brandt------------------------------------------------------------------ 2022c9a583bSTodd Brandt 2032c9a583bSTodd Brandt The best way to gauge the health of a system is to run a series of 2042c9a583bSTodd Brandt suspend/resumes over an extended period and analyze the behavior. This can be 2052c9a583bSTodd Brandt accomplished with sleepgraph's -multi argument. You specify two numbers: the 2062c9a583bSTodd Brandt number of tests to run OR the duration in days, hours, or minutes, and the 2072c9a583bSTodd Brandt delay in seconds between them. For instance, -multi 20 5: execute 20 tests with 2082c9a583bSTodd Brandt a 5 second delay between each, or -multi 24h 0: execute tests over a 24 hour 2092c9a583bSTodd Brandt period with no delay between tests. You can include any other options you like 2102c9a583bSTodd Brandt to generate the data you want. It's most useful to collect dev mode timelines 2112c9a583bSTodd Brandt as the kprobes don't alter the performance much and you get more insight. 2122c9a583bSTodd Brandt 2132c9a583bSTodd Brandt On completion, the output folder contains a series of folders for the 2142c9a583bSTodd Brandt individual test data and a set of summary pages in the root. The summary.html 2152c9a583bSTodd Brandt file is a tabular list of the tests with relevant info and links. The 2162c9a583bSTodd Brandt summary-issue.html and summary-devices.html files include data taken from 2172c9a583bSTodd Brandt all tests on kernel issues and device performance. The folder looks like this: 2182c9a583bSTodd Brandt 2192c9a583bSTodd Brandt suspend-xN-{date}-{time}: 2202c9a583bSTodd Brandt summary.html 2212c9a583bSTodd Brandt summary-issues.html 2222c9a583bSTodd Brandt summary-devices.html 2232c9a583bSTodd Brandt suspend-{date}-{time} (1) 2242c9a583bSTodd Brandt suspend-{date}-{time} (2) 2252c9a583bSTodd Brandt ... 2262c9a583bSTodd Brandt 2272c9a583bSTodd Brandt These are the relevant arguments to use for testing: 2282c9a583bSTodd Brandt 2292c9a583bSTodd Brandt -m mode 2302c9a583bSTodd Brandt Mode to initiate for suspend e.g. mem, freeze, standby (default: mem). 2312c9a583bSTodd Brandt 2322c9a583bSTodd Brandt -rtcwake t 2332c9a583bSTodd Brandt Use rtcwake to autoresume after t seconds (default: 15). 2342c9a583bSTodd Brandt 2352c9a583bSTodd Brandt -gzip (optional) 2362c9a583bSTodd Brandt Gzip the trace and dmesg logs to save space. The tool can also read in 2372c9a583bSTodd Brandt gzipped logs for processing. This reduces the multitest folder size. 2382c9a583bSTodd Brandt 2392c9a583bSTodd Brandt -dev (optional) 2402c9a583bSTodd Brandt Add kernel source calls and threads to the timeline (default: disabled). 2412c9a583bSTodd Brandt 2422c9a583bSTodd Brandt -multi n d 2432c9a583bSTodd Brandt Execute n consecutive tests at d seconds intervals. The outputs will be 2442c9a583bSTodd Brandt created in a new subdirectory: suspend-xN-{date}-{time}. When the multitest 2452c9a583bSTodd Brandt run is done, the -summary command is called automatically to create summary 2462c9a583bSTodd Brandt html files for all the data (unless you use -skiphtml). -skiphtml will 2472c9a583bSTodd Brandt speed up the testing by not creating timelines or summary html files. You 2482c9a583bSTodd Brandt can then run the tool again at a later time with -summary and -genhtml to 2492c9a583bSTodd Brandt create the timelines. 2502c9a583bSTodd Brandt 2512c9a583bSTodd Brandt -skiphtml (optional) 2522c9a583bSTodd Brandt Run the test and capture the trace logs, but skip the timeline and summary 2532c9a583bSTodd Brandt html generation. This can greatly speed up overall testing. You can then 2542c9a583bSTodd Brandt copy the data to a faster host machine and run -summary -genhtml to 2552c9a583bSTodd Brandt generate the timelines and summary. 2562c9a583bSTodd Brandt 2572c9a583bSTodd Brandt These are the relevant commands to use after testing is complete: 2582c9a583bSTodd Brandt 2592c9a583bSTodd Brandt -summary indir 2602c9a583bSTodd Brandt Generate or regenerate the summary for a -multi test run. Creates three 2612c9a583bSTodd Brandt files: summary.html, summary-issues.html, and summary-devices.html in the 2622c9a583bSTodd Brandt current folder. summary.html is a table of tests with relevant info sorted 2632c9a583bSTodd Brandt by kernel/host/mode, and links to the test html files. summary-issues.html 2642c9a583bSTodd Brandt is a list of kernel issues found in dmesg from all the tests. 2652c9a583bSTodd Brandt summary-devices.html is a list of devices and times from all the tests. 2662c9a583bSTodd Brandt 2672c9a583bSTodd Brandt -genhtml 2682c9a583bSTodd Brandt Used with -summary to regenerate any missing html timelines from their 2692c9a583bSTodd Brandt dmesg and ftrace logs. This will require a significant amount of time if 2702c9a583bSTodd Brandt there are thousands of tests. 2712c9a583bSTodd Brandt 2722c9a583bSTodd BrandtUsage Examples 2732c9a583bSTodd Brandt_______________ 2742c9a583bSTodd Brandt 2752c9a583bSTodd Brandt A multitest is initiated like this: 2762c9a583bSTodd Brandt 2772c9a583bSTodd Brandt %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0 2782c9a583bSTodd Brandt 2792c9a583bSTodd Brandt or you can skip timeline generation in order to speed things up 2802c9a583bSTodd Brandt 2812c9a583bSTodd Brandt %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0 -skiphtml 2822c9a583bSTodd Brandt 2832c9a583bSTodd Brandt The tool will produce an output folder with all the test subfolders inside. 2842c9a583bSTodd Brandt Each test subfolder contains the dmesg/ftrace logs and/or the html timeline 2852c9a583bSTodd Brandt depending on whether you used the -skiphtml option. The root folder contains 2862c9a583bSTodd Brandt the summary.html files. 2872c9a583bSTodd Brandt 2882c9a583bSTodd Brandt The summary for an existing multitest is generated like this: 2892c9a583bSTodd Brandt 2902c9a583bSTodd Brandt %> cd suspend-x2000-{date}-{time} 2912c9a583bSTodd Brandt %> sleepgraph.py -summary . 2922c9a583bSTodd Brandt 2932c9a583bSTodd Brandt or if you need to generate the html timelines you can use -genhtml 2942c9a583bSTodd Brandt 2952c9a583bSTodd Brandt %> cd suspend-xN-{date}-{time} 2962c9a583bSTodd Brandt %> sleepgraph.py -summary . -genhtml 297d5a5e4ecSTodd Brandt 298d5a5e4ecSTodd Brandt------------------------------------------------------------------ 299d5a5e4ecSTodd Brandt| CONFIGURATION FILES | 300d5a5e4ecSTodd Brandt------------------------------------------------------------------ 301d5a5e4ecSTodd Brandt 302d5a5e4ecSTodd Brandt Since 4.0 we've moved to using config files in lieu of command line options. 303d5a5e4ecSTodd Brandt The config folder contains a collection of typical use cases. 304d5a5e4ecSTodd Brandt There are corresponding configs for other power modes: 305d5a5e4ecSTodd Brandt 306d5a5e4ecSTodd Brandt Simple suspend/resume with basic timeline (mem/freeze/standby) 307d5a5e4ecSTodd Brandt config/suspend.cfg 308d5a5e4ecSTodd Brandt config/freeze.cfg 309d5a5e4ecSTodd Brandt config/standby.cfg 310d5a5e4ecSTodd Brandt 311d5a5e4ecSTodd Brandt Dev mode suspend/resume with dev timeline (mem/freeze/standby) 312d5a5e4ecSTodd Brandt config/suspend-dev.cfg 313d5a5e4ecSTodd Brandt config/freeze-dev.cfg 314d5a5e4ecSTodd Brandt config/standby-dev.cfg 315d5a5e4ecSTodd Brandt 316d5a5e4ecSTodd Brandt Simple suspend/resume with timeline and callgraph (mem/freeze/standby) 317d5a5e4ecSTodd Brandt config/suspend-callgraph.cfg 318d5a5e4ecSTodd Brandt config/freeze-callgraph.cfg 319d5a5e4ecSTodd Brandt config/standby-callgraph.cfg 320d5a5e4ecSTodd Brandt 321d5a5e4ecSTodd Brandt Sample proc mode x2 run using mem suspend 322d5a5e4ecSTodd Brandt config/suspend-x2-proc.cfg 323d5a5e4ecSTodd Brandt 324d5a5e4ecSTodd Brandt Sample for editing timeline funcs (moves internal functions into config) 325d5a5e4ecSTodd Brandt config/custom-timeline-functions.cfg 326d5a5e4ecSTodd Brandt 327d5a5e4ecSTodd Brandt Sample debug config for serio subsystem 328d5a5e4ecSTodd Brandt config/debug-serio-suspend.cfg 329d5a5e4ecSTodd Brandt 330d5a5e4ecSTodd Brandt 331d5a5e4ecSTodd BrandtUsage Examples 332d5a5e4ecSTodd Brandt______________ 333d5a5e4ecSTodd Brandt 334d5a5e4ecSTodd Brandt Run a simple mem suspend: 335d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -config config/suspend.cfg 336d5a5e4ecSTodd Brandt 337d5a5e4ecSTodd Brandt Run a mem suspend with callgraph data: 338d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -config config/suspend-callgraph.cfg 339d5a5e4ecSTodd Brandt 340d5a5e4ecSTodd Brandt Run a mem suspend with dev mode detail: 341d5a5e4ecSTodd Brandt %> sudo ./sleepgraph.py -config config/suspend-dev.cfg 342d5a5e4ecSTodd Brandt 343d5a5e4ecSTodd Brandt 344d5a5e4ecSTodd BrandtConfig File Options 345d5a5e4ecSTodd Brandt___________________ 346d5a5e4ecSTodd Brandt 347d5a5e4ecSTodd Brandt [Settings] 348d5a5e4ecSTodd Brandt 349d5a5e4ecSTodd Brandt # Verbosity: print verbose messages (def: false) 350d5a5e4ecSTodd Brandt verbose: false 351d5a5e4ecSTodd Brandt 352d5a5e4ecSTodd Brandt # Suspend Mode: e.g. standby, mem, freeze, disk (def: mem) 353d5a5e4ecSTodd Brandt mode: mem 354d5a5e4ecSTodd Brandt 355d5a5e4ecSTodd Brandt # Output Directory Format: {hostname}, {date}, {time} give current values 356d5a5e4ecSTodd Brandt output-dir: suspend-{hostname}-{date}-{time} 357d5a5e4ecSTodd Brandt 358d5a5e4ecSTodd Brandt # Automatic Wakeup: use rtcwake to wakeup after X seconds (def: infinity) 359d5a5e4ecSTodd Brandt rtcwake: 15 360d5a5e4ecSTodd Brandt 361d5a5e4ecSTodd Brandt # Add Logs: add the dmesg and ftrace log to the html output (def: false) 362d5a5e4ecSTodd Brandt addlogs: false 363d5a5e4ecSTodd Brandt 364d5a5e4ecSTodd Brandt # Sus/Res Gap: insert a gap between sus & res in the timeline (def: false) 365d5a5e4ecSTodd Brandt srgap: false 366d5a5e4ecSTodd Brandt 367d5a5e4ecSTodd Brandt # Custom Command: Command to execute in lieu of suspend (def: "") 368d5a5e4ecSTodd Brandt command: echo mem > /sys/power/state 369d5a5e4ecSTodd Brandt 370d5a5e4ecSTodd Brandt # Proc mode: graph user processes and cpu usage in the timeline (def: false) 371d5a5e4ecSTodd Brandt proc: false 372d5a5e4ecSTodd Brandt 373d5a5e4ecSTodd Brandt # Dev mode: graph source functions in the timeline (def: false) 374d5a5e4ecSTodd Brandt dev: false 375d5a5e4ecSTodd Brandt 376d5a5e4ecSTodd Brandt # Suspend/Resume x2: run 2 suspend/resumes back to back (def: false) 377d5a5e4ecSTodd Brandt x2: false 378d5a5e4ecSTodd Brandt 379d5a5e4ecSTodd Brandt # x2 Suspend Delay: time delay between the two test runs in ms (def: 0 ms) 380d5a5e4ecSTodd Brandt x2delay: 0 381d5a5e4ecSTodd Brandt 382d5a5e4ecSTodd Brandt # Pre Suspend Delay: nclude an N ms delay before (1st) suspend (def: 0 ms) 383d5a5e4ecSTodd Brandt predelay: 0 384d5a5e4ecSTodd Brandt 385d5a5e4ecSTodd Brandt # Post Resume Delay: include an N ms delay after (last) resume (def: 0 ms) 386d5a5e4ecSTodd Brandt postdelay: 0 387d5a5e4ecSTodd Brandt 388d5a5e4ecSTodd Brandt # Min Device Length: graph only dev callbacks longer than min (def: 0.001 ms) 389d5a5e4ecSTodd Brandt mindev: 0.001 390d5a5e4ecSTodd Brandt 391d5a5e4ecSTodd Brandt # Callgraph: gather ftrace callgraph data on all timeline events (def: false) 392d5a5e4ecSTodd Brandt callgraph: false 393d5a5e4ecSTodd Brandt 394d5a5e4ecSTodd Brandt # Expand Callgraph: pre-expand the callgraph treeviews in html (def: false) 395d5a5e4ecSTodd Brandt expandcg: false 396d5a5e4ecSTodd Brandt 397d5a5e4ecSTodd Brandt # Min Callgraph Length: show callgraphs only if longer than min (def: 1 ms) 398d5a5e4ecSTodd Brandt mincg: 1 399d5a5e4ecSTodd Brandt 400d5a5e4ecSTodd Brandt # Timestamp Precision: number of sig digits in timestamps (0:S, [3:ms], 6:us) 401d5a5e4ecSTodd Brandt timeprec: 3 402d5a5e4ecSTodd Brandt 403d5a5e4ecSTodd Brandt # Device Filter: show only devs whose name/driver includes one of these strings 404d5a5e4ecSTodd Brandt devicefilter: _cpu_up,_cpu_down,i915,usb 405d5a5e4ecSTodd Brandt 406d5a5e4ecSTodd Brandt # Override default timeline entries: 407d5a5e4ecSTodd Brandt # Do not use the internal default functions for timeline entries (def: false) 408d5a5e4ecSTodd Brandt # Set this to true if you intend to only use the ones defined in the config 409d5a5e4ecSTodd Brandt override-timeline-functions: true 410d5a5e4ecSTodd Brandt 411d5a5e4ecSTodd Brandt # Override default dev timeline entries: 412d5a5e4ecSTodd Brandt # Do not use the internal default functions for dev timeline entries (def: false) 413d5a5e4ecSTodd Brandt # Set this to true if you intend to only use the ones defined in the config 414d5a5e4ecSTodd Brandt override-dev-timeline-functions: true 415d5a5e4ecSTodd Brandt 416d5a5e4ecSTodd Brandt # Call Loop Max Gap (dev mode only) 417d5a5e4ecSTodd Brandt # merge loops of the same call if each is less than maxgap apart (def: 100us) 418d5a5e4ecSTodd Brandt callloop-maxgap: 0.0001 419d5a5e4ecSTodd Brandt 420d5a5e4ecSTodd Brandt # Call Loop Max Length (dev mode only) 421d5a5e4ecSTodd Brandt # merge loops of the same call if each is less than maxlen in length (def: 5ms) 422d5a5e4ecSTodd Brandt callloop-maxlen: 0.005 423d5a5e4ecSTodd Brandt 424d5a5e4ecSTodd Brandt------------------------------------------------------------------ 425d5a5e4ecSTodd Brandt| CUSTOM TIMELINE ENTRIES | 426d5a5e4ecSTodd Brandt------------------------------------------------------------------ 427d5a5e4ecSTodd Brandt 428d5a5e4ecSTodd BrandtAdding or Editing Timeline Functions 429d5a5e4ecSTodd Brandt____________________________________ 430d5a5e4ecSTodd Brandt 431d5a5e4ecSTodd Brandt The tool uses an array of function names to fill out empty spaces in the 432d5a5e4ecSTodd Brandt timeline where device callbacks don't appear. For instance, in suspend_prepare 433d5a5e4ecSTodd Brandt the tool adds the sys_sync and freeze_processes calls as virtual device blocks 434d5a5e4ecSTodd Brandt in the timeline to show you where the time is going. These calls should fill 435d5a5e4ecSTodd Brandt the timeline with contiguous data so that most kernel execution is covered. 436d5a5e4ecSTodd Brandt 437d5a5e4ecSTodd Brandt It is possible to add new function calls to the timeline by adding them to 438d5a5e4ecSTodd Brandt the config. It's also possible to copy the internal timeline functions into 439d5a5e4ecSTodd Brandt the config so that you can override and edit them. Place them in the 440d5a5e4ecSTodd Brandt timeline_functions_ARCH section with the name of your architecture appended. 441d5a5e4ecSTodd Brandt i.e. for x86_64: [timeline_functions_x86_64] 442d5a5e4ecSTodd Brandt 443d5a5e4ecSTodd Brandt Use the override-timeline-functions option if you only want to use your 444d5a5e4ecSTodd Brandt custom calls, or leave it false to append them to the internal ones. 445d5a5e4ecSTodd Brandt 446d5a5e4ecSTodd Brandt This section includes a list of functions (set using kprobes) which use both 447d5a5e4ecSTodd Brandt symbol data and function arg data. The args are pulled directly from the 448d5a5e4ecSTodd Brandt stack using this architecture's registers and stack formatting. Each entry 449d5a5e4ecSTodd Brandt can include up to four pieces of info: The function name, a format string, 450d5a5e4ecSTodd Brandt an argument list, and a color. But only a function name is required. 451d5a5e4ecSTodd Brandt 452d5a5e4ecSTodd Brandt For a full example config, see config/custom-timeline-functions.cfg. It pulls 453d5a5e4ecSTodd Brandt all the internal timeline functions into the config and allows you to edit 454d5a5e4ecSTodd Brandt them. 455d5a5e4ecSTodd Brandt 456d5a5e4ecSTodd Brandt Entry format: 457d5a5e4ecSTodd Brandt 458d5a5e4ecSTodd Brandt function: format{fn_arg1}_{fn_arg2} fn_arg1 fn_arg2 ... [color=purple] 459d5a5e4ecSTodd Brandt 460d5a5e4ecSTodd Brandt Required Arguments: 461d5a5e4ecSTodd Brandt 462d5a5e4ecSTodd Brandt function: The symbol name for the function you want probed, this is the 463d5a5e4ecSTodd Brandt minimum required for an entry, it will show up as the function 464d5a5e4ecSTodd Brandt name with no arguments. 465d5a5e4ecSTodd Brandt 466d5a5e4ecSTodd Brandt example: _cpu_up: 467d5a5e4ecSTodd Brandt 468d5a5e4ecSTodd Brandt Optional Arguments: 469d5a5e4ecSTodd Brandt 470d5a5e4ecSTodd Brandt format: The format to display the data on the timeline in. Use braces to 471d5a5e4ecSTodd Brandt enclose the arg names. 472d5a5e4ecSTodd Brandt 473d5a5e4ecSTodd Brandt example: CPU_ON[{cpu}] 474d5a5e4ecSTodd Brandt 475d5a5e4ecSTodd Brandt color: The color of the entry block in the timeline. The default color is 476d5a5e4ecSTodd Brandt transparent, so the entry shares the phase color. The color is an 477d5a5e4ecSTodd Brandt html color string, either a word, or an RGB. 478d5a5e4ecSTodd Brandt 479d5a5e4ecSTodd Brandt example: [color=#CC00CC] 480d5a5e4ecSTodd Brandt 481d5a5e4ecSTodd Brandt arglist: A list of arguments from registers/stack addresses. See URL: 482d5a5e4ecSTodd Brandt https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt 483d5a5e4ecSTodd Brandt 484d5a5e4ecSTodd Brandt example: cpu=%di:s32 485d5a5e4ecSTodd Brandt 486d5a5e4ecSTodd Brandt Here is a full example entry. It displays cpu resume calls in the timeline 487d5a5e4ecSTodd Brandt in orange. They will appear as CPU_ON[0], CPU_ON[1], etc. 488d5a5e4ecSTodd Brandt 489d5a5e4ecSTodd Brandt [timeline_functions_x86_64] 490d5a5e4ecSTodd Brandt _cpu_up: CPU_ON[{cpu}] cpu=%di:s32 [color=orange] 491d5a5e4ecSTodd Brandt 492d5a5e4ecSTodd Brandt 493d5a5e4ecSTodd BrandtAdding or Editing Dev Mode Timeline Source Functions 494d5a5e4ecSTodd Brandt____________________________________________________ 495d5a5e4ecSTodd Brandt 496d5a5e4ecSTodd Brandt In dev mode, the tool uses an array of function names to monitor source 497d5a5e4ecSTodd Brandt execution within the timeline entries. 498d5a5e4ecSTodd Brandt 499d5a5e4ecSTodd Brandt The function calls are displayed inside the main device/call blocks in the 500d5a5e4ecSTodd Brandt timeline. However, if a function call is not within a main timeline event, 501d5a5e4ecSTodd Brandt it will spawn an entirely new event named after the caller's kernel thread. 502d5a5e4ecSTodd Brandt These asynchronous kernel threads will populate in a separate section 503d5a5e4ecSTodd Brandt beneath the main device/call section. 504d5a5e4ecSTodd Brandt 505d5a5e4ecSTodd Brandt The tool has a set of hard coded calls which focus on the most common use 506d5a5e4ecSTodd Brandt cases: msleep, udelay, schedule_timeout, mutex_lock_slowpath, etc. These are 507d5a5e4ecSTodd Brandt the functions that add a hardcoded time delay to the suspend/resume path. 508d5a5e4ecSTodd Brandt The tool also includes some common functions native to important 509d5a5e4ecSTodd Brandt subsystems: ata, i915, and ACPI, etc. 510d5a5e4ecSTodd Brandt 511d5a5e4ecSTodd Brandt It is possible to add new function calls to the dev timeline by adding them 512d5a5e4ecSTodd Brandt to the config. It's also possible to copy the internal dev timeline 513d5a5e4ecSTodd Brandt functions into the config so that you can override and edit them. Place them 514d5a5e4ecSTodd Brandt in the dev_timeline_functions_ARCH section with the name of your architecture 515d5a5e4ecSTodd Brandt appended. i.e. for x86_64: [dev_timeline_functions_x86_64] 516d5a5e4ecSTodd Brandt 517d5a5e4ecSTodd Brandt Use the override-dev-timeline-functions option if you only want to use your 518d5a5e4ecSTodd Brandt custom calls, or leave it false to append them to the internal ones. 519d5a5e4ecSTodd Brandt 520d5a5e4ecSTodd Brandt The format is the same as the timeline_functions_x86_64 section. It's a 521d5a5e4ecSTodd Brandt list of functions (set using kprobes) which use both symbol data and function 522d5a5e4ecSTodd Brandt arg data. The args are pulled directly from the stack using this 523d5a5e4ecSTodd Brandt architecture's registers and stack formatting. Each entry can include up 524d5a5e4ecSTodd Brandt to four pieces of info: The function name, a format string, an argument list, 525d5a5e4ecSTodd Brandt and a color. But only the function name is required. 526d5a5e4ecSTodd Brandt 527d5a5e4ecSTodd Brandt For a full example config, see config/custom-timeline-functions.cfg. It pulls 528d5a5e4ecSTodd Brandt all the internal dev timeline functions into the config and allows you to edit 529d5a5e4ecSTodd Brandt them. 530d5a5e4ecSTodd Brandt 531d5a5e4ecSTodd Brandt Here is a full example entry. It displays the ATA port reset calls as 532d5a5e4ecSTodd Brandt ataN_port_reset in the timeline. This is where most of the SATA disk resume 533d5a5e4ecSTodd Brandt time goes, so it can be helpful to see the low level call. 534d5a5e4ecSTodd Brandt 535d5a5e4ecSTodd Brandt [dev_timeline_functions_x86_64] 536d5a5e4ecSTodd Brandt ata_eh_recover: ata{port}_port_reset port=+36(%di):s32 [color=#CC00CC] 537d5a5e4ecSTodd Brandt 538d5a5e4ecSTodd Brandt 539d5a5e4ecSTodd BrandtVerifying your custom functions 540d5a5e4ecSTodd Brandt_______________________________ 541d5a5e4ecSTodd Brandt 542d5a5e4ecSTodd Brandt Once you have a set of functions (kprobes) defined, it can be useful to 543d5a5e4ecSTodd Brandt perform a quick check to see if you formatted them correctly and if the system 544d5a5e4ecSTodd Brandt actually supports them. To do this, run the tool with your config file 545d5a5e4ecSTodd Brandt and the -status option. The tool will go through all the kprobes (both 546d5a5e4ecSTodd Brandt custom and internal if you haven't overridden them) and actually attempts 547d5a5e4ecSTodd Brandt to set them in ftrace. It will then print out success or fail for you. 548d5a5e4ecSTodd Brandt 549d5a5e4ecSTodd Brandt Note that kprobes which don't actually exist in the kernel won't stop the 550d5a5e4ecSTodd Brandt tool, they just wont show up. 551d5a5e4ecSTodd Brandt 552d5a5e4ecSTodd Brandt For example: 553d5a5e4ecSTodd Brandt 554d5a5e4ecSTodd Brandt sudo ./sleepgraph.py -config config/custom-timeline-functions.cfg -status 555d5a5e4ecSTodd Brandt Checking this system (myhostname)... 556d5a5e4ecSTodd Brandt have root access: YES 557d5a5e4ecSTodd Brandt is sysfs mounted: YES 558d5a5e4ecSTodd Brandt is "mem" a valid power mode: YES 559d5a5e4ecSTodd Brandt is ftrace supported: YES 560d5a5e4ecSTodd Brandt are kprobes supported: YES 561d5a5e4ecSTodd Brandt timeline data source: FTRACE (all trace events found) 562d5a5e4ecSTodd Brandt is rtcwake supported: YES 563d5a5e4ecSTodd Brandt verifying timeline kprobes work: 564d5a5e4ecSTodd Brandt _cpu_down: YES 565d5a5e4ecSTodd Brandt _cpu_up: YES 566d5a5e4ecSTodd Brandt acpi_pm_finish: YES 567d5a5e4ecSTodd Brandt acpi_pm_prepare: YES 568d5a5e4ecSTodd Brandt freeze_kernel_threads: YES 569d5a5e4ecSTodd Brandt freeze_processes: YES 570d5a5e4ecSTodd Brandt sys_sync: YES 571d5a5e4ecSTodd Brandt thaw_processes: YES 572d5a5e4ecSTodd Brandt verifying dev kprobes work: 573d5a5e4ecSTodd Brandt __const_udelay: YES 574d5a5e4ecSTodd Brandt __mutex_lock_slowpath: YES 575d5a5e4ecSTodd Brandt acpi_os_stall: YES 576d5a5e4ecSTodd Brandt acpi_ps_parse_aml: YES 577d5a5e4ecSTodd Brandt intel_opregion_init: NO 578d5a5e4ecSTodd Brandt intel_opregion_register: NO 579d5a5e4ecSTodd Brandt intel_opregion_setup: NO 580d5a5e4ecSTodd Brandt msleep: YES 581d5a5e4ecSTodd Brandt schedule_timeout: YES 582d5a5e4ecSTodd Brandt schedule_timeout_uninterruptible: YES 583d5a5e4ecSTodd Brandt usleep_range: YES 584d5a5e4ecSTodd Brandt 585d5a5e4ecSTodd Brandt 586d5a5e4ecSTodd Brandt------------------------------------------------------------------ 587d5a5e4ecSTodd Brandt| TESTING ON CONSUMER LINUX OPERATING SYSTEMS | 588d5a5e4ecSTodd Brandt------------------------------------------------------------------ 589d5a5e4ecSTodd Brandt 590d5a5e4ecSTodd BrandtAndroid 591d5a5e4ecSTodd Brandt_______ 592d5a5e4ecSTodd Brandt 593d5a5e4ecSTodd Brandt The easiest way to execute on an android device is to run the android.sh 594d5a5e4ecSTodd Brandt script on the device, then pull the ftrace log back to the host and run 595d5a5e4ecSTodd Brandt sleepgraph.py on it. 596d5a5e4ecSTodd Brandt 597d5a5e4ecSTodd Brandt Here are the steps: 598d5a5e4ecSTodd Brandt 599d5a5e4ecSTodd Brandt [download and install the tool on the device] 600d5a5e4ecSTodd Brandt 601d5a5e4ecSTodd Brandt host%> wget https://raw.githubusercontent.com/intel/pm-graph/master/tools/android.sh 602d5a5e4ecSTodd Brandt host%> adb connect 192.168.1.6 603d5a5e4ecSTodd Brandt host%> adb root 604d5a5e4ecSTodd Brandt # push the script to a writeable location 605d5a5e4ecSTodd Brandt host%> adb push android.sh /sdcard/ 606d5a5e4ecSTodd Brandt 607d5a5e4ecSTodd Brandt [check whether the tool will run on your device] 608d5a5e4ecSTodd Brandt 609d5a5e4ecSTodd Brandt host%> adb shell 610d5a5e4ecSTodd Brandt dev%> cd /sdcard 611d5a5e4ecSTodd Brandt dev%> sh android.sh status 612d5a5e4ecSTodd Brandt host : asus_t100 613d5a5e4ecSTodd Brandt kernel : 3.14.0-i386-dirty 614d5a5e4ecSTodd Brandt modes : freeze mem 615d5a5e4ecSTodd Brandt rtcwake : supported 616d5a5e4ecSTodd Brandt ftrace : supported 617d5a5e4ecSTodd Brandt trace events { 618d5a5e4ecSTodd Brandt suspend_resume: found 619d5a5e4ecSTodd Brandt device_pm_callback_end: found 620d5a5e4ecSTodd Brandt device_pm_callback_start: found 621d5a5e4ecSTodd Brandt } 622d5a5e4ecSTodd Brandt # the above is what you see on a system that's properly patched 623d5a5e4ecSTodd Brandt 624d5a5e4ecSTodd Brandt [execute the suspend] 625d5a5e4ecSTodd Brandt 626d5a5e4ecSTodd Brandt # NOTE: The suspend will only work if the screen isn't timed out, 627d5a5e4ecSTodd Brandt # so you have to press some keys first to wake it up b4 suspend) 628d5a5e4ecSTodd Brandt dev%> sh android.sh suspend mem 629d5a5e4ecSTodd Brandt ------------------------------------ 630d5a5e4ecSTodd Brandt Suspend/Resume timing test initiated 631d5a5e4ecSTodd Brandt ------------------------------------ 632d5a5e4ecSTodd Brandt hostname : asus_t100 633d5a5e4ecSTodd Brandt kernel : 3.14.0-i386-dirty 634d5a5e4ecSTodd Brandt mode : mem 635d5a5e4ecSTodd Brandt ftrace out : /mnt/shell/emulated/0/ftrace.txt 636d5a5e4ecSTodd Brandt dmesg out : /mnt/shell/emulated/0/dmesg.txt 637d5a5e4ecSTodd Brandt log file : /mnt/shell/emulated/0/log.txt 638d5a5e4ecSTodd Brandt ------------------------------------ 639d5a5e4ecSTodd Brandt INITIALIZING FTRACE........DONE 640d5a5e4ecSTodd Brandt STARTING FTRACE 641d5a5e4ecSTodd Brandt SUSPEND START @ 21:24:02 (rtcwake in 10 seconds) 642d5a5e4ecSTodd Brandt <adb connection will now terminate> 643d5a5e4ecSTodd Brandt 644d5a5e4ecSTodd Brandt [retrieve the data from the device] 645d5a5e4ecSTodd Brandt 646d5a5e4ecSTodd Brandt # I find that you have to actually kill the adb process and 647d5a5e4ecSTodd Brandt # reconnect sometimes in order for the connection to work post-suspend 648d5a5e4ecSTodd Brandt host%> adb connect 192.168.1.6 649d5a5e4ecSTodd Brandt # (required) get the ftrace data, this is the most important piece 650d5a5e4ecSTodd Brandt host%> adb pull /sdcard/ftrace.txt 651d5a5e4ecSTodd Brandt # (optional) get the dmesg data, this is for debugging 652d5a5e4ecSTodd Brandt host%> adb pull /sdcard/dmesg.txt 653d5a5e4ecSTodd Brandt # (optional) get the log, which just lists some test times for comparison 654d5a5e4ecSTodd Brandt host%> adb pull /sdcard/log.txt 655d5a5e4ecSTodd Brandt 656d5a5e4ecSTodd Brandt [create an output html file using sleepgraph.py] 657d5a5e4ecSTodd Brandt 658d5a5e4ecSTodd Brandt host%> sleepgraph.py -ftrace ftrace.txt 659d5a5e4ecSTodd Brandt 660d5a5e4ecSTodd Brandt You should now have an output.html with the android data, enjoy! 661