1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) 2 3====================================================== 4Discovering Linux kernel subsystems used by a workload 5====================================================== 6 7:Authors: - Shuah Khan <skhan@linuxfoundation.org> 8 - Shefali Sharma <sshefali021@gmail.com> 9:maintained-by: Shuah Khan <skhan@linuxfoundation.org> 10 11Key Points 12========== 13 14 * Understanding system resources necessary to build and run a workload 15 is important. 16 * Linux tracing and strace can be used to discover the system resources 17 in use by a workload. The completeness of the system usage information 18 depends on the completeness of coverage of a workload. 19 * Performance and security of the operating system can be analyzed with 20 the help of tools such as: 21 `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_, 22 `stress-ng <https://www.mankier.com/1/stress-ng>`_, 23 `paxtest <https://github.com/opntr/paxtest-freebsd>`_. 24 * Once we discover and understand the workload needs, we can focus on them 25 to avoid regressions and use it to evaluate safety considerations. 26 27Methodology 28=========== 29 30`strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_ is a 31diagnostic, instructional, and debugging tool and can be used to discover 32the system resources in use by a workload. Once we discover and understand 33the workload needs, we can focus on them to avoid regressions and use it 34to evaluate safety considerations. We use strace tool to trace workloads. 35 36This method of tracing using strace tells us the system calls invoked by 37the workload and doesn't include all the system calls that can be invoked 38by it. In addition, this tracing method tells us just the code paths within 39these system calls that are invoked. As an example, if a workload opens a 40file and reads from it successfully, then the success path is the one that 41is traced. Any error paths in that system call will not be traced. If there 42is a workload that provides full coverage of a workload then the method 43outlined here will trace and find all possible code paths. The completeness 44of the system usage information depends on the completeness of coverage of a 45workload. 46 47The goal is tracing a workload on a system running a default kernel without 48requiring custom kernel installs. 49 50How do we gather fine-grained system information? 51================================================= 52 53strace tool can be used to trace system calls made by a process and signals 54it receives. System calls are the fundamental interface between an 55application and the operating system kernel. They enable a program to 56request services from the kernel. For instance, the open() system call in 57Linux is used to provide access to a file in the file system. strace enables 58us to track all the system calls made by an application. It lists all the 59system calls made by a process and their resulting output. 60 61You can generate profiling data combining strace and perf record tools to 62record the events and information associated with a process. This provides 63insight into the process. "perf annotate" tool generates the statistics of 64each instruction of the program. This document goes over the details of how 65to gather fine-grained information on a workload's usage of system resources. 66 67We used strace to trace the perf, stress-ng, paxtest workloads to illustrate 68our methodology to discover resources used by a workload. This process can 69be applied to trace other workloads. 70 71Getting the system ready for tracing 72==================================== 73 74Before we can get started we will show you how to get your system ready. 75We assume that you have a Linux distribution running on a physical system 76or a virtual machine. Most distributions will include strace command. Let’s 77install other tools that aren’t usually included to build Linux kernel. 78Please note that the following works on Debian based distributions. You 79might have to find equivalent packages on other Linux distributions. 80 81Install tools to build Linux kernel and tools in kernel repository. 82scripts/ver_linux is a good way to check if your system already has 83the necessary tools:: 84 85 sudo apt-get install build-essential flex bison yacc 86 sudo apt install libelf-dev systemtap-sdt-dev libslang2-dev libperl-dev libdw-dev 87 88cscope is a good tool to browse kernel sources. Let's install it now:: 89 90 sudo apt-get install cscope 91 92Install stress-ng and paxtest:: 93 94 apt-get install stress-ng 95 apt-get install paxtest 96 97Workload overview 98================= 99 100As mentioned earlier, we used strace to trace perf bench, stress-ng and 101paxtest workloads to show how to analyze a workload and identify Linux 102subsystems used by these workloads. Let's start with an overview of these 103three workloads to get a better understanding of what they do and how to 104use them. 105 106perf bench (all) workload 107------------------------- 108 109The perf bench command contains multiple multi-threaded microkernel 110benchmarks for executing different subsystems in the Linux kernel and 111system calls. This allows us to easily measure the impact of changes, 112which can help mitigate performance regressions. It also acts as a common 113benchmarking framework, enabling developers to easily create test cases, 114integrate transparently, and use performance-rich tooling subsystems. 115 116Stress-ng netdev stressor workload 117---------------------------------- 118 119stress-ng is used for performing stress testing on the kernel. It allows 120you to exercise various physical subsystems of the computer, as well as 121interfaces of the OS kernel, using "stressor-s". They are available for 122CPU, CPU cache, devices, I/O, interrupts, file system, memory, network, 123operating system, pipelines, schedulers, and virtual machines. Please refer 124to the `stress-ng man-page <https://www.mankier.com/1/stress-ng>`_ to 125find the description of all the available stressor-s. The netdev stressor 126starts specified number (N) of workers that exercise various netdevice 127ioctl commands across all the available network devices. 128 129paxtest kiddie workload 130----------------------- 131 132paxtest is a program that tests buffer overflows in the kernel. It tests 133kernel enforcements over memory usage. Generally, execution in some memory 134segments makes buffer overflows possible. It runs a set of programs that 135attempt to subvert memory usage. It is used as a regression test suite for 136PaX, but might be useful to test other memory protection patches for the 137kernel. We used paxtest kiddie mode which looks for simple vulnerabilities. 138 139What is strace and how do we use it? 140==================================== 141 142As mentioned earlier, strace which is a useful diagnostic, instructional, 143and debugging tool and can be used to discover the system resources in use 144by a workload. It can be used: 145 146 * To see how a process interacts with the kernel. 147 * To see why a process is failing or hanging. 148 * For reverse engineering a process. 149 * To find the files on which a program depends. 150 * For analyzing the performance of an application. 151 * For troubleshooting various problems related to the operating system. 152 153In addition, strace can generate run-time statistics on times, calls, and 154errors for each system call and report a summary when program exits, 155suppressing the regular output. This attempts to show system time (CPU time 156spent running in the kernel) independent of wall clock time. We plan to use 157these features to get information on workload system usage. 158 159strace command supports basic, verbose, and stats modes. strace command when 160run in verbose mode gives more detailed information about the system calls 161invoked by a process. 162 163Running strace -c generates a report of the percentage of time spent in each 164system call, the total time in seconds, the microseconds per call, the total 165number of calls, the count of each system call that has failed with an error 166and the type of system call made. 167 168 * Usage: strace <command we want to trace> 169 * Verbose mode usage: strace -v <command> 170 * Gather statistics: strace -c <command> 171 172We used the “-c” option to gather fine-grained run-time statistics in use 173by three workloads we have chose for this analysis. 174 175 * perf 176 * stress-ng 177 * paxtest 178 179What is cscope and how do we use it? 180==================================== 181 182Now let’s look at `cscope <https://cscope.sourceforge.net/>`_, a command 183line tool for browsing C, C++ or Java code-bases. We can use it to find 184all the references to a symbol, global definitions, functions called by a 185function, functions calling a function, text strings, regular expression 186patterns, files including a file. 187 188We can use cscope to find which system call belongs to which subsystem. 189This way we can find the kernel subsystems used by a process when it is 190executed. 191 192Let’s checkout the latest Linux repository and build cscope database:: 193 194 git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux 195 cd linux 196 cscope -R -p10 # builds cscope.out database before starting browse session 197 cscope -d -p10 # starts browse session on cscope.out database 198 199Note: Run "cscope -R -p10" to build the database and "cscope -d -p10" to 200enter into the browsing session. cscope by default uses the cscope.out 201database. To get out of this mode press ctrl+d. -p option is used to 202specify the number of file path components to display. -p10 is optimal 203for browsing kernel sources. 204 205Alternatively, the kernel build system can generate the cscope database:: 206 207 make cscope 208 209To exclude directories from the generated database, pass IGNORE_DIRS to 210the cscope target. For example, to exclude Documentation/, run:: 211 212 make IGNORE_DIRS="Documentation" cscope 213 214What is perf and how do we use it? 215================================== 216 217Perf is an analysis tool based on Linux 2.6+ systems, which abstracts the 218CPU hardware difference in performance measurement in Linux, and provides 219a simple command line interface. Perf is based on the perf_events interface 220exported by the kernel. It is very useful for profiling the system and 221finding performance bottlenecks in an application. 222 223If you haven't already checked out the Linux mainline repository, you can do 224so and then build kernel and perf tool:: 225 226 git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux 227 cd linux 228 make -j3 all 229 cd tools/perf 230 make 231 232Note: The perf command can be built without building the kernel in the 233repository and can be run on older kernels. However matching the kernel 234and perf revisions gives more accurate information on the subsystem usage. 235 236We used "perf stat" and "perf bench" options. For a detailed information on 237the perf tool, run "perf -h". 238 239perf stat 240--------- 241The perf stat command generates a report of various hardware and software 242events. It does so with the help of hardware counter registers found in 243modern CPUs that keep the count of these activities. "perf stat cal" shows 244stats for cal command. 245 246Perf bench 247---------- 248The perf bench command contains multiple multi-threaded microkernel 249benchmarks for executing different subsystems in the Linux kernel and 250system calls. This allows us to easily measure the impact of changes, 251which can help mitigate performance regressions. It also acts as a common 252benchmarking framework, enabling developers to easily create test cases, 253integrate transparently, and use performance-rich tooling. 254 255"perf bench all" runs all available benchmarks in the perf bench 256framework. The exact set of benchmarks depends on the perf version and on 257the features enabled when perf was built. 258 259To list the benchmark collections available on the current system, run:: 260 261 perf bench 262 263To list benchmarks in a collection, run:: 264 265 perf bench <collection> 266 267For example, to list the benchmarks in the mem collection, run:: 268 269 perf bench mem 270 271What is stress-ng and how do we use it? 272======================================= 273 274As mentioned earlier, stress-ng is used for performing stress testing on 275the kernel. It allows you to exercise various physical subsystems of the 276computer, as well as interfaces of the OS kernel, using stressor-s. They 277are available for CPU, CPU cache, devices, I/O, interrupts, file system, 278memory, network, operating system, pipelines, schedulers, and virtual 279machines. 280 281The netdev stressor starts N workers that exercise various netdevice ioctl 282commands across all the available network devices. The following ioctls are 283exercised: 284 285 * SIOCGIFCONF, SIOCGIFINDEX, SIOCGIFNAME, SIOCGIFFLAGS 286 * SIOCGIFADDR, SIOCGIFNETMASK, SIOCGIFMETRIC, SIOCGIFMTU 287 * SIOCGIFHWADDR, SIOCGIFMAP, SIOCGIFTXQLEN 288 289The following command runs the stressor:: 290 291 stress-ng --netdev 1 -t 60 --metrics 292 293We can use the perf record command to record the events and information 294associated with a process. This command records the profiling data in the 295perf.data file in the same directory. 296 297Using the following commands you can record the events associated with the 298netdev stressor, view the generated report perf.data and annotate the output 299to view the statistics of each instruction of the program:: 300 301 perf record -- stress-ng --netdev 1 -t 60 --metrics 302 perf report 303 perf annotate 304 305What is paxtest and how do we use it? 306===================================== 307 308paxtest is a program that tests buffer overflows in the kernel. It tests 309kernel enforcements over memory usage. Generally, execution in some memory 310segments makes buffer overflows possible. It runs a set of programs that 311attempt to subvert memory usage. It is used as a regression test suite for 312PaX, and will be useful to test other memory protection patches for the 313kernel. 314 315paxtest provides kiddie and blackhat modes. The paxtest kiddie mode runs 316in normal mode, whereas the blackhat mode tries to get around the protection 317of the kernel testing for vulnerabilities. We focus on the kiddie mode here 318and combine "paxtest kiddie" run with "perf record" to collect CPU stack 319traces for the paxtest kiddie run to see which function is calling other 320functions in the performance profile. Then the "dwarf" (DWARF's Call Frame 321Information) mode can be used to unwind the stack. 322 323The following command can be used to view resulting report in call-graph 324format:: 325 326 perf record --call-graph dwarf paxtest kiddie 327 perf report --stdio 328 329Tracing workloads 330================= 331 332Now that we understand the workloads, let's start tracing them. 333 334Tracing perf bench all workload 335------------------------------- 336 337Run the following command to trace perf bench all workload:: 338 339 strace -c perf bench all 340 341**System Calls made by the workload** 342 343The below table shows the system calls invoked by the workload, number of 344times each system call is invoked, and the corresponding Linux subsystem. 345 346+-------------------+-----------+-----------------+-------------------------+ 347| System Call | # calls | Linux Subsystem | System Call (API) | 348+===================+===========+=================+=========================+ 349| getppid | 10000001 | Process Mgmt | sys_getpid() | 350+-------------------+-----------+-----------------+-------------------------+ 351| clone | 1077 | Process Mgmt. | sys_clone() | 352+-------------------+-----------+-----------------+-------------------------+ 353| prctl | 23 | Process Mgmt. | sys_prctl() | 354+-------------------+-----------+-----------------+-------------------------+ 355| prlimit64 | 7 | Process Mgmt. | sys_prlimit64() | 356+-------------------+-----------+-----------------+-------------------------+ 357| getpid | 10 | Process Mgmt. | sys_getpid() | 358+-------------------+-----------+-----------------+-------------------------+ 359| uname | 3 | Process Mgmt. | sys_uname() | 360+-------------------+-----------+-----------------+-------------------------+ 361| sysinfo | 1 | Process Mgmt. | sys_sysinfo() | 362+-------------------+-----------+-----------------+-------------------------+ 363| getuid | 1 | Process Mgmt. | sys_getuid() | 364+-------------------+-----------+-----------------+-------------------------+ 365| getgid | 1 | Process Mgmt. | sys_getgid() | 366+-------------------+-----------+-----------------+-------------------------+ 367| geteuid | 1 | Process Mgmt. | sys_geteuid() | 368+-------------------+-----------+-----------------+-------------------------+ 369| getegid | 1 | Process Mgmt. | sys_getegid() | 370+-------------------+-----------+-----------------+-------------------------+ 371| close | 49951 | Filesystem | sys_close() | 372+-------------------+-----------+-----------------+-------------------------+ 373| pipe | 604 | Filesystem | sys_pipe() | 374+-------------------+-----------+-----------------+-------------------------+ 375| openat | 48560 | Filesystem | sys_openat() | 376+-------------------+-----------+-----------------+-------------------------+ 377| fstat | 8338 | Filesystem | sys_fstat() | 378+-------------------+-----------+-----------------+-------------------------+ 379| stat | 1573 | Filesystem | sys_stat() | 380+-------------------+-----------+-----------------+-------------------------+ 381| pread64 | 9646 | Filesystem | sys_pread64() | 382+-------------------+-----------+-----------------+-------------------------+ 383| getdents64 | 1873 | Filesystem | sys_getdents64() | 384+-------------------+-----------+-----------------+-------------------------+ 385| access | 3 | Filesystem | sys_access() | 386+-------------------+-----------+-----------------+-------------------------+ 387| lstat | 1880 | Filesystem | sys_lstat() | 388+-------------------+-----------+-----------------+-------------------------+ 389| lseek | 6 | Filesystem | sys_lseek() | 390+-------------------+-----------+-----------------+-------------------------+ 391| ioctl | 3 | Filesystem | sys_ioctl() | 392+-------------------+-----------+-----------------+-------------------------+ 393| dup2 | 1 | Filesystem | sys_dup2() | 394+-------------------+-----------+-----------------+-------------------------+ 395| execve | 2 | Filesystem | sys_execve() | 396+-------------------+-----------+-----------------+-------------------------+ 397| fcntl | 8779 | Filesystem | sys_fcntl() | 398+-------------------+-----------+-----------------+-------------------------+ 399| statfs | 1 | Filesystem | sys_statfs() | 400+-------------------+-----------+-----------------+-------------------------+ 401| epoll_create | 2 | Filesystem | sys_epoll_create() | 402+-------------------+-----------+-----------------+-------------------------+ 403| epoll_ctl | 64 | Filesystem | sys_epoll_ctl() | 404+-------------------+-----------+-----------------+-------------------------+ 405| newfstatat | 8318 | Filesystem | sys_newfstatat() | 406+-------------------+-----------+-----------------+-------------------------+ 407| eventfd2 | 192 | Filesystem | sys_eventfd2() | 408+-------------------+-----------+-----------------+-------------------------+ 409| mmap | 243 | Memory Mgmt. | sys_mmap() | 410+-------------------+-----------+-----------------+-------------------------+ 411| mprotect | 32 | Memory Mgmt. | sys_mprotect() | 412+-------------------+-----------+-----------------+-------------------------+ 413| brk | 21 | Memory Mgmt. | sys_brk() | 414+-------------------+-----------+-----------------+-------------------------+ 415| munmap | 128 | Memory Mgmt. | sys_munmap() | 416+-------------------+-----------+-----------------+-------------------------+ 417| set_mempolicy | 156 | Memory Mgmt. | sys_set_mempolicy() | 418+-------------------+-----------+-----------------+-------------------------+ 419| set_tid_address | 1 | Process Mgmt. | sys_set_tid_address() | 420+-------------------+-----------+-----------------+-------------------------+ 421| set_robust_list | 1 | Futex | sys_set_robust_list() | 422+-------------------+-----------+-----------------+-------------------------+ 423| futex | 341 | Futex | sys_futex() | 424+-------------------+-----------+-----------------+-------------------------+ 425| sched_getaffinity | 79 | Scheduler | sys_sched_getaffinity() | 426+-------------------+-----------+-----------------+-------------------------+ 427| sched_setaffinity | 223 | Scheduler | sys_sched_setaffinity() | 428+-------------------+-----------+-----------------+-------------------------+ 429| socketpair | 202 | Network | sys_socketpair() | 430+-------------------+-----------+-----------------+-------------------------+ 431| rt_sigprocmask | 21 | Signal | sys_rt_sigprocmask() | 432+-------------------+-----------+-----------------+-------------------------+ 433| rt_sigaction | 36 | Signal | sys_rt_sigaction() | 434+-------------------+-----------+-----------------+-------------------------+ 435| rt_sigreturn | 2 | Signal | sys_rt_sigreturn() | 436+-------------------+-----------+-----------------+-------------------------+ 437| wait4 | 889 | Time | sys_wait4() | 438+-------------------+-----------+-----------------+-------------------------+ 439| clock_nanosleep | 37 | Time | sys_clock_nanosleep() | 440+-------------------+-----------+-----------------+-------------------------+ 441| capget | 4 | Capability | sys_capget() | 442+-------------------+-----------+-----------------+-------------------------+ 443 444Tracing stress-ng netdev stressor workload 445------------------------------------------ 446 447Run the following command to trace stress-ng netdev stressor workload:: 448 449 strace -c stress-ng --netdev 1 -t 60 --metrics 450 451**System Calls made by the workload** 452 453The below table shows the system calls invoked by the workload, number of 454times each system call is invoked, and the corresponding Linux subsystem. 455 456+-------------------+-----------+-----------------+-------------------------+ 457| System Call | # calls | Linux Subsystem | System Call (API) | 458+===================+===========+=================+=========================+ 459| openat | 74 | Filesystem | sys_openat() | 460+-------------------+-----------+-----------------+-------------------------+ 461| close | 75 | Filesystem | sys_close() | 462+-------------------+-----------+-----------------+-------------------------+ 463| read | 58 | Filesystem | sys_read() | 464+-------------------+-----------+-----------------+-------------------------+ 465| fstat | 20 | Filesystem | sys_fstat() | 466+-------------------+-----------+-----------------+-------------------------+ 467| flock | 10 | Filesystem | sys_flock() | 468+-------------------+-----------+-----------------+-------------------------+ 469| write | 7 | Filesystem | sys_write() | 470+-------------------+-----------+-----------------+-------------------------+ 471| getdents64 | 8 | Filesystem | sys_getdents64() | 472+-------------------+-----------+-----------------+-------------------------+ 473| pread64 | 8 | Filesystem | sys_pread64() | 474+-------------------+-----------+-----------------+-------------------------+ 475| lseek | 1 | Filesystem | sys_lseek() | 476+-------------------+-----------+-----------------+-------------------------+ 477| access | 2 | Filesystem | sys_access() | 478+-------------------+-----------+-----------------+-------------------------+ 479| getcwd | 1 | Filesystem | sys_getcwd() | 480+-------------------+-----------+-----------------+-------------------------+ 481| execve | 1 | Filesystem | sys_execve() | 482+-------------------+-----------+-----------------+-------------------------+ 483| mmap | 61 | Memory Mgmt. | sys_mmap() | 484+-------------------+-----------+-----------------+-------------------------+ 485| munmap | 3 | Memory Mgmt. | sys_munmap() | 486+-------------------+-----------+-----------------+-------------------------+ 487| mprotect | 20 | Memory Mgmt. | sys_mprotect() | 488+-------------------+-----------+-----------------+-------------------------+ 489| mlock | 2 | Memory Mgmt. | sys_mlock() | 490+-------------------+-----------+-----------------+-------------------------+ 491| brk | 3 | Memory Mgmt. | sys_brk() | 492+-------------------+-----------+-----------------+-------------------------+ 493| rt_sigaction | 21 | Signal | sys_rt_sigaction() | 494+-------------------+-----------+-----------------+-------------------------+ 495| rt_sigprocmask | 1 | Signal | sys_rt_sigprocmask() | 496+-------------------+-----------+-----------------+-------------------------+ 497| sigaltstack | 1 | Signal | sys_sigaltstack() | 498+-------------------+-----------+-----------------+-------------------------+ 499| rt_sigreturn | 1 | Signal | sys_rt_sigreturn() | 500+-------------------+-----------+-----------------+-------------------------+ 501| getpid | 8 | Process Mgmt. | sys_getpid() | 502+-------------------+-----------+-----------------+-------------------------+ 503| prlimit64 | 5 | Process Mgmt. | sys_prlimit64() | 504+-------------------+-----------+-----------------+-------------------------+ 505| arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() | 506+-------------------+-----------+-----------------+-------------------------+ 507| sysinfo | 2 | Process Mgmt. | sys_sysinfo() | 508+-------------------+-----------+-----------------+-------------------------+ 509| getuid | 2 | Process Mgmt. | sys_getuid() | 510+-------------------+-----------+-----------------+-------------------------+ 511| uname | 1 | Process Mgmt. | sys_uname() | 512+-------------------+-----------+-----------------+-------------------------+ 513| setpgid | 1 | Process Mgmt. | sys_setpgid() | 514+-------------------+-----------+-----------------+-------------------------+ 515| getrusage | 1 | Process Mgmt. | sys_getrusage() | 516+-------------------+-----------+-----------------+-------------------------+ 517| geteuid | 1 | Process Mgmt. | sys_geteuid() | 518+-------------------+-----------+-----------------+-------------------------+ 519| getppid | 1 | Process Mgmt. | sys_getppid() | 520+-------------------+-----------+-----------------+-------------------------+ 521| sendto | 3 | Network | sys_sendto() | 522+-------------------+-----------+-----------------+-------------------------+ 523| connect | 1 | Network | sys_connect() | 524+-------------------+-----------+-----------------+-------------------------+ 525| socket | 1 | Network | sys_socket() | 526+-------------------+-----------+-----------------+-------------------------+ 527| clone | 1 | Process Mgmt. | sys_clone() | 528+-------------------+-----------+-----------------+-------------------------+ 529| set_tid_address | 1 | Process Mgmt. | sys_set_tid_address() | 530+-------------------+-----------+-----------------+-------------------------+ 531| wait4 | 2 | Time | sys_wait4() | 532+-------------------+-----------+-----------------+-------------------------+ 533| alarm | 1 | Time | sys_alarm() | 534+-------------------+-----------+-----------------+-------------------------+ 535| set_robust_list | 1 | Futex | sys_set_robust_list() | 536+-------------------+-----------+-----------------+-------------------------+ 537 538Tracing paxtest kiddie workload 539------------------------------- 540 541Run the following command to trace paxtest kiddie workload:: 542 543 strace -c paxtest kiddie 544 545**System Calls made by the workload** 546 547The below table shows the system calls invoked by the workload, number of 548times each system call is invoked, and the corresponding Linux subsystem. 549 550+-------------------+-----------+-----------------+----------------------+ 551| System Call | # calls | Linux Subsystem | System Call (API) | 552+===================+===========+=================+======================+ 553| read | 3 | Filesystem | sys_read() | 554+-------------------+-----------+-----------------+----------------------+ 555| write | 11 | Filesystem | sys_write() | 556+-------------------+-----------+-----------------+----------------------+ 557| close | 41 | Filesystem | sys_close() | 558+-------------------+-----------+-----------------+----------------------+ 559| stat | 24 | Filesystem | sys_stat() | 560+-------------------+-----------+-----------------+----------------------+ 561| fstat | 2 | Filesystem | sys_fstat() | 562+-------------------+-----------+-----------------+----------------------+ 563| pread64 | 6 | Filesystem | sys_pread64() | 564+-------------------+-----------+-----------------+----------------------+ 565| access | 1 | Filesystem | sys_access() | 566+-------------------+-----------+-----------------+----------------------+ 567| pipe | 1 | Filesystem | sys_pipe() | 568+-------------------+-----------+-----------------+----------------------+ 569| dup2 | 24 | Filesystem | sys_dup2() | 570+-------------------+-----------+-----------------+----------------------+ 571| execve | 1 | Filesystem | sys_execve() | 572+-------------------+-----------+-----------------+----------------------+ 573| fcntl | 26 | Filesystem | sys_fcntl() | 574+-------------------+-----------+-----------------+----------------------+ 575| openat | 14 | Filesystem | sys_openat() | 576+-------------------+-----------+-----------------+----------------------+ 577| rt_sigaction | 7 | Signal | sys_rt_sigaction() | 578+-------------------+-----------+-----------------+----------------------+ 579| rt_sigreturn | 38 | Signal | sys_rt_sigreturn() | 580+-------------------+-----------+-----------------+----------------------+ 581| clone | 38 | Process Mgmt. | sys_clone() | 582+-------------------+-----------+-----------------+----------------------+ 583| wait4 | 44 | Time | sys_wait4() | 584+-------------------+-----------+-----------------+----------------------+ 585| mmap | 7 | Memory Mgmt. | sys_mmap() | 586+-------------------+-----------+-----------------+----------------------+ 587| mprotect | 3 | Memory Mgmt. | sys_mprotect() | 588+-------------------+-----------+-----------------+----------------------+ 589| munmap | 1 | Memory Mgmt. | sys_munmap() | 590+-------------------+-----------+-----------------+----------------------+ 591| brk | 3 | Memory Mgmt. | sys_brk() | 592+-------------------+-----------+-----------------+----------------------+ 593| getpid | 1 | Process Mgmt. | sys_getpid() | 594+-------------------+-----------+-----------------+----------------------+ 595| getuid | 1 | Process Mgmt. | sys_getuid() | 596+-------------------+-----------+-----------------+----------------------+ 597| getgid | 1 | Process Mgmt. | sys_getgid() | 598+-------------------+-----------+-----------------+----------------------+ 599| geteuid | 2 | Process Mgmt. | sys_geteuid() | 600+-------------------+-----------+-----------------+----------------------+ 601| getegid | 1 | Process Mgmt. | sys_getegid() | 602+-------------------+-----------+-----------------+----------------------+ 603| getppid | 1 | Process Mgmt. | sys_getppid() | 604+-------------------+-----------+-----------------+----------------------+ 605| arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() | 606+-------------------+-----------+-----------------+----------------------+ 607 608Conclusion 609========== 610 611This document is intended to be used as a guide on how to gather fine-grained 612information on the resources in use by workloads using strace. 613 614References 615========== 616 617 * `Discovery Linux Kernel Subsystems used by OpenAPS <https://elisa.tech/blog/2022/02/02/discovery-linux-kernel-subsystems-used-by-openaps>`_ 618 * `ELISA-White-Papers-Discovering Linux kernel subsystems used by a workload <https://github.com/elisa-tech/ELISA-White-Papers/blob/master/Processes/Discovering_Linux_kernel_subsystems_used_by_a_workload.md>`_ 619 * `strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_ 620 * `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_ 621 * `paxtest README <https://github.com/opntr/paxtest-freebsd/blob/hardenedbsd/0.9.14-hbsd/README>`_ 622 * `stress-ng <https://www.mankier.com/1/stress-ng>`_ 623 * `Monitoring and managing system status and performance <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/index>`_ 624