1.. SPDX-License-Identifier: GPL-2.0 2 3===================================== 4Using Propeller with the Linux kernel 5===================================== 6 7This enables Propeller build support for the kernel when using Clang 8compiler. Propeller is a profile-guided optimization (PGO) method used 9to optimize binary executables. Like AutoFDO, it utilizes hardware 10sampling to gather information about the frequency of execution of 11different code paths within a binary. Unlike AutoFDO, this information 12is then used right before linking phase to optimize (among others) 13block layout within and across functions. 14 15A few important notes about adopting Propeller optimization: 16 17#. Although it can be used as a standalone optimization step, it is 18 strongly recommended to apply Propeller on top of AutoFDO, 19 AutoFDO+ThinLTO or Instrument FDO. The rest of this document 20 assumes this paradigm. 21 22#. Propeller uses another round of profiling on top of 23 AutoFDO/AutoFDO+ThinLTO/iFDO. The whole build process involves 24 "build-afdo - train-afdo - build-propeller - train-propeller - 25 build-optimized". 26 27#. Propeller requires LLVM 19 release or later for Clang/Clang++ 28 and the linker(ld.lld). 29 30#. In addition to LLVM toolchain, Propeller requires a profiling 31 conversion tool: https://github.com/google/autofdo with a release 32 after v0.30.1: https://github.com/google/autofdo/releases/tag/v0.30.1. 33 34The Propeller optimization process involves the following steps: 35 36#. Initial building: Build the AutoFDO or AutoFDO+ThinLTO binary as 37 you would normally do, but with a set of compile-time / link-time 38 flags, so that a special metadata section is created within the 39 kernel binary. The special section is only intend to be used by the 40 profiling tool, it is not part of the runtime image, nor does it 41 change kernel run time text sections. 42 43#. Profiling: The above kernel is then run with a representative 44 workload to gather execution frequency data. This data is collected 45 using hardware sampling, via perf. Propeller is most effective on 46 platforms supporting advanced PMU features like LBR on Intel 47 machines. This step is the same as profiling the kernel for AutoFDO 48 (the exact perf parameters can be different). 49 50#. Propeller profile generation: Perf output file is converted to a 51 pair of Propeller profiles via an offline tool. 52 53#. Optimized build: Build the AutoFDO or AutoFDO+ThinLTO optimized 54 binary as you would normally do, but with a compile-time / 55 link-time flag to pick up the Propeller compile time and link time 56 profiles. This build step uses 3 profiles - the AutoFDO profile, 57 the Propeller compile-time profile and the Propeller link-time 58 profile. 59 60#. Deployment: The optimized kernel binary is deployed and used 61 in production environments, providing improved performance 62 and reduced latency. 63 64Preparation 65=========== 66 67Configure the kernel with:: 68 69 CONFIG_AUTOFDO_CLANG=y 70 CONFIG_PROPELLER_CLANG=y 71 72Customization 73============= 74 75The default CONFIG_PROPELLER_CLANG setting covers kernel space objects 76for Propeller builds. One can, however, enable or disable Propeller build 77for individual files and directories by adding a line similar to the 78following to the respective kernel Makefile: 79 80- For enabling a single file (e.g. foo.o):: 81 82 PROPELLER_PROFILE_foo.o := y 83 84- For enabling all files in one directory:: 85 86 PROPELLER_PROFILE := y 87 88- For disabling one file:: 89 90 PROPELLER_PROFILE_foo.o := n 91 92- For disabling all files in one directory:: 93 94 PROPELLER__PROFILE := n 95 96 97Workflow 98======== 99 100Here is an example workflow for building an AutoFDO+Propeller kernel: 101 1021) Assuming an AutoFDO profile is already collected following 103 instructions in the AutoFDO document, build the kernel on the host 104 machine, with AutoFDO and Propeller build configs :: 105 106 CONFIG_AUTOFDO_CLANG=y 107 CONFIG_PROPELLER_CLANG=y 108 109 and :: 110 111 $ make LLVM=1 CLANG_AUTOFDO_PROFILE=<autofdo-profile-name> 112 1132) Install the kernel on the test machine. 114 1153) Run the load tests. The '-c' option in perf specifies the sample 116 event period. We suggest using a suitable prime number, like 500009, 117 for this purpose. 118 119 - For Intel platforms:: 120 121 $ perf record -e BR_INST_RETIRED.NEAR_TAKEN:k -a -N -b -c <count> -o <perf_file> -- <loadtest> 122 123 - For AMD platforms:: 124 125 $ perf record --pfm-event RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k -a -N -b -c <count> -o <perf_file> -- <loadtest> 126 127 Note you can repeat the above steps to collect multiple <perf_file>s. 128 1294) (Optional) Download the raw perf file(s) to the host machine. 130 1315) Use the create_llvm_prof tool (https://github.com/google/autofdo) to 132 generate Propeller profile. :: 133 134 $ create_llvm_prof --binary=<vmlinux> --profile=<perf_file> 135 --format=propeller --propeller_output_module_name 136 --out=<propeller_profile_prefix>_cc_profile.txt 137 --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt 138 139 "<propeller_profile_prefix>" can be something like "/home/user/dir/any_string". 140 141 This command generates a pair of Propeller profiles: 142 "<propeller_profile_prefix>_cc_profile.txt" and 143 "<propeller_profile_prefix>_ld_profile.txt". 144 145 If there are more than 1 perf_file collected in the previous step, 146 you can create a temp list file "<perf_file_list>" with each line 147 containing one perf file name and run:: 148 149 $ create_llvm_prof --binary=<vmlinux> --profile=@<perf_file_list> 150 --format=propeller --propeller_output_module_name 151 --out=<propeller_profile_prefix>_cc_profile.txt 152 --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt 153 1546) Rebuild the kernel using the AutoFDO and Propeller 155 profiles. :: 156 157 CONFIG_AUTOFDO_CLANG=y 158 CONFIG_PROPELLER_CLANG=y 159 160 and :: 161 162 $ make LLVM=1 CLANG_AUTOFDO_PROFILE=<profile_file> CLANG_PROPELLER_PROFILE_PREFIX=<propeller_profile_prefix> 163