1.\" 2.\" SPDX-License-Identifier: BSD-2-Clause 3.\" 4.\" Copyright (c) 2025 Mateusz Piotrowski <0mp@FreeBSD.org> 5.\" 6.Dd July 16, 2025 7.Dt DTRACE_FBT 4 8.Os 9.Sh NAME 10.Nm dtrace_fbt 11.Nd a DTrace provider for dynamic kernel tracing based on function boundaries 12.Sh SYNOPSIS 13.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry 14.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return 15.Sh DESCRIPTION 16The Function Boundary Tracing 17.Pq Nm fbt 18provider instruments the entry and return of almost every kernel function 19corresponding to an 20.Xr elf 5 21symbol in the kernel and loaded kernel modules. 22.Pp 23.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry 24fires whenever the 25.Ar function 26is called. 27.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return 28fires when the 29.Ar function 30returns. 31.Pp 32The 33.Ar module 34in the probe description is either the name of the loaded kernel module 35or 36.Ql kernel 37for functions compiled into the kernel. 38.Ss Function Boundary Instrumentation 39The 40.Nm fbt 41will always instrument a function's entry, but 42its return will be intsrumented so long as it can find a 43.Ql ret 44instruction. 45.Pp 46In some cases, 47.Nm fbt 48cannot instrument a function's entry and/or return. 49Refer to subsection 50.Sx Frame Pointer 51for more details. 52.Ss Probe Arguments 53The arguments of the entry probe 54.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry 55are the arguments of the traced function call. 56.Bl -column -offset indent "Entry Probe Argument" "Definition" 57.It Sy Entry Probe Argument Ta Sy Definition 58.It Fa args[0] Ta Function's first argument, typed 59.Pq e.g., Xr malloc 9 Ap s Ft size_t Fa size 60.It Fa args[1] Ta Function's second argument, typed 61.Pq e.g., Xr malloc 9 Ap s Ft struct malloc_type Fa *type 62.It Fa args[2] Ta Function's third argument, typed 63.Pq e.g., Xr malloc 9 Ap s Ft int Fa flags 64.It Fa ... Ta ... 65.El 66.Pp 67The arguments of the return probe 68.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return 69are 70.Fa args[0] 71.Po 72the offset of the firing return instruction within the function; 73useful to tell apart two different return statements in a single function 74.Pc 75and 76.Fa args[1] 77.Pq the return value, if any . 78.Bl -column -offset indent "Return Probe Argument" "Definition" 79.It Sy Return Probe Argument Ta Sy Definition 80.It Fa args[0] Ta Offset of the traced return instruction 81.It Fa args[1] Ta Function's return value 82.Po e.g., a kernel virtual address if returning from a successful 83.Xr malloc 9 84.Pc 85.El 86.Pp 87Subsection 88.Sx Example 2 : Getting Details About Probe's Arguments 89shows how to get probe's argument count and types directly with 90.Xr dtrace 1 91without having to resort to the reading function's source code 92or documentation. 93.Sh EXAMPLES 94.Ss Example 1 : Listing Available FBT Probes 95The following example shows how to list all the available 96.Nm fbt 97probes. 98.Bd -literal -offset 2n 99# dtrace -l -P fbt 100 ID PROVIDER MODULE FUNCTION NAME 101[...] 10231868 fbt kernel hammer_time entry 10331869 fbt kernel hammer_time return 104[...] 105.Ed 106.Pp 107Since 108.Fn hammer_time 109is a part of the kernel and not a separate loaded module, the 110.Ar module 111column displays 112.Ql kernel . 113.Ss Example 2 : Getting Details About Probe's Arguments 114The following example shows how to generate a program stability report of 115.Xr malloc 9 Ap s 116entry and return probes. 117Those reports are useful to view 118the probe's number of arguments and their types. 119.Bd -literal -offset 2n 120# dtrace -l -v -n fbt::malloc:entry 121[...] 122 Argument Types 123 args[0]: size_t 124 args[1]: struct malloc_type * 125 args[2]: int 126.Ed 127.Pp 128The count and types of 129.Nm fbt Ns Cm \&::malloc:entry 130arguments 131match the function signature of 132.Xr malloc 9 : 133.Va args[0] 134is 135.Ft size_t , 136.Va args[1] 137is 138.Ft "struct malloc_type *" , 139and 140.Va "args[2]" 141is 142.Ft int . 143.Bd -literal -offset 2n 144# dtrace -l -v -n fbt::malloc:return 145[...] 146 Argument Types 147 args[0]: int 148 args[1]: void * 149.Ed 150.Pp 151The 152.Cm return 153probe reports two arguments and their types: 154the return instruction offset 155.Pq the usual Ft int 156and the function's return value, which in this case is 157.Ft void * , 158as 159.Xr malloc 9 160returns a kernel virtual address. 161.Ss Example 3 : Counting Kernel Slab Memory Allocation by Function 162.Bd -literal -offset 2n 163# dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }' 164dtrace: description 'fbt::kmem*:entry ' matched 47 probes 165^C 166 kmem_alloc_contig 1 167 kmem_alloc_contig_domainset 1 168 kmem_cache_reap_active 1 169 kmem_alloc_contig_pages 2 170 kmem_free 2 171 kmem_std_destructor 19 172 kmem_std_constructor 26 173 kmem_cache_free 151 174 kmem_cache_alloc 181 175.Ed 176.Ss Example 4 : Counting Kernel Slab Memory Allocation by Calling Function 177.Bd -literal -offset 2n 178# dtrace -q -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\en", @); }' 179^C 180 kernel`contigmalloc+0x33 1 181 kernel`free+0xd3 1 182 kernel`kmem_alloc_contig+0x29 1 183kernel`kmem_alloc_contig_domainset+0x19a 1 184 zfs.ko`arc_reap_cb_check+0x16 1 185.Ed 186.Ss Example 5 : Counting Kernel malloc()'s by Calling Function 187.Bd -literal -offset 2n 188# dtrace -q -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%45a %@16d\en", @); }' 189^C 190 kernel`devclass_get_devices+0xa8 1 191 kernel`sys_ioctl+0xb7 1 192 dtrace.ko`dtrace_ioctl+0x15c1 1 193 dtrace.ko`dtrace_ioctl+0x972 2 194 dtrace.ko`dtrace_dof_create+0x35 2 195 kernel`kern_poll_kfds+0x2f0 4 196 kernel`kern_poll_kfds+0x28a 19 197.Ed 198.Ss Example 6 : Counting Kernel malloc()'s by Kernel Stack Trace 199.Bd -literal -offset 2n 200# dtrace -q -n 'fbt::malloc:entry { @[stack()] = count(); }' 201^C 202 dtrace.ko`dtrace_dof_create+0x35 203 dtrace.ko`dtrace_ioctl+0x827 204 kernel`devfs_ioctl+0xd1 205 kernel`VOP_IOCTL_APV+0x2a 206 kernel`vn_ioctl+0xb6 207 kernel`devfs_ioctl_f+0x1e 208 kernel`kern_ioctl+0x286 209 kernel`sys_ioctl+0x12f 210 kernel`amd64_syscall+0x169 211 kernel`0xffffffff81092b0b 212 2 213.Ed 214.Ss Example 7 : Summarizing vmem_alloc()'s by Arena Name and Size Distribution 215.Bd -literal -offset 2n 216# dtrace -q -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }' 217^C 218 219 kernel arena dom 220 value ------------- Distribution ------------- count 221 2048 | 0 222 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4 223 8192 |@@@@@@@@@@@@@ 2 224 16384 | 0 225.Ed 226.Ss Example 8 : Measuring Total Time Spent Executing a Function 227This DTrace script measures the total time spent in 228.Fn vm_page* 229kernel functions. 230The 231.Fn quantize 232aggregation organizes the measurements into power-of-two buckets, 233providing a time distribution in nanoseconds for each function. 234.Bd -literal -offset 2n 235fbt::vm_page*:entry { 236 self->start = timestamp; 237} 238 239fbt::vm_page*:return /self->start/ { 240 @[probefunc] = quantize(timestamp - self->start); 241 self->start = 0; 242} 243.Ed 244.Sh SEE ALSO 245.Xr dtrace 1 , 246.Xr dtrace_kinst 4 , 247.Xr tracing 7 248.Rs 249.%A Brendan Gregg 250.%A Jim Mauro 251.%B DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD 252.%I Prentice Hall 253.%P pp. 898\(en903 254.%D 2011 255.%U https://www.brendangregg.com/dtracebook/ 256.Re 257.Rs 258.%B The illumos Dynamic Tracing Guide 259.%O Chapter fbt Provider 260.%D 2008 261.%U https://illumos.org/books/dtrace/chp-fbt.html#chp-fbt 262.Re 263.Sh AUTHORS 264This manual page was written by 265.An Mateusz Piotrowski Aq Mt 0mp@FreeBSD.org . 266.Sh CAVEATS 267.Ss Stability and Portability 268.Nm fbt 269probes are by definition tightly coupled to kernel code; if the code underlying 270a script changes, the script may fail to run or may produce incorrect results. 271Scripts written for one version of 272.Fx 273might not work on others, 274and almost certainly will not work on other operating systems. 275.Pp 276Individual 277.Nm fbt 278probes often do not correspond nicely to logical system events. 279For example, consider a DTrace script which prints the destination 280address of every IP packet as the kernel hands them over 281to the network card driver (NIC). 282An 283.Nm fbt Ns -based 284implementation of such a script is a discouragingly difficult task: 285it involves instrumenting at least four different functions in different parts 286of the IPv4 and IPv6 code. 287At the same time, with the 288.Xr dtrace_ip 4 289provider the script is a simple one-liner: 290.Dl dtrace -n 'ip:::send {printf("%s", args[2]->ip_daddr);}' 291.Pp 292Make sure to review available 293.Xr dtrace 1 294providers first 295before implementing a custom script with the 296.Nm fbt 297provider. 298If none of the DTrace providers offer the desired probes, 299consider adding new statically-defined tracing probes 300.Pq Xr SDT 9 . 301.Ss Frame Pointer 302Inline functions are not instrumentable by 303.Nm fbt 304as they lack a frame pointer. 305A developer might explicitly disable inlining by adding the 306.Ql __noinline 307attribute to a function definition, 308but of course this requires a recompilation of the kernel. 309Building the kernel with 310.Fl fno-omit-frame-pointer 311is another way of preserving frame pointers. 312Note, that sometimes compilers will omit the frame pointer in leaf functions, 313even when configured with 314.Fl fno-omit-frame-pointer . 315.Pp 316Function returns via a tail call are also not instrumentable by 317.Nm fbt . 318As a result, 319a function might have an entry probe 320and a mix of instrumented and uninstrumentable returns. 321.Pp 322Use 323.Xr dtrace_kinst 4 324to trace arbitrary instructions inside kernel functions 325and work around some of the 326limitations 327of 328.Nm fbt . 329.Ss Tracing DTrace 330The 331.Nm fbt 332provider cannot attach to functions inside DTrace provider kernel modules. 333