1b9d64941SMark Johnston.\" Copyright (c) 2013-2015 Mark Johnston <markj@freebsd.org> 298491bfaSMark Johnston.\" All rights reserved. 398491bfaSMark Johnston.\" 498491bfaSMark Johnston.\" Redistribution and use in source and binary forms, with or without 598491bfaSMark Johnston.\" modification, are permitted provided that the following conditions 698491bfaSMark Johnston.\" are met: 798491bfaSMark Johnston.\" 1. Redistributions of source code must retain the above copyright 898491bfaSMark Johnston.\" notice, this list of conditions and the following disclaimer. 998491bfaSMark Johnston.\" 2. Redistributions in binary form must reproduce the above copyright 1098491bfaSMark Johnston.\" notice, this list of conditions and the following disclaimer in the 1198491bfaSMark Johnston.\" documentation and/or other materials provided with the distribution. 1298491bfaSMark Johnston.\" 1398491bfaSMark Johnston.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 1498491bfaSMark Johnston.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 1598491bfaSMark Johnston.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 1698491bfaSMark Johnston.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 1798491bfaSMark Johnston.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 1898491bfaSMark Johnston.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 1998491bfaSMark Johnston.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 2098491bfaSMark Johnston.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 2198491bfaSMark Johnston.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 2298491bfaSMark Johnston.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 2398491bfaSMark Johnston.\" SUCH DAMAGE. 2498491bfaSMark Johnston.\" 2530ed58c1SMark Johnston.Dd April 18, 2015 2698491bfaSMark Johnston.Dt SDT 9 2798491bfaSMark Johnston.Os 2898491bfaSMark Johnston.Sh NAME 2998491bfaSMark Johnston.Nm SDT 3098491bfaSMark Johnston.Nd a DTrace framework for adding statically-defined tracing probes 3198491bfaSMark Johnston.Sh SYNOPSIS 3271f3caafSSteven Hartland.In sys/param.h 3371f3caafSSteven Hartland.In sys/queue.h 3498491bfaSMark Johnston.In sys/sdt.h 3598491bfaSMark Johnston.Fn SDT_PROVIDER_DECLARE prov 3698491bfaSMark Johnston.Fn SDT_PROVIDER_DEFINE prov 3798491bfaSMark Johnston.Fn SDT_PROBE_DECLARE prov mod func name 38d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE prov mod func name 39d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE0 prov mod func name 40d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE1 prov mod func name arg0 41d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE2 prov mod func name arg0 arg1 42d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE3 prov mod func name arg0 arg1 arg2 43d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE4 prov mod func name arg0 arg1 arg2 arg3 44d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE5 prov mod func name arg0 arg1 arg2 arg3 arg4 45d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE6 prov mod func name arg0 arg1 arg2 arg3 arg4 arg5 46d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE7 prov mod func name arg0 arg1 arg2 arg3 arg4 arg5 \ 4798491bfaSMark Johnston arg6 48d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE0_XLATE prov mod func name 49d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE1_XLATE prov mod func name arg0 xarg0 50d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE2_XLATE prov mod func name arg0 xarg0 arg1 xarg1 51d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE3_XLATE prov mod func name arg0 xarg0 arg1 xarg1 \ 52c6cb04a4SMark Johnston arg2 xarg2 53d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE4_XLATE prov mod func name arg0 xarg0 arg1 xarg1 \ 54c6cb04a4SMark Johnston arg2 xarg2 arg3 xarg3 55d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE5_XLATE prov mod func name arg0 xarg0 arg1 xarg1 \ 56c6cb04a4SMark Johnston arg2 xarg2 arg3 xarg3 arg4 xarg4 57d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE6_XLATE prov mod func name arg0 xarg0 arg1 xarg1 \ 58c6cb04a4SMark Johnston arg2 xarg2 arg3 xarg3 arg4 xarg4 arg5 xarg5 59d9fae5abSAndriy Gapon.Fn SDT_PROBE_DEFINE7_XLATE prov mod func name arg0 xarg0 arg1 xarg1 \ 60c6cb04a4SMark Johnston arg2 xarg2 arg3 xarg3 arg4 xarg4 arg5 xarg5 arg6 xarg6 6198491bfaSMark Johnston.Fn SDT_PROBE0 prov mod func name 6298491bfaSMark Johnston.Fn SDT_PROBE1 prov mod func name arg0 6398491bfaSMark Johnston.Fn SDT_PROBE2 prov mod func name arg0 arg1 6498491bfaSMark Johnston.Fn SDT_PROBE3 prov mod func name arg0 arg1 arg2 6598491bfaSMark Johnston.Fn SDT_PROBE4 prov mod func name arg0 arg1 arg2 arg3 6698491bfaSMark Johnston.Fn SDT_PROBE5 prov mod func name arg0 arg1 arg2 arg3 arg4 6798491bfaSMark Johnston.Fn SDT_PROBE6 prov mod func name arg0 arg1 arg2 arg3 arg4 arg5 6898491bfaSMark Johnston.Fn SDT_PROBE7 prov mod func name arg0 arg1 arg2 arg3 arg4 arg5 arg6 6998491bfaSMark Johnston.Sh DESCRIPTION 7098491bfaSMark JohnstonThe 7198491bfaSMark Johnston.Nm 7298491bfaSMark Johnstonmacros allow programmers to define static trace points in kernel code. 7398491bfaSMark JohnstonThese trace points are used by the 7498491bfaSMark Johnston.Nm 7598491bfaSMark Johnstonframework to create DTrace probes, allowing the code to be instrumented 7698491bfaSMark Johnstonusing 7798491bfaSMark Johnston.Xr dtrace 1 . 7898491bfaSMark JohnstonBy default, 7998491bfaSMark Johnston.Nm 8098491bfaSMark Johnstontrace points are disabled and have no effect on the surrounding code. 8198491bfaSMark JohnstonWhen a DTrace probe corresponding to a given trace point is enabled, threads 8298491bfaSMark Johnstonthat execute the trace point will call a handler and cause the probe to fire. 8398491bfaSMark JohnstonMoreover, trace points can take arguments, making it possible to pass data 8498491bfaSMark Johnstonto the DTrace framework when an enabled probe fires. 8598491bfaSMark Johnston.Pp 8698491bfaSMark JohnstonMultiple trace points may correspond to a single DTrace probe, allowing 8798491bfaSMark Johnstonprogrammers to create DTrace probes that correspond to logical system events 8898491bfaSMark Johnstonrather than tying probes to specific code execution paths. 8998491bfaSMark JohnstonFor instance, a DTrace probe corresponding to the arrival of an IP packet into 9098491bfaSMark Johnstonthe network stack may be defined using two 9198491bfaSMark Johnston.Nm 9298491bfaSMark Johnstontrace points: one for IPv4 packets and one for IPv6 packets. 9398491bfaSMark Johnston.Pp 9498491bfaSMark JohnstonIn addition to defining DTrace probes, the 9598491bfaSMark Johnston.Nm 9698491bfaSMark Johnstonmacros allow programmers to define new DTrace providers, making it possible to 9798491bfaSMark Johnstonnamespace logically-related probes. 9898491bfaSMark JohnstonAn example is FreeBSD's sctp provider, which contains 9998491bfaSMark Johnston.Nm 10098491bfaSMark Johnstonprobes for FreeBSD's 10198491bfaSMark Johnston.Xr sctp 4 10298491bfaSMark Johnstonimplementation. 10398491bfaSMark Johnston.Pp 10498491bfaSMark JohnstonThe 10598491bfaSMark Johnston.Fn SDT_PROVIDER_DECLARE 10698491bfaSMark Johnstonand 10798491bfaSMark Johnston.Fn SDT_PROVIDER_DEFINE 10898491bfaSMark Johnstonmacros are used respectively to declare and define a DTrace provider named 10998491bfaSMark Johnston.Ar prov 11098491bfaSMark Johnstonwith the 11198491bfaSMark Johnston.Nm 11298491bfaSMark Johnstonframework. 11398491bfaSMark JohnstonA provider need only be defined once; however, the provider must be declared 11498491bfaSMark Johnstonbefore defining any 11598491bfaSMark Johnston.Nm 11698491bfaSMark Johnstonprobes belonging to that provider. 11798491bfaSMark Johnston.Pp 11898491bfaSMark JohnstonSimilarly, the 11998491bfaSMark Johnston.Fn SDT_PROBE_DECLARE 12098491bfaSMark Johnstonand 12198491bfaSMark Johnston.Fn SDT_PROBE_DEFINE* 12298491bfaSMark Johnstonmacros are used to declare and define DTrace probes using the 12398491bfaSMark Johnston.Nm 12498491bfaSMark Johnstonframework. 12598491bfaSMark JohnstonOnce a probe has been defined, trace points for that probe may be added to 12698491bfaSMark Johnstonkernel code. 12798491bfaSMark JohnstonDTrace probe identifiers consist of a provider, module, function and name, all 12898491bfaSMark Johnstonof which may be specified in the 12998491bfaSMark Johnston.Nm 13098491bfaSMark Johnstonprobe definition. 13198491bfaSMark JohnstonNote that probes should not specify a module name: the module name of a probe is 13298491bfaSMark Johnstonused to determine whether or not it should be destroyed when a kernel module is 13398491bfaSMark Johnstonunloaded. 13498491bfaSMark JohnstonSee the 13598491bfaSMark Johnston.Sx BUGS 13698491bfaSMark Johnstonsection. 13798491bfaSMark JohnstonNote in particular that probes must not be defined across multiple kernel 13898491bfaSMark Johnstonmodules. 139d9fae5abSAndriy Gapon.Pp 140d9fae5abSAndriy GaponIf 14198491bfaSMark Johnston.Ql - 142d9fae5abSAndriy Gaponcharacter (dash) is wanted in a probe name, 143d9fae5abSAndriy Gaponthen it should be represented as 144d9fae5abSAndriy Gapon.Ql __ 145d9fae5abSAndriy Gapon(double underscore) in the probe 14698491bfaSMark Johnston.Ar name 147d9fae5abSAndriy Gaponparameter passed to various 148d9fae5abSAndriy Gapon.Fn SDT_* 149d9fae5abSAndriy Gaponmacros, 150d9fae5abSAndriy Gaponbecause of technical reasons 151d9fae5abSAndriy Gapon(a dash is not valid in C identifiers). 15298491bfaSMark Johnston.Pp 15398491bfaSMark JohnstonThe 15498491bfaSMark Johnston.Fn SDT_PROBE_DEFINE* 15598491bfaSMark Johnstonmacros also allow programmers to declare the types of the arguments that are 15698491bfaSMark Johnstonpassed to probes. 15798491bfaSMark JohnstonThis is optional; if the argument types are omitted (through use of the 15898491bfaSMark Johnston.Fn SDT_PROBE_DEFINE 15998491bfaSMark Johnstonmacro), users wishing to make use of the arguments will have to manually cast 16098491bfaSMark Johnstonthem to the correct types in their D scripts. 16198491bfaSMark JohnstonIt is strongly recommended that probe definitions include a declaration of their 16298491bfaSMark Johnstonargument types. 16398491bfaSMark Johnston.Pp 16498491bfaSMark JohnstonThe 165c6cb04a4SMark Johnston.Fn SDT_PROBE_DEFINE*_XLATE 166c6cb04a4SMark Johnstonmacros are used for probes whose argument types are to be dynamically translated 167c6cb04a4SMark Johnstonto the types specified by the corresponding 168c6cb04a4SMark Johnston.Ar xarg 169c6cb04a4SMark Johnstonarguments. 170c6cb04a4SMark JohnstonThis is mainly useful when porting probe definitions from other operating 171c6cb04a4SMark Johnstonsystems. 172c6cb04a4SMark JohnstonAs seen by 173c6cb04a4SMark Johnston.Xr dtrace 1 , 174c6cb04a4SMark Johnstonthe arguments of a probe defined using these macros will have types which match 175c6cb04a4SMark Johnstonthe 176c6cb04a4SMark Johnston.Ar xarg 177c6cb04a4SMark Johnstontypes in the probe definition. 178c6cb04a4SMark JohnstonHowever, the arguments passed in at the trace point will have types matching the 179c6cb04a4SMark Johnstonnative argument types in the probe definition, and thus the native type is 180c6cb04a4SMark Johnstondynamically translated to the translated type. 181c6cb04a4SMark JohnstonSo long as an appropriate translator is defined in 182c6cb04a4SMark Johnston.Pa /usr/lib/dtrace , 183c6cb04a4SMark Johnstonscripts making use of the probe need not concern themselves with the underlying 184c6cb04a4SMark Johnstontype of a given 185c6cb04a4SMark Johnston.Nm 186c6cb04a4SMark Johnstonprobe argument. 187c6cb04a4SMark Johnston.Pp 188c6cb04a4SMark JohnstonThe 18998491bfaSMark Johnston.Fn SDT_PROBE* 19098491bfaSMark Johnstonmacros are used to create 19198491bfaSMark Johnston.Nm 19298491bfaSMark Johnstontrace points. 19398491bfaSMark JohnstonThey are meant to be added to executable code and can be used to instrument the 19498491bfaSMark Johnstoncode in which they are called. 19530ed58c1SMark Johnston.Sh PROVIDERS 19630ed58c1SMark JohnstonA number of kernel DTrace providers are available. 19730ed58c1SMark JohnstonIn general, these providers define stable interfaces and should be treated as 19830ed58c1SMark Johnstonsuch: existing D scripts may be broken if a probe is renamed or its arguments 19930ed58c1SMark Johnstonare modified. 20030ed58c1SMark JohnstonHowever, it is often useful to define ad-hoc 20130ed58c1SMark Johnston.Nm 20230ed58c1SMark Johnstonprobes for debugging a subsystem or driver. 20330ed58c1SMark JohnstonSimilarly, a developer may wish to provide a group of 20430ed58c1SMark Johnston.Nm 20530ed58c1SMark Johnstonprobes without committing to their future stability. 20630ed58c1SMark JohnstonSuch probes should be added to the 20730ed58c1SMark Johnston.Ql sdt 20830ed58c1SMark Johnstonprovider instead of defining a new provider. 20998491bfaSMark Johnston.Sh EXAMPLES 21030ed58c1SMark JohnstonThe DTrace providers available on the current system can be listed with 21130ed58c1SMark Johnston.Bd -literal -offset indent 21230ed58c1SMark Johnstondtrace -l | sed 1d | awk '{print $2}' | sort -u 21330ed58c1SMark Johnston.Ed 21430ed58c1SMark Johnston.Pp 21530ed58c1SMark JohnstonA detailed list of the probes offered by a given provider can be obtained by 21630ed58c1SMark Johnstonspecifying the provider using the 21730ed58c1SMark Johnston.Fl P 21830ed58c1SMark Johnstonflag. 21930ed58c1SMark JohnstonFor example, to view the probes and argument types for the 22030ed58c1SMark Johnston.Ql sched 22130ed58c1SMark Johnstonprovider, run 22230ed58c1SMark Johnston.Bd -literal -offset indent 22330ed58c1SMark Johnstondtrace -lv -P sched 22430ed58c1SMark Johnston.Ed 22530ed58c1SMark Johnston.Pp 22698491bfaSMark JohnstonThe following probe definition will create a DTrace probe called 227d85550b2SMark Johnston.Ql icmp:::receive-unreachable , 22898491bfaSMark Johnstonwhich would hypothetically be triggered when the kernel receives an ICMP packet 22998491bfaSMark Johnstonof type Destination Unreachable: 23098491bfaSMark Johnston.Bd -literal -offset indent 23198491bfaSMark JohnstonSDT_PROVIDER_DECLARE(icmp); 23298491bfaSMark Johnston 233d85550b2SMark JohnstonSDT_PROBE_DEFINE1(icmp, , , receive__unreachable, 234c6cb04a4SMark Johnston "struct icmp *"); 23598491bfaSMark Johnston 23698491bfaSMark Johnston.Ed 237c6cb04a4SMark JohnstonThis particular probe would take a single argument: a pointer to the struct 238c6cb04a4SMark Johnstoncontaining the ICMP header for the packet. 23998491bfaSMark JohnstonNote that the module name of this probe is not specified. 24098491bfaSMark Johnston.Pp 24198491bfaSMark JohnstonConsider a DTrace probe which fires when the network stack receives an IP 24298491bfaSMark Johnstonpacket. 24398491bfaSMark JohnstonSuch a probe would be defined by multiple tracepoints: 24498491bfaSMark Johnston.Bd -literal -offset indent 245d7a53a3bSMark JohnstonSDT_PROBE_DEFINE3(ip, , , receive, "struct ifnet *", 246c6cb04a4SMark Johnston "struct ip *", "struct ip6_hdr *"); 24798491bfaSMark Johnston 24898491bfaSMark Johnstonint 24998491bfaSMark Johnstonip_input(struct mbuf *m) 25098491bfaSMark Johnston{ 25198491bfaSMark Johnston struct ip *ip; 25298491bfaSMark Johnston ... 25398491bfaSMark Johnston ip = mtod(m, struct ip *); 254c6cb04a4SMark Johnston SDT_PROBE3(ip, , , receive, m->m_pkthdr.rcvif, ip, NULL); 25598491bfaSMark Johnston ... 25698491bfaSMark Johnston} 25798491bfaSMark Johnston 25898491bfaSMark Johnstonint 25998491bfaSMark Johnstonip6_input(struct mbuf *m) 26098491bfaSMark Johnston{ 26198491bfaSMark Johnston struct ip6_hdr *ip6; 26298491bfaSMark Johnston ... 26398491bfaSMark Johnston ip6 = mtod(m, struct ip6_hdr *); 264c6cb04a4SMark Johnston SDT_PROBE3(ip, , , receive, m->m_pkthdr.rcvif, NULL, ip6); 26598491bfaSMark Johnston ... 26698491bfaSMark Johnston} 26798491bfaSMark Johnston 26898491bfaSMark Johnston.Ed 26998491bfaSMark JohnstonIn particular, the probe should fire when the kernel receives either an IPv4 27098491bfaSMark Johnstonpacket or an IPv6 packet. 271c6cb04a4SMark Johnston.Pp 272c6cb04a4SMark JohnstonConsider the ICMP probe discussed above. 273c6cb04a4SMark JohnstonWe note that its second argument is of type 274c6cb04a4SMark Johnston.Ar struct icmp , 275c6cb04a4SMark Johnstonwhich is a type defined in the FreeBSD kernel to represent the ICMP header of 276c6cb04a4SMark Johnstonan ICMP packet, defined in RFC 792. 277c6cb04a4SMark JohnstonLinux has a corresponding type, 278c6cb04a4SMark Johnston.Ar struct icmphdr , 279c6cb04a4SMark Johnstonfor the same purpose, but its field names differ from FreeBSD's 280c6cb04a4SMark Johnston.Ar struct icmp . 281c6cb04a4SMark JohnstonSimilarly, illumos defines the 282c6cb04a4SMark Johnston.Ar icmph_t 283c6cb04a4SMark Johnstontype, again with different field names. 284c6cb04a4SMark JohnstonEven with the 285c6cb04a4SMark Johnston.Ql icmp:::pkt-receive 286c6cb04a4SMark Johnstonprobes defined in all three operating systems, 287c6cb04a4SMark Johnstonone would still have to write OS-specific scripts to extract a given field out 288c6cb04a4SMark Johnstonof the ICMP header argument. 289c6cb04a4SMark JohnstonDynamically-translated types solve this problem: one can define an 290c6cb04a4SMark JohnstonOS-independent 291c6cb04a4SMark Johnston.Xr c 7 292c6cb04a4SMark Johnstonstruct to represent an ICMP header, say 293c6cb04a4SMark Johnston.Ar struct icmp_hdr_dt , 294c6cb04a4SMark Johnstonand define translators from each of the three OS-specific types to 295c6cb04a4SMark Johnston.Ar struct icmp_hdr_dt , 296c6cb04a4SMark Johnstonall in the 297c6cb04a4SMark Johnston.Xr dtrace 1 298c6cb04a4SMark Johnstonlibrary path. 299c6cb04a4SMark JohnstonThen the FreeBSD probe above can be defined with: 300c6cb04a4SMark Johnston.Bd -literal -offset indent 301d7a53a3bSMark JohnstonSDT_PROBE_DEFINE1_XLATE(ip, , , receive, "struct icmp *", 302c6cb04a4SMark Johnston "struct icmp_hdr_dt *"); 303c6cb04a4SMark Johnston.Ed 30498491bfaSMark Johnston.Sh SEE ALSO 305b9d64941SMark Johnston.Xr dtrace 1 , 306*4b4ad3a2SMark Johnston.Xr dtrace_io 4 , 307*4b4ad3a2SMark Johnston.Xr dtrace_ip 4 , 308*4b4ad3a2SMark Johnston.Xr dtrace_proc 4 , 309*4b4ad3a2SMark Johnston.Xr dtrace_sched 4 , 310*4b4ad3a2SMark Johnston.Xr dtrace_tcp 4 , 311*4b4ad3a2SMark Johnston.Xr dtrace_udp 4 31298491bfaSMark Johnston.Sh AUTHORS 31398491bfaSMark Johnston.An -nosplit 31498491bfaSMark JohnstonDTrace and the 31598491bfaSMark Johnston.Nm 31698491bfaSMark Johnstonframework were originally ported to FreeBSD from Solaris by 3178a7314fcSBaptiste Daroussin.An John Birrell Aq Mt jb@FreeBSD.org . 31898491bfaSMark JohnstonThis manual page was written by 3198a7314fcSBaptiste Daroussin.An Mark Johnston Aq Mt markj@FreeBSD.org . 32098491bfaSMark Johnston.Sh BUGS 32198491bfaSMark JohnstonThe 32298491bfaSMark Johnston.Nm 323d85550b2SMark Johnstonmacros allow the module and function names of a probe to be specified as part of 324d85550b2SMark Johnstona probe definition. 325d85550b2SMark JohnstonThe DTrace framework uses the module name of probes to determine which probes 326d85550b2SMark Johnstonshould be destroyed when a kernel module is unloaded, so the module 32798491bfaSMark Johnstonname of a probe should match the name of the module in which its defined. 32898491bfaSMark Johnston.Nm 32998491bfaSMark Johnstonwill set the module name properly if it is left unspecified in the probe 33098491bfaSMark Johnstondefinition; see the 33198491bfaSMark Johnston.Sx EXAMPLES 33298491bfaSMark Johnstonsection. 33398491bfaSMark Johnston.Pp 33498491bfaSMark JohnstonOne of the goals of the original 33598491bfaSMark Johnston.Nm 33698491bfaSMark Johnstonimplementation (and by extension, of FreeBSD's port) is that inactive 33798491bfaSMark Johnston.Nm 33898491bfaSMark Johnstonprobes should have no performance impact. 33998491bfaSMark JohnstonThis is unfortunately not the case; 34098491bfaSMark Johnston.Nm 34198491bfaSMark Johnstontrace points will add a small but non-zero amount of latency to the code 34298491bfaSMark Johnstonin which they are defined. 34398491bfaSMark JohnstonA more sophisticated implementation of the probes will help alleviate this 34498491bfaSMark Johnstonproblem. 345