1.\" Copyright (c) 2001 John H. Baldwin <jhb@FreeBSD.org> 2.\" 3.\" Redistribution and use in source and binary forms, with or without 4.\" modification, are permitted provided that the following conditions 5.\" are met: 6.\" 1. Redistributions of source code must retain the above copyright 7.\" notice, this list of conditions and the following disclaimer. 8.\" 2. Redistributions in binary form must reproduce the above copyright 9.\" notice, this list of conditions and the following disclaimer in the 10.\" documentation and/or other materials provided with the distribution. 11.\" 12.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 13.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 14.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 15.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 16.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 17.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 18.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 19.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 20.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 21.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 22.\" SUCH DAMAGE. 23.\" 24.Dd March 26, 2021 25.Dt KTR 4 26.Os 27.Sh NAME 28.Nm ktr 29.Nd kernel tracing facility 30.Sh SYNOPSIS 31.Cd options KTR 32.Cd options ALQ 33.Cd options KTR_ALQ 34.Cd options KTR_COMPILE=(KTR_LOCK|KTR_INTR|KTR_PROC) 35.Cd options KTR_CPUMASK=0x3 36.Cd options KTR_ENTRIES=8192 37.Cd options KTR_MASK=(KTR_INTR|KTR_PROC) 38.Cd options KTR_VERBOSE 39.Sh DESCRIPTION 40The 41.Nm 42facility allows kernel events to be logged while the kernel executes so that 43they can be examined later when debugging. 44The only mandatory option to enable 45.Nm 46is 47.Dq Li options KTR . 48.Pp 49The 50.Dv KTR_ENTRIES 51option sets the size of the buffer of events. 52The size of the buffer in the currently running kernel can be found via the 53sysctl 54.Va debug.ktr.entries . 55By default the buffer contains 1024 entries. 56.Ss Event Masking 57Event levels can be enabled or disabled to trim excessive and overly verbose 58logging. 59First, a mask of events is specified at compile time via the 60.Dv KTR_COMPILE 61option to limit which events are actually compiled into the kernel. 62The default value for this option is for all events to be enabled. 63.Pp 64Secondly, the actual events logged while the kernel runs can be further 65masked via the run time event mask. 66The 67.Dv KTR_MASK 68option sets the default value of the run time event mask. 69The runtime event mask can also be set by the 70.Xr loader 8 71via the 72.Va debug.ktr.mask 73environment variable. 74It can also be examined and set after booting via the 75.Va debug.ktr.mask 76sysctl. 77By default the run time mask is set to block any tracing. 78The definitions of the event mask bits can be found in 79.In sys/ktr_class.h . 80.Pp 81Furthermore, there is a CPU event mask whose default value can be changed via 82the 83.Dv KTR_CPUMASK 84option. 85When two or more parameters to 86.Dv KTR_CPUMASK , 87are used, it is important they are not separated by whitespace. 88A CPU must have the bit corresponding to its logical id set in this bitmask 89for events that occur on it to be logged. 90This mask can be set by the 91.Xr loader 8 92via the 93.Va debug.ktr.cpumask 94environment variable. 95It can also be examined and set after booting via the 96.Va debug.ktr.cpumask 97sysctl. 98By default, only CPUs specified in 99.Dv KTR_CPUMASK 100will log events. 101See 102.Pa sys/conf/NOTES 103for more information. 104.Ss Verbose Mode 105By default, events are only logged to the internal buffer for examination 106later, but if the verbose flag is set then they are dumped to the kernel 107console as well. 108This flag can also be set from the loader via the 109.Va debug.ktr.verbose 110environment variable, or it can be examined and set after booting via the 111.Va debug.ktr.verbose 112sysctl. 113If the flag is set to zero, which is the default, then verbose output is 114disabled. 115If the flag is set to one, then the contents of the log message and the CPU 116number are printed to the kernel console. 117If the flag is greater than one, then the filename and line number of the 118event are output to the console in addition to the log message and the CPU 119number. 120The 121.Dv KTR_VERBOSE 122option sets the flag to one. 123.Ss Examining the Events 124The KTR buffer can be examined from within 125.Xr ddb 4 126via the 127.Ic show ktr Op Cm /vV 128command. 129This command displays the contents of the trace buffer one page at a time. 130At the 131.Dq Li --more-- 132prompt, the Enter key displays one more entry and prompts again. 133The spacebar displays another page of entries. 134Any other key quits. 135By default the timestamp, filename, and line number are not displayed with 136each log entry. 137If the 138.Cm /v 139modifier is specified, then they are displayed in addition to the normal 140output. 141If the 142.Cm /V 143modifier is specified, then just the timestamp is displayed in 144addition to the normal output. 145Note that the events are displayed in reverse chronological order. 146That is, the most recent events are displayed first. 147.Ss Logging ktr to Disk 148The 149.Dv KTR_ALQ 150option can be used to log 151.Nm 152entries to disk for post analysis using the 153.Xr ktrdump 8 154utility. 155This option depends on the 156.Dv ALQ 157option. 158Due to the potentially high volume of trace messages the trace mask should be 159selected carefully. 160This feature is configured through a group of sysctls. 161.Bl -tag -width ".Va debug.ktr.alq_enable" 162.It Va debug.ktr.alq_file 163displays or sets the file that 164.Nm 165will log to. 166By default its value is 167.Pa /tmp/ktr.out . 168If the file name is changed while 169.Nm 170is enabled it will not take effect until 171the next invocation. 172.It Va debug.ktr.alq_enable 173enables logging of 174.Nm 175entries to disk if it is set to one. 176Setting this to 0 will terminate logging to disk and revert to 177logging to the normal ktr ring buffer. 178Data is not sent to the ring buffer while logging to disk. 179.It Va debug.ktr.alq_max 180is the maximum number of entries that will be recorded to disk, or 0 for 181infinite. 182This is helpful for limiting the number of particularly high frequency entries 183that are recorded. 184.It Va debug.ktr.alq_depth 185determines the number of entries in the write buffer. 186This is the buffer that holds entries before they are written to disk and 187defaults to the value of the 188.Dv KTR_ENTRIES 189option. 190.It Va debug.ktr.alq_failed 191records the number of times we failed to write an entry due to overflowing the 192write buffer. 193This may happen if the frequency of the logged 194.Nm 195messages outpaces the depth 196of the queue. 197.It Va debug.ktr.alq_cnt 198records the number of entries that have currently been written to disk. 199.El 200.Sh SEE ALSO 201.Xr ktrdump 8 , 202.Xr alq 9 , 203.Xr ktr 9 204.Sh HISTORY 205The KTR kernel tracing facility first appeared in 206.Bsx 3.0 207and was imported into 208.Fx 5.0 . 209