1.\" Copyright (c) 2001 John H. Baldwin <jhb@FreeBSD.org> 2.\" 3.\" Redistribution and use in source and binary forms, with or without 4.\" modification, are permitted provided that the following conditions 5.\" are met: 6.\" 1. Redistributions of source code must retain the above copyright 7.\" notice, this list of conditions and the following disclaimer. 8.\" 2. Redistributions in binary form must reproduce the above copyright 9.\" notice, this list of conditions and the following disclaimer in the 10.\" documentation and/or other materials provided with the distribution. 11.\" 12.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 13.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 14.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 15.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 16.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 17.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 18.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 19.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 20.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 21.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 22.\" SUCH DAMAGE. 23.\" 24.\" $FreeBSD$ 25.\" 26.Dd March 26, 2021 27.Dt KTR 4 28.Os 29.Sh NAME 30.Nm ktr 31.Nd kernel tracing facility 32.Sh SYNOPSIS 33.Cd options KTR 34.Cd options ALQ 35.Cd options KTR_ALQ 36.Cd options KTR_COMPILE=(KTR_LOCK|KTR_INTR|KTR_PROC) 37.Cd options KTR_CPUMASK=0x3 38.Cd options KTR_ENTRIES=8192 39.Cd options KTR_MASK=(KTR_INTR|KTR_PROC) 40.Cd options KTR_VERBOSE 41.Sh DESCRIPTION 42The 43.Nm 44facility allows kernel events to be logged while the kernel executes so that 45they can be examined later when debugging. 46The only mandatory option to enable 47.Nm 48is 49.Dq Li options KTR . 50.Pp 51The 52.Dv KTR_ENTRIES 53option sets the size of the buffer of events. 54The size of the buffer in the currently running kernel can be found via the 55sysctl 56.Va debug.ktr.entries . 57By default the buffer contains 1024 entries. 58.Ss Event Masking 59Event levels can be enabled or disabled to trim excessive and overly verbose 60logging. 61First, a mask of events is specified at compile time via the 62.Dv KTR_COMPILE 63option to limit which events are actually compiled into the kernel. 64The default value for this option is for all events to be enabled. 65.Pp 66Secondly, the actual events logged while the kernel runs can be further 67masked via the run time event mask. 68The 69.Dv KTR_MASK 70option sets the default value of the run time event mask. 71The runtime event mask can also be set by the 72.Xr loader 8 73via the 74.Va debug.ktr.mask 75environment variable. 76It can also be examined and set after booting via the 77.Va debug.ktr.mask 78sysctl. 79By default the run time mask is set to block any tracing. 80The definitions of the event mask bits can be found in 81.In sys/ktr_class.h . 82.Pp 83Furthermore, there is a CPU event mask whose default value can be changed via 84the 85.Dv KTR_CPUMASK 86option. 87When two or more parameters to 88.Dv KTR_CPUMASK , 89are used, it is important they are not separated by whitespace. 90A CPU must have the bit corresponding to its logical id set in this bitmask 91for events that occur on it to be logged. 92This mask can be set by the 93.Xr loader 8 94via the 95.Va debug.ktr.cpumask 96environment variable. 97It can also be examined and set after booting via the 98.Va debug.ktr.cpumask 99sysctl. 100By default, only CPUs specified in 101.Dv KTR_CPUMASK 102will log events. 103See 104.Pa sys/conf/NOTES 105for more information. 106.Ss Verbose Mode 107By default, events are only logged to the internal buffer for examination 108later, but if the verbose flag is set then they are dumped to the kernel 109console as well. 110This flag can also be set from the loader via the 111.Va debug.ktr.verbose 112environment variable, or it can be examined and set after booting via the 113.Va debug.ktr.verbose 114sysctl. 115If the flag is set to zero, which is the default, then verbose output is 116disabled. 117If the flag is set to one, then the contents of the log message and the CPU 118number are printed to the kernel console. 119If the flag is greater than one, then the filename and line number of the 120event are output to the console in addition to the log message and the CPU 121number. 122The 123.Dv KTR_VERBOSE 124option sets the flag to one. 125.Ss Examining the Events 126The KTR buffer can be examined from within 127.Xr ddb 4 128via the 129.Ic show ktr Op Cm /vV 130command. 131This command displays the contents of the trace buffer one page at a time. 132At the 133.Dq Li --more-- 134prompt, the Enter key displays one more entry and prompts again. 135The spacebar displays another page of entries. 136Any other key quits. 137By default the timestamp, filename, and line number are not displayed with 138each log entry. 139If the 140.Cm /v 141modifier is specified, then they are displayed in addition to the normal 142output. 143If the 144.Cm /V 145modifier is specified, then just the timestamp is displayed in 146addition to the normal output. 147Note that the events are displayed in reverse chronological order. 148That is, the most recent events are displayed first. 149.Ss Logging ktr to Disk 150The 151.Dv KTR_ALQ 152option can be used to log 153.Nm 154entries to disk for post analysis using the 155.Xr ktrdump 8 156utility. 157This option depends on the 158.Dv ALQ 159option. 160Due to the potentially high volume of trace messages the trace mask should be 161selected carefully. 162This feature is configured through a group of sysctls. 163.Bl -tag -width ".Va debug.ktr.alq_enable" 164.It Va debug.ktr.alq_file 165displays or sets the file that 166.Nm 167will log to. 168By default its value is 169.Pa /tmp/ktr.out . 170If the file name is changed while 171.Nm 172is enabled it will not take effect until 173the next invocation. 174.It Va debug.ktr.alq_enable 175enables logging of 176.Nm 177entries to disk if it is set to one. 178Setting this to 0 will terminate logging to disk and revert to 179logging to the normal ktr ring buffer. 180Data is not sent to the ring buffer while logging to disk. 181.It Va debug.ktr.alq_max 182is the maximum number of entries that will be recorded to disk, or 0 for 183infinite. 184This is helpful for limiting the number of particularly high frequency entries 185that are recorded. 186.It Va debug.ktr.alq_depth 187determines the number of entries in the write buffer. 188This is the buffer that holds entries before they are written to disk and 189defaults to the value of the 190.Dv KTR_ENTRIES 191option. 192.It Va debug.ktr.alq_failed 193records the number of times we failed to write an entry due to overflowing the 194write buffer. 195This may happen if the frequency of the logged 196.Nm 197messages outpaces the depth 198of the queue. 199.It Va debug.ktr.alq_cnt 200records the number of entries that have currently been written to disk. 201.El 202.Sh SEE ALSO 203.Xr ktrdump 8 , 204.Xr alq 9 , 205.Xr ktr 9 206.Sh HISTORY 207The KTR kernel tracing facility first appeared in 208.Bsx 3.0 209and was imported into 210.Fx 5.0 . 211