12e5fc9fdSJohn Baldwin.\" Copyright (c) 2001 John H. Baldwin <jhb@FreeBSD.org> 22e5fc9fdSJohn Baldwin.\" 32e5fc9fdSJohn Baldwin.\" Redistribution and use in source and binary forms, with or without 42e5fc9fdSJohn Baldwin.\" modification, are permitted provided that the following conditions 52e5fc9fdSJohn Baldwin.\" are met: 62e5fc9fdSJohn Baldwin.\" 1. Redistributions of source code must retain the above copyright 72e5fc9fdSJohn Baldwin.\" notice, this list of conditions and the following disclaimer. 82e5fc9fdSJohn Baldwin.\" 2. Redistributions in binary form must reproduce the above copyright 92e5fc9fdSJohn Baldwin.\" notice, this list of conditions and the following disclaimer in the 102e5fc9fdSJohn Baldwin.\" documentation and/or other materials provided with the distribution. 112e5fc9fdSJohn Baldwin.\" 122e5fc9fdSJohn Baldwin.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 132e5fc9fdSJohn Baldwin.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 142e5fc9fdSJohn Baldwin.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 152e5fc9fdSJohn Baldwin.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 162e5fc9fdSJohn Baldwin.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 172e5fc9fdSJohn Baldwin.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 182e5fc9fdSJohn Baldwin.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 192e5fc9fdSJohn Baldwin.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 202e5fc9fdSJohn Baldwin.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 212e5fc9fdSJohn Baldwin.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 222e5fc9fdSJohn Baldwin.\" SUCH DAMAGE. 232e5fc9fdSJohn Baldwin.\" 24*55baee8cSBryan Drewery.Dd March 26, 2021 252e5fc9fdSJohn Baldwin.Dt KTR 4 262e5fc9fdSJohn Baldwin.Os 272e5fc9fdSJohn Baldwin.Sh NAME 282e5fc9fdSJohn Baldwin.Nm ktr 292e5fc9fdSJohn Baldwin.Nd kernel tracing facility 302e5fc9fdSJohn Baldwin.Sh SYNOPSIS 312e5fc9fdSJohn Baldwin.Cd options KTR 321fbff2d1SSuleiman Souhlal.Cd options ALQ 337229a4a2SJeff Roberson.Cd options KTR_ALQ 342e5fc9fdSJohn Baldwin.Cd options KTR_COMPILE=(KTR_LOCK|KTR_INTR|KTR_PROC) 352e5fc9fdSJohn Baldwin.Cd options KTR_CPUMASK=0x3 362e5fc9fdSJohn Baldwin.Cd options KTR_ENTRIES=8192 372e5fc9fdSJohn Baldwin.Cd options KTR_MASK=(KTR_INTR|KTR_PROC) 382e5fc9fdSJohn Baldwin.Cd options KTR_VERBOSE 392e5fc9fdSJohn Baldwin.Sh DESCRIPTION 402e5fc9fdSJohn BaldwinThe 412e5fc9fdSJohn Baldwin.Nm 422e5fc9fdSJohn Baldwinfacility allows kernel events to be logged while the kernel executes so that 432e5fc9fdSJohn Baldwinthey can be examined later when debugging. 442e5fc9fdSJohn BaldwinThe only mandatory option to enable 452e5fc9fdSJohn Baldwin.Nm 468d458e5bSRuslan Ermilovis 478d458e5bSRuslan Ermilov.Dq Li options KTR . 482e5fc9fdSJohn Baldwin.Pp 498d458e5bSRuslan ErmilovThe 508d458e5bSRuslan Ermilov.Dv KTR_ENTRIES 518d458e5bSRuslan Ermilovoption sets the size of the buffer of events. 522e5fc9fdSJohn BaldwinThe size of the buffer in the currently running kernel can be found via the 5340fcb8d3SKevin Losysctl 548d458e5bSRuslan Ermilov.Va debug.ktr.entries . 552e5fc9fdSJohn BaldwinBy default the buffer contains 1024 entries. 562e5fc9fdSJohn Baldwin.Ss Event Masking 572e5fc9fdSJohn BaldwinEvent levels can be enabled or disabled to trim excessive and overly verbose 582e5fc9fdSJohn Baldwinlogging. 598d458e5bSRuslan ErmilovFirst, a mask of events is specified at compile time via the 608d458e5bSRuslan Ermilov.Dv KTR_COMPILE 612e5fc9fdSJohn Baldwinoption to limit which events are actually compiled into the kernel. 622e5fc9fdSJohn BaldwinThe default value for this option is for all events to be enabled. 632e5fc9fdSJohn Baldwin.Pp 642e5fc9fdSJohn BaldwinSecondly, the actual events logged while the kernel runs can be further 652e5fc9fdSJohn Baldwinmasked via the run time event mask. 668d458e5bSRuslan ErmilovThe 678d458e5bSRuslan Ermilov.Dv KTR_MASK 688d458e5bSRuslan Ermilovoption sets the default value of the run time event mask. 692e5fc9fdSJohn BaldwinThe runtime event mask can also be set by the 702e5fc9fdSJohn Baldwin.Xr loader 8 712e5fc9fdSJohn Baldwinvia the 728d458e5bSRuslan Ermilov.Va debug.ktr.mask 732e5fc9fdSJohn Baldwinenvironment variable. 742e5fc9fdSJohn BaldwinIt can also be examined and set after booting via the 758d458e5bSRuslan Ermilov.Va debug.ktr.mask 762e5fc9fdSJohn Baldwinsysctl. 777d6f679aSKevin LoBy default the run time mask is set to block any tracing. 782e5fc9fdSJohn BaldwinThe definitions of the event mask bits can be found in 79*55baee8cSBryan Drewery.In sys/ktr_class.h . 802e5fc9fdSJohn Baldwin.Pp 8154dc0e41SJeff RobersonFurthermore, there is a CPU event mask whose default value can be changed via 828d458e5bSRuslan Ermilovthe 838d458e5bSRuslan Ermilov.Dv KTR_CPUMASK 848d458e5bSRuslan Ermilovoption. 8516a7f9e2SGlen BarberWhen two or more parameters to 8616a7f9e2SGlen Barber.Dv KTR_CPUMASK , 8716a7f9e2SGlen Barberare used, it is important they are not separated by whitespace. 882e5fc9fdSJohn BaldwinA CPU must have the bit corresponding to its logical id set in this bitmask 892e5fc9fdSJohn Baldwinfor events that occur on it to be logged. 902e5fc9fdSJohn BaldwinThis mask can be set by the 912e5fc9fdSJohn Baldwin.Xr loader 8 922e5fc9fdSJohn Baldwinvia the 938d458e5bSRuslan Ermilov.Va debug.ktr.cpumask 942e5fc9fdSJohn Baldwinenvironment variable. 952e5fc9fdSJohn BaldwinIt can also be examined and set after booting via the 968d458e5bSRuslan Ermilov.Va debug.ktr.cpumask 972e5fc9fdSJohn Baldwinsysctl. 9816a7f9e2SGlen BarberBy default, only CPUs specified in 9916a7f9e2SGlen Barber.Dv KTR_CPUMASK 10016a7f9e2SGlen Barberwill log events. 10116a7f9e2SGlen BarberSee 10216a7f9e2SGlen Barber.Pa sys/conf/NOTES 10316a7f9e2SGlen Barberfor more information. 104d962d52aSRuslan Ermilov.Ss Verbose Mode 1052e5fc9fdSJohn BaldwinBy default, events are only logged to the internal buffer for examination 1062e5fc9fdSJohn Baldwinlater, but if the verbose flag is set then they are dumped to the kernel 1072e5fc9fdSJohn Baldwinconsole as well. 1082e5fc9fdSJohn BaldwinThis flag can also be set from the loader via the 1098d458e5bSRuslan Ermilov.Va debug.ktr.verbose 1102e5fc9fdSJohn Baldwinenvironment variable, or it can be examined and set after booting via the 1118d458e5bSRuslan Ermilov.Va debug.ktr.verbose 1122e5fc9fdSJohn Baldwinsysctl. 1132e5fc9fdSJohn BaldwinIf the flag is set to zero, which is the default, then verbose output is 1142e5fc9fdSJohn Baldwindisabled. 1152e5fc9fdSJohn BaldwinIf the flag is set to one, then the contents of the log message and the CPU 1162e5fc9fdSJohn Baldwinnumber are printed to the kernel console. 1172e5fc9fdSJohn BaldwinIf the flag is greater than one, then the filename and line number of the 1182e5fc9fdSJohn Baldwinevent are output to the console in addition to the log message and the CPU 1192e5fc9fdSJohn Baldwinnumber. 1208d458e5bSRuslan ErmilovThe 1218d458e5bSRuslan Ermilov.Dv KTR_VERBOSE 1228d458e5bSRuslan Ermilovoption sets the flag to one. 1232e5fc9fdSJohn Baldwin.Ss Examining the Events 1242e5fc9fdSJohn BaldwinThe KTR buffer can be examined from within 1252e5fc9fdSJohn Baldwin.Xr ddb 4 1262e5fc9fdSJohn Baldwinvia the 127f6b93040SJulian Elischer.Ic show ktr Op Cm /vV 1282e5fc9fdSJohn Baldwincommand. 1292e5fc9fdSJohn BaldwinThis command displays the contents of the trace buffer one page at a time. 1302e5fc9fdSJohn BaldwinAt the 1318d458e5bSRuslan Ermilov.Dq Li --more-- 1322e5fc9fdSJohn Baldwinprompt, the Enter key displays one more entry and prompts again. 1332e5fc9fdSJohn BaldwinThe spacebar displays another page of entries. 1342e5fc9fdSJohn BaldwinAny other key quits. 1352e5fc9fdSJohn BaldwinBy default the timestamp, filename, and line number are not displayed with 1362e5fc9fdSJohn Baldwineach log entry. 1372e5fc9fdSJohn BaldwinIf the 1388d458e5bSRuslan Ermilov.Cm /v 1392e5fc9fdSJohn Baldwinmodifier is specified, then they are displayed in addition to the normal 1402e5fc9fdSJohn Baldwinoutput. 141f6b93040SJulian ElischerIf the 142f6b93040SJulian Elischer.Cm /V 143f6b93040SJulian Elischermodifier is specified, then just the timestamp is displayed in 144f6b93040SJulian Elischeraddition to the normal output. 1452e5fc9fdSJohn BaldwinNote that the events are displayed in reverse chronological order. 1462e5fc9fdSJohn BaldwinThat is, the most recent events are displayed first. 147d962d52aSRuslan Ermilov.Ss Logging ktr to Disk 1487229a4a2SJeff RobersonThe 1497229a4a2SJeff Roberson.Dv KTR_ALQ 150d962d52aSRuslan Ermilovoption can be used to log 151d962d52aSRuslan Ermilov.Nm 152d962d52aSRuslan Ermiloventries to disk for post analysis using the 1537229a4a2SJeff Roberson.Xr ktrdump 8 1547229a4a2SJeff Robersonutility. 1551fbff2d1SSuleiman SouhlalThis option depends on the 1561fbff2d1SSuleiman Souhlal.Dv ALQ 1571fbff2d1SSuleiman Souhlaloption. 1587229a4a2SJeff RobersonDue to the potentially high volume of trace messages the trace mask should be 1597229a4a2SJeff Robersonselected carefully. 1607229a4a2SJeff RobersonThis feature is configured through a group of sysctls. 161d962d52aSRuslan Ermilov.Bl -tag -width ".Va debug.ktr.alq_enable" 162d962d52aSRuslan Ermilov.It Va debug.ktr.alq_file 163d962d52aSRuslan Ermilovdisplays or sets the file that 164d962d52aSRuslan Ermilov.Nm 165d962d52aSRuslan Ermilovwill log to. 166d962d52aSRuslan ErmilovBy default its value is 167d962d52aSRuslan Ermilov.Pa /tmp/ktr.out . 168d962d52aSRuslan ErmilovIf the file name is changed while 169d962d52aSRuslan Ermilov.Nm 170d962d52aSRuslan Ermilovis enabled it will not take effect until 1717229a4a2SJeff Robersonthe next invocation. 172d962d52aSRuslan Ermilov.It Va debug.ktr.alq_enable 173d962d52aSRuslan Ermilovenables logging of 174d962d52aSRuslan Ermilov.Nm 175d962d52aSRuslan Ermiloventries to disk if it is set to one. 176f6b93040SJulian ElischerSetting this to 0 will terminate logging to disk and revert to 177f6b93040SJulian Elischerlogging to the normal ktr ring buffer. 178f6b93040SJulian ElischerData is not sent to the ring buffer while logging to disk. 179d962d52aSRuslan Ermilov.It Va debug.ktr.alq_max 1807229a4a2SJeff Robersonis the maximum number of entries that will be recorded to disk, or 0 for 1817229a4a2SJeff Robersoninfinite. 1827229a4a2SJeff RobersonThis is helpful for limiting the number of particularly high frequency entries 1837229a4a2SJeff Robersonthat are recorded. 184d962d52aSRuslan Ermilov.It Va debug.ktr.alq_depth 1857229a4a2SJeff Robersondetermines the number of entries in the write buffer. 1867229a4a2SJeff RobersonThis is the buffer that holds entries before they are written to disk and 1877229a4a2SJeff Robersondefaults to the value of the 1887229a4a2SJeff Roberson.Dv KTR_ENTRIES 1897229a4a2SJeff Robersonoption. 190d962d52aSRuslan Ermilov.It Va debug.ktr.alq_failed 1917229a4a2SJeff Robersonrecords the number of times we failed to write an entry due to overflowing the 1927229a4a2SJeff Robersonwrite buffer. 193d962d52aSRuslan ErmilovThis may happen if the frequency of the logged 194d962d52aSRuslan Ermilov.Nm 195d962d52aSRuslan Ermilovmessages outpaces the depth 1967229a4a2SJeff Robersonof the queue. 197d962d52aSRuslan Ermilov.It Va debug.ktr.alq_cnt 1987229a4a2SJeff Robersonrecords the number of entries that have currently been written to disk. 199d962d52aSRuslan Ermilov.El 2002e5fc9fdSJohn Baldwin.Sh SEE ALSO 201d962d52aSRuslan Ermilov.Xr ktrdump 8 , 2021fbff2d1SSuleiman Souhlal.Xr alq 9 , 203d962d52aSRuslan Ermilov.Xr ktr 9 2042e5fc9fdSJohn Baldwin.Sh HISTORY 2052e5fc9fdSJohn BaldwinThe KTR kernel tracing facility first appeared in 206a867da6fSJohn Baldwin.Bsx 3.0 207a867da6fSJohn Baldwinand was imported into 2082e5fc9fdSJohn Baldwin.Fx 5.0 . 209