12e5fc9fdSJohn Baldwin.\" Copyright (c) 2001 John H. Baldwin <jhb@FreeBSD.org> 22e5fc9fdSJohn Baldwin.\" All rights reserved. 32e5fc9fdSJohn Baldwin.\" 42e5fc9fdSJohn Baldwin.\" Redistribution and use in source and binary forms, with or without 52e5fc9fdSJohn Baldwin.\" modification, are permitted provided that the following conditions 62e5fc9fdSJohn Baldwin.\" are met: 72e5fc9fdSJohn Baldwin.\" 1. Redistributions of source code must retain the above copyright 82e5fc9fdSJohn Baldwin.\" notice, this list of conditions and the following disclaimer. 92e5fc9fdSJohn Baldwin.\" 2. Redistributions in binary form must reproduce the above copyright 102e5fc9fdSJohn Baldwin.\" notice, this list of conditions and the following disclaimer in the 112e5fc9fdSJohn Baldwin.\" documentation and/or other materials provided with the distribution. 122e5fc9fdSJohn Baldwin.\" 132e5fc9fdSJohn Baldwin.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 142e5fc9fdSJohn Baldwin.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 152e5fc9fdSJohn Baldwin.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 162e5fc9fdSJohn Baldwin.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 172e5fc9fdSJohn Baldwin.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 182e5fc9fdSJohn Baldwin.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 192e5fc9fdSJohn Baldwin.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 202e5fc9fdSJohn Baldwin.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 212e5fc9fdSJohn Baldwin.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 222e5fc9fdSJohn Baldwin.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 232e5fc9fdSJohn Baldwin.\" SUCH DAMAGE. 242e5fc9fdSJohn Baldwin.\" 252e5fc9fdSJohn Baldwin.\" $FreeBSD$ 262e5fc9fdSJohn Baldwin.\" 27*7d6f679aSKevin Lo.Dd October 19, 2012 282e5fc9fdSJohn Baldwin.Dt KTR 4 292e5fc9fdSJohn Baldwin.Os 302e5fc9fdSJohn Baldwin.Sh NAME 312e5fc9fdSJohn Baldwin.Nm ktr 322e5fc9fdSJohn Baldwin.Nd kernel tracing facility 332e5fc9fdSJohn Baldwin.Sh SYNOPSIS 342e5fc9fdSJohn Baldwin.Cd options KTR 351fbff2d1SSuleiman Souhlal.Cd options ALQ 367229a4a2SJeff Roberson.Cd options KTR_ALQ 372e5fc9fdSJohn Baldwin.Cd options KTR_COMPILE=(KTR_LOCK|KTR_INTR|KTR_PROC) 382e5fc9fdSJohn Baldwin.Cd options KTR_CPUMASK=0x3 392e5fc9fdSJohn Baldwin.Cd options KTR_ENTRIES=8192 402e5fc9fdSJohn Baldwin.Cd options KTR_MASK=(KTR_INTR|KTR_PROC) 412e5fc9fdSJohn Baldwin.Cd options KTR_VERBOSE 422e5fc9fdSJohn Baldwin.Sh DESCRIPTION 432e5fc9fdSJohn BaldwinThe 442e5fc9fdSJohn Baldwin.Nm 452e5fc9fdSJohn Baldwinfacility allows kernel events to be logged while the kernel executes so that 462e5fc9fdSJohn Baldwinthey can be examined later when debugging. 472e5fc9fdSJohn BaldwinThe only mandatory option to enable 482e5fc9fdSJohn Baldwin.Nm 498d458e5bSRuslan Ermilovis 508d458e5bSRuslan Ermilov.Dq Li options KTR . 512e5fc9fdSJohn Baldwin.Pp 528d458e5bSRuslan ErmilovThe 538d458e5bSRuslan Ermilov.Dv KTR_ENTRIES 548d458e5bSRuslan Ermilovoption sets the size of the buffer of events. 552e5fc9fdSJohn BaldwinThe size of the buffer in the currently running kernel can be found via the 562e5fc9fdSJohn Baldwinread-only sysctl 578d458e5bSRuslan Ermilov.Va debug.ktr.entries . 582e5fc9fdSJohn BaldwinBy default the buffer contains 1024 entries. 592e5fc9fdSJohn Baldwin.Ss Event Masking 602e5fc9fdSJohn BaldwinEvent levels can be enabled or disabled to trim excessive and overly verbose 612e5fc9fdSJohn Baldwinlogging. 628d458e5bSRuslan ErmilovFirst, a mask of events is specified at compile time via the 638d458e5bSRuslan Ermilov.Dv KTR_COMPILE 642e5fc9fdSJohn Baldwinoption to limit which events are actually compiled into the kernel. 652e5fc9fdSJohn BaldwinThe default value for this option is for all events to be enabled. 662e5fc9fdSJohn Baldwin.Pp 672e5fc9fdSJohn BaldwinSecondly, the actual events logged while the kernel runs can be further 682e5fc9fdSJohn Baldwinmasked via the run time event mask. 698d458e5bSRuslan ErmilovThe 708d458e5bSRuslan Ermilov.Dv KTR_MASK 718d458e5bSRuslan Ermilovoption sets the default value of the run time event mask. 722e5fc9fdSJohn BaldwinThe runtime event mask can also be set by the 732e5fc9fdSJohn Baldwin.Xr loader 8 742e5fc9fdSJohn Baldwinvia the 758d458e5bSRuslan Ermilov.Va debug.ktr.mask 762e5fc9fdSJohn Baldwinenvironment variable. 772e5fc9fdSJohn BaldwinIt can also be examined and set after booting via the 788d458e5bSRuslan Ermilov.Va debug.ktr.mask 792e5fc9fdSJohn Baldwinsysctl. 80*7d6f679aSKevin LoBy default the run time mask is set to block any tracing. 812e5fc9fdSJohn BaldwinThe definitions of the event mask bits can be found in 82fe08efe6SRuslan Ermilov.In sys/ktr.h . 832e5fc9fdSJohn Baldwin.Pp 8454dc0e41SJeff RobersonFurthermore, there is a CPU event mask whose default value can be changed via 858d458e5bSRuslan Ermilovthe 868d458e5bSRuslan Ermilov.Dv KTR_CPUMASK 878d458e5bSRuslan Ermilovoption. 882e5fc9fdSJohn BaldwinA CPU must have the bit corresponding to its logical id set in this bitmask 892e5fc9fdSJohn Baldwinfor events that occur on it to be logged. 902e5fc9fdSJohn BaldwinThis mask can be set by the 912e5fc9fdSJohn Baldwin.Xr loader 8 922e5fc9fdSJohn Baldwinvia the 938d458e5bSRuslan Ermilov.Va debug.ktr.cpumask 942e5fc9fdSJohn Baldwinenvironment variable. 952e5fc9fdSJohn BaldwinIt can also be examined and set after booting via the 968d458e5bSRuslan Ermilov.Va debug.ktr.cpumask 972e5fc9fdSJohn Baldwinsysctl. 982e5fc9fdSJohn BaldwinBy default events on all CPUs are enabled. 99d962d52aSRuslan Ermilov.Ss Verbose Mode 1002e5fc9fdSJohn BaldwinBy default, events are only logged to the internal buffer for examination 1012e5fc9fdSJohn Baldwinlater, but if the verbose flag is set then they are dumped to the kernel 1022e5fc9fdSJohn Baldwinconsole as well. 1032e5fc9fdSJohn BaldwinThis flag can also be set from the loader via the 1048d458e5bSRuslan Ermilov.Va debug.ktr.verbose 1052e5fc9fdSJohn Baldwinenvironment variable, or it can be examined and set after booting via the 1068d458e5bSRuslan Ermilov.Va debug.ktr.verbose 1072e5fc9fdSJohn Baldwinsysctl. 1082e5fc9fdSJohn BaldwinIf the flag is set to zero, which is the default, then verbose output is 1092e5fc9fdSJohn Baldwindisabled. 1102e5fc9fdSJohn BaldwinIf the flag is set to one, then the contents of the log message and the CPU 1112e5fc9fdSJohn Baldwinnumber are printed to the kernel console. 1122e5fc9fdSJohn BaldwinIf the flag is greater than one, then the filename and line number of the 1132e5fc9fdSJohn Baldwinevent are output to the console in addition to the log message and the CPU 1142e5fc9fdSJohn Baldwinnumber. 1158d458e5bSRuslan ErmilovThe 1168d458e5bSRuslan Ermilov.Dv KTR_VERBOSE 1178d458e5bSRuslan Ermilovoption sets the flag to one. 1182e5fc9fdSJohn Baldwin.Ss Examining the Events 1192e5fc9fdSJohn BaldwinThe KTR buffer can be examined from within 1202e5fc9fdSJohn Baldwin.Xr ddb 4 1212e5fc9fdSJohn Baldwinvia the 122f6b93040SJulian Elischer.Ic show ktr Op Cm /vV 1232e5fc9fdSJohn Baldwincommand. 1242e5fc9fdSJohn BaldwinThis command displays the contents of the trace buffer one page at a time. 1252e5fc9fdSJohn BaldwinAt the 1268d458e5bSRuslan Ermilov.Dq Li --more-- 1272e5fc9fdSJohn Baldwinprompt, the Enter key displays one more entry and prompts again. 1282e5fc9fdSJohn BaldwinThe spacebar displays another page of entries. 1292e5fc9fdSJohn BaldwinAny other key quits. 1302e5fc9fdSJohn BaldwinBy default the timestamp, filename, and line number are not displayed with 1312e5fc9fdSJohn Baldwineach log entry. 1322e5fc9fdSJohn BaldwinIf the 1338d458e5bSRuslan Ermilov.Cm /v 1342e5fc9fdSJohn Baldwinmodifier is specified, then they are displayed in addition to the normal 1352e5fc9fdSJohn Baldwinoutput. 136f6b93040SJulian ElischerIf the 137f6b93040SJulian Elischer.Cm /V 138f6b93040SJulian Elischermodifier is specified, then just the timestamp is displayed in 139f6b93040SJulian Elischeraddition to the normal output. 1402e5fc9fdSJohn BaldwinNote that the events are displayed in reverse chronological order. 1412e5fc9fdSJohn BaldwinThat is, the most recent events are displayed first. 142d962d52aSRuslan Ermilov.Ss Logging ktr to Disk 1437229a4a2SJeff RobersonThe 1447229a4a2SJeff Roberson.Dv KTR_ALQ 145d962d52aSRuslan Ermilovoption can be used to log 146d962d52aSRuslan Ermilov.Nm 147d962d52aSRuslan Ermiloventries to disk for post analysis using the 1487229a4a2SJeff Roberson.Xr ktrdump 8 1497229a4a2SJeff Robersonutility. 1501fbff2d1SSuleiman SouhlalThis option depends on the 1511fbff2d1SSuleiman Souhlal.Dv ALQ 1521fbff2d1SSuleiman Souhlaloption. 1537229a4a2SJeff RobersonDue to the potentially high volume of trace messages the trace mask should be 1547229a4a2SJeff Robersonselected carefully. 1557229a4a2SJeff RobersonThis feature is configured through a group of sysctls. 156d962d52aSRuslan Ermilov.Bl -tag -width ".Va debug.ktr.alq_enable" 157d962d52aSRuslan Ermilov.It Va debug.ktr.alq_file 158d962d52aSRuslan Ermilovdisplays or sets the file that 159d962d52aSRuslan Ermilov.Nm 160d962d52aSRuslan Ermilovwill log to. 161d962d52aSRuslan ErmilovBy default its value is 162d962d52aSRuslan Ermilov.Pa /tmp/ktr.out . 163d962d52aSRuslan ErmilovIf the file name is changed while 164d962d52aSRuslan Ermilov.Nm 165d962d52aSRuslan Ermilovis enabled it will not take effect until 1667229a4a2SJeff Robersonthe next invocation. 167d962d52aSRuslan Ermilov.It Va debug.ktr.alq_enable 168d962d52aSRuslan Ermilovenables logging of 169d962d52aSRuslan Ermilov.Nm 170d962d52aSRuslan Ermiloventries to disk if it is set to one. 171f6b93040SJulian ElischerSetting this to 0 will terminate logging to disk and revert to 172f6b93040SJulian Elischerlogging to the normal ktr ring buffer. 173f6b93040SJulian ElischerData is not sent to the ring buffer while logging to disk. 174d962d52aSRuslan Ermilov.It Va debug.ktr.alq_max 1757229a4a2SJeff Robersonis the maximum number of entries that will be recorded to disk, or 0 for 1767229a4a2SJeff Robersoninfinite. 1777229a4a2SJeff RobersonThis is helpful for limiting the number of particularly high frequency entries 1787229a4a2SJeff Robersonthat are recorded. 179d962d52aSRuslan Ermilov.It Va debug.ktr.alq_depth 1807229a4a2SJeff Robersondetermines the number of entries in the write buffer. 1817229a4a2SJeff RobersonThis is the buffer that holds entries before they are written to disk and 1827229a4a2SJeff Robersondefaults to the value of the 1837229a4a2SJeff Roberson.Dv KTR_ENTRIES 1847229a4a2SJeff Robersonoption. 185d962d52aSRuslan Ermilov.It Va debug.ktr.alq_failed 1867229a4a2SJeff Robersonrecords the number of times we failed to write an entry due to overflowing the 1877229a4a2SJeff Robersonwrite buffer. 188d962d52aSRuslan ErmilovThis may happen if the frequency of the logged 189d962d52aSRuslan Ermilov.Nm 190d962d52aSRuslan Ermilovmessages outpaces the depth 1917229a4a2SJeff Robersonof the queue. 192d962d52aSRuslan Ermilov.It Va debug.ktr.alq_cnt 1937229a4a2SJeff Robersonrecords the number of entries that have currently been written to disk. 194d962d52aSRuslan Ermilov.El 1952e5fc9fdSJohn Baldwin.Sh SEE ALSO 196d962d52aSRuslan Ermilov.Xr ktrdump 8 , 1971fbff2d1SSuleiman Souhlal.Xr alq 9 , 198d962d52aSRuslan Ermilov.Xr ktr 9 1992e5fc9fdSJohn Baldwin.Sh HISTORY 2002e5fc9fdSJohn BaldwinThe KTR kernel tracing facility first appeared in 201a867da6fSJohn Baldwin.Bsx 3.0 202a867da6fSJohn Baldwinand was imported into 2032e5fc9fdSJohn Baldwin.Fx 5.0 . 204