xref: /freebsd/share/man/man4/ddb.4 (revision 5ca8e32633c4ffbbcd6762e5888b6a4ba0708c6c)
1.\"
2.\" Mach Operating System
3.\" Copyright (c) 1991,1990 Carnegie Mellon University
4.\" Copyright (c) 2007 Robert N. M. Watson
5.\" All Rights Reserved.
6.\"
7.\" Permission to use, copy, modify and distribute this software and its
8.\" documentation is hereby granted, provided that both the copyright
9.\" notice and this permission notice appear in all copies of the
10.\" software, derivative works or modified versions, and any portions
11.\" thereof, and that both notices appear in supporting documentation.
12.\"
13.\" CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS"
14.\" CONDITION.  CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND FOR
15.\" ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
16.\"
17.\" Carnegie Mellon requests users of this software to return to
18.\"
19.\"  Software Distribution Coordinator  or  Software.Distribution@CS.CMU.EDU
20.\"  School of Computer Science
21.\"  Carnegie Mellon University
22.\"  Pittsburgh PA 15213-3890
23.\"
24.\" any improvements or extensions that they make and grant Carnegie Mellon
25.\" the rights to redistribute these changes.
26.\"
27.Dd November 10, 2022
28.Dt DDB 4
29.Os
30.Sh NAME
31.Nm ddb
32.Nd interactive kernel debugger
33.Sh SYNOPSIS
34In order to enable kernel debugging facilities include:
35.Bd -ragged -offset indent
36.Cd options KDB
37.Cd options DDB
38.Ed
39.Pp
40To prevent activation of the debugger on kernel
41.Xr panic 9 :
42.Bd -ragged -offset indent
43.Cd options KDB_UNATTENDED
44.Ed
45.Pp
46In order to print a stack trace of the current thread on the console
47for a panic:
48.Bd -ragged -offset indent
49.Cd options KDB_TRACE
50.Ed
51.Pp
52To print the numerical value of symbols in addition to the symbolic
53representation, define:
54.Bd -ragged -offset indent
55.Cd options DDB_NUMSYM
56.Ed
57.Pp
58To enable the
59.Xr gdb 4
60backend, so that remote debugging with
61.Xr kgdb 1 Pq Pa ports/devel/gdb
62is possible, include:
63.Bd -ragged -offset indent
64.Cd options GDB
65.Ed
66.Sh DESCRIPTION
67The
68.Nm
69kernel debugger is an interactive debugger with a syntax inspired by
70.Xr gdb 1 Pq Pa ports/devel/gdb .
71If linked into the running kernel,
72it can be invoked locally with the
73.Ql debug
74.Xr keymap 5
75action, usually mapped to Ctrl+Alt+Esc, or by setting the
76.Va debug.kdb.enter
77sysctl to 1.
78The debugger is also invoked on kernel
79.Xr panic 9
80if the
81.Va debug.debugger_on_panic
82.Xr sysctl 8
83MIB variable is set non-zero,
84which is the default
85unless the
86.Dv KDB_UNATTENDED
87option is specified.
88Similarly, if the
89.Va debug.debugger_on_recursive_panic
90variable is set to
91.Dv 1 ,
92then the debugger will be invoked on a recursive kernel panic.
93This variable has a default value of
94.Dv 0 ,
95and has no effect if
96.Va debug.debugger_on_panic
97is already set non-zero.
98.Pp
99The current location is called
100.Va dot .
101The
102.Va dot
103is displayed with
104a hexadecimal format at a prompt.
105The commands
106.Ic examine
107and
108.Ic write
109update
110.Va dot
111to the address of the last line
112examined or the last location modified, and set
113.Va next
114to the address of
115the next location to be examined or changed.
116Other commands do not change
117.Va dot ,
118and set
119.Va next
120to be the same as
121.Va dot .
122.Pp
123The general command syntax is:
124.Ar command Ns Op Li / Ns Ar modifier
125.Oo Ar addr Oc Ns Op , Ns Ar count
126.Pp
127A blank line repeats the previous command from the address
128.Va next
129with
130count 1 and no modifiers.
131Specifying
132.Ar addr
133sets
134.Va dot
135to the address.
136Omitting
137.Ar addr
138uses
139.Va dot .
140A missing
141.Ar count
142is taken
143to be 1 for printing commands or infinity for stack traces.
144A
145.Ar count
146of -1 is equivalent to a missing
147.Ar count .
148Options that are supplied but not supported by the given
149.Ar command
150are usually ignored.
151.Pp
152The
153.Nm
154debugger has a pager feature (like the
155.Xr more 1
156command)
157for the output.
158If an output line exceeds the number set in the
159.Va lines
160variable, it displays
161.Dq Li --More--
162and waits for a response.
163The valid responses for it are:
164.Pp
165.Bl -tag -compact -width ".Li SPC"
166.It Li SPC
167one more page
168.It Li RET
169one more line
170.It Li q
171abort the current command, and return to the command input mode
172.El
173.Pp
174Finally,
175.Nm
176provides a small (currently 10 items) command history, and offers
177simple
178.Nm emacs Ns -style
179command line editing capabilities.
180In addition to
181the
182.Nm emacs
183control keys, the usual ANSI arrow keys may be used to browse through the
184history buffer, and move the cursor within the current line.
185.Sh COMMANDS
186.Ss COMMON DEBUGGER COMMANDS
187.Bl -tag -width indent -compact
188.It Ic help
189Print a short summary of the available commands and command
190abbreviations.
191.Pp
192.It Xo
193.Ic examine Ns Op Li / Ns Cm AISabcdghilmorsuxz ...
194.Oo Ar addr Oc Ns Op , Ns Ar count
195.Xc
196.It Xo
197.Ic x       Ns Op Li / Ns Cm AISabcdghilmorsuxz ...
198.Oo Ar addr Oc Ns Op , Ns Ar count
199.Xc
200Display the addressed locations according to the formats in the modifier.
201Multiple modifier formats display multiple locations.
202If no format is specified, the last format specified for this command
203is used.
204.Pp
205The format characters are:
206.Bl -tag -compact -width indent
207.It Cm b
208look at by bytes (8 bits)
209.It Cm h
210look at by half words (16 bits)
211.It Cm l
212look at by long words (32 bits)
213.It Cm g
214look at by quad words (64 bits)
215.It Cm a
216print the location being displayed
217.It Cm A
218print the location with a line number if possible
219.It Cm x
220display in unsigned hex
221.It Cm z
222display in signed hex
223.It Cm o
224display in unsigned octal
225.It Cm d
226display in signed decimal
227.It Cm u
228display in unsigned decimal
229.It Cm r
230display in current radix, signed
231.It Cm c
232display low 8 bits as a character.
233Non-printing characters are displayed as an octal escape code (e.g.,
234.Ql \e000 ) .
235.It Cm s
236display the null-terminated string at the location.
237Non-printing characters are displayed as octal escapes.
238.It Cm m
239display in unsigned hex with character dump at the end of each line.
240The location is also displayed in hex at the beginning of each line.
241.It Cm i
242display as a disassembled instruction
243.It Cm I
244display as a disassembled instruction with possible alternate formats
245depending on the machine.
246On i386, this selects the alternate format for the instruction decoding
247(16 bits in a 32-bit code segment and vice versa).
248.It Cm S
249display a symbol name for the pointer stored at the address
250.El
251.Pp
252.It Ic xf
253Examine forward:
254execute an
255.Ic examine
256command with the last specified parameters to it
257except that the next address displayed by it is used as the start address.
258.Pp
259.It Ic xb
260Examine backward:
261execute an
262.Ic examine
263command with the last specified parameters to it
264except that the last start address subtracted by the size displayed by it
265is used as the start address.
266.Pp
267.It Ic print Ns Op Li / Ns Cm acdoruxz
268.It Ic p Ns Op Li / Ns Cm acdoruxz
269Print
270.Ar addr Ns s
271according to the modifier character (as described above for
272.Cm examine ) .
273Valid formats are:
274.Cm a , x , z , o , d , u , r ,
275and
276.Cm c .
277If no modifier is specified, the last one specified to it is used.
278The argument
279.Ar addr
280can be a string, in which case it is printed as it is.
281For example:
282.Bd -literal -offset indent
283print/x "eax = " $eax "\enecx = " $ecx "\en"
284.Ed
285.Pp
286will print like:
287.Bd -literal -offset indent
288eax = xxxxxx
289ecx = yyyyyy
290.Ed
291.Pp
292.It Ic pprint Ns Oo Li / Ns Cm d depth Oc Oo Ar name Oc
293Pretty-print symbol specified by
294.Ar name
295using CTF debugging data. Works for all symbols exported by the kernel and loaded kernel modules.
296.Pp
297If the
298.Cm d
299modifier has been specified, contents of structs nested up to
300.Ar depth
301levels deep will also be included in the output.
302.Ed
303.Pp
304.It Ic pprint struct Ns Oo Li / Ns Cm d depth Ic Oc Oo Ar name Oc Ns Op Ns Ar addr
305Print memory at
306.Ar addr
307as struct
308.Ar name Ns .
309Works for all structs defined by the kernel and loaded kernel modules.
310.Pp
311If the
312.Cm d
313modifier has been specified, contents of structs nested up to
314.Ar depth
315levels deep will also be included in the output.
316.Ed
317.Pp
318.It Xo
319.Ic write Ns Op Li / Ns Cm bhl
320.Ar addr expr1 Op Ar expr2 ...
321.Xc
322.It Xo
323.Ic w Ns Op Li / Ns Cm bhl
324.Ar addr expr1 Op Ar expr2 ...
325.Xc
326Write the expressions specified after
327.Ar addr
328on the command line at succeeding locations starting with
329.Ar addr .
330The write unit size can be specified in the modifier with a letter
331.Cm b
332(byte),
333.Cm h
334(half word) or
335.Cm l
336(long word) respectively.
337If omitted,
338long word is assumed.
339.Pp
340.Sy Warning :
341since there is no delimiter between expressions, strange
342things may happen.
343It is best to enclose each expression in parentheses.
344.Pp
345.It Ic set Li $ Ns Ar variable Oo Li = Oc Ar expr
346Set the named variable or register with the value of
347.Ar expr .
348Valid variable names are described below.
349.Pp
350.It Ic break Ns Oo Li / Ns Cm u Oc Oo Ar addr Oc Ns Op , Ns Ar count
351.It Ic b     Ns Oo Li / Ns Cm u Oc Oo Ar addr Oc Ns Op , Ns Ar count
352Set a break point at
353.Ar addr .
354If
355.Ar count
356is supplied, the
357.Ic continue
358command will not stop at this break point on the first
359.Ar count
360\- 1 times that it is hit.
361If the break point is set, a break point number is
362printed with
363.Ql # .
364This number can be used in deleting the break point
365or adding conditions to it.
366.Pp
367If the
368.Cm u
369modifier is specified, this command sets a break point in user
370address space.
371Without the
372.Cm u
373option, the address is considered to be in the kernel
374space, and a wrong space address is rejected with an error message.
375This modifier can be used only if it is supported by machine dependent
376routines.
377.Pp
378.Sy Warning :
379If a user text is shadowed by a normal user space debugger,
380user space break points may not work correctly.
381Setting a break
382point at the low-level code paths may also cause strange behavior.
383.Pp
384.It Ic delete Op Ar addr
385.It Ic d      Op Ar addr
386.It Ic delete Li # Ns Ar number
387.It Ic d      Li # Ns Ar number
388Delete the specified break point.
389The break point can be specified by a
390break point number with
391.Ql # ,
392or by using the same
393.Ar addr
394specified in the original
395.Ic break
396command, or by omitting
397.Ar addr
398to get the default address of
399.Va dot .
400.Pp
401.It Ic halt
402Halt the system.
403.Pp
404.It Ic watch Oo Ar addr Oc Ns Op , Ns Ar size
405Set a watchpoint for a region.
406Execution stops when an attempt to modify the region occurs.
407The
408.Ar size
409argument defaults to 4.
410If you specify a wrong space address, the request is rejected
411with an error message.
412.Pp
413.Sy Warning :
414Attempts to watch wired kernel memory
415may cause unrecoverable error in some systems such as i386.
416Watchpoints on user addresses work best.
417.Pp
418.It Ic hwatch Oo Ar addr Oc Ns Op , Ns Ar size
419Set a hardware watchpoint for a region if supported by the
420architecture.
421Execution stops when an attempt to modify the region occurs.
422The
423.Ar size
424argument defaults to 4.
425.Pp
426.Sy Warning :
427The hardware debug facilities do not have a concept of separate
428address spaces like the watch command does.
429Use
430.Ic hwatch
431for setting watchpoints on kernel address locations only, and avoid
432its use on user mode address spaces.
433.Pp
434.It Ic dhwatch Oo Ar addr Oc Ns Op , Ns Ar size
435Delete specified hardware watchpoint.
436.Pp
437.It Ic kill Ar sig pid
438Send signal
439.Ar sig
440to process
441.Ar pid .
442The signal is acted on upon returning from the debugger.
443This command can be used to kill a process causing resource contention
444in the case of a hung system.
445See
446.Xr signal 3
447for a list of signals.
448Note that the arguments are reversed relative to
449.Xr kill 2 .
450.Pp
451.It Ic step Ns Oo Li / Ns Cm p Oc Ns Op , Ns Ar count
452.It Ic s    Ns Oo Li / Ns Cm p Oc Ns Op , Ns Ar count
453Single step
454.Ar count
455times.
456If the
457.Cm p
458modifier is specified, print each instruction at each step.
459Otherwise, only print the last instruction.
460.Pp
461.Sy Warning :
462depending on machine type, it may not be possible to
463single-step through some low-level code paths or user space code.
464On machines with software-emulated single-stepping (e.g., pmax),
465stepping through code executed by interrupt handlers will probably
466do the wrong thing.
467.Pp
468.It Ic continue Ns Op Li / Ns Cm c
469.It Ic c Ns Op Li / Ns Cm c
470Continue execution until a breakpoint or watchpoint.
471If the
472.Cm c
473modifier is specified, count instructions while executing.
474Some machines (e.g., pmax) also count loads and stores.
475.Pp
476.Sy Warning :
477when counting, the debugger is really silently single-stepping.
478This means that single-stepping on low-level code may cause strange
479behavior.
480.Pp
481.It Ic until Ns Op Li / Ns Cm p
482Stop at the next call or return instruction.
483If the
484.Cm p
485modifier is specified, print the call nesting depth and the
486cumulative instruction count at each call or return.
487Otherwise,
488only print when the matching return is hit.
489.Pp
490.It Ic next Ns Op Li / Ns Cm p
491.It Ic match Ns Op Li / Ns Cm p
492Stop at the matching return instruction.
493If the
494.Cm p
495modifier is specified, print the call nesting depth and the
496cumulative instruction count at each call or return.
497Otherwise, only print when the matching return is hit.
498.Pp
499.It Xo
500.Ic trace Ns Op Li / Ns Cm u
501.Op Ar pid | tid Ns
502.Op , Ns Ar count
503.Xc
504.It Xo
505.Ic t Ns Op Li / Ns Cm u
506.Op Ar pid | tid Ns
507.Op , Ns Ar count
508.Xc
509.It Xo
510.Ic where Ns Op Li / Ns Cm u
511.Op Ar pid | tid Ns
512.Op , Ns Ar count
513.Xc
514.It Xo
515.Ic bt Ns Op Li / Ns Cm u
516.Op Ar pid | tid Ns
517.Op , Ns Ar count
518.Xc
519Stack trace.
520The
521.Cm u
522option traces user space; if omitted,
523.Ic trace
524only traces
525kernel space.
526The optional argument
527.Ar count
528is the number of frames to be traced.
529If
530.Ar count
531is omitted, all frames are printed.
532.Pp
533.Sy Warning :
534User space stack trace is valid
535only if the machine dependent code supports it.
536.Pp
537.It Xo
538.Ic search Ns Op Li / Ns Cm bhl
539.Ar addr
540.Ar value
541.Op Ar mask Ns
542.Op , Ns Ar count
543.Xc
544Search memory for
545.Ar value .
546The optional
547.Ar count
548argument limits the search.
549.\"
550.Pp
551.It Xo
552.Ic Ic reboot Ns Op Li / Ns Cm s
553.Op Ar seconds
554.Xc
555.It Xo
556.Ic Ic reset Ns Op Li / Ns Cm s
557.Op Ar seconds
558.Xc
559Hard reset the system.
560If the optional argument
561.Ar seconds
562is given, the debugger will wait for this long, at most a week,
563before rebooting.
564When the
565.Cm s
566modifier is given, the command will skip running any registered shutdown
567handlers and attempt the most basic reset.
568.Pp
569.It Ic thread Ar addr | tid
570Switch the debugger to the thread with ID
571.Ar tid ,
572if the argument is a decimal number, or address
573.Ar addr ,
574otherwise.
575.Pp
576.It Ic watchdog Op Ar exp
577Program the
578.Xr watchdog 4
579timer to fire in
580.Pf 2^ Ar exp
581seconds.
582If no argument is provided, the watchdog timer is disabled.
583.El
584.Ss SPECIALIZED HELPER COMMANDS
585.Bl -tag -width indent -compact
586.It Xo
587.Ic findstack
588.Ar addr
589.Xc
590Prints the address of the thread whose kernel-mode stack contains
591.Ar addr ,
592if any.
593.Pp
594.It Ic show Cm active trace
595.It acttrace
596Show a stack trace for every thread running on a CPU.
597.Pp
598.It Ic show Cm all procs Ns Op Li / Ns Cm a
599.It Ic ps Ns Op Li / Ns Cm a
600Display all process information.
601The process information may not be shown if it is not
602supported in the machine, or the bottom of the stack of the
603target process is not in the main memory at that time.
604The
605.Cm a
606modifier will print command line arguments for each process.
607.\"
608.Pp
609.It Ic show Cm all trace
610.It Ic alltrace
611Show a stack trace for every thread in the system.
612.Pp
613.It Ic show Cm all ttys
614Show all TTY's within the system.
615Output is similar to
616.Xr pstat 8 ,
617but also includes the address of the TTY structure.
618.\"
619.Pp
620.It Ic show Cm all vnets
621Show the same output as "show vnet" does, but lists all
622virtualized network stacks within the system.
623.\"
624.Pp
625.It Ic show Cm allchains
626Show the same information like "show lockchain" does, but
627for every thread in the system.
628.\"
629.Pp
630.It Ic show Cm alllocks
631Show all locks that are currently held.
632This command is only available if
633.Xr witness 4
634is included in the kernel.
635.\"
636.Pp
637.It Ic show Cm allpcpu
638The same as "show pcpu", but for every CPU present in the system.
639.\"
640.Pp
641.It Ic show Cm allrman
642Show information related with resource management, including
643interrupt request lines, DMA request lines, I/O ports, I/O memory
644addresses, and Resource IDs.
645.\"
646.Pp
647.It Ic show Cm apic
648Dump data about APIC IDT vector mappings.
649.\"
650.Pp
651.It Ic show Cm badstacks
652Walk the
653.Xr witness 4
654graph and print any lock-order violations.
655This command is only available if
656.Xr witness 4
657is included in the kernel.
658.\"
659.Pp
660.It Ic show Cm breaks
661Show breakpoints set with the "break" command.
662.\"
663.Pp
664.It Ic show Cm bio Ar addr
665Show information about the bio structure
666.Vt struct bio
667present at
668.Ar addr .
669See the
670.Pa sys/bio.h
671header file and
672.Xr g_bio 9
673for more details on the exact meaning of the structure fields.
674.\"
675.Pp
676.It Ic show Cm buffer Ar addr
677Show information about the buf structure
678.Vt struct buf
679present at
680.Ar addr .
681See the
682.Pa sys/buf.h
683header file for more details on the exact meaning of the structure fields.
684.\"
685.Pp
686.It Ic show Cm callout Ar addr
687Show information about the callout structure
688.Vt struct callout
689present at
690.Ar addr .
691.\"
692.Pp
693.It Ic show Cm cdev Op Ar addr
694Show the internal devfs state of the cdev structure located at
695.Ar addr .
696If no argument is provided, show the list of all created cdevs, consisting of
697the devfs node name and the
698.Vt struct cdev
699address.
700.\"
701.Pp
702.It Ic show Cm conifhk
703Lists hooks currently waiting for completion in
704.Fn run_interrupt_driven_config_hooks .
705.\"
706.Pp
707.It Ic show Cm cpusets
708Print numbered root and assigned CPU affinity sets.
709See
710.Xr cpuset 2
711for more details.
712.\"
713.Pp
714.It Ic show Cm cyrixreg
715Show registers specific to the Cyrix processor.
716.\"
717.Pp
718.It Ic show Cm devmap
719Prints the contents of the static device mapping table.
720Currently only available on the
721ARM
722architecture.
723.\"
724.Pp
725.It Ic show Cm domain Ar addr
726Print protocol domain structure
727.Vt struct domain
728at address
729.Ar addr .
730See the
731.Pa sys/domain.h
732header file for more details on the exact meaning of the structure fields.
733.\"
734.Pp
735.It Ic show Cm ffs Op Ar addr
736Show brief information about ffs mount at the address
737.Ar addr ,
738if argument is given.
739Otherwise, provides the summary about each ffs mount.
740.\"
741.Pp
742.It Ic show Cm file Ar addr
743Show information about the file structure
744.Vt struct file
745present at address
746.Ar addr .
747.\"
748.Pp
749.It Ic show Cm files
750Show information about every file structure in the system.
751.\"
752.Pp
753.It Ic show Cm freepages
754Show the number of physical pages in each of the free lists.
755.\"
756.Pp
757.It Ic show Cm geom Op Ar addr
758If the
759.Ar addr
760argument is not given, displays the entire GEOM topology.
761If
762.Ar addr
763is given, displays details about the given GEOM object (class, geom,
764provider or consumer).
765.\"
766.Pp
767.It Ic show Cm idt
768Show IDT layout.
769The first column specifies the IDT vector.
770The second one is the name of the interrupt/trap handler.
771Those functions are machine dependent.
772.\"
773.Pp
774.It Ic show Cm igi_list Ar addr
775Show information about the IGMP structure
776.Vt struct igmp_ifsoftc
777present at
778.Ar addr .
779.\"
780.Pp
781.It Ic show Cm iosched Ar addr
782Show information about the I/O scheduler
783.Vt struct cam_iosched_softc
784located at
785.Ar addr .
786.\"
787.Pp
788.It Ic show Cm inodedeps Op Ar addr
789Show brief information about each inodedep structure.
790If
791.Ar addr
792is given, only inodedeps belonging to the fs located at the
793supplied address are shown.
794.\"
795.Pp
796.It Ic show Cm inpcb Ar addr
797Show information on IP Control Block
798.Vt struct in_pcb
799present at
800.Ar addr .
801.\"
802.Pp
803.It Ic show Cm intr
804Dump information about interrupt handlers.
805.\"
806.Pp
807.It Ic show Cm intrcnt
808Dump the interrupt statistics.
809.\"
810.Pp
811.It Ic show Cm irqs
812Show interrupt lines and their respective kernel threads.
813.\"
814.Pp
815.It Ic show Cm ktr Ns Op Li / Ns Cm a Ns Cm v Ns Cm V
816Print the contents of the
817.Xr ktr 4
818trace buffer.
819The
820.Cm v
821modifier will request fully verbose output, causing the file, line number, and
822timestamp to be printed for each trace entry.
823The
824.Cm V
825modifier will request only the timestamps to be printed.
826The
827.Cm a
828modifier will request that the output be unpaginated.
829.\"
830.Pp
831.It Ic show Cm lapic
832Show information from the local APIC registers for this CPU.
833.\"
834.Pp
835.It Ic show Cm lock Ar addr
836Show lock structure.
837The output format is as follows:
838.Bl -tag -width "flags"
839.It Ic class :
840Class of the lock.
841Possible types include
842.Xr mutex 9 ,
843.Xr rmlock 9 ,
844.Xr rwlock 9 ,
845.Xr sx 9 .
846.It Ic name :
847Name of the lock.
848.It Ic flags :
849Flags passed to the lock initialization function.
850.Em flags
851values are lock class specific.
852.It Ic state :
853Current state of a lock.
854.Em state
855values are lock class specific.
856.It Ic owner :
857Lock owner.
858.El
859.\"
860.Pp
861.It Ic show Cm lockchain Ar addr
862Show all threads a particular thread at address
863.Ar addr
864is waiting on based on non-spin locks.
865.\"
866.Pp
867.It Ic show Cm lockedbufs
868Show the same information as "show buf", but for every locked
869.Vt struct buf
870object.
871.\"
872.Pp
873.It Ic show Cm lockedvnods
874List all locked vnodes in the system.
875.\"
876.Pp
877.It Ic show Cm locks
878Prints all locks that are currently acquired.
879This command is only available if
880.Xr witness 4
881is included in the kernel.
882.\"
883.Pp
884.It Ic show Cm locktree
885.\"
886.Pp
887.It Ic show Cm malloc Ns Op Li / Ns Cm i
888Prints
889.Xr malloc 9
890memory allocator statistics.
891If the
892.Cm i
893modifier is specified, format output as machine-parseable comma-separated
894values ("CSV").
895The output columns are as follows:
896.Pp
897.Bl -tag -compact -offset indent -width "Requests"
898.It Ic Type
899Specifies a type of memory.
900It is the same as a description string used while defining the
901given memory type with
902.Xr MALLOC_DECLARE 9 .
903.It Ic InUse
904Number of memory allocations of the given type, for which
905.Xr free 9
906has not been called yet.
907.It Ic MemUse
908Total memory consumed by the given allocation type.
909.It Ic Requests
910Number of memory allocation requests for the given
911memory type.
912.El
913.Pp
914The same information can be gathered in userspace with
915.Dq Nm vmstat Fl m .
916.\"
917.Pp
918.It Ic show Cm map Ns Oo Li / Ns Cm f Oc Ar addr
919Prints the VM map at
920.Ar addr .
921If the
922.Cm f
923modifier is specified the
924complete map is printed.
925.\"
926.Pp
927.It Ic show Cm msgbuf
928Print the system's message buffer.
929It is the same output as in the
930.Dq Nm dmesg
931case.
932It is useful if you got a kernel panic, attached a serial cable
933to the machine and want to get the boot messages from before the
934system hang.
935.\"
936.Pp
937.It Ic show Cm mount Op Ar addr
938Displays details about the mount point located at
939.Ar addr .
940If no
941.Ar addr
942is specified,
943displays short info about all currently mounted file systems.
944.\"
945.Pp
946.It Ic show Cm object Ns Oo Li / Ns Cm f Oc Ar addr
947Prints the VM object at
948.Ar addr .
949If the
950.Cm f
951option is specified the
952complete object is printed.
953.\"
954.Pp
955.It Ic show Cm panic
956Print the panic message if set.
957.\"
958.Pp
959.It Ic show Cm page
960Show statistics on VM pages.
961.\"
962.Pp
963.It Ic show Cm pageq
964Show statistics on VM page queues.
965.\"
966.Pp
967.It Ic show Cm pciregs
968Print PCI bus registers.
969The same information can be gathered in userspace by running
970.Dq Nm pciconf Fl lv .
971.\"
972.Pp
973.It Ic show Cm pcpu
974Print current processor state.
975The output format is as follows:
976.Pp
977.Bl -tag -compact -offset indent -width "spin locks held:"
978.It Ic cpuid
979Processor identifier.
980.It Ic curthread
981Thread pointer, process identifier and the name of the process.
982.It Ic curpcb
983Control block pointer.
984.It Ic fpcurthread
985FPU thread pointer.
986.It Ic idlethread
987Idle thread pointer.
988.It Ic APIC ID
989CPU identifier coming from APIC.
990.It Ic currentldt
991LDT pointer.
992.It Ic spin locks held
993Names of spin locks held.
994.El
995.\"
996.Pp
997.It Ic show Cm pgrpdump
998Dump process groups present within the system.
999.\"
1000.Pp
1001.It Ic show Cm prison Op Ar addr
1002Show the prison structure located at
1003.Ar addr .
1004If no
1005.Ar addr
1006argument is specified, show information about all prisons in the system.
1007.\"
1008.Pp
1009.It Ic show Cm proc Op Ar addr
1010Show information about the process structure located at address
1011.Ar addr ,
1012or the current process if no argument is specified.
1013.\"
1014.Pp
1015.It Ic show Cm procvm Op Ar addr
1016Show process virtual memory layout for the process located at
1017.Ar addr ,
1018or the current process if no argument is specified.
1019.\"
1020.Pp
1021.It Ic show Cm protosw Ar addr
1022Print protocol switch structure
1023.Vt struct protosw
1024at address
1025.Ar addr .
1026.\"
1027.Pp
1028.It Ic show Cm registers Ns Op Li / Ns Cm u
1029Display the register set.
1030If the
1031.Cm u
1032modifier is specified, the register contents of the thread's previous
1033trapframe are displayed instead.
1034Usually, this corresponds to the saved state from userspace.
1035.\"
1036.Pp
1037.It Ic show Cm rman Ar addr
1038Show resource manager object
1039.Vt struct rman
1040at address
1041.Ar addr .
1042Addresses of particular pointers can be gathered with "show allrman"
1043command.
1044.\"
1045.Pp
1046.It Ic show Cm route Ar addr
1047Show route table result for destination
1048.Ar addr .
1049At this time, INET and INET6 formatted addresses are supported.
1050.\"
1051.Pp
1052.It Ic show Cm routetable Oo Ar af Oc
1053Show full route table or tables.
1054If
1055.Ar af
1056is specified, show only routes for the given numeric address family.
1057If no argument is specified, dump the route table for all address families.
1058.\"
1059.Pp
1060.It Ic show Cm rtc
1061Show real time clock value.
1062Useful for long debugging sessions.
1063.\"
1064.Pp
1065.It Ic show Cm sleepchain
1066Deprecated.
1067Now an alias for
1068.Ic show Cm lockchain .
1069.\"
1070.Pp
1071.It Ic show Cm sleepq Ar addr
1072.It Ic show Cm sleepqueue Ar addr
1073Show the
1074.Xr sleepqueue 9
1075structure located at
1076.Ar addr .
1077.\"
1078.Pp
1079.It Ic show Cm sockbuf Ar addr
1080Show the socket buffer
1081.Va struct sockbuf
1082located at
1083.Ar addr .
1084.\"
1085.Pp
1086.It Ic show Cm socket Ar addr
1087Show the socket object
1088.Vt struct socket
1089located at
1090.Ar addr .
1091.\"
1092.Pp
1093.It Ic show Cm sysregs
1094Show system registers (e.g.,
1095.Li cr0-4
1096on i386.)
1097Not present on some platforms.
1098.\"
1099.Pp
1100.It Ic show Cm tcpcb Ar addr
1101Print TCP control block
1102.Vt struct tcpcb
1103lying at address
1104.Ar addr .
1105For exact interpretation of output, visit
1106.Pa netinet/tcp.h
1107header file.
1108.\"
1109.Pp
1110.It Ic show Cm thread Op Ar addr | tid
1111If no
1112.Ar addr
1113or
1114.Ar tid
1115is specified, show detailed information about current thread.
1116Otherwise, print information about the thread with ID
1117.Ar tid
1118or kernel address
1119.Ar addr .
1120(If the argument is a decimal number, it is assumed to be a tid.)
1121.\"
1122.Pp
1123.It Ic show Cm threads
1124Show all threads within the system.
1125Output format is as follows:
1126.Pp
1127.Bl -tag -compact -offset indent -width "Second column"
1128.It Ic First column
1129Thread identifier (TID)
1130.It Ic Second column
1131Thread structure address
1132.It Ic Third column
1133Backtrace.
1134.El
1135.\"
1136.Pp
1137.It Ic show Cm tty Ar addr
1138Display the contents of a TTY structure in a readable form.
1139.\"
1140.Pp
1141.It Ic show Cm turnstile Ar addr
1142Show turnstile
1143.Vt struct turnstile
1144structure at address
1145.Ar addr .
1146Turnstiles are structures used within the
1147.Fx
1148kernel to implement
1149synchronization primitives which, while holding a specific type of lock, cannot
1150sleep or context switch to another thread.
1151Currently, those are:
1152.Xr mutex 9 ,
1153.Xr rwlock 9 ,
1154.Xr rmlock 9 .
1155.\"
1156.Pp
1157.It Ic show Cm uma Ns Op Li / Ns Cm i
1158Show UMA allocator statistics.
1159If the
1160.Cm i
1161modifier is specified, format output as machine-parseable comma-separated
1162values ("CSV").
1163The output contains the following columns:
1164.Pp
1165.Bl -tag -compact -offset indent -width "Total Mem"
1166.It Cm "Zone"
1167Name of the UMA zone.
1168The same string that was passed to
1169.Xr uma_zcreate 9
1170as a first argument.
1171.It Cm "Size"
1172Size of a given memory object (slab).
1173.It Cm "Used"
1174Number of slabs being currently used.
1175.It Cm "Free"
1176Number of free slabs within the UMA zone.
1177.It Cm "Requests"
1178Number of allocations requests to the given zone.
1179.It Cm "Total Mem"
1180Total memory in use (either allocated or free) by a zone, in bytes.
1181.It Cm "XFree"
1182Number of free slabs within the UMA zone that were freed on a different NUMA
1183domain than allocated.
1184(The count in the
1185.Cm "Free"
1186column is inclusive of
1187.Cm "XFree" . )
1188.El
1189.Pp
1190The same information might be gathered in the userspace
1191with the help of
1192.Dq Nm vmstat Fl z .
1193.\"
1194.Pp
1195.It Ic show Cm unpcb Ar addr
1196Shows UNIX domain socket private control block
1197.Vt struct unpcb
1198present at the address
1199.Ar addr .
1200.\"
1201.Pp
1202.It Ic show Cm vmochk
1203Prints, whether the internal VM objects are in a map somewhere
1204and none have zero ref counts.
1205.\"
1206.Pp
1207.It Ic show Cm vmopag
1208Walk the list of VM objects in the system, printing the indices and physical
1209addresses of the VM pages belonging to each object.
1210.\"
1211.Pp
1212.It Ic show Cm vnet Ar addr
1213Prints virtualized network stack
1214.Vt struct vnet
1215structure present at the address
1216.Ar addr .
1217.\"
1218.Pp
1219.It Ic show Cm vnode Ar addr
1220Prints vnode
1221.Vt struct vnode
1222structure lying at
1223.Ar addr .
1224For the exact interpretation of the output, look at the
1225.Pa sys/vnode.h
1226header file.
1227.\"
1228.Pp
1229.It Ic show Cm vnodebufs Ar addr
1230Shows clean/dirty buffer lists of the vnode located at
1231.Ar addr .
1232.\"
1233.Pp
1234.It Ic show Cm vpath Ar addr
1235Walk the namecache to lookup the pathname of the vnode located at
1236.Ar addr .
1237.\"
1238.Pp
1239.It Ic show Cm watches
1240Displays all watchpoints.
1241Shows watchpoints set with "watch" command.
1242.\"
1243.Pp
1244.It Ic show Cm witness
1245Shows information about lock acquisition coming from the
1246.Xr witness 4
1247subsystem.
1248.El
1249.Ss OFFLINE DEBUGGING COMMANDS
1250.Bl -tag -width indent -compact
1251.It Ic dump
1252Initiate a kernel core dump to the device(s) configured by
1253.Xr dumpon 8 .
1254.Pp
1255.It Ic gdb
1256Switches to remote GDB mode.
1257In remote GDB mode, another machine is required that runs
1258.Xr gdb 1 Pq Pa ports/devel/gdb
1259using the remote debug feature, with a connection to the serial
1260console port on the target machine.
1261.Pp
1262.It Ic netdump Fl s Ar server Oo Fl g Ar gateway Fl c Ar client Fl i Ar iface Oc
1263Configure
1264.Xr netdump 4
1265with the provided parameters, and immediately perform a netdump.
1266.Pp
1267There are some known limitations.
1268Principally,
1269.Xr netdump 4
1270only supports IPv4 at this time.
1271The address arguments to the
1272.Ic netdump
1273command must be dotted decimal IPv4 addresses.
1274(Hostnames are not supported.)
1275At present, the command only works if the machine is in a panic state.
1276Finally, the
1277.Nm
1278.Ic netdump
1279command does not provide any way to configure compression or encryption.
1280.Pp
1281.It Ic netgdb Fl s Ar server Oo Fl g Ar gateway Fl c Ar client Fl i Ar iface Oc
1282Initiate a
1283.Xr netgdb 4
1284session with the provided parameters.
1285.Pp
1286.Ic netgdb
1287has identical limitations to
1288.Ic netdump .
1289.Pp
1290.It Ic capture on
1291.It Ic capture off
1292.It Ic capture reset
1293.It Ic capture status
1294.Nm
1295supports a basic output capture facility, which can be used to retrieve the
1296results of debugging commands from userspace using
1297.Xr sysctl 3 .
1298.Ic capture on
1299enables output capture;
1300.Ic capture off
1301disables capture.
1302.Ic capture reset
1303will clear the capture buffer and disable capture.
1304.Ic capture status
1305will report current buffer use, buffer size, and disposition of output
1306capture.
1307.Pp
1308Userspace processes may inspect and manage
1309.Nm
1310capture state using
1311.Xr sysctl 8 :
1312.Pp
1313.Va debug.ddb.capture.bufsize
1314may be used to query or set the current capture buffer size.
1315.Pp
1316.Va debug.ddb.capture.maxbufsize
1317may be used to query the compile-time limit on the capture buffer size.
1318.Pp
1319.Va debug.ddb.capture.bytes
1320may be used to query the number of bytes of output currently in the capture
1321buffer.
1322.Pp
1323.Va debug.ddb.capture.data
1324returns the contents of the buffer as a string to an appropriately privileged
1325process.
1326.Pp
1327This facility is particularly useful in concert with the scripting and
1328.Xr textdump 4
1329facilities, allowing scripted debugging output to be captured and
1330committed to disk as part of a textdump for later analysis.
1331The contents of the capture buffer may also be inspected in a kernel core dump
1332using
1333.Xr kgdb 1 Pq Pa ports/devel/gdb .
1334.Pp
1335.It Ic run
1336.It Ic script
1337.It Ic scripts
1338.It Ic unscript
1339Run, define, list, and delete scripts.
1340See the
1341.Sx SCRIPTING
1342section for more information on the scripting facility.
1343.Pp
1344.It Ic textdump dump
1345.It Ic textdump set
1346.It Ic textdump status
1347.It Ic textdump unset
1348Use the
1349.Ic textdump dump
1350command to immediately perform a textdump.
1351More information may be found in
1352.Xr textdump 4 .
1353The
1354.Ic textdump set
1355command may be used to force the next kernel core dump to be a textdump
1356rather than a traditional memory dump or minidump.
1357.Ic textdump status
1358reports whether a textdump has been scheduled.
1359.Ic textdump unset
1360cancels a request to perform a textdump as the next kernel core dump.
1361.El
1362.Sh VARIABLES
1363The debugger accesses registers and variables as
1364.Li $ Ns Ar name .
1365Register names are as in the
1366.Dq Ic show Cm registers
1367command.
1368Some variables are suffixed with numbers, and may have some modifier
1369following a colon immediately after the variable name.
1370For example, register variables can have a
1371.Cm u
1372modifier to indicate user register (e.g.,
1373.Dq Li $eax:u ) .
1374.Pp
1375Built-in variables currently supported are:
1376.Pp
1377.Bl -tag -width ".Va tabstops" -compact
1378.It Va radix
1379Input and output radix.
1380.It Va maxoff
1381Addresses are printed as
1382.Dq Ar symbol Ns Li + Ns Ar offset
1383unless
1384.Ar offset
1385is greater than
1386.Va maxoff .
1387.It Va maxwidth
1388The width of the displayed line.
1389.It Va lines
1390The number of lines.
1391It is used by the built-in pager.
1392Setting it to 0 disables paging.
1393.It Va tabstops
1394Tab stop width.
1395.It Va work Ns Ar xx
1396Work variable;
1397.Ar xx
1398can take values from 0 to 31.
1399.El
1400.Sh EXPRESSIONS
1401Most expression operators in C are supported except
1402.Ql ~ ,
1403.Ql ^ ,
1404and unary
1405.Ql & .
1406Special rules in
1407.Nm
1408are:
1409.Bl -tag -width ".No Identifiers"
1410.It Identifiers
1411The name of a symbol is translated to the value of the symbol, which
1412is the address of the corresponding object.
1413.Ql \&.
1414and
1415.Ql \&:
1416can be used in the identifier.
1417If supported by an object format dependent routine,
1418.Sm off
1419.Oo Ar filename : Oc Ar func : lineno ,
1420.Sm on
1421.Oo Ar filename : Oc Ns Ar variable ,
1422and
1423.Oo Ar filename : Oc Ns Ar lineno
1424can be accepted as a symbol.
1425.It Numbers
1426Radix is determined by the first two letters:
1427.Ql 0x :
1428hex,
1429.Ql 0o :
1430octal,
1431.Ql 0t :
1432decimal; otherwise, follow current radix.
1433.It Li \&.
1434.Va dot
1435.It Li +
1436.Va next
1437.It Li ..
1438address of the start of the last line examined.
1439Unlike
1440.Va dot
1441or
1442.Va next ,
1443this is only changed by
1444.Ic examine
1445or
1446.Ic write
1447command.
1448.It Li '
1449last address explicitly specified.
1450.It Li $ Ns Ar variable
1451Translated to the value of the specified variable.
1452It may be followed by a
1453.Ql \&:
1454and modifiers as described above.
1455.It Ar a Ns Li # Ns Ar b
1456A binary operator which rounds up the left hand side to the next
1457multiple of right hand side.
1458.It Li * Ns Ar expr
1459Indirection.
1460It may be followed by a
1461.Ql \&:
1462and modifiers as described above.
1463.El
1464.Sh SCRIPTING
1465.Nm
1466supports a basic scripting facility to allow automating tasks or responses to
1467specific events.
1468Each script consists of a list of DDB commands to be executed sequentially,
1469and is assigned a unique name.
1470Certain script names have special meaning, and will be automatically run on
1471various
1472.Nm
1473events if scripts by those names have been defined.
1474.Pp
1475The
1476.Ic script
1477command may be used to define a script by name.
1478Scripts consist of a series of
1479.Nm
1480commands separated with the
1481.Ql \&;
1482character.
1483For example:
1484.Bd -literal -offset indent
1485script kdb.enter.panic=bt; show pcpu
1486script lockinfo=show alllocks; show lockedvnods
1487.Ed
1488.Pp
1489The
1490.Ic scripts
1491command lists currently defined scripts.
1492.Pp
1493The
1494.Ic run
1495command execute a script by name.
1496For example:
1497.Bd -literal -offset indent
1498run lockinfo
1499.Ed
1500.Pp
1501The
1502.Ic unscript
1503command may be used to delete a script by name.
1504For example:
1505.Bd -literal -offset indent
1506unscript kdb.enter.panic
1507.Ed
1508.Pp
1509These functions may also be performed from userspace using the
1510.Xr ddb 8
1511command.
1512.Pp
1513Certain scripts are run automatically, if defined, for specific
1514.Nm
1515events.
1516The follow scripts are run when various events occur:
1517.Bl -tag -width kdb.enter.powerfail
1518.It Va kdb.enter.acpi
1519The kernel debugger was entered as a result of an
1520.Xr acpi 4
1521event.
1522.It Va kdb.enter.bootflags
1523The kernel debugger was entered at boot as a result of the debugger boot
1524flag being set.
1525.It Va kdb.enter.break
1526The kernel debugger was entered as a result of a serial or console break.
1527.It Va kdb.enter.cam
1528The kernel debugger was entered as a result of a
1529.Xr CAM 4
1530event.
1531.It Va kdb.enter.mac
1532The kernel debugger was entered as a result of an assertion failure in the
1533.Xr mac_test 4
1534module of the
1535TrustedBSD MAC Framework.
1536.It Va kdb.enter.netgraph
1537The kernel debugger was entered as a result of a
1538.Xr netgraph 4
1539event.
1540.It Va kdb.enter.panic
1541.Xr panic 9
1542was called.
1543.It Va kdb.enter.powerpc
1544The kernel debugger was entered as a result of an unimplemented interrupt
1545type on the powerpc platform.
1546.It Va kdb.enter.sysctl
1547The kernel debugger was entered as a result of the
1548.Va debug.kdb.enter
1549sysctl being set.
1550.It Va kdb.enter.unionfs
1551The kernel debugger was entered as a result of an assertion failure in the
1552union file system.
1553.It Va kdb.enter.unknown
1554The kernel debugger was entered, but no reason has been set.
1555.It Va kdb.enter.vfslock
1556The kernel debugger was entered as a result of a VFS lock violation.
1557.It Va kdb.enter.watchdog
1558The kernel debugger was entered as a result of a watchdog firing.
1559.It Va kdb.enter.witness
1560The kernel debugger was entered as a result of a
1561.Xr witness 4
1562violation.
1563.El
1564.Pp
1565In the event that none of these scripts is found,
1566.Nm
1567will attempt to execute a default script:
1568.Bl -tag -width kdb.enter.powerfail
1569.It Va kdb.enter.default
1570The kernel debugger was entered, but a script exactly matching the reason for
1571entering was not defined.
1572This can be used as a catch-all to handle cases not specifically of interest;
1573for example,
1574.Va kdb.enter.witness
1575might be defined to have special handling, and
1576.Va kdb.enter.default
1577might be defined to simply panic and reboot.
1578.El
1579.Sh HINTS
1580On machines with an ISA expansion bus, a simple NMI generation card can be
1581constructed by connecting a push button between the A01 and B01 (CHCHK# and
1582GND) card fingers.
1583Momentarily shorting these two fingers together may cause the bridge chipset to
1584generate an NMI, which causes the kernel to pass control to
1585.Nm .
1586Some bridge chipsets do not generate a NMI on CHCHK#, so your mileage may vary.
1587The NMI allows one to break into the debugger on a wedged machine to
1588diagnose problems.
1589Other bus' bridge chipsets may be able to generate NMI using bus specific
1590methods.
1591There are many PCI and PCIe add-in cards which can generate NMI for
1592debugging.
1593Modern server systems typically use IPMI to generate signals to enter the
1594debugger.
1595The
1596.Va devel/ipmitool
1597port can be used to send the
1598.Cd chassis power diag
1599command which delivers an NMI to the processor.
1600Embedded systems often use JTAG for debugging, but rarely use it in
1601combination with
1602.Nm .
1603.Pp
1604Serial consoles can break to the debugger by sending a BREAK
1605condition on the serial line.
1606This requires a kernel built with
1607.Cd options BREAK_TO_DEBUGGER
1608is specified in the kernel.
1609Most terminal emulation programs can send a break sequence with a
1610special key sequence or menu selection.
1611Sending the break can be difficult or even happen spuriously in some setups.
1612An alternative method is to build a kernel with
1613.Cd options ALT_BREAK_TO_DEBUGGER
1614then the sequence of CR TILDE CTRL-B enters the debugger;
1615CR TILDE CTRL-P causes a panic; and
1616CR TILDE CTRL-R causes an immediate reboot.
1617In all these sequences, CR represents Carriage Return and is usually
1618sent by pressing the Enter or Return key.
1619TILDE is the ASCII tilde character (~).
1620CTRL-x is Control x, sent by pressing the Control key, then x, then releasing
1621both.
1622.Pp
1623The break-to-debugger behavior can be enabled by setting
1624.Xr sysctl 8
1625.Va debug.kdb.break_to_debugger
1626to 1.
1627The alt-break-to-debugger behavior can be enabled by setting
1628.Xr sysctl 8
1629.Va debug.kdb.alt_break_to_debugger
1630to 1.
1631The debugger can be entered by setting
1632.Xr sysctl 8
1633.Va debug.kdb.enter
1634to 1.
1635.Pp
1636Output can be interrupted, paused, and resumed with the control
1637characters CTRL-C, CTRL-S, and CTRL-Q.
1638Because these control characters are received as in-band data from the
1639console, there is an input buffer, and once that buffer fills
1640.Nm
1641must either stop responding to control characters or drop additional
1642input while continuing to search for control characters.
1643This behavior is controlled by the tunable
1644.Xr sysctl 8
1645.Va debug.ddb.prioritize_control_input ,
1646which defaults to 1.
1647The input buffer size is 512 bytes.
1648.Sh FILES
1649Header files mentioned in this manual page can be found below
1650.Pa /usr/include
1651directory.
1652.Pp
1653.Bl -dash -compact
1654.It
1655.Pa sys/buf.h
1656.It
1657.Pa sys/domain.h
1658.It
1659.Pa netinet/in_pcb.h
1660.It
1661.Pa sys/socket.h
1662.It
1663.Pa sys/vnode.h
1664.El
1665.Sh SEE ALSO
1666.Xr gdb 1 Pq Pa ports/devel/gdb ,
1667.Xr kgdb 1 Pq Pa ports/devel/gdb ,
1668.Xr acpi 4 ,
1669.Xr CAM 4 ,
1670.Xr gdb 4 ,
1671.Xr mac_ddb 4 ,
1672.Xr mac_test 4 ,
1673.Xr netgraph 4 ,
1674.Xr textdump 4 ,
1675.Xr witness 4 ,
1676.Xr ddb 8 ,
1677.Xr sysctl 8 ,
1678.Xr panic 9
1679.Sh HISTORY
1680The
1681.Nm
1682debugger was developed for Mach, and ported to
1683.Bx 386 0.1 .
1684This manual page translated from
1685.Xr man 7
1686macros by
1687.An Garrett Wollman .
1688.Pp
1689.An Robert N. M. Watson
1690added support for
1691.Nm
1692output capture,
1693.Xr textdump 4
1694and scripting in
1695.Fx 7.1 .
1696