1.\" 2.\" Mach Operating System 3.\" Copyright (c) 1991,1990 Carnegie Mellon University 4.\" Copyright (c) 2007 Robert N. M. Watson 5.\" All Rights Reserved. 6.\" 7.\" Permission to use, copy, modify and distribute this software and its 8.\" documentation is hereby granted, provided that both the copyright 9.\" notice and this permission notice appear in all copies of the 10.\" software, derivative works or modified versions, and any portions 11.\" thereof, and that both notices appear in supporting documentation. 12.\" 13.\" CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS" 14.\" CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND FOR 15.\" ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. 16.\" 17.\" Carnegie Mellon requests users of this software to return to 18.\" 19.\" Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU 20.\" School of Computer Science 21.\" Carnegie Mellon University 22.\" Pittsburgh PA 15213-3890 23.\" 24.\" any improvements or extensions that they make and grant Carnegie Mellon 25.\" the rights to redistribute these changes. 26.\" 27.Dd May 28, 2025 28.Dt DDB 4 29.Os 30.Sh NAME 31.Nm ddb 32.Nd interactive kernel debugger 33.Sh SYNOPSIS 34In order to enable kernel debugging facilities include: 35.Bd -ragged -offset indent 36.Cd options KDB 37.Cd options DDB 38.Ed 39.Pp 40To prevent activation of the debugger on kernel 41.Xr panic 9 : 42.Bd -ragged -offset indent 43.Cd options KDB_UNATTENDED 44.Ed 45.Pp 46In order to print a stack trace of the current thread on the console 47for a panic: 48.Bd -ragged -offset indent 49.Cd options KDB_TRACE 50.Ed 51.Pp 52To print the numerical value of symbols in addition to the symbolic 53representation, define: 54.Bd -ragged -offset indent 55.Cd options DDB_NUMSYM 56.Ed 57.Pp 58To enable the 59.Xr gdb 4 60backend, so that remote debugging with 61.Xr kgdb 1 Pq Pa ports/devel/gdb 62is possible, include: 63.Bd -ragged -offset indent 64.Cd options GDB 65.Ed 66.Sh DESCRIPTION 67The 68.Nm 69kernel debugger is an interactive debugger with a syntax inspired by 70.Xr gdb 1 Pq Pa ports/devel/gdb . 71If linked into the running kernel, 72it can be invoked locally with the 73.Ql debug 74.Xr keymap 5 75action, usually mapped to Ctrl+Alt+Esc, or by setting the 76.Va debug.kdb.enter 77sysctl to 1. 78The debugger is also invoked on kernel 79.Xr panic 9 80if the 81.Va debug.debugger_on_panic 82.Xr sysctl 8 83MIB variable is set non-zero, 84which is the default 85unless the 86.Dv KDB_UNATTENDED 87option is specified. 88Similarly, if the 89.Va debug.debugger_on_recursive_panic 90variable is set to 91.Dv 1 , 92then the debugger will be invoked on a recursive kernel panic. 93This variable has a default value of 94.Dv 0 , 95and has no effect if 96.Va debug.debugger_on_panic 97is already set non-zero. 98.Pp 99The current location is called 100.Va dot . 101The 102.Va dot 103is displayed with 104a hexadecimal format at a prompt. 105The commands 106.Ic examine 107and 108.Ic write 109update 110.Va dot 111to the address of the last line 112examined or the last location modified, and set 113.Va next 114to the address of 115the next location to be examined or changed. 116Other commands do not change 117.Va dot , 118and set 119.Va next 120to be the same as 121.Va dot . 122.Pp 123The general command syntax is: 124.Ar command Ns Op Li / Ns Ar modifier 125.Oo Ar addr Oc Ns Op , Ns Ar count 126.Pp 127A blank line repeats the previous command from the address 128.Va next 129with 130count 1 and no modifiers. 131Specifying 132.Ar addr 133sets 134.Va dot 135to the address. 136Omitting 137.Ar addr 138uses 139.Va dot . 140A missing 141.Ar count 142is taken 143to be 1 for printing commands or infinity for stack traces. 144A 145.Ar count 146of -1 is equivalent to a missing 147.Ar count . 148Options that are supplied but not supported by the given 149.Ar command 150are usually ignored. 151.Pp 152The 153.Nm 154debugger has a pager feature (like the 155.Xr more 1 156command) 157for the output. 158If an output line exceeds the number set in the 159.Va lines 160variable, it displays 161.Dq Li --More-- 162and waits for a response. 163The valid responses for it are: 164.Pp 165.Bl -tag -compact -width ".Li SPC" 166.It Li SPC 167one more page 168.It Li RET 169one more line 170.It Li q 171abort the current command, and return to the command input mode 172.El 173.Pp 174Finally, 175.Nm 176provides a small (currently 10 items) command history, and offers 177simple 178.Nm emacs Ns -style 179command line editing capabilities. 180In addition to 181the 182.Nm emacs 183control keys, the usual ANSI arrow keys may be used to browse through the 184history buffer, and move the cursor within the current line. 185.Sh COMMANDS 186.Ss COMMON DEBUGGER COMMANDS 187.Bl -tag -width indent -compact 188.It Ic help 189Print a short summary of the available commands and command 190abbreviations. 191.Pp 192.It Xo 193.Ic examine Ns Op Li / Ns Cm AISabcdghilmorsuxz ... 194.Oo Ar addr Oc Ns Op , Ns Ar count 195.Xc 196.It Xo 197.Ic x Ns Op Li / Ns Cm AISabcdghilmorsuxz ... 198.Oo Ar addr Oc Ns Op , Ns Ar count 199.Xc 200Display the addressed locations according to the formats in the modifier. 201Multiple modifier formats display multiple locations. 202If no format is specified, the last format specified for this command 203is used. 204.Pp 205The format characters are: 206.Bl -tag -compact -width indent 207.It Cm b 208look at by bytes (8 bits) 209.It Cm h 210look at by half words (16 bits) 211.It Cm l 212look at by long words (32 bits) 213.It Cm g 214look at by quad words (64 bits) 215.It Cm a 216print the location being displayed 217.It Cm A 218print the location with a line number if possible 219.It Cm x 220display in unsigned hex 221.It Cm z 222display in signed hex 223.It Cm o 224display in unsigned octal 225.It Cm d 226display in signed decimal 227.It Cm u 228display in unsigned decimal 229.It Cm r 230display in current radix, signed 231.It Cm c 232display low 8 bits as a character. 233Non-printing characters are displayed as an octal escape code (e.g., 234.Ql \e000 ) . 235.It Cm s 236display the null-terminated string at the location. 237Non-printing characters are displayed as octal escapes. 238.It Cm m 239display in unsigned hex with character dump at the end of each line. 240The location is also displayed in hex at the beginning of each line. 241.It Cm i 242display as a disassembled instruction 243.It Cm I 244display as a disassembled instruction with possible alternate formats 245depending on the machine. 246On i386, this selects the alternate format for the instruction decoding 247(16 bits in a 32-bit code segment and vice versa). 248.It Cm S 249display a symbol name for the pointer stored at the address 250.El 251.Pp 252.It Ic xf 253Examine forward: 254execute an 255.Ic examine 256command with the last specified parameters to it 257except that the next address displayed by it is used as the start address. 258.Pp 259.It Ic xb 260Examine backward: 261execute an 262.Ic examine 263command with the last specified parameters to it 264except that the last start address subtracted by the size displayed by it 265is used as the start address. 266.Pp 267.It Ic print Ns Op Li / Ns Cm acdoruxz 268.It Ic p Ns Op Li / Ns Cm acdoruxz 269Print 270.Ar addr Ns s 271according to the modifier character (as described above for 272.Cm examine ) . 273Valid formats are: 274.Cm a , x , z , o , d , u , r , 275and 276.Cm c . 277If no modifier is specified, the last one specified to it is used. 278The argument 279.Ar addr 280can be a string, in which case it is printed as it is. 281For example: 282.Bd -literal -offset indent 283print/x "eax = " $eax "\enecx = " $ecx "\en" 284.Ed 285.Pp 286will print like: 287.Bd -literal -offset indent 288eax = xxxxxx 289ecx = yyyyyy 290.Ed 291.Pp 292.It Ic pprint Ns Oo Li / Ns Cm d depth Oc Oo Ar name Oc 293Pretty-print symbol specified by 294.Ar name 295using CTF debugging data. Works for all symbols exported by the kernel and loaded kernel modules. 296.Pp 297If the 298.Cm d 299modifier has been specified, contents of structs nested up to 300.Ar depth 301levels deep will also be included in the output. 302.Pp 303.It Ic pprint struct Ns Oo Li / Ns Cm d depth Ic Oc Oo Ar name Oc Ns Op Ns Ar addr 304Print memory at 305.Ar addr 306as struct 307.Ar name Ns . 308Works for all structs defined by the kernel and loaded kernel modules. 309.Pp 310If the 311.Cm d 312modifier has been specified, contents of structs nested up to 313.Ar depth 314levels deep will also be included in the output. 315.Pp 316.It Xo 317.Ic write Ns Op Li / Ns Cm bhl 318.Ar addr expr1 Op Ar expr2 ... 319.Xc 320.It Xo 321.Ic w Ns Op Li / Ns Cm bhl 322.Ar addr expr1 Op Ar expr2 ... 323.Xc 324Write the expressions specified after 325.Ar addr 326on the command line at succeeding locations starting with 327.Ar addr . 328The write unit size can be specified in the modifier with a letter 329.Cm b 330(byte), 331.Cm h 332(half word) or 333.Cm l 334(long word) respectively. 335If omitted, 336long word is assumed. 337.Pp 338.Sy Warning : 339since there is no delimiter between expressions, strange 340things may happen. 341It is best to enclose each expression in parentheses. 342.Pp 343.It Ic set Li $ Ns Ar variable Oo Li = Oc Ar expr 344Set the named variable or register with the value of 345.Ar expr . 346Valid variable names are described below. 347.Pp 348.It Ic break Ns Oo Li / Ns Cm u Oc Oo Ar addr Oc Ns Op , Ns Ar count 349.It Ic b Ns Oo Li / Ns Cm u Oc Oo Ar addr Oc Ns Op , Ns Ar count 350Set a break point at 351.Ar addr . 352If 353.Ar count 354is supplied, the 355.Ic continue 356command will not stop at this break point on the first 357.Ar count 358\- 1 times that it is hit. 359If the break point is set, a break point number is 360printed with 361.Ql # . 362This number can be used in deleting the break point 363or adding conditions to it. 364.Pp 365If the 366.Cm u 367modifier is specified, this command sets a break point in user 368address space. 369Without the 370.Cm u 371option, the address is considered to be in the kernel 372space, and a wrong space address is rejected with an error message. 373This modifier can be used only if it is supported by machine dependent 374routines. 375.Pp 376.Sy Warning : 377If a user text is shadowed by a normal user space debugger, 378user space break points may not work correctly. 379Setting a break 380point at the low-level code paths may also cause strange behavior. 381.Pp 382.It Ic delete Op Ar addr 383.It Ic d Op Ar addr 384.It Ic delete Li # Ns Ar number 385.It Ic d Li # Ns Ar number 386Delete the specified break point. 387The break point can be specified by a 388break point number with 389.Ql # , 390or by using the same 391.Ar addr 392specified in the original 393.Ic break 394command, or by omitting 395.Ar addr 396to get the default address of 397.Va dot . 398.Pp 399.It Ic halt 400Halt the system. 401.Pp 402.It Ic watch Oo Ar addr Oc Ns Op , Ns Ar size 403Set a watchpoint for a region. 404Execution stops when an attempt to modify the region occurs. 405The 406.Ar size 407argument defaults to 4. 408If you specify a wrong space address, the request is rejected 409with an error message. 410.Pp 411.Sy Warning : 412Attempts to watch wired kernel memory 413may cause unrecoverable error in some systems such as i386. 414Watchpoints on user addresses work best. 415.Pp 416.It Ic hwatch Oo Ar addr Oc Ns Op , Ns Ar size 417Set a hardware watchpoint for a region if supported by the 418architecture. 419Execution stops when an attempt to modify the region occurs. 420The 421.Ar size 422argument defaults to 4. 423.Pp 424.Sy Warning : 425The hardware debug facilities do not have a concept of separate 426address spaces like the watch command does. 427Use 428.Ic hwatch 429for setting watchpoints on kernel address locations only, and avoid 430its use on user mode address spaces. 431.Pp 432.It Ic dhwatch Oo Ar addr Oc Ns Op , Ns Ar size 433Delete specified hardware watchpoint. 434.Pp 435.It Ic kill Ar sig pid 436Send signal 437.Ar sig 438to process 439.Ar pid . 440The signal is acted on upon returning from the debugger. 441This command can be used to kill a process causing resource contention 442in the case of a hung system. 443See 444.Xr signal 3 445for a list of signals. 446Note that the arguments are reversed relative to 447.Xr kill 2 . 448.Pp 449.It Ic step Ns Oo Li / Ns Cm p Oc Ns Op , Ns Ar count 450.It Ic s Ns Oo Li / Ns Cm p Oc Ns Op , Ns Ar count 451Single step 452.Ar count 453times. 454If the 455.Cm p 456modifier is specified, print each instruction at each step. 457Otherwise, only print the last instruction. 458.Pp 459.Sy Warning : 460depending on machine type, it may not be possible to 461single-step through some low-level code paths or user space code. 462On machines with software-emulated single-stepping (e.g., pmax), 463stepping through code executed by interrupt handlers will probably 464do the wrong thing. 465.Pp 466.It Ic continue Ns Op Li / Ns Cm c 467.It Ic c Ns Op Li / Ns Cm c 468Continue execution until a breakpoint or watchpoint. 469If the 470.Cm c 471modifier is specified, count instructions while executing. 472Some machines (e.g., pmax) also count loads and stores. 473.Pp 474.Sy Warning : 475when counting, the debugger is really silently single-stepping. 476This means that single-stepping on low-level code may cause strange 477behavior. 478.Pp 479.It Ic until Ns Op Li / Ns Cm p 480Stop at the next call or return instruction. 481If the 482.Cm p 483modifier is specified, print the call nesting depth and the 484cumulative instruction count at each call or return. 485Otherwise, 486only print when the matching return is hit. 487.Pp 488.It Ic next Ns Op Li / Ns Cm p 489.It Ic match Ns Op Li / Ns Cm p 490Stop at the matching return instruction. 491If the 492.Cm p 493modifier is specified, print the call nesting depth and the 494cumulative instruction count at each call or return. 495Otherwise, only print when the matching return is hit. 496.Pp 497.It Xo 498.Ic trace Ns Op Li / Ns Cm u 499.Op Ar pid | tid Ns 500.Op , Ns Ar count 501.Xc 502.It Xo 503.Ic t Ns Op Li / Ns Cm u 504.Op Ar pid | tid Ns 505.Op , Ns Ar count 506.Xc 507.It Xo 508.Ic where Ns Op Li / Ns Cm u 509.Op Ar pid | tid Ns 510.Op , Ns Ar count 511.Xc 512.It Xo 513.Ic bt Ns Op Li / Ns Cm u 514.Op Ar pid | tid Ns 515.Op , Ns Ar count 516.Xc 517Stack trace. 518The 519.Cm u 520option traces user space; if omitted, 521.Ic trace 522only traces 523kernel space. 524The optional argument 525.Ar count 526is the number of frames to be traced. 527If 528.Ar count 529is omitted, all frames are printed. 530.Pp 531.Sy Warning : 532User space stack trace is valid 533only if the machine dependent code supports it. 534.Pp 535.It Xo 536.Ic search Ns Op Li / Ns Cm bhl 537.Ar addr 538.Ar value 539.Op Ar mask Ns 540.Op , Ns Ar count 541.Xc 542Search memory for 543.Ar value . 544The optional 545.Ar count 546argument limits the search. 547.\" 548.Pp 549.It Xo 550.Ic Ic reboot Ns Op Li / Ns Cm s 551.Op Ar seconds 552.Xc 553.It Xo 554.Ic Ic reset Ns Op Li / Ns Cm s 555.Op Ar seconds 556.Xc 557Hard reset the system. 558If the optional argument 559.Ar seconds 560is given, the debugger will wait for this long, at most a week, 561before rebooting. 562When the 563.Cm s 564modifier is given, the command will skip running any registered shutdown 565handlers and attempt the most basic reset. 566.Pp 567.It Ic thread Ar addr | tid 568Switch the debugger to the thread with ID 569.Ar tid , 570if the argument is a decimal number, or address 571.Ar addr , 572otherwise. 573.Pp 574.It Ic watchdog Op Ar exp 575Program the 576.Xr watchdog 4 577timer to fire in 578.Pf 2^ Ar exp 579seconds. 580If no argument is provided, the watchdog timer is disabled. 581.El 582.Ss SPECIALIZED HELPER COMMANDS 583.Bl -tag -width indent -compact 584.It Xo 585.Ic findstack 586.Ar addr 587.Xc 588Prints the address of the thread whose kernel-mode stack contains 589.Ar addr , 590if any. 591.Pp 592.It Ic show Cm active trace 593.It Ic acttrace 594Show a stack trace for every thread running on a CPU. 595.Pp 596.It Ic show Cm all procs Ns Op Li / Ns Cm a 597.It Ic ps Ns Op Li / Ns Cm a 598Display all process information. 599The process information may not be shown if it is not 600supported in the machine, or the bottom of the stack of the 601target process is not in the main memory at that time. 602The 603.Cm a 604modifier will print command line arguments for each process. 605.\" 606.Pp 607.It Ic show Cm all tcpcbs Ns Op Li / Ns Cm l 608Show the same output as "show tcpcb" does, but for all 609TCP control blocks within the system. 610Using the 611.Cm l 612modifier will limit the output to TCP control blocks, which are locked. 613.\" 614.Pp 615.It Ic show Cm all trace 616.It Ic alltrace 617Show a stack trace for every thread in the system. 618.Pp 619.It Ic show Cm all ttys 620Show all TTY's within the system. 621Output is similar to 622.Xr pstat 8 , 623but also includes the address of the TTY structure. 624.\" 625.Pp 626.It Ic show Cm all vnets 627Show the same output as "show vnet" does, but lists all 628virtualized network stacks within the system. 629.\" 630.Pp 631.It Ic show Cm allchains 632Show the same information like "show lockchain" does, but 633for every thread in the system. 634.\" 635.Pp 636.It Ic show Cm alllocks 637Show all locks that are currently held. 638This command is only available if 639.Xr witness 4 640is included in the kernel. 641.\" 642.Pp 643.It Ic show Cm allpcpu 644The same as "show pcpu", but for every CPU present in the system. 645.\" 646.Pp 647.It Ic show Cm allrman 648Show information related with resource management, including 649interrupt request lines, DMA request lines, I/O ports, I/O memory 650addresses, and Resource IDs. 651.\" 652.Pp 653.It Ic show Cm apic 654Dump data about APIC IDT vector mappings. 655.\" 656.Pp 657.It Ic show Cm badstacks 658Walk the 659.Xr witness 4 660graph and print any lock-order violations. 661This command is only available if 662.Xr witness 4 663is included in the kernel. 664.\" 665.Pp 666.It Ic show Cm breaks 667Show breakpoints set with the "break" command. 668.\" 669.Pp 670.It Ic show Cm bio Ar addr 671Show information about the bio structure 672.Vt struct bio 673present at 674.Ar addr . 675See the 676.Pa sys/bio.h 677header file and 678.Xr g_bio 9 679for more details on the exact meaning of the structure fields. 680.\" 681.Pp 682.It Ic show Cm buffer Ar addr 683Show information about the buf structure 684.Vt struct buf 685present at 686.Ar addr . 687See the 688.Pa sys/buf.h 689header file for more details on the exact meaning of the structure fields. 690.\" 691.Pp 692.It Ic show Cm callout Ar addr 693Show information about the callout structure 694.Vt struct callout 695present at 696.Ar addr . 697.\" 698.Pp 699.It Ic show Cm cdev Op Ar addr 700Show the internal devfs state of the cdev structure located at 701.Ar addr . 702If no argument is provided, show the list of all created cdevs, consisting of 703the devfs node name and the 704.Vt struct cdev 705address. 706.\" 707.Pp 708.It Ic show Cm conifhk 709Lists hooks currently waiting for completion in 710.Fn run_interrupt_driven_config_hooks . 711.\" 712.Pp 713.It Ic show Cm cpusets 714Print numbered root and assigned CPU affinity sets. 715See 716.Xr cpuset 2 717for more details. 718.\" 719.Pp 720.It Ic show Cm cyrixreg 721Show registers specific to the Cyrix processor. 722.\" 723.Pp 724.It Ic show Cm devmap 725Prints the contents of the static device mapping table. 726Currently only available on the 727ARM 728architecture. 729.\" 730.Pp 731.It Ic show Cm domain Ar addr 732Print protocol domain structure 733.Vt struct domain 734at address 735.Ar addr . 736See the 737.Pa sys/domain.h 738header file for more details on the exact meaning of the structure fields. 739.\" 740.Pp 741.It Ic show Cm ffs Op Ar addr 742Show brief information about ffs mount at the address 743.Ar addr , 744if argument is given. 745Otherwise, provides the summary about each ffs mount. 746.\" 747.Pp 748.It Ic show Cm file Ar addr 749Show information about the file structure 750.Vt struct file 751present at address 752.Ar addr . 753.\" 754.Pp 755.It Ic show Cm files 756Show information about every file structure in the system. 757.\" 758.Pp 759.It Ic show Cm freepages 760Show the number of physical pages in each of the free lists. 761.\" 762.Pp 763.It Ic show Cm geom Op Ar addr 764If the 765.Ar addr 766argument is not given, displays the entire GEOM topology. 767If 768.Ar addr 769is given, displays details about the given GEOM object (class, geom, 770provider or consumer). 771.\" 772.Pp 773.It Ic show Cm idt 774Show IDT layout. 775The first column specifies the IDT vector. 776The second one is the name of the interrupt/trap handler. 777Those functions are machine dependent. 778.\" 779.Pp 780.It Ic show Cm igi_list Ar addr 781Show information about the IGMP structure 782.Vt struct igmp_ifsoftc 783present at 784.Ar addr . 785.\" 786.Pp 787.It Ic show Cm iosched Ar addr 788Show information about the I/O scheduler 789.Vt struct cam_iosched_softc 790located at 791.Ar addr . 792.\" 793.Pp 794.It Ic show Cm inodedeps Op Ar addr 795Show brief information about each inodedep structure. 796If 797.Ar addr 798is given, only inodedeps belonging to the fs located at the 799supplied address are shown. 800.\" 801.Pp 802.It Ic show Cm inpcb Ar addr 803Show information on IP Control Block 804.Vt struct in_pcb 805present at 806.Ar addr . 807.\" 808.Pp 809.It Ic show Cm intr 810Dump information about interrupt handlers. 811.\" 812.Pp 813.It Ic show Cm intrcnt 814Dump the interrupt statistics. 815.\" 816.Pp 817.It Ic show Cm irqs 818Show interrupt lines and their respective kernel threads. 819.\" 820.Pp 821.It Ic show Cm ktr Ns Op Li / Ns Cm a Ns Cm v Ns Cm V 822Print the contents of the 823.Xr ktr 4 824trace buffer. 825The 826.Cm v 827modifier will request fully verbose output, causing the file, line number, and 828timestamp to be printed for each trace entry. 829The 830.Cm V 831modifier will request only the timestamps to be printed. 832The 833.Cm a 834modifier will request that the output be unpaginated. 835.\" 836.Pp 837.It Ic show Cm lapic 838Show information from the local APIC registers for this CPU. 839.\" 840.Pp 841.It Ic show Cm lock Ar addr 842Show lock structure. 843The output format is as follows: 844.Bl -tag -width "flags" 845.It Ic class : 846Class of the lock. 847Possible types include 848.Xr mutex 9 , 849.Xr rmlock 9 , 850.Xr rwlock 9 , 851.Xr sx 9 . 852.It Ic name : 853Name of the lock. 854.It Ic flags : 855Flags passed to the lock initialization function. 856.Em flags 857values are lock class specific. 858.It Ic state : 859Current state of a lock. 860.Em state 861values are lock class specific. 862.It Ic owner : 863Lock owner. 864.El 865.\" 866.Pp 867.It Ic show Cm lockchain Ar addr 868Show all threads a particular thread at address 869.Ar addr 870is waiting on based on non-spin locks. 871.\" 872.Pp 873.It Ic show Cm lockedbufs 874Show the same information as "show buf", but for every locked 875.Vt struct buf 876object. 877.\" 878.Pp 879.It Ic show Cm lockedvnods 880List all locked vnodes in the system. 881.\" 882.Pp 883.It Ic show Cm locks 884Prints all locks that are currently acquired. 885This command is only available if 886.Xr witness 4 887is included in the kernel. 888.\" 889.Pp 890.It Ic show Cm locktree 891.\" 892.Pp 893.It Ic show Cm malloc Ns Op Li / Ns Cm i 894Prints 895.Xr malloc 9 896memory allocator statistics. 897If the 898.Cm i 899modifier is specified, format output as machine-parseable comma-separated 900values ("CSV"). 901The output columns are as follows: 902.Pp 903.Bl -tag -compact -offset indent -width "Requests" 904.It Ic Type 905Specifies a type of memory. 906It is the same as a description string used while defining the 907given memory type with 908.Xr MALLOC_DECLARE 9 . 909.It Ic InUse 910Number of memory allocations of the given type, for which 911.Xr free 9 912has not been called yet. 913.It Ic MemUse 914Total memory consumed by the given allocation type. 915.It Ic Requests 916Number of memory allocation requests for the given 917memory type. 918.El 919.Pp 920The same information can be gathered in userspace with 921.Dq Nm vmstat Fl m . 922.\" 923.Pp 924.It Ic show Cm map Ns Oo Li / Ns Cm f Oc Ar addr 925Prints the VM map at 926.Ar addr . 927If the 928.Cm f 929modifier is specified the 930complete map is printed. 931.\" 932.Pp 933.It Ic show Cm msgbuf 934Print the system's message buffer. 935It is the same output as in the 936.Dq Nm dmesg 937case. 938It is useful if you got a kernel panic, attached a serial cable 939to the machine and want to get the boot messages from before the 940system hang. 941.\" 942.Pp 943.It Ic show Cm mount Op Ar addr 944Displays details about the mount point located at 945.Ar addr . 946If no 947.Ar addr 948is specified, 949displays short info about all currently mounted file systems. 950.\" 951.Pp 952.It Ic show Cm object Ns Oo Li / Ns Cm f Oc Ar addr 953Prints the VM object at 954.Ar addr . 955If the 956.Cm f 957option is specified the 958complete object is printed. 959.\" 960.Pp 961.It Ic show Cm panic 962Print the panic message if set. 963.\" 964.Pp 965.It Ic show Cm page 966Show statistics on VM pages. 967.\" 968.Pp 969.It Ic show Cm pageq 970Show statistics on VM page queues. 971.\" 972.Pp 973.It Ic show Cm pciregs 974Print PCI bus registers. 975The same information can be gathered in userspace by running 976.Dq Nm pciconf Fl lv . 977.\" 978.Pp 979.It Ic show Cm pcpu 980Print current processor state. 981The output format is as follows: 982.Pp 983.Bl -tag -compact -offset indent -width "spin locks held:" 984.It Ic cpuid 985Processor identifier. 986.It Ic curthread 987Thread pointer, process identifier and the name of the process. 988.It Ic curpcb 989Control block pointer. 990.It Ic fpcurthread 991FPU thread pointer. 992.It Ic idlethread 993Idle thread pointer. 994.It Ic APIC ID 995CPU identifier coming from APIC. 996.It Ic currentldt 997LDT pointer. 998.It Ic spin locks held 999Names of spin locks held. 1000.El 1001.\" 1002.Pp 1003.It Ic show Cm pgrpdump 1004Dump process groups present within the system. 1005.\" 1006.Pp 1007.It Ic show Cm prison Op Ar addr 1008Show the prison structure located at 1009.Ar addr . 1010If no 1011.Ar addr 1012argument is specified, show information about all prisons in the system. 1013.\" 1014.Pp 1015.It Ic show Cm proc Op Ar addr 1016Show information about the process structure located at address 1017.Ar addr , 1018or the current process if no argument is specified. 1019.\" 1020.Pp 1021.It Ic show Cm procvm Op Ar addr 1022Show process virtual memory layout for the process located at 1023.Ar addr , 1024or the current process if no argument is specified. 1025.\" 1026.Pp 1027.It Ic show Cm protosw Ar addr 1028Print protocol switch structure 1029.Vt struct protosw 1030at address 1031.Ar addr . 1032.\" 1033.Pp 1034.It Ic show Cm registers Ns Op Li / Ns Cm u 1035Display the register set. 1036If the 1037.Cm u 1038modifier is specified, the register contents of the thread's previous 1039trapframe are displayed instead. 1040Usually, this corresponds to the saved state from userspace. 1041.\" 1042.Pp 1043.It Ic show Cm rman Ar addr 1044Show resource manager object 1045.Vt struct rman 1046at address 1047.Ar addr . 1048Addresses of particular pointers can be gathered with "show allrman" 1049command. 1050.\" 1051.Pp 1052.It Ic show Cm route Ar addr 1053Show route table result for destination 1054.Ar addr . 1055At this time, INET and INET6 formatted addresses are supported. 1056.\" 1057.Pp 1058.It Ic show Cm routetable Oo Ar af Oc 1059Show full route table or tables. 1060If 1061.Ar af 1062is specified, show only routes for the given numeric address family. 1063If no argument is specified, dump the route table for all address families. 1064.\" 1065.Pp 1066.It Ic show Cm rtc 1067Show real time clock value. 1068Useful for long debugging sessions. 1069.\" 1070.Pp 1071.It Ic show Cm sleepchain 1072Deprecated. 1073Now an alias for 1074.Ic show Cm lockchain . 1075.\" 1076.Pp 1077.It Ic show Cm sleepq Ar addr 1078.It Ic show Cm sleepqueue Ar addr 1079Show the 1080.Xr sleepqueue 9 1081structure located at 1082.Ar addr . 1083.\" 1084.Pp 1085.It Ic show Cm sockbuf Ar addr 1086Show the socket buffer 1087.Va struct sockbuf 1088located at 1089.Ar addr . 1090.\" 1091.Pp 1092.It Ic show Cm socket Ar addr 1093Show the socket object 1094.Vt struct socket 1095located at 1096.Ar addr . 1097.\" 1098.Pp 1099.It Ic show Cm sysregs 1100Show system registers (e.g., 1101.Li cr0-4 1102on i386.) 1103Not present on some platforms. 1104.\" 1105.Pp 1106.It Ic show Cm tcpcb Ar addr 1107Print TCP control block 1108.Vt struct tcpcb 1109lying at address 1110.Ar addr . 1111For exact interpretation of output, visit 1112.Pa netinet/tcp.h 1113header file. 1114.\" 1115.Pp 1116.It Ic show Cm thread Op Ar addr | tid 1117If no 1118.Ar addr 1119or 1120.Ar tid 1121is specified, show detailed information about current thread. 1122Otherwise, print information about the thread with ID 1123.Ar tid 1124or kernel address 1125.Ar addr . 1126(If the argument is a decimal number, it is assumed to be a tid.) 1127.\" 1128.Pp 1129.It Ic show Cm threads 1130Show all threads within the system. 1131Output format is as follows: 1132.Pp 1133.Bl -tag -compact -offset indent -width "Second column" 1134.It Ic First column 1135Thread identifier (TID) 1136.It Ic Second column 1137Thread structure address 1138.It Ic Third column 1139Backtrace. 1140.El 1141.\" 1142.Pp 1143.It Ic show Cm tty Ar addr 1144Display the contents of a TTY structure in a readable form. 1145.\" 1146.Pp 1147.It Ic show Cm turnstile Ar addr 1148Show turnstile 1149.Vt struct turnstile 1150structure at address 1151.Ar addr . 1152Turnstiles are structures used within the 1153.Fx 1154kernel to implement 1155synchronization primitives which, while holding a specific type of lock, cannot 1156sleep or context switch to another thread. 1157Currently, those are: 1158.Xr mutex 9 , 1159.Xr rwlock 9 , 1160.Xr rmlock 9 . 1161.\" 1162.Pp 1163.It Ic show Cm uma Ns Op Li / Ns Cm i 1164Show UMA allocator statistics. 1165If the 1166.Cm i 1167modifier is specified, format output as machine-parseable comma-separated 1168values ("CSV"). 1169The output contains the following columns: 1170.Pp 1171.Bl -tag -compact -offset indent -width "Total Mem" 1172.It Cm "Zone" 1173Name of the UMA zone. 1174The same string that was passed to 1175.Xr uma_zcreate 9 1176as a first argument. 1177.It Cm "Size" 1178Size of a given memory object (slab). 1179.It Cm "Used" 1180Number of slabs being currently used. 1181.It Cm "Free" 1182Number of free slabs within the UMA zone. 1183.It Cm "Requests" 1184Number of allocations requests to the given zone. 1185.It Cm "Total Mem" 1186Total memory in use (either allocated or free) by a zone, in bytes. 1187.It Cm "XFree" 1188Number of free slabs within the UMA zone that were freed on a different NUMA 1189domain than allocated. 1190(The count in the 1191.Cm "Free" 1192column is inclusive of 1193.Cm "XFree" . ) 1194.El 1195.Pp 1196The same information might be gathered in the userspace 1197with the help of 1198.Dq Nm vmstat Fl z . 1199.\" 1200.Pp 1201.It Ic show Cm unpcb Ar addr 1202Shows UNIX domain socket private control block 1203.Vt struct unpcb 1204present at the address 1205.Ar addr . 1206.\" 1207.Pp 1208.It Ic show Cm vmochk 1209Prints, whether the internal VM objects are in a map somewhere 1210and none have zero ref counts. 1211.\" 1212.Pp 1213.It Ic show Cm vmopag 1214Walk the list of VM objects in the system, printing the indices and physical 1215addresses of the VM pages belonging to each object. 1216.\" 1217.Pp 1218.It Ic show Cm vnet Ar addr 1219Prints virtualized network stack 1220.Vt struct vnet 1221structure present at the address 1222.Ar addr . 1223.\" 1224.Pp 1225.It Ic show Cm vnode Ar addr 1226Prints vnode 1227.Vt struct vnode 1228structure lying at 1229.Ar addr . 1230For the exact interpretation of the output, look at the 1231.Pa sys/vnode.h 1232header file. 1233.\" 1234.Pp 1235.It Ic show Cm vnodebufs Ar addr 1236Shows clean/dirty buffer lists of the vnode located at 1237.Ar addr . 1238.\" 1239.Pp 1240.It Ic show Cm vpath Ar addr 1241Walk the namecache to lookup the pathname of the vnode located at 1242.Ar addr . 1243.\" 1244.Pp 1245.It Ic show Cm watches 1246Displays all watchpoints. 1247Shows watchpoints set with "watch" command. 1248.\" 1249.Pp 1250.It Ic show Cm witness 1251Shows information about lock acquisition coming from the 1252.Xr witness 4 1253subsystem. 1254.El 1255.Ss OFFLINE DEBUGGING COMMANDS 1256.Bl -tag -width indent -compact 1257.It Ic dump 1258Initiate a kernel core dump to the device(s) configured by 1259.Xr dumpon 8 . 1260.Pp 1261.It Ic gdb 1262Switches to remote GDB mode. 1263In remote GDB mode, another machine is required that runs 1264.Xr gdb 1 Pq Pa ports/devel/gdb 1265using the remote debug feature, with a connection to the serial 1266console port on the target machine. 1267.Pp 1268.It Ic netdump Fl s Ar server Oo Fl g Ar gateway Fl c Ar client Fl i Ar iface Oc 1269Configure 1270.Xr netdump 4 1271with the provided parameters, and immediately perform a netdump. 1272.Pp 1273There are some known limitations. 1274Principally, 1275.Xr netdump 4 1276only supports IPv4 at this time. 1277The address arguments to the 1278.Ic netdump 1279command must be dotted decimal IPv4 addresses. 1280(Hostnames are not supported.) 1281At present, the command only works if the machine is in a panic state. 1282Finally, the 1283.Nm 1284.Ic netdump 1285command does not provide any way to configure compression or encryption. 1286.Pp 1287.It Ic netgdb Fl s Ar server Oo Fl g Ar gateway Fl c Ar client Fl i Ar iface Oc 1288Initiate a 1289.Xr netgdb 4 1290session with the provided parameters. 1291.Pp 1292.Ic netgdb 1293has identical limitations to 1294.Ic netdump . 1295.Pp 1296.It Ic capture on 1297.It Ic capture off 1298.It Ic capture reset 1299.It Ic capture status 1300.Nm 1301supports a basic output capture facility, which can be used to retrieve the 1302results of debugging commands from userspace using 1303.Xr sysctl 3 . 1304.Ic capture on 1305enables output capture; 1306.Ic capture off 1307disables capture. 1308.Ic capture reset 1309will clear the capture buffer and disable capture. 1310.Ic capture status 1311will report current buffer use, buffer size, and disposition of output 1312capture. 1313.Pp 1314Userspace processes may inspect and manage 1315.Nm 1316capture state using 1317.Xr sysctl 8 : 1318.Pp 1319.Va debug.ddb.capture.bufsize 1320may be used to query or set the current capture buffer size. 1321.Pp 1322.Va debug.ddb.capture.maxbufsize 1323may be used to query the compile-time limit on the capture buffer size. 1324.Pp 1325.Va debug.ddb.capture.bytes 1326may be used to query the number of bytes of output currently in the capture 1327buffer. 1328.Pp 1329.Va debug.ddb.capture.data 1330returns the contents of the buffer as a string to an appropriately privileged 1331process. 1332.Pp 1333This facility is particularly useful in concert with the scripting and 1334.Xr textdump 4 1335facilities, allowing scripted debugging output to be captured and 1336committed to disk as part of a textdump for later analysis. 1337The contents of the capture buffer may also be inspected in a kernel core dump 1338using 1339.Xr kgdb 1 Pq Pa ports/devel/gdb . 1340.Pp 1341.It Ic run 1342.It Ic script 1343.It Ic scripts 1344.It Ic unscript 1345Run, define, list, and delete scripts. 1346See the 1347.Sx SCRIPTING 1348section for more information on the scripting facility. 1349.Pp 1350.It Ic textdump dump 1351.It Ic textdump set 1352.It Ic textdump status 1353.It Ic textdump unset 1354Use the 1355.Ic textdump dump 1356command to immediately perform a textdump. 1357More information may be found in 1358.Xr textdump 4 . 1359The 1360.Ic textdump set 1361command may be used to force the next kernel core dump to be a textdump 1362rather than a traditional memory dump or minidump. 1363.Ic textdump status 1364reports whether a textdump has been scheduled. 1365.Ic textdump unset 1366cancels a request to perform a textdump as the next kernel core dump. 1367.El 1368.Sh VARIABLES 1369The debugger accesses registers and variables as 1370.Li $ Ns Ar name . 1371Register names are as in the 1372.Dq Ic show Cm registers 1373command. 1374Some variables are suffixed with numbers, and may have some modifier 1375following a colon immediately after the variable name. 1376For example, register variables can have a 1377.Cm u 1378modifier to indicate user register (e.g., 1379.Dq Li $eax:u ) . 1380.Pp 1381Built-in variables currently supported are: 1382.Pp 1383.Bl -tag -width ".Va tabstops" -compact 1384.It Va radix 1385Input and output radix. 1386.It Va maxoff 1387Addresses are printed as 1388.Dq Ar symbol Ns Li + Ns Ar offset 1389unless 1390.Ar offset 1391is greater than 1392.Va maxoff . 1393.It Va maxwidth 1394The width of the displayed line. 1395.It Va lines 1396The number of lines. 1397It is used by the built-in pager. 1398Setting it to 0 disables paging. 1399.It Va tabstops 1400Tab stop width. 1401.It Va work Ns Ar xx 1402Work variable; 1403.Ar xx 1404can take values from 0 to 31. 1405.El 1406.Sh EXPRESSIONS 1407Most expression operators in C are supported except 1408.Ql ~ , 1409.Ql ^ , 1410and unary 1411.Ql & . 1412Special rules in 1413.Nm 1414are: 1415.Bl -tag -width ".No Identifiers" 1416.It Identifiers 1417The name of a symbol is translated to the value of the symbol, which 1418is the address of the corresponding object. 1419.Ql \&. 1420and 1421.Ql \&: 1422can be used in the identifier. 1423If supported by an object format dependent routine, 1424.Sm off 1425.Oo Ar filename : Oc Ar func : lineno , 1426.Sm on 1427.Oo Ar filename : Oc Ns Ar variable , 1428and 1429.Oo Ar filename : Oc Ns Ar lineno 1430can be accepted as a symbol. 1431.It Numbers 1432Radix is determined by the first two letters: 1433.Ql 0x : 1434hex, 1435.Ql 0o : 1436octal, 1437.Ql 0t : 1438decimal; otherwise, follow current radix. 1439.It Li \&. 1440.Va dot 1441.It Li + 1442.Va next 1443.It Li .. 1444address of the start of the last line examined. 1445Unlike 1446.Va dot 1447or 1448.Va next , 1449this is only changed by 1450.Ic examine 1451or 1452.Ic write 1453command. 1454.It Li ' 1455last address explicitly specified. 1456.It Li $ Ns Ar variable 1457Translated to the value of the specified variable. 1458It may be followed by a 1459.Ql \&: 1460and modifiers as described above. 1461.It Ar a Ns Li # Ns Ar b 1462A binary operator which rounds up the left hand side to the next 1463multiple of right hand side. 1464.It Li * Ns Ar expr 1465Indirection. 1466It may be followed by a 1467.Ql \&: 1468and modifiers as described above. 1469.El 1470.Sh SCRIPTING 1471.Nm 1472supports a basic scripting facility to allow automating tasks or responses to 1473specific events. 1474Each script consists of a list of DDB commands to be executed sequentially, 1475and is assigned a unique name. 1476Certain script names have special meaning, and will be automatically run on 1477various 1478.Nm 1479events if scripts by those names have been defined. 1480.Pp 1481The 1482.Ic script 1483command may be used to define a script by name. 1484Scripts consist of a series of 1485.Nm 1486commands separated with the 1487.Ql \&; 1488character. 1489For example: 1490.Bd -literal -offset indent 1491script kdb.enter.panic=bt; show pcpu 1492script lockinfo=show alllocks; show lockedvnods 1493.Ed 1494.Pp 1495The 1496.Ic scripts 1497command lists currently defined scripts. 1498.Pp 1499The 1500.Ic run 1501command execute a script by name. 1502For example: 1503.Bd -literal -offset indent 1504run lockinfo 1505.Ed 1506.Pp 1507The 1508.Ic unscript 1509command may be used to delete a script by name. 1510For example: 1511.Bd -literal -offset indent 1512unscript kdb.enter.panic 1513.Ed 1514.Pp 1515These functions may also be performed from userspace using the 1516.Xr ddb 8 1517command. 1518.Pp 1519Certain scripts are run automatically, if defined, for specific 1520.Nm 1521events. 1522The follow scripts are run when various events occur: 1523.Bl -tag -width kdb.enter.powerfail 1524.It Va kdb.enter.acpi 1525The kernel debugger was entered as a result of an 1526.Xr acpi 4 1527event. 1528.It Va kdb.enter.bootflags 1529The kernel debugger was entered at boot as a result of the debugger boot 1530flag being set. 1531.It Va kdb.enter.break 1532The kernel debugger was entered as a result of a serial or console break. 1533.It Va kdb.enter.cam 1534The kernel debugger was entered as a result of a 1535.Xr CAM 4 1536event. 1537.It Va kdb.enter.mac 1538The kernel debugger was entered as a result of an assertion failure in the 1539.Xr mac_test 4 1540module of the 1541TrustedBSD MAC Framework. 1542.It Va kdb.enter.netgraph 1543The kernel debugger was entered as a result of a 1544.Xr netgraph 4 1545event. 1546.It Va kdb.enter.panic 1547.Xr panic 9 1548was called. 1549.It Va kdb.enter.powerpc 1550The kernel debugger was entered as a result of an unimplemented interrupt 1551type on the powerpc platform. 1552.It Va kdb.enter.sysctl 1553The kernel debugger was entered as a result of the 1554.Va debug.kdb.enter 1555sysctl being set. 1556.It Va kdb.enter.unionfs 1557The kernel debugger was entered as a result of an assertion failure in the 1558union file system. 1559.It Va kdb.enter.unknown 1560The kernel debugger was entered, but no reason has been set. 1561.It Va kdb.enter.vfslock 1562The kernel debugger was entered as a result of a VFS lock violation. 1563.It Va kdb.enter.watchdog 1564The kernel debugger was entered as a result of a watchdog firing. 1565.It Va kdb.enter.witness 1566The kernel debugger was entered as a result of a 1567.Xr witness 4 1568violation. 1569.El 1570.Pp 1571In the event that none of these scripts is found, 1572.Nm 1573will attempt to execute a default script: 1574.Bl -tag -width kdb.enter.powerfail 1575.It Va kdb.enter.default 1576The kernel debugger was entered, but a script exactly matching the reason for 1577entering was not defined. 1578This can be used as a catch-all to handle cases not specifically of interest; 1579for example, 1580.Va kdb.enter.witness 1581might be defined to have special handling, and 1582.Va kdb.enter.default 1583might be defined to simply panic and reboot. 1584.El 1585.Sh HINTS 1586On machines with an ISA expansion bus, a simple NMI generation card can be 1587constructed by connecting a push button between the A01 and B01 (CHCHK# and 1588GND) card fingers. 1589Momentarily shorting these two fingers together may cause the bridge chipset to 1590generate an NMI, which causes the kernel to pass control to 1591.Nm . 1592Some bridge chipsets do not generate a NMI on CHCHK#, so your mileage may vary. 1593The NMI allows one to break into the debugger on a wedged machine to 1594diagnose problems. 1595Other bus' bridge chipsets may be able to generate NMI using bus specific 1596methods. 1597There are many PCI and PCIe add-in cards which can generate NMI for 1598debugging. 1599Modern server systems typically use IPMI to generate signals to enter the 1600debugger. 1601The 1602.Va devel/ipmitool 1603port can be used to send the 1604.Cd chassis power diag 1605command which delivers an NMI to the processor. 1606Embedded systems often use JTAG for debugging, but rarely use it in 1607combination with 1608.Nm . 1609.Pp 1610Serial consoles can break to the debugger by sending a BREAK 1611condition on the serial line. 1612This requires a kernel built with 1613.Cd options BREAK_TO_DEBUGGER 1614is specified in the kernel. 1615Most terminal emulation programs can send a break sequence with a 1616special key sequence or menu selection. 1617Sending the break can be difficult or even happen spuriously in some setups. 1618An alternative method is to build a kernel with 1619.Cd options ALT_BREAK_TO_DEBUGGER 1620then the sequence of CR TILDE CTRL-B enters the debugger; 1621CR TILDE CTRL-P causes a panic; and 1622CR TILDE CTRL-R causes an immediate reboot. 1623In all these sequences, CR represents Carriage Return and is usually 1624sent by pressing the Enter or Return key. 1625TILDE is the ASCII tilde character (~). 1626CTRL-x is Control x, sent by pressing the Control key, then x, then releasing 1627both. 1628.Pp 1629The break-to-debugger behavior can be enabled by setting 1630.Xr sysctl 8 1631.Va debug.kdb.break_to_debugger 1632to 1. 1633The alt-break-to-debugger behavior can be enabled by setting 1634.Xr sysctl 8 1635.Va debug.kdb.alt_break_to_debugger 1636to 1. 1637The debugger can be entered by setting 1638.Xr sysctl 8 1639.Va debug.kdb.enter 1640to 1. 1641.Pp 1642Output can be interrupted, paused, and resumed with the control 1643characters CTRL-C, CTRL-S, and CTRL-Q. 1644Because these control characters are received as in-band data from the 1645console, there is an input buffer, and once that buffer fills 1646.Nm 1647must either stop responding to control characters or drop additional 1648input while continuing to search for control characters. 1649This behavior is controlled by the tunable 1650.Xr sysctl 8 1651.Va debug.ddb.prioritize_control_input , 1652which defaults to 1. 1653The input buffer size is 512 bytes. 1654.Sh FILES 1655Header files mentioned in this manual page can be found below 1656.Pa /usr/include 1657directory. 1658.Pp 1659.Bl -dash -compact 1660.It 1661.Pa sys/buf.h 1662.It 1663.Pa sys/domain.h 1664.It 1665.Pa netinet/in_pcb.h 1666.It 1667.Pa sys/socket.h 1668.It 1669.Pa sys/vnode.h 1670.El 1671.Sh SEE ALSO 1672.Xr gdb 1 Pq Pa ports/devel/gdb , 1673.Xr kgdb 1 Pq Pa ports/devel/gdb , 1674.Xr acpi 4 , 1675.Xr CAM 4 , 1676.Xr gdb 4 , 1677.Xr mac_ddb 4 , 1678.Xr mac_test 4 , 1679.Xr netgraph 4 , 1680.Xr textdump 4 , 1681.Xr witness 4 , 1682.Xr ddb 8 , 1683.Xr sysctl 8 , 1684.Xr panic 9 1685.Sh HISTORY 1686The 1687.Nm 1688debugger was developed for Mach, and ported to 1689.Bx 386 0.1 . 1690This manual page translated from 1691.Xr man 7 1692macros by 1693.An Garrett Wollman . 1694.Pp 1695.An Robert N. M. Watson 1696added support for 1697.Nm 1698output capture, 1699.Xr textdump 4 1700and scripting in 1701.Fx 7.1 . 1702