1.\" Copyright (c) 2014-2018 Devin Teske 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 14.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16.\" DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, 17.\" INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 18.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 19.\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, 21.\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 22.\" ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 23.\" POSSIBILITY OF SUCH DAMAGE. 24.\" 25.Dd February 9, 2018 26.Dt DWATCH 1 27.Os 28.Sh NAME 29.Nm dwatch 30.Nd watch processes as they trigger a particular DTrace probe 31.Sh SYNOPSIS 32.Nm 33.Op Fl 1defFmnPqRvVwxy 34.Op Fl B Ar num 35.Op Fl E Ar code 36.Op Fl g Ar group 37.Op Fl j Ar jail 38.Op Fl k Ar name 39.Op Fl K Ar num 40.Op Fl N Ar count 41.Op Fl o Ar file 42.Op Fl O Ar cmd 43.Op Fl p Ar pid 44.Op Fl r Ar regex 45.Op Fl t Ar test 46.Op Fl T Ar time 47.Op Fl u Ar user 48.Op Fl X Ar profile 49.Op Fl z Ar regex 50.Op Fl - 51.Op probe[,...] 52.Op args ... 53.Nm 54.Fl l 55.Op Fl fmnPqy 56.Op Fl r Ar regex 57.Op probe ... 58.Nm 59.Fl Q 60.Op Fl 1qy 61.Op Fl r Ar regex 62.Sh DESCRIPTION 63The 64.Nm 65utility uses 66.Xr dtrace 1 67to display process info when a given DTrace probe point is triggered. 68Only the root user or users with 69.Xr sudo 8 Pq Pa ports/security/sudo 70access can run this command. 71.Pp 72.Nm 73automates the process of generating DTrace scripts to coalesce trace output by 74date/time, 75process info, 76and 77.Op optionally 78probe-specific data. 79.Pp 80Output format without options is: 81.Pp 82.Dl date/time uid.gid execname[pid]: psargs 83.Pp 84For example, 85the command 86.Ql dwatch BEGIN 87produces: 88.Pp 89.Dl INFO Watching 'dtrace:::BEGIN' ... 90.Dl 2017 May 29 08:23:20 0.0 dtrace[60671]: dtrace -s /dev/stdin 91.Pp 92The 93.Fl F 94option causes 95.Nm 96to instead coalesce trace output by date/time, 97process info, 98and probe traversal. 99.Pp 100Output format with the 101.Ql Fl F 102option is: 103.Pp 104.Dl date/time uid.gid execname[pid]: {->,<-, |} prov:mod:func:name ... 105.Pp 106For example, 107the command 108.Ql dwatch -F BEGIN 109produces: 110.Pp 111.Dl INFO Watching 'dtrace:::BEGIN' ... 112.Dl 2017 May 29 21:34:41 0.0 dtrace[86593]: | dtrace:::BEGIN ... 113.Pp 114The 115.Fl R 116option causes 117.Nm 118to display a process tree containing the parent, 119grandparent, 120and ancestor process info. 121.Pp 122Output format with the 123.Ql Fl R 124option is: 125.Pp 126.Dl date/time uid0.gid0 execname[pid0]: psargs0 127.Dl " -+= pid3 uid3.gid3 psargs3" 128.Dl " \e\\-+= pid2 uid2.gid2 psargs2" 129.Dl " \e\\-+= pid1 uid1.gid1 psargs1" 130.Dl " \e\\-+= pid0 uid0.guid0 psargs0" 131.Pp 132For example, 133the command 134.Ql dwatch -R BEGIN 135produces: 136.Pp 137.Dl INFO Watching 'dtrace:::BEGIN' ... 138.Dl 2017 May 29 21:38:54 0.0 dtrace[86899]: dtrace -s /dev/stdin 139.Dl " -+= 86855 604.604 -bash" 140.Dl " \e\\-+= 86857 604.604 /bin/sh /usr/sbin/dwatch -R BEGIN" 141.Dl " \e\\-+= 86897 0.0 sudo dtrace -s /dev/stdin" 142.Dl " \e\\-+= 86899 0.0 dtrace -s /dev/stdin" 143.Pp 144Of particular interest is the ability to filter using regular expressions. 145The 146.Ql Fl g Ar group , 147.Ql Fl p Ar pid , 148.Ql Fl r Ar regex , 149.Ql Fl u Ar user , 150and 151.Ql Fl z Ar regex 152options can be combined with 153.Ql Fl R 154to match on parent process criteria as well as current process info. 155.Pp 156In contrast, 157the 158.Ql Fl j Ar jail , 159and 160.Ql Fl k Ar name 161options apply only to the current process even if 162.Ql Fl R 163is given. 164.Pp 165The 166.Ql Fl E Ar code 167option gives the ability to customize probe-specific data. 168For example, 169the command: 170.Pp 171.Dl dwatch -E 'printf("%s", copyinstr(arg0))' chdir 172.Pp 173displays the path argument sent to 174.Xr chdir 2 175calls. 176.Pp 177Profiles can be written for more complex routines and/or convenience. 178To list available profiles use the 179.Ql Fl Q 180option. 181Use the 182.Ql Fl X Ar profile 183option to use a particular profile. 184.Pp 185For example, 186the command 187.Ql dwatch -X kill 188displays arguments sent to 189.Xr kill 2 . 190.Sh OPTIONS 191If a 192.Ar probe 193argument does not contain colon 194.Pq Qo Li ":" Qc 195and none of 196.Ql Fl P , 197.Ql Fl m , 198.Ql Fl f , 199or 200.Ql Fl n 201are given, 202the probe argument is intelligently mapped to its most-likely value. 203Use 204.Ql Nm Fl l Ar name 205to see what probes will match a given name. 206.Pp 207Multiple probes must be given as a single 208.Pq quoted 209argument, 210separated by comma and/or whitespace. 211Any/all arguments following said probes will be passed to 212.Xr dtrace 1 213unmodified. 214.Bl -tag -width "-c count" 215.It Fl 1 216Print one line per process/profile 217.Pq Default; disables Ql Fl R . 218.It Fl B Ar num 219Maximum number of arguments to display 220.Pq Default 64 . 221.It Fl d 222Debug. 223Send 224.Xr dtrace 1 225script to stdout instead of executing. 226.It Fl e 227Exit after compiling request but prior to enabling probes. 228.It Fl E Ar code 229DTrace 230.Ar code 231for event details. 232If `-', 233read from stdin. 234This allows customization of what is printed after date/time and process info. 235By default, 236the name and arguments of the program triggering the probe are shown. 237Can be specified multiple times. 238.It Fl f 239Enable probes matching the specified function names. 240.It Fl F 241Coalesce trace output by probe. 242.It Fl g Ar group 243Group filter. 244Only show processes matching 245.Ar group 246name/gid. 247This can be an 248.Xr awk 1 249regular expression to match a numerical gid. 250.It Fl j Ar jail 251Jail filter. 252Only show processes matching 253.Ar jail 254name/jid. 255.It Fl k Ar name 256Only show processes matching 257.Ar name . 258Can also be of the format 259.Ql Li name* 260to indicate 261.Dq Li begins with , 262.Ql Li *name 263to indicate 264.Dq Li ends with , 265or 266.Ql Li *name* 267to indicate 268.Dq Li contains . 269Can be specified multiple times. 270.It Fl K Ar num 271Maximum directory depth to display 272.Pq Default 64 . 273.It Fl l 274List available probes on standard output and exit. 275.It Fl m 276Enable probes matching the specified module names. 277.It Fl X Ar profile 278Load profile from DWATCH_PROFILES_PATH. 279.It Fl n 280Enable probes matching the specified probe names. 281.It Fl N Ar count 282Exit after 283.Ar count 284matching entries 285.Pq Default 0 for disabled . 286.It Fl o Ar file 287Set output file. 288If 289.Ql Li - , 290the path 291.Ql Li /dev/stdout 292is used. 293.It Fl O Ar cmd 294Execute 295.Ar cmd 296for each event. 297This can be any valid 298.Xr sh 1 299command. 300The environment variables 301.Ql Li $TAG 302and 303.Ql Li $DETAILS 304are set for the given 305.Ar cmd . 306.It Fl p Ar pid 307Process id filter. 308Only show processes with matching 309.Ar pid . 310This can be an 311.Xr awk 1 312regular expression. 313.It Fl P 314Enable probe matching the specified provider name. 315.It Fl q 316Quiet. 317Hide informational messages and all dtrace(1) errors. 318.It Fl Q 319List available profiles in DWATCH_PROFILES_PATH and exit. 320.It Fl r Ar regex 321Filter. 322Only show blocks matching 323.Xr awk 1 324regular expression. 325.It Fl R 326Show parent, 327grandparent, 328and ancestor of process. 329.It Fl t Ar test 330Test clause 331.Pq predicate 332to limit events 333.Pq Default none . 334Can be specified multiple times. 335.It Fl T Ar time 336Timeout. 337The format is 338.Ql Li #[smhd] 339or just 340.Ql Li # 341for seconds. 342.It Fl u Ar user 343User filter. 344Only show processes matching 345.Ar user 346name/uid. 347This can be an 348.Xr awk 1 349regular expression to match a numerical UID. 350.It Fl v 351Verbose. 352Show all errors from 353.Xr dtrace 1 . 354.It Fl V 355Report 356.Nm 357version on standard output and exit. 358.It Fl w 359Permit destructive actions 360.Pq copyout*, stop, panic, etc. . 361.It Fl x 362Trace. 363Print 364.Ql Li <probe-id> 365when a probe is triggered. 366.It Fl y 367Always treat stdout as console 368.Pq enable colors/columns/etc. . 369.It Fl z Ar regex 370Only show processes matching 371.Xr awk 1 372regular expression. 373.El 374.Sh PROFILES 375Profiles customize the data printed during events. 376Profiles are loaded from a colon-separated list of directories in 377.Ev DWATCH_PROFILES_PATH . 378This is an incomplete list of profiles with basic descriptions: 379.Bl -tag -width "vop_readdir" 380.It chmod 381Print mode and path from 382.Xr chmod 2 , 383.Xr lchmod 2 , 384.Xr fchmodat 2 385.It errno 386Print non-zero errno results from system calls 387.It io 388Print disk I/O details provided by 389.Xr dtrace_io 4 390.It ip 391Print IPv4 and IPv6 details provided by 392.Xr dtrace_ip 4 393.It kill 394Print signal and pid from 395.Xr kill 2 396.It nanosleep 397Print requested time from 398.Xr nanosleep 2 399.It open 400Print path from 401.Xr open 2 , 402.Xr openat 2 403.It proc 404Print process execution details provided by 405.Xr dtrace_proc 4 406.It proc-signal 407Print process signal details provided by 408.Xr dtrace_proc 4 409.It rw 410Print buffer contents from 411.Xr read 2 , 412.Xr write 2 413.It sched 414Print CPU scheduling details provided by 415.Xr dtrace_sched 4 416.It tcp 417Print TCP address/port details provided by 418.Xr dtrace_tcp 4 419.It tcp-io 420Print TCP I/O details provided by 421.Xr dtrace_tcp 4 422.It udp 423Print UDP I/O details provided by 424.Xr dtrace_udp 4 425.It vop_create 426Print filesystem paths being created by 427.Xr VOP_CREATE 9 428.It vop_lookup 429Print filesystem paths being looked-up by 430.Xr VOP_LOOKUP 9 431.It vop_mkdir 432Print directory paths being created by 433.Xr VOP_MKDIR 9 434.It vop_mknod 435Print device node paths being created by 436.Xr VOP_MKNOD 9 437.It vop_readdir 438Print directory paths being read by 439.Xr VOP_READDIR 9 440.It vop_remove 441Print filesystem paths being removed by 442.Xr VOP_REMOVE 9 443.It vop_rename 444Print filesystem paths being renamed by 445.Xr VOP_RENAME 9 446.It vop_rmdir 447Print directory paths being removed by 448.Xr VOP_RMDIR 9 449.It vop_symlink 450Print symlink paths being created by 451.Xr VOP_SYMLINK 9 452.El 453.Sh ENVIRONMENT 454These environment variables affect the execution of 455.Nm : 456.Bl -tag -width "DWATCH_PROFILES_PATH" 457.It Ev DWATCH_PROFILES_PATH 458If 459.Ev DWATCH_PROFILES_PATH 460is set, 461.Nm 462searches for profiles in the colon-separated list of directories in that 463variable instead of the default 464.Ql Li /usr/libexec/dwatch:/usr/local/libexec/dwatch . 465If set to NULL, 466profiles are not loaded. 467.El 468.Sh EXIT STATUS 469.Ex -std 470.Sh EXAMPLES 471Watch processes entering system CPU scheduler. 472.Bd -literal -offset indent 473dwatch on-cpu 474.Ed 475.Pp 476List available profiles, 477one line per profile. 478.Bd -literal -offset indent 479dwatch -1 -Q 480.Ed 481.Pp 482Do not execute 483.Xr dtrace 1 484but display script on stdout and exit. 485.Bd -literal -offset indent 486dwatch -d fsync 487.Ed 488.Pp 489Compile and test but do not execute code generated with given probe. 490.Bd -literal -offset indent 491dwatch -e test_probe 492.Ed 493.Pp 494Print argument one being passed to each call of zfs_sync(). 495.Bd -literal -offset indent 496dwatch -E 'printf("%i", arg1)' zfs_sync 497.Ed 498.Pp 499Watch all functions named 500.Ql Li read . 501.Bd -literal -offset indent 502dwatch -f read 503.Ed 504.Pp 505Watch all probe traversal. 506.Bd -literal -offset indent 507dwatch -F : 508.Ed 509.Pp 510Watch syscall probe traversal. 511.Bd -literal -offset indent 512dwatch -F syscall 513.Ed 514.Pp 515Display only processes belonging to wheel super-group. 516.Bd -literal -offset indent 517dwatch -g wheel execve 518.Ed 519.Pp 520Display only processes belonging to groups 521.Ql Li daemon 522or 523.Ql Li nobody . 524.Bd -literal -offset indent 525dwatch -g '1|65534' execve 526.Ed 527.Pp 528Ignore jails, 529displaying only base system processes. 530.Bd -literal -offset indent 531dwatch -j 0 execve 532.Ed 533.Pp 534Display only processes running inside the jail named 535.Ql Li myjail . 536.Bd -literal -offset indent 537dwatch -j myjail execve 538.Ed 539.Pp 540Watch syscall traversal by ruby processes. 541.Bd -literal -offset indent 542dwatch -k 'ruby*' -F syscall 543.Ed 544.Pp 545Watch syscall traversal by processes containing 546.Ql Li daemon 547in their name. 548.Bd -literal -offset indent 549dwatch -k '*daemon*' -F syscall 550.Ed 551.Pp 552Watch signals being passed to 553.Xr kill 2 . 554.Bd -literal -offset indent 555dwatch -X kill 556.Ed 557.Pp 558Watch signals being passed between 559.Xr bash 1 560and 561.Xr vi 1 . 562.Bd -literal -offset indent 563dwatch -k bash -k vi -X kill 564.Ed 565.Pp 566Display a list of unique functions available. 567.Bd -literal -offset indent 568dwatch -l -f 569.Ed 570.Pp 571List available probes for functions ending in 572.Ql Li read . 573.Bd -literal -offset indent 574dwatch -l -f '*read' 575.Ed 576.Pp 577List available probes ending in 578.Dq Li read . 579.Bd -literal -offset indent 580dwatch -l -r 'read$' 581.Ed 582.Pp 583Display a list of unique providers. 584.Bd -literal -offset indent 585dwatch -l -P 586.Ed 587.Pp 588Watch paths being removed by 589.Xr VOP_REMOVE 9 . 590.Bd -literal -offset indent 591dwatch -X vop_remove 592.Ed 593.Pp 594Watch the name 595.Ql Li read 596instead of the function 597.Ql Li read . 598The 599.Nm 600selection algorithm will commonly favor the function named 601.Ql Li read 602when not given a type 603.Pq using So Fl P Sc , So Fl m Sc , So Fl f Sc , or So Fl n Sc 604because there are more probes matching the function named 605.Ql Li read 606than probes matching 607.Ql Li read 608for any other type. 609.Bd -literal -offset indent 610dwatch -n read 611.Ed 612.Pp 613Display the first process to call 614.Xr kill 2 615and then exit. 616.Bd -literal -offset indent 617dwatch -N 1 kill 618.Ed 619.Pp 620Watch processes forked by pid 1234. 621.Bd -literal -offset indent 622dwatch -p 1234 execve 623.Ed 624.Pp 625Watch processes forked by either pid 1234 or pid 5678. 626.Bd -literal -offset indent 627dwatch -p '1234|5678' execve 628.Ed 629.Pp 630Watch the provider 631.Ql Li random 632instead of the function 633.Ql Li random . 634The 635.Nm 636selection algorithm will commonly favor the function named 637.Ql Li random 638when not given a type 639.Pq using So Fl P Sc , So Fl m Sc , So Fl f Sc , or So Fl n Sc 640because there are more probes matching the function named 641.Ql Li random 642than probes matching the provider named 643.Ql Li random . 644.Bd -literal -offset indent 645dwatch -P random 646.Ed 647.Pp 648Display available profiles matching 649.Ql Li vop . 650.Bd -literal -offset indent 651dwatch -Q -r vop 652.Ed 653.Pp 654Watch 655.Xr VOP_LOOKUP 9 656paths containing 657.Ql Li /lib/ . 658.Bd -literal -offset indent 659dwatch -r /lib/ -X vop_lookup 660.Ed 661.Pp 662Show process tree for each command as it is executed. 663.Bd -literal -offset indent 664dwatch -R execve 665.Ed 666.Pp 667Watch processes forked by pid 1234 or children thereof. 668.Bd -literal -offset indent 669dwatch -R -p 1234 execve 670.Ed 671.Pp 672Display processes calling 673.Xr write 2 674with 675.Dq nbytes 676less than 10. 677.Bd -literal -offset indent 678dwatch -t 'arg2<10' -E 'printf("%d",arg2)' write 679.Ed 680.Pp 681Display 682.Xr write 2 683buffer when 684.Dq execname 685is not 686.Ql Li dtrace 687and 688.Dq nbytes 689is less than 10. 690.Bd -literal -offset indent 691dwatch -X write -t 'execname != "dtrace" && this->nbytes < 10' 692.Ed 693.Pp 694Watch 695.Ql Li statfs 696for 5 minutes and exit. 697.Bd -literal -offset indent 698dwatch -T 5m statfs 699.Ed 700.Pp 701Display only processes belonging to the root super-user. 702.Bd -literal -offset indent 703dwatch -u root execve 704.Ed 705.Pp 706Display only processes belonging to users 707.Ql Li daemon 708or 709.Ql Li nobody . 710.Bd -literal -offset indent 711dwatch -u '1|65534' execve 712.Ed 713.Pp 714Print version and exit. 715.Bd -literal -offset indent 716dwatch -V 717.Ed 718.Pp 719View the first 100 scheduler preemptions. 720.Bd -literal -offset indent 721dwatch -y -N 100 preempt | less -R 722.Ed 723.Pp 724Display processes matching either 725.Dq Li mkdir 726or 727.Dq Li rmdir . 728.Bd -literal -offset indent 729dwatch -z '(mk|rm)dir' execve 730.Ed 731.Pp 732Run a command and watch network activity only while that command runs. 733.Bd -literal -offset indent 734dwatch -X tcp -- -c "nc -zvw10 google.com 22" 735.Ed 736.Pp 737Watch 738.Xr open 2 739and 740.Xr openat 2 741calls only while pid 1234 is active. 742.Bd -literal -offset indent 743dwatch -X open -- -p 1234 744.Ed 745.Pp 746Watch probe traversal for a given command. 747Note that 748.Dq Li -c true 749is passed to 750.Xr dtrace 1 751since it appears after the 752.Nm 753probe argument. 754.Bd -literal -offset indent 755dwatch -F 'pid$target:::entry' -c true 756.Ed 757.Sh SEE ALSO 758.Xr dtrace 1 759.Sh HISTORY 760.Nm 761first appeared in 762.Fx 11.2 . 763.Sh AUTHORS 764.An Devin Teske Aq Mt dteske@FreeBSD.org 765