1.\" Copyright (c) 2014-2018 Devin Teske 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 14.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16.\" DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, 17.\" INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 18.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 19.\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, 21.\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 22.\" ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 23.\" POSSIBILITY OF SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd February 9, 2018 28.Dt DWATCH 1 29.Os 30.Sh NAME 31.Nm dwatch 32.Nd watch processes as they trigger a particular DTrace probe 33.Sh SYNOPSIS 34.Nm 35.Op Fl 1defFmnPqRvVwxy 36.Op Fl B Ar num 37.Op Fl E Ar code 38.Op Fl g Ar group 39.Op Fl j Ar jail 40.Op Fl k Ar name 41.Op Fl K Ar num 42.Op Fl N Ar count 43.Op Fl o Ar file 44.Op Fl O Ar cmd 45.Op Fl p Ar pid 46.Op Fl r Ar regex 47.Op Fl t Ar test 48.Op Fl T Ar time 49.Op Fl u Ar user 50.Op Fl X Ar profile 51.Op Fl z Ar regex 52.Op Fl - 53.Op probe[,...] 54.Op args ... 55.Nm 56.Fl l 57.Op Fl fmnPqy 58.Op Fl r Ar regex 59.Op probe ... 60.Nm 61.Fl Q 62.Op Fl 1qy 63.Op Fl r Ar regex 64.Sh DESCRIPTION 65The 66.Nm 67utility uses 68.Xr dtrace 1 69to display process info when a given DTrace probe point is triggered. 70Only the root user or users with 71.Xr sudo 8 72access can run this command. 73.Pp 74.Nm 75automates the process of generating DTrace scripts to coalesce trace output by 76date/time, 77process info, 78and 79.Op optionally 80probe-specific data. 81.Pp 82Output format without options is: 83.Pp 84.Dl date/time uid.gid execname[pid]: psargs 85.Pp 86For example, 87the command 88.Ql dwatch BEGIN 89produces: 90.Pp 91.Dl INFO Watching 'dtrace:::BEGIN' ... 92.Dl 2017 May 29 08:23:20 0.0 dtrace[60671]: dtrace -s /dev/stdin 93.Pp 94The 95.Fl F 96option causes 97.Nm 98to instead coalesce trace output by date/time, 99process info, 100and probe traversal. 101.Pp 102Output format with the 103.Ql Fl F 104option is: 105.Pp 106.Dl date/time uid.gid execname[pid]: {->,<-, |} prov:mod:func:name ... 107.Pp 108For example, 109the command 110.Ql dwatch -F BEGIN 111produces: 112.Pp 113.Dl INFO Watching 'dtrace:::BEGIN' ... 114.Dl 2017 May 29 21:34:41 0.0 dtrace[86593]: | dtrace:::BEGIN ... 115.Pp 116The 117.Fl R 118option causes 119.Nm 120to display a process tree containing the parent, 121grandparent, 122and ancestor process info. 123.Pp 124Output format with the 125.Ql Fl R 126option is: 127.Pp 128.Dl date/time uid0.gid0 execname[pid0]: psargs0 129.Dl " -+= pid3 uid3.gid3 psargs3" 130.Dl " \e\\-+= pid2 uid2.gid2 psargs2" 131.Dl " \e\\-+= pid1 uid1.gid1 psargs1" 132.Dl " \e\\-+= pid0 uid0.guid0 psargs0" 133.Pp 134For example, 135the command 136.Ql dwatch -R BEGIN 137produces: 138.Pp 139.Dl INFO Watching 'dtrace:::BEGIN' ... 140.Dl 2017 May 29 21:38:54 0.0 dtrace[86899]: dtrace -s /dev/stdin 141.Dl " -+= 86855 604.604 -bash" 142.Dl " \e\\-+= 86857 604.604 /bin/sh /usr/sbin/dwatch -R BEGIN" 143.Dl " \e\\-+= 86897 0.0 sudo dtrace -s /dev/stdin" 144.Dl " \e\\-+= 86899 0.0 dtrace -s /dev/stdin" 145.Pp 146Of particular interest is the ability to filter using regular expressions. 147The 148.Ql Fl g Ar group , 149.Ql Fl p Ar pid , 150.Ql Fl r Ar regex , 151.Ql Fl u Ar user , 152and 153.Ql Fl z Ar regex 154options can be combined with 155.Ql Fl R 156to match on parent process criteria as well as current process info. 157.Pp 158In contrast, 159the 160.Ql Fl j Ar jail , 161and 162.Ql Fl k Ar name 163options apply only to the current process even if 164.Ql Fl R 165is given. 166.Pp 167The 168.Ql Fl E Ar code 169option gives the ability to customize probe-specific data. 170For example, 171the command: 172.Pp 173.Dl dwatch -E 'printf("%s", copyinstr(arg0))' chdir 174.Pp 175displays the path argument sent to 176.Xr chdir 2 177calls. 178.Pp 179Profiles can be written for more complex routines and/or convenience. 180To list available profiles use the 181.Ql Fl Q 182option. 183Use the 184.Ql Fl X Ar profile 185option to use a particular profile. 186.Pp 187For example, 188the command 189.Ql dwatch -X kill 190displays arguments sent to 191.Xr kill 2 . 192.Sh OPTIONS 193If a 194.Ar probe 195argument does not contain colon 196.Pq Qo Li ":" Qc 197and none of 198.Ql Fl P , 199.Ql Fl m , 200.Ql Fl f , 201or 202.Ql Fl n 203are given, 204the probe argument is intelligently mapped to its most-likely value. 205Use 206.Ql Nm Fl l Ar name 207to see what probes will match a given name. 208.Pp 209Multiple probes must be given as a single 210.Pq quoted 211argument, 212separated by comma and/or whitespace. 213Any/all arguments following said probes will be passed to 214.Xr dtrace 1 215unmodified. 216.Bl -tag -width "-c count" 217.It Fl 1 218Print one line per process/profile 219.Pq Default; disables Ql Fl R . 220.It Fl B Ar num 221Maximum number of arguments to display 222.Pq Default 64 . 223.It Fl d 224Debug. 225Send 226.Xr dtrace 1 227script to stdout instead of executing. 228.It Fl e 229Exit after compiling request but prior to enabling probes. 230.It Fl E Ar code 231DTrace 232.Ar code 233for event details. 234If `-', 235read from stdin. 236This allows customization of what is printed after date/time and process info. 237By default, 238the name and arguments of the program triggering the probe are shown. 239Can be specified multiple times. 240.It Fl f 241Enable probes matching the specified function names. 242.It Fl F 243Coalesce trace output by probe. 244.It Fl g Ar group 245Group filter. 246Only show processes matching 247.Ar group 248name/gid. 249This can be an 250.Xr awk 1 251regular expression to match a numerical gid. 252.It Fl j Ar jail 253Jail filter. 254Only show processes matching 255.Ar jail 256name/jid. 257.It Fl k Ar name 258Only show processes matching 259.Ar name . 260Can also be of the format 261.Ql Li name* 262to indicate 263.Dq Li begins with , 264.Ql Li *name 265to indicate 266.Dq Li ends with , 267or 268.Ql Li *name* 269to indicate 270.Dq Li contains . 271Can be specified multiple times. 272.It Fl K Ar num 273Maximum directory depth to display 274.Pq Default 64 . 275.It Fl l 276List available probes on standard output and exit. 277.It Fl m 278Enable probes matching the specified module names. 279.It Fl X Ar profile 280Load profile from DWATCH_PROFILES_PATH. 281.It Fl n 282Enable probes matching the specified probe names. 283.It Fl N Ar count 284Exit after 285.Ar count 286matching entries 287.Pq Default 0 for disabled . 288.It Fl o Ar file 289Set output file. 290If 291.Ql Li - , 292the path 293.Ql Li /dev/stdout 294is used. 295.It Fl O Ar cmd 296Execute 297.Ar cmd 298for each event. 299This can be any valid 300.Xr sh 1 301command. 302The environment variables 303.Ql Li $TAG 304and 305.Ql Li $DETAILS 306are set for the given 307.Ar cmd . 308.It Fl p Ar pid 309Process id filter. 310Only show processes with matching 311.Ar pid . 312This can be an 313.Xr awk 1 314regular expression. 315.It Fl P 316Enable probe matching the specified provider name. 317.It Fl q 318Quiet. 319Hide informational messages and all dtrace(1) errors. 320.It Fl Q 321List available profiles in DWATCH_PROFILES_PATH and exit. 322.It Fl r Ar regex 323Filter. 324Only show blocks matching 325.Xr awk 1 326regular expression. 327.It Fl R 328Show parent, 329grandparent, 330and ancestor of process. 331.It Fl t Ar test 332Test clause 333.Pq predicate 334to limit events 335.Pq Default none . 336Can be specified multiple times. 337.It Fl T Ar time 338Timeout. 339The format is 340.Ql Li #[smhd] 341or just 342.Ql Li # 343for seconds. 344.It Fl u Ar user 345User filter. 346Only show processes matching 347.Ar user 348name/uid. 349This can be an 350.Xr awk 1 351regular expression to match a numerical UID. 352.It Fl v 353Verbose. 354Show all errors from 355.Xr dtrace 1 . 356.It Fl V 357Report 358.Nm 359version on standard output and exit. 360.It Fl w 361Permit destructive actions 362.Pq copyout*, stop, panic, etc. . 363.It Fl x 364Trace. 365Print 366.Ql Li <probe-id> 367when a probe is triggered. 368.It Fl y 369Always treat stdout as console 370.Pq enable colors/columns/etc. . 371.It Fl z Ar regex 372Only show processes matching 373.Xr awk 1 374regular expression. 375.El 376.Sh PROFILES 377Profiles customize the data printed during events. 378Profiles are loaded from a colon-separated list of directories in 379.Ev DWATCH_PROFILES_PATH . 380This is an incomplete list of profiles with basic descriptions: 381.Bl -tag -width "vop_readdir" 382.It chmod 383Print mode and path from 384.Xr chmod 2 , 385.Xr lchmod 2 , 386.Xr fchmodat 2 387.It errno 388Print non-zero errno results from system calls 389.It io 390Print disk I/O details provided by 391.Xr dtrace_io 4 392.It ip 393Print IPv4 and IPv6 details provided by 394.Xr dtrace_ip 4 395.It kill 396Print signal and pid from 397.Xr kill 2 398.It nanosleep 399Print requested time from 400.Xr nanosleep 2 401.It open 402Print path from 403.Xr open 2 , 404.Xr openat 2 405.It proc 406Print process execution details provided by 407.Xr dtrace_proc 4 408.It proc-signal 409Print process signal details provided by 410.Xr dtrace_proc 4 411.It rw 412Print buffer contents from 413.Xr read 2 , 414.Xr write 2 415.It sched 416Print CPU scheduling details provided by 417.Xr dtrace_sched 4 418.It tcp 419Print TCP address/port details provided by 420.Xr dtrace_tcp 4 421.It tcp-io 422Print TCP I/O details provided by 423.Xr dtrace_tcp 4 424.It udp 425Print UDP I/O details provided by 426.Xr dtrace_udp 4 427.It vop_create 428Print filesystem paths being created by 429.Xr VOP_CREATE 9 430.It vop_lookup 431Print filesystem paths being looked-up by 432.Xr VOP_LOOKUP 9 433.It vop_mkdir 434Print directory paths being created by 435.Xr VOP_MKDIR 9 436.It vop_mknod 437Print device node paths being created by 438.Xr VOP_MKNOD 9 439.It vop_readdir 440Print directory paths being read by 441.Xr VOP_READDIR 9 442.It vop_remove 443Print filesystem paths being removed by 444.Xr VOP_REMOVE 9 445.It vop_rename 446Print filesystem paths being renamed by 447.Xr VOP_RENAME 9 448.It vop_rmdir 449Print directory paths being removed by 450.Xr VOP_RMDIR 9 451.It vop_symlink 452Print symlink paths being created by 453.Xr VOP_SYMLINK 9 454.El 455.Sh ENVIRONMENT 456These environment variables affect the execution of 457.Nm : 458.Bl -tag -width "DWATCH_PROFILES_PATH" 459.It Ev DWATCH_PROFILES_PATH 460If 461.Ev DWATCH_PROFILES_PATH 462is set, 463.Nm 464searches for profiles in the colon-separated list of directories in that 465variable instead of the default 466.Ql Li /usr/libexec/dwatch:/usr/local/libexec/dwatch . 467If set to NULL, 468profiles are not loaded. 469.El 470.Sh EXIT STATUS 471.Ex -std 472.Sh EXAMPLES 473Watch processes entering system CPU scheduler. 474.Bd -literal -offset indent 475dwatch on-cpu 476.Ed 477.Pp 478List available profiles, 479one line per profile. 480.Bd -literal -offset indent 481dwatch -1 -Q 482.Ed 483.Pp 484Do not execute 485.Xr dtrace 1 486but display script on stdout and exit. 487.Bd -literal -offset indent 488dwatch -d fsync 489.Ed 490.Pp 491Compile and test but do not execute code generated with given probe. 492.Bd -literal -offset indent 493dwatch -e test_probe 494.Ed 495.Pp 496Print argument one being passed to each call of zfs_sync(). 497.Bd -literal -offset indent 498dwatch -E 'printf("%i", arg1)' zfs_sync 499.Ed 500.Pp 501Watch all functions named 502.Ql Li read . 503.Bd -literal -offset indent 504dwatch -f read 505.Ed 506.Pp 507Watch all probe traversal. 508.Bd -literal -offset indent 509dwatch -F : 510.Ed 511.Pp 512Watch syscall probe traversal. 513.Bd -literal -offset indent 514dwatch -F syscall 515.Ed 516.Pp 517Display only processes belonging to wheel super-group. 518.Bd -literal -offset indent 519dwatch -g wheel execve 520.Ed 521.Pp 522Display only processes belonging to groups 523.Ql Li daemon 524or 525.Ql Li nobody . 526.Bd -literal -offset indent 527dwatch -g '1|65534' execve 528.Ed 529.Pp 530Ignore jails, 531displaying only base system processes. 532.Bd -literal -offset indent 533dwatch -j 0 execve 534.Ed 535.Pp 536Display only processes running inside the jail named 537.Ql Li myjail . 538.Bd -literal -offset indent 539dwatch -j myjail execve 540.Ed 541.Pp 542Watch syscall traversal by ruby processes. 543.Bd -literal -offset indent 544dwatch -k 'ruby*' -F syscall 545.Ed 546.Pp 547Watch syscall traversal by processes containing 548.Ql Li daemon 549in their name. 550.Bd -literal -offset indent 551dwatch -k '*daemon*' -F syscall 552.Ed 553.Pp 554Watch signals being passed to 555.Xr kill 2 . 556.Bd -literal -offset indent 557dwatch -X kill 558.Ed 559.Pp 560Watch signals being passed between 561.Xr bash 1 562and 563.Xr vi 1 . 564.Bd -literal -offset indent 565dwatch -k bash -k vi -X kill 566.Ed 567.Pp 568Display a list of unique functions available. 569.Bd -literal -offset indent 570dwatch -l -f 571.Ed 572.Pp 573List available probes for functions ending in 574.Ql Li read . 575.Bd -literal -offset indent 576dwatch -l -f '*read' 577.Ed 578.Pp 579List available probes ending in 580.Dq Li read . 581.Bd -literal -offset indent 582dwatch -l -r 'read$' 583.Ed 584.Pp 585Display a list of unique providers. 586.Bd -literal -offset indent 587dwatch -l -P 588.Ed 589.Pp 590Watch paths being removed by 591.Xr VOP_REMOVE 9 . 592.Bd -literal -offset indent 593dwatch -X vop_remove 594.Ed 595.Pp 596Watch the name 597.Ql Li read 598instead of the function 599.Ql Li read . 600The 601.Nm 602selection algorithm will commonly favor the function named 603.Ql Li read 604when not given a type 605.Pq using So Fl P Sc , So Fl m Sc , So Fl f Sc , or So Fl n Sc 606because there are more probes matching the function named 607.Ql Li read 608than probes matching 609.Ql Li read 610for any other type. 611.Bd -literal -offset indent 612dwatch -n read 613.Ed 614.Pp 615Display the first process to call 616.Xr kill 2 617and then exit. 618.Bd -literal -offset indent 619dwatch -N 1 kill 620.Ed 621.Pp 622Watch processes forked by pid 1234. 623.Bd -literal -offset indent 624dwatch -p 1234 execve 625.Ed 626.Pp 627Watch processes forked by either pid 1234 or pid 5678. 628.Bd -literal -offset indent 629dwatch -p '1234|5678' execve 630.Ed 631.Pp 632Watch the provider 633.Ql Li random 634instead of the function 635.Ql Li random . 636The 637.Nm 638selection algorithm will commonly favor the function named 639.Ql Li random 640when not given a type 641.Pq using So Fl P Sc , So Fl m Sc , So Fl f Sc , or So Fl n Sc 642because there are more probes matching the function named 643.Ql Li random 644than probes matching the provider named 645.Ql Li random . 646.Bd -literal -offset indent 647dwatch -P random 648.Ed 649.Pp 650Display available profiles matching 651.Ql Li vop . 652.Bd -literal -offset indent 653dwatch -Q -r vop 654.Ed 655.Pp 656Watch 657.Xr VOP_LOOKUP 9 658paths containing 659.Ql Li /lib/ . 660.Bd -literal -offset indent 661dwatch -r /lib/ -X vop_lookup 662.Ed 663.Pp 664Show process tree for each command as it is executed. 665.Bd -literal -offset indent 666dwatch -R execve 667.Ed 668.Pp 669Watch processes forked by pid 1234 or children thereof. 670.Bd -literal -offset indent 671dwatch -R -p 1234 execve 672.Ed 673.Pp 674Display processes calling 675.Xr write 2 676with 677.Dq nbytes 678less than 10. 679.Bd -literal -offset indent 680dwatch -t 'arg2<10' -E 'printf("%d",arg2)' write 681.Ed 682.Pp 683Display 684.Xr write 2 685buffer when 686.Dq execname 687is not 688.Ql Li dtrace 689and 690.Dq nbytes 691is less than 10. 692.Bd -literal -offset indent 693dwatch -X write -t 'execname != "dtrace" && this->nbytes < 10' 694.Ed 695.Pp 696Watch 697.Ql Li statfs 698for 5 minutes and exit. 699.Bd -literal -offset indent 700dwatch -T 5m statfs 701.Ed 702.Pp 703Display only processes belonging to the root super-user. 704.Bd -literal -offset indent 705dwatch -u root execve 706.Ed 707.Pp 708Display only processes belonging to users 709.Ql Li daemon 710or 711.Ql Li nobody . 712.Bd -literal -offset indent 713dwatch -u '1|65534' execve 714.Ed 715.Pp 716Print version and exit. 717.Bd -literal -offset indent 718dwatch -V 719.Ed 720.Pp 721View the first 100 scheduler preemptions. 722.Bd -literal -offset indent 723dwatch -y -N 100 preempt | less -R 724.Ed 725.Pp 726Display processes matching either 727.Dq Li mkdir 728or 729.Dq Li rmdir . 730.Bd -literal -offset indent 731dwatch -z '(mk|rm)dir' execve 732.Ed 733.Pp 734Run a command and watch network activity only while that command runs. 735.Bd -literal -offset indent 736dwatch -X tcp -- -c "nc -zvw10 google.com 22" 737.Ed 738.Pp 739Watch 740.Xr open 2 741and 742.Xr openat 2 743calls only while pid 1234 is active. 744.Bd -literal -offset indent 745dwatch -X open -- -p 1234 746.Ed 747.Pp 748Watch probe traversal for a given command. 749Note that 750.Dq Li -c true 751is passed to 752.Xr dtrace 1 753since it appears after the 754.Nm 755probe argument. 756.Bd -literal -offset indent 757dwatch -F 'pid$target:::entry' -c true 758.Ed 759.Sh SEE ALSO 760.Xr dtrace 1 761.Sh HISTORY 762.Nm 763first appeared in 764.Fx 11.2 . 765.Sh AUTHORS 766.An Devin Teske Aq Mt dteske@FreeBSD.org 767