en Copyright 2015 Java http://kernelsources.org:8080/source/history/freebsd/cddl/usr.sbin/dwatch/libexec/dtmalloc#434283fda99e89af20c3fe95fde427624ae96d04 dwatch: add nine diagnostic modules; grow errno, io, proc, schedGrow the module collection to answer, each with a single command, thefirst questions asked when diagnosing a sick system: why is myapplication stalling, where is the kernel fighting over locks, whatfile could it not find, why is this process getting EPERM, what killedmy process, will that fatal signal actually leave a core behind, whatwas my process stuck on, where is my kernel memory going, who iscreating or entering jails, is the network slow because TCP isresending, how long did my thread wait to run, and is the disk itselfslow. Every module keeps to the house style: invocation-nameoverloading through hard links, predicate-only D with inline lookuptables (no if-statements), and stable providers only (syscall, proc,sched, io, dtmalloc, and the lockstat, vfs, priv, and mib SDTproviders), so the modules remain drop-in compatible with olderreleases (the one documented exception is noted below). No kernelchanges: new and extended profiles under cddl/usr.sbin/dwatch/libexecplus one libdtrace inline table (priv.d).slow (slow-fsync, slow-open, slow-read, slow-syscall, slow-write, orany slow-NAME by new link) records syscall entry timestamps inthread-local storage and prints, at return, any call whose latencymeets a threshold (DWATCH_SLOW_MS, default 100), naming the syscall,the elapsed time to the microsecond, and any errno returned. The bareprofile watches a curated set of filesystem-related calls expected tobe fast; slow-syscall watches everything; unrecognized invocationnames fall through to syscall::NAME:return with the matching entryprobe derived mechanically from the return probe list.lock (lock-adaptive, lock-block, lock-lockmgr, lock-rw, lock-spin,lock-sx, lock-thread) rides the dtrace_lockstat(4) block and spinprobes, printing the held-off thread (free from the standard eventtag), the holdoff duration, the lock class, the lo_name of the lockthrough a single cast of arg0 to struct lock_object (the first memberof every kernel lock), and reader/writer intent on the probes thatreport it. Holdoffs shorter than DWATCH_LOCK_MS (default 1; 0 showseverything) are suppressed.namei (namei-enoent, namei-entry, namei-failure) records the pathnameat vfs:namei:lookup:entry and reports it with the result at return.Unlike the vop_lookup profile, which reconstructs paths from the namecache one component at a time, this sees the whole path exactly as theprocess requested it. namei-enoent hunts file-not-found storms -- thesingle most common use of truss(1) -- without stopping the victim.priv (priv-err, priv-ok) watches priv_check(9) verdicts, naming theexact privilege denied -- something no syscall tracer can see, becauseby the time EPERM surfaces the priv(9) value is gone. The number isdecoded by priv_string[], a new libdtrace inline table in the errno.dand signal.d tradition, mechanically generated from sys/priv.h (247entries) and installed to /usr/lib/dtrace where dtrace(1) auto-loadsit; on older releases it is a drop-in file like the module itself.coredump (coredump-top) watches for delivery of signals whose defaultaction produces a core, per the SIGPROP_CORE entries of the sigproptblin kern_sig.c, and renders a verdict the same way and in the sameorder the kernel will decide it: ignored or caught per the target'sstruct sigacts, then the coredump() gauntlet of kern.coredump,kern.sugid_coredump vs P_SUGID, procctl(2) PROC_TRACE_CTL, andRLIMIT_CORE -- the sysctl knobs read live through kernel globals.Where a coredump-worthy signal will produce no core, the verdict saysprecisely which policy ate it. coredump-top maintains a cumulativecatalog of coredump-worthy signals by process and signal, refreshedevery 3 seconds in the style of systop; combine the event profile with`-O cmd' to capture state as each event occurs.hang (hang-top) pairs sched:::sleep with sched:::wakeup through atid-keyed timestamp array and prints, as each thread wakes, any sleepthat meets a threshold (DWATCH_HANG_MS, default 1000), naming thesleeper in the details and the waker in the standard event tag. Thisis the blocking the slow module structurally cannot see: a syscallthat never returns never reports its latency, while hang reports themoment the wait ends, with the full duration. hang-top maintains acumulative catalog of long sleeps by process (count and maximum) inthe style of coredump-top.jail (jail-attach, jail-get, jail-remove, jail-set) watches the jailmanagement plane -- jail(2), jail_set(2), jail_get(2), jail_attach(2),and jail_remove(2) -- naming the operation, the jail id (taken fromthe entry argument for attach/remove, from the return value for theothers), and any errno. Complements the dwatch `-j jail' filter,which scopes any profile to processes inside one jail; this watcheswho manipulates jails, from any jail or none.dtmalloc (dtmalloc-top, or any dtmalloc-NAME by new link) rides thedtmalloc provider (one malloc and one free probe per malloc(9) type).The event profile prints allocations and frees meeting a sizethreshold (DWATCH_MALLOC_MIN, default 65536) -- who is allocating hugekernel buffers. dtmalloc-top maintains a running catalog of net bytesand outstanding allocation balance by type, sorted by net bytes soleak suspects rise: a type that climbs without bound while the systemis in steady state is the suspect. The catalog reflects activitysince the watch began, and is honest about caches holding what theyallocate.mib (tcp-retransmit, or any mib-NAME by new link) rides theper-counter mib SDT probes of the network stack. The tcp-retransmitprofile curates the counters that signal send-path congestion or loss-- data packet retransmissions, unnecessary retransmissions,retransmit timer expirations, and connections dropped by retransmitexhaustion -- decoded through an inline description table, answering"is this network slow because TCP is resending?" as events withprocess context rather than netstat(1) deltas. NB: the mib probesexist only in kernels built with options KDTRACE_MIB_SDT (default in-CURRENT via std.debug); the module documents this and dtrace(1)refuses the script elsewhere, making the dependency self-announcing.Four existing modules gain personalities. proc growsproc-signal-fatal, filtering signal-send to signals whose defaultdisposition terminates the receiver, most-notably includingkernel-generated SIGSEGV/SIGBUS/SIGILL/SIGFPE that no kill(2) watcherwill ever see. errno now reads its invocation name: errno-NAME showsonly syscalls returning that errno, where NAME is a symbolic name fromerrno.d or a number; links are installed for errno-EACCES,errno-ECAPMODE, errno-ENOENT, errno-ENOTCAPABLE, and errno-EPERM (thelatter pairs covering capsicum(4) capability-mode violations), and anyother errno needs only a new link. sched grows sched-latency,recording a timestamp at sched:::enqueue keyed by tid and printing atsched:::on-cpu any run-queue wait meeting a threshold (DWATCH_SCHED_MS,default 10) -- the literal measurement of scheduler delay on a systemwith idle CPU that still feels sluggish. io grows io-slow, pairingio:::start with io:::done through a bio-keyed timestamp array andprinting any request that meets a threshold (DWATCH_IO_MS, default100), naming the device, command, size, and elapsed time; watchedagainst zvols and a pool's leaf vdevs this brackets where in a ZFSstack the time is going, without touching unstable providers.Document all of the above plus the DWATCH_HANG_MS, DWATCH_IO_MS,DWATCH_LOCK_MS, DWATCH_MALLOC_MIN, DWATCH_SCHED_MS, and DWATCH_SLOW_MSknobs in dwatch(1).All 46 new invocation names were exercised through `dwatch -d' with aprofile-path sandbox emulating the installed hard links: every onesources cleanly and emits the intended D -- probe selection per alias,entry/return and sleep/wakeup pairing through thread-local and globalassociative arrays, threshold and mask predicates picking up theirknobs, aggregation clauses and printa column layout in the -topprofiles, multi-line predicate rendering, and `-t' correctlydisplacing each module's default test were verified by inspection ofthe generated scripts. Invocations untouched by this pass generate Didentical to their previous output. Modules pass sh -n, fit 80columns, and dwatch.1 passes mandoc -Tlint with no new warnings. Avalidation harness performing a `dwatch -e' compile per profileagainst the live kernel globs every staged profile for runs whereverthe dtrace device is present.Reviewed by: markjDifferential Revision: https://reviews.freebsd.org/D58093 List of files: /freebsd/cddl/usr.sbin/dwatch/libexec/dtmalloc Wed, 29 Jul 2026 01:32:57 +0200 Devin Teske <dteske@FreeBSD.org>