1.\" SPDX-License-Identifier: CDDL-1.0 2.\" 3.\" CDDL HEADER START 4.\" 5.\" The contents of this file are subject to the terms of the 6.\" Common Development and Distribution License (the "License"). 7.\" You may not use this file except in compliance with the License. 8.\" 9.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 10.\" or https://opensource.org/licenses/CDDL-1.0. 11.\" See the License for the specific language governing permissions 12.\" and limitations under the License. 13.\" 14.\" When distributing Covered Code, include this CDDL HEADER in each 15.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 16.\" If applicable, add the following below this CDDL HEADER, with the 17.\" fields enclosed by brackets "[]" replaced with your own identifying 18.\" information: Portions Copyright [yyyy] [name of copyright owner] 19.\" 20.\" CDDL HEADER END 21.\" 22.\" Copyright (c) 2007, Sun Microsystems, Inc. All Rights Reserved. 23.\" Copyright (c) 2012, 2018 by Delphix. All rights reserved. 24.\" Copyright (c) 2012 Cyril Plisko. All Rights Reserved. 25.\" Copyright (c) 2017 Datto Inc. 26.\" Copyright (c) 2018 George Melikov. All Rights Reserved. 27.\" Copyright 2017 Nexenta Systems, Inc. 28.\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved. 29.\" 30.Dd March 16, 2022 31.Dt ZPOOL-IOSTAT 8 32.Os 33. 34.Sh NAME 35.Nm zpool-iostat 36.Nd display logical I/O statistics for ZFS storage pools 37.Sh SYNOPSIS 38.Nm zpool 39.Cm iostat 40.Op Oo Oo Fl c Ar SCRIPT Oc Oo Fl lq Oc Oc Ns | Ns Fl rw 41.Op Fl T Sy u Ns | Ns Sy d 42.Op Fl ghHLnpPvy 43.Oo Ar pool Ns … Ns | Ns Oo Ar pool vdev Ns … Oc Ns | Ns Ar vdev Ns … Oc 44.Op Ar interval Op Ar count 45. 46.Sh DESCRIPTION 47Displays logical I/O statistics for the given pools/vdevs. 48Physical I/O statistics may be observed via 49.Xr iostat 1 . 50If writes are located nearby, they may be merged into a single 51larger operation. 52Additional I/O may be generated depending on the level of vdev redundancy. 53To filter output, you may pass in a list of pools, a pool and list of vdevs 54in that pool, or a list of any vdevs from any pool. 55If no items are specified, statistics for every pool in the system are shown. 56When given an 57.Ar interval , 58the statistics are printed every 59.Ar interval 60seconds until killed. 61If 62.Fl n 63flag is specified the headers are displayed only once, otherwise they are 64displayed periodically. 65If 66.Ar count 67is specified, the command exits after 68.Ar count 69reports are printed. 70The first report printed is always the statistics since boot regardless of 71whether 72.Ar interval 73and 74.Ar count 75are passed. 76However, this behavior can be suppressed with the 77.Fl y 78flag. 79Also note that the units of 80.Sy K , 81.Sy M , 82.Sy G Ns … 83that are printed in the report are in base 1024. 84To get the raw values, use the 85.Fl p 86flag. 87.Bl -tag -width Ds 88.It Fl c Op Ar SCRIPT1 Ns Oo , Ns Ar SCRIPT2 Oc Ns … 89Run a script (or scripts) on each vdev and include the output as a new column 90in the 91.Nm zpool Cm iostat 92output. 93Users can run any script found in their 94.Pa ~/.zpool.d 95directory or from the system 96.Pa /etc/zfs/zpool.d 97directory. 98Script names containing the slash 99.Pq Sy / 100character are not allowed. 101The default search path can be overridden by setting the 102.Sy ZPOOL_SCRIPTS_PATH 103environment variable. 104A privileged user can only run 105.Fl c 106if they have the 107.Sy ZPOOL_SCRIPTS_AS_ROOT 108environment variable set. 109If a script requires the use of a privileged command, like 110.Xr smartctl 8 , 111then it's recommended you allow the user access to it in 112.Pa /etc/sudoers 113or add the user to the 114.Pa /etc/sudoers.d/zfs 115file. 116.Pp 117If 118.Fl c 119is passed without a script name, it prints a list of all scripts. 120.Fl c 121also sets verbose mode 122.No \&( Ns Fl v Ns No \&) . 123.Pp 124Script output should be in the form of "name=value". 125The column name is set to "name" and the value is set to "value". 126Multiple lines can be used to output multiple columns. 127The first line of output not in the 128"name=value" format is displayed without a column title, 129and no more output after that is displayed. 130This can be useful for printing error messages. 131Blank or NULL values are printed as a '-' to make output AWKable. 132.Pp 133The following environment variables are set before running each script: 134.Bl -tag -compact -width "VDEV_ENC_SYSFS_PATH" 135.It Sy VDEV_PATH 136Full path to the vdev 137.It Sy VDEV_UPATH 138Underlying path to the vdev 139.Pq Pa /dev/sd* . 140For use with device mapper, multipath, or partitioned vdevs. 141.It Sy VDEV_ENC_SYSFS_PATH 142The sysfs path to the enclosure for the vdev (if any). 143.El 144.It Fl T Sy u Ns | Ns Sy d 145Display a time stamp. 146Specify 147.Sy u 148for a printed representation of the internal representation of time. 149See 150.Xr time 1 . 151Specify 152.Sy d 153for standard date format. 154See 155.Xr date 1 . 156.It Fl g 157Display vdev GUIDs instead of the normal device names. 158These GUIDs can be used in place of device names for the zpool 159detach/offline/remove/replace commands. 160.It Fl H 161Scripted mode. 162Do not display headers, and separate fields by a 163single tab instead of arbitrary space. 164.It Fl L 165Display real paths for vdevs resolving all symbolic links. 166This can be used to look up the current block device name regardless of the 167.Pa /dev/disk/ 168path used to open it. 169.It Fl n 170Print headers only once when passed 171.It Fl p 172Display numbers in parsable (exact) values. 173Time values are in nanoseconds. 174.It Fl P 175Display full paths for vdevs instead of only the last component of the path. 176This can be used in conjunction with the 177.Fl L 178flag. 179.It Fl r 180Print request size histograms for the leaf vdev's I/O. 181This includes histograms of individual I/O (ind) and aggregate I/O (agg). 182These stats can be useful for observing how well I/O aggregation is working. 183Note that TRIM I/O may exceed 16M, but will be counted as 16M. 184.It Fl v 185Verbose statistics Reports usage statistics for individual vdevs within the 186pool, in addition to the pool-wide statistics. 187.It Fl y 188Normally the first line of output reports the statistics since boot: 189suppress it. 190.It Fl w 191Display latency histograms: 192.Bl -tag -compact -width "asyncq_read/write" 193.It Sy total_wait 194Total I/O time (queuing + disk I/O time). 195.It Sy disk_wait 196Disk I/O time (time reading/writing the disk). 197.It Sy syncq_wait 198Amount of time I/O spent in synchronous priority queues. 199Does not include disk time. 200.It Sy asyncq_wait 201Amount of time I/O spent in asynchronous priority queues. 202Does not include disk time. 203.It Sy scrub 204Amount of time I/O spent in scrub queue. 205Does not include disk time. 206.It Sy rebuild 207Amount of time I/O spent in rebuild queue. 208Does not include disk time. 209.El 210.It Fl l 211Include average latency statistics: 212.Bl -tag -compact -width "asyncq_read/write" 213.It Sy total_wait 214Average total I/O time (queuing + disk I/O time). 215.It Sy disk_wait 216Average disk I/O time (time reading/writing the disk). 217.It Sy syncq_wait 218Average amount of time I/O spent in synchronous priority queues. 219Does not include disk time. 220.It Sy asyncq_wait 221Average amount of time I/O spent in asynchronous priority queues. 222Does not include disk time. 223.It Sy scrub 224Average queuing time in scrub queue. 225Does not include disk time. 226.It Sy trim 227Average queuing time in trim queue. 228Does not include disk time. 229.It Sy rebuild 230Average queuing time in rebuild queue. 231Does not include disk time. 232.El 233.It Fl q 234Include active queue statistics. 235Each priority queue has both pending 236.Sy ( pend ) 237and active 238.Sy ( activ ) 239I/O requests. 240Pending requests are waiting to be issued to the disk, 241and active requests have been issued to disk and are waiting for completion. 242These stats are broken out by priority queue: 243.Bl -tag -compact -width "asyncq_read/write" 244.It Sy syncq_read/write 245Current number of entries in synchronous priority 246queues. 247.It Sy asyncq_read/write 248Current number of entries in asynchronous priority queues. 249.It Sy scrubq_read 250Current number of entries in scrub queue. 251.It Sy trimq_write 252Current number of entries in trim queue. 253.It Sy rebuildq_write 254Current number of entries in rebuild queue. 255.El 256.Pp 257All queue statistics are instantaneous measurements of the number of 258entries in the queues. 259If you specify an interval, 260the measurements will be sampled from the end of the interval. 261.El 262. 263.Sh EXAMPLES 264.\" These are, respectively, examples 13, 16 from zpool.8 265.\" Make sure to update them bidirectionally 266.Ss Example 13 : No Adding Cache Devices to a ZFS Pool 267The following command adds two disks for use as cache devices to a ZFS storage 268pool: 269.Dl # Nm zpool Cm add Ar pool Sy cache Pa sdc sdd 270.Pp 271Once added, the cache devices gradually fill with content from main memory. 272Depending on the size of your cache devices, it could take over an hour for 273them to fill. 274Capacity and reads can be monitored using the 275.Cm iostat 276subcommand as follows: 277.Dl # Nm zpool Cm iostat Fl v Ar pool 5 278. 279.Ss Example 16 : No Adding output columns 280Additional columns can be added to the 281.Nm zpool Cm status No and Nm zpool Cm iostat No output with Fl c . 282.Bd -literal -compact -offset Ds 283.No # Nm zpool Cm status Fl c Pa vendor , Ns Pa model , Ns Pa size 284 NAME STATE READ WRITE CKSUM vendor model size 285 tank ONLINE 0 0 0 286 mirror-0 ONLINE 0 0 0 287 U1 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 288 U10 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 289 U11 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 290 U12 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 291 U13 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 292 U14 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 293 294.No # Nm zpool Cm iostat Fl vc Pa size 295 capacity operations bandwidth 296pool alloc free read write read write size 297---------- ----- ----- ----- ----- ----- ----- ---- 298rpool 14.6G 54.9G 4 55 250K 2.69M 299 sda1 14.6G 54.9G 4 55 250K 2.69M 70G 300---------- ----- ----- ----- ----- ----- ----- ---- 301.Ed 302. 303.Sh SEE ALSO 304.Xr iostat 1 , 305.Xr smartctl 8 , 306.Xr zpool-list 8 , 307.Xr zpool-status 8 308