1.\" 2.\" CDDL HEADER START 3.\" 4.\" The contents of this file are subject to the terms of the 5.\" Common Development and Distribution License (the "License"). 6.\" You may not use this file except in compliance with the License. 7.\" 8.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9.\" or https://opensource.org/licenses/CDDL-1.0. 10.\" See the License for the specific language governing permissions 11.\" and limitations under the License. 12.\" 13.\" When distributing Covered Code, include this CDDL HEADER in each 14.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15.\" If applicable, add the following below this CDDL HEADER, with the 16.\" fields enclosed by brackets "[]" replaced with your own identifying 17.\" information: Portions Copyright [yyyy] [name of copyright owner] 18.\" 19.\" CDDL HEADER END 20.\" 21.\" Copyright (c) 2007, Sun Microsystems, Inc. All Rights Reserved. 22.\" Copyright (c) 2012, 2018 by Delphix. All rights reserved. 23.\" Copyright (c) 2012 Cyril Plisko. All Rights Reserved. 24.\" Copyright (c) 2017 Datto Inc. 25.\" Copyright (c) 2018 George Melikov. All Rights Reserved. 26.\" Copyright 2017 Nexenta Systems, Inc. 27.\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved. 28.\" 29.Dd March 16, 2022 30.Dt ZPOOL-IOSTAT 8 31.Os 32. 33.Sh NAME 34.Nm zpool-iostat 35.Nd display logical I/O statistics for ZFS storage pools 36.Sh SYNOPSIS 37.Nm zpool 38.Cm iostat 39.Op Oo Oo Fl c Ar SCRIPT Oc Oo Fl lq Oc Oc Ns | Ns Fl rw 40.Op Fl T Sy u Ns | Ns Sy d 41.Op Fl ghHLnpPvy 42.Oo Ar pool Ns … Ns | Ns Oo Ar pool vdev Ns … Oc Ns | Ns Ar vdev Ns … Oc 43.Op Ar interval Op Ar count 44. 45.Sh DESCRIPTION 46Displays logical I/O statistics for the given pools/vdevs. 47Physical I/O statistics may be observed via 48.Xr iostat 1 . 49If writes are located nearby, they may be merged into a single 50larger operation. 51Additional I/O may be generated depending on the level of vdev redundancy. 52To filter output, you may pass in a list of pools, a pool and list of vdevs 53in that pool, or a list of any vdevs from any pool. 54If no items are specified, statistics for every pool in the system are shown. 55When given an 56.Ar interval , 57the statistics are printed every 58.Ar interval 59seconds until killed. 60If 61.Fl n 62flag is specified the headers are displayed only once, otherwise they are 63displayed periodically. 64If 65.Ar count 66is specified, the command exits after 67.Ar count 68reports are printed. 69The first report printed is always the statistics since boot regardless of 70whether 71.Ar interval 72and 73.Ar count 74are passed. 75However, this behavior can be suppressed with the 76.Fl y 77flag. 78Also note that the units of 79.Sy K , 80.Sy M , 81.Sy G Ns … 82that are printed in the report are in base 1024. 83To get the raw values, use the 84.Fl p 85flag. 86.Bl -tag -width Ds 87.It Fl c Op Ar SCRIPT1 Ns Oo , Ns Ar SCRIPT2 Oc Ns … 88Run a script (or scripts) on each vdev and include the output as a new column 89in the 90.Nm zpool Cm iostat 91output. 92Users can run any script found in their 93.Pa ~/.zpool.d 94directory or from the system 95.Pa /etc/zfs/zpool.d 96directory. 97Script names containing the slash 98.Pq Sy / 99character are not allowed. 100The default search path can be overridden by setting the 101.Sy ZPOOL_SCRIPTS_PATH 102environment variable. 103A privileged user can only run 104.Fl c 105if they have the 106.Sy ZPOOL_SCRIPTS_AS_ROOT 107environment variable set. 108If a script requires the use of a privileged command, like 109.Xr smartctl 8 , 110then it's recommended you allow the user access to it in 111.Pa /etc/sudoers 112or add the user to the 113.Pa /etc/sudoers.d/zfs 114file. 115.Pp 116If 117.Fl c 118is passed without a script name, it prints a list of all scripts. 119.Fl c 120also sets verbose mode 121.No \&( Ns Fl v Ns No \&) . 122.Pp 123Script output should be in the form of "name=value". 124The column name is set to "name" and the value is set to "value". 125Multiple lines can be used to output multiple columns. 126The first line of output not in the 127"name=value" format is displayed without a column title, 128and no more output after that is displayed. 129This can be useful for printing error messages. 130Blank or NULL values are printed as a '-' to make output AWKable. 131.Pp 132The following environment variables are set before running each script: 133.Bl -tag -compact -width "VDEV_ENC_SYSFS_PATH" 134.It Sy VDEV_PATH 135Full path to the vdev 136.It Sy VDEV_UPATH 137Underlying path to the vdev 138.Pq Pa /dev/sd* . 139For use with device mapper, multipath, or partitioned vdevs. 140.It Sy VDEV_ENC_SYSFS_PATH 141The sysfs path to the enclosure for the vdev (if any). 142.El 143.It Fl T Sy u Ns | Ns Sy d 144Display a time stamp. 145Specify 146.Sy u 147for a printed representation of the internal representation of time. 148See 149.Xr time 2 . 150Specify 151.Sy d 152for standard date format. 153See 154.Xr date 1 . 155.It Fl g 156Display vdev GUIDs instead of the normal device names. 157These GUIDs can be used in place of device names for the zpool 158detach/offline/remove/replace commands. 159.It Fl H 160Scripted mode. 161Do not display headers, and separate fields by a 162single tab instead of arbitrary space. 163.It Fl L 164Display real paths for vdevs resolving all symbolic links. 165This can be used to look up the current block device name regardless of the 166.Pa /dev/disk/ 167path used to open it. 168.It Fl n 169Print headers only once when passed 170.It Fl p 171Display numbers in parsable (exact) values. 172Time values are in nanoseconds. 173.It Fl P 174Display full paths for vdevs instead of only the last component of the path. 175This can be used in conjunction with the 176.Fl L 177flag. 178.It Fl r 179Print request size histograms for the leaf vdev's I/O. 180This includes histograms of individual I/O (ind) and aggregate I/O (agg). 181These stats can be useful for observing how well I/O aggregation is working. 182Note that TRIM I/O may exceed 16M, but will be counted as 16M. 183.It Fl v 184Verbose statistics Reports usage statistics for individual vdevs within the 185pool, in addition to the pool-wide statistics. 186.It Fl y 187Normally the first line of output reports the statistics since boot: 188suppress it. 189.It Fl w 190Display latency histograms: 191.Bl -tag -compact -width "asyncq_read/write" 192.It Sy total_wait 193Total I/O time (queuing + disk I/O time). 194.It Sy disk_wait 195Disk I/O time (time reading/writing the disk). 196.It Sy syncq_wait 197Amount of time I/O spent in synchronous priority queues. 198Does not include disk time. 199.It Sy asyncq_wait 200Amount of time I/O spent in asynchronous priority queues. 201Does not include disk time. 202.It Sy scrub 203Amount of time I/O spent in scrub queue. 204Does not include disk time. 205.It Sy rebuild 206Amount of time I/O spent in rebuild queue. 207Does not include disk time. 208.El 209.It Fl l 210Include average latency statistics: 211.Bl -tag -compact -width "asyncq_read/write" 212.It Sy total_wait 213Average total I/O time (queuing + disk I/O time). 214.It Sy disk_wait 215Average disk I/O time (time reading/writing the disk). 216.It Sy syncq_wait 217Average amount of time I/O spent in synchronous priority queues. 218Does not include disk time. 219.It Sy asyncq_wait 220Average amount of time I/O spent in asynchronous priority queues. 221Does not include disk time. 222.It Sy scrub 223Average queuing time in scrub queue. 224Does not include disk time. 225.It Sy trim 226Average queuing time in trim queue. 227Does not include disk time. 228.It Sy rebuild 229Average queuing time in rebuild queue. 230Does not include disk time. 231.El 232.It Fl q 233Include active queue statistics. 234Each priority queue has both pending 235.Sy ( pend ) 236and active 237.Sy ( activ ) 238I/O requests. 239Pending requests are waiting to be issued to the disk, 240and active requests have been issued to disk and are waiting for completion. 241These stats are broken out by priority queue: 242.Bl -tag -compact -width "asyncq_read/write" 243.It Sy syncq_read/write 244Current number of entries in synchronous priority 245queues. 246.It Sy asyncq_read/write 247Current number of entries in asynchronous priority queues. 248.It Sy scrubq_read 249Current number of entries in scrub queue. 250.It Sy trimq_write 251Current number of entries in trim queue. 252.It Sy rebuildq_write 253Current number of entries in rebuild queue. 254.El 255.Pp 256All queue statistics are instantaneous measurements of the number of 257entries in the queues. 258If you specify an interval, 259the measurements will be sampled from the end of the interval. 260.El 261. 262.Sh EXAMPLES 263.\" These are, respectively, examples 13, 16 from zpool.8 264.\" Make sure to update them bidirectionally 265.Ss Example 13 : No Adding Cache Devices to a ZFS Pool 266The following command adds two disks for use as cache devices to a ZFS storage 267pool: 268.Dl # Nm zpool Cm add Ar pool Sy cache Pa sdc sdd 269.Pp 270Once added, the cache devices gradually fill with content from main memory. 271Depending on the size of your cache devices, it could take over an hour for 272them to fill. 273Capacity and reads can be monitored using the 274.Cm iostat 275subcommand as follows: 276.Dl # Nm zpool Cm iostat Fl v Ar pool 5 277. 278.Ss Example 16 : No Adding output columns 279Additional columns can be added to the 280.Nm zpool Cm status No and Nm zpool Cm iostat No output with Fl c . 281.Bd -literal -compact -offset Ds 282.No # Nm zpool Cm status Fl c Pa vendor , Ns Pa model , Ns Pa size 283 NAME STATE READ WRITE CKSUM vendor model size 284 tank ONLINE 0 0 0 285 mirror-0 ONLINE 0 0 0 286 U1 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 287 U10 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 288 U11 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 289 U12 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 290 U13 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 291 U14 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 292 293.No # Nm zpool Cm iostat Fl vc Pa size 294 capacity operations bandwidth 295pool alloc free read write read write size 296---------- ----- ----- ----- ----- ----- ----- ---- 297rpool 14.6G 54.9G 4 55 250K 2.69M 298 sda1 14.6G 54.9G 4 55 250K 2.69M 70G 299---------- ----- ----- ----- ----- ----- ----- ---- 300.Ed 301. 302.Sh SEE ALSO 303.Xr iostat 1 , 304.Xr smartctl 8 , 305.Xr zpool-list 8 , 306.Xr zpool-status 8 307