1.\" 2.\" CDDL HEADER START 3.\" 4.\" The contents of this file are subject to the terms of the 5.\" Common Development and Distribution License (the "License"). 6.\" You may not use this file except in compliance with the License. 7.\" 8.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9.\" or https://opensource.org/licenses/CDDL-1.0. 10.\" See the License for the specific language governing permissions 11.\" and limitations under the License. 12.\" 13.\" When distributing Covered Code, include this CDDL HEADER in each 14.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15.\" If applicable, add the following below this CDDL HEADER, with the 16.\" fields enclosed by brackets "[]" replaced with your own identifying 17.\" information: Portions Copyright [yyyy] [name of copyright owner] 18.\" 19.\" CDDL HEADER END 20.\" 21.\" Copyright (c) 2007, Sun Microsystems, Inc. All Rights Reserved. 22.\" Copyright (c) 2012, 2018 by Delphix. All rights reserved. 23.\" Copyright (c) 2012 Cyril Plisko. All Rights Reserved. 24.\" Copyright (c) 2017 Datto Inc. 25.\" Copyright (c) 2018 George Melikov. All Rights Reserved. 26.\" Copyright 2017 Nexenta Systems, Inc. 27.\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved. 28.\" Copyright (c) 2024, Klara Inc. 29.\" 30.Dd February 28, 2024 31.Dt ZPOOL-EVENTS 8 32.Os 33. 34.Sh NAME 35.Nm zpool-events 36.Nd list recent events generated by kernel 37.Sh SYNOPSIS 38.Nm zpool 39.Cm events 40.Op Fl vHf 41.Op Ar pool 42.Nm zpool 43.Cm events 44.Fl c 45. 46.Sh DESCRIPTION 47Lists all recent events generated by the ZFS kernel modules. 48These events are consumed by the 49.Xr zed 8 50and used to automate administrative tasks such as replacing a failed device 51with a hot spare. 52For more information about the subclasses and event payloads 53that can be generated see 54.Sx EVENTS 55and the following sections. 56. 57.Sh OPTIONS 58.Bl -tag -compact -width Ds 59.It Fl c 60Clear all previous events. 61.It Fl f 62Follow mode. 63.It Fl H 64Scripted mode. 65Do not display headers, and separate fields by a 66single tab instead of arbitrary space. 67.It Fl v 68Print the entire payload for each event. 69.El 70. 71.Sh EVENTS 72These are the different event subclasses. 73The full event name would be 74.Sy ereport.fs.zfs.\& Ns Em SUBCLASS , 75but only the last part is listed here. 76.Pp 77.Bl -tag -compact -width "vdev.bad_guid_sum" 78.It Sy checksum 79Issued when a checksum error has been detected. 80.It Sy io 81Issued when there is an I/O error in a vdev in the pool. 82.It Sy data 83Issued when there have been data errors in the pool. 84.It Sy deadman 85Issued when an I/O request is determined to be "hung", this can be caused 86by lost completion events due to flaky hardware or drivers. 87See 88.Sy zfs_deadman_failmode 89in 90.Xr zfs 4 91for additional information regarding "hung" I/O detection and configuration. 92.It Sy delay 93Issued when a completed I/O request exceeds the maximum allowed time 94specified by the 95.Sy zio_slow_io_ms 96module parameter. 97This can be an indicator of problems with the underlying storage device. 98The number of delay events is ratelimited by the 99.Sy zfs_slow_io_events_per_second 100module parameter. 101.It Sy dio_verify 102Issued when there was a checksum verify error after a Direct I/O write has been 103issued. 104This event can only take place if the module parameter 105.Sy zfs_vdev_direct_write_verify 106is not set to zero. 107See 108.Xr zfs 4 109for more details on the 110.Sy zfs_vdev_direct_write_verify 111module paramter. 112.It Sy config 113Issued every time a vdev change have been done to the pool. 114.It Sy zpool 115Issued when a pool cannot be imported. 116.It Sy zpool.destroy 117Issued when a pool is destroyed. 118.It Sy zpool.export 119Issued when a pool is exported. 120.It Sy zpool.import 121Issued when a pool is imported. 122.It Sy zpool.reguid 123Issued when a REGUID (new unique identifier for the pool have been regenerated) 124have been detected. 125.It Sy vdev.unknown 126Issued when the vdev is unknown. 127Such as trying to clear device errors on a vdev that have failed/been kicked 128from the system/pool and is no longer available. 129.It Sy vdev.open_failed 130Issued when a vdev could not be opened (because it didn't exist for example). 131.It Sy vdev.corrupt_data 132Issued when corrupt data have been detected on a vdev. 133.It Sy vdev.no_replicas 134Issued when there are no more replicas to sustain the pool. 135This would lead to the pool being 136.Em DEGRADED . 137.It Sy vdev.bad_guid_sum 138Issued when a missing device in the pool have been detected. 139.It Sy vdev.too_small 140Issued when the system (kernel) have removed a device, and ZFS 141notices that the device isn't there any more. 142This is usually followed by a 143.Sy probe_failure 144event. 145.It Sy vdev.bad_label 146Issued when the label is OK but invalid. 147.It Sy vdev.bad_ashift 148Issued when the ashift alignment requirement has increased. 149.It Sy vdev.remove 150Issued when a vdev is detached from a mirror (or a spare detached from a 151vdev where it have been used to replace a failed drive - only works if 152the original drive have been re-added). 153.It Sy vdev.clear 154Issued when clearing device errors in a pool. 155Such as running 156.Nm zpool Cm clear 157on a device in the pool. 158.It Sy vdev.check 159Issued when a check to see if a given vdev could be opened is started. 160.It Sy vdev.spare 161Issued when a spare have kicked in to replace a failed device. 162.It Sy vdev.autoexpand 163Issued when a vdev can be automatically expanded. 164.It Sy io_failure 165Issued when there is an I/O failure in a vdev in the pool. 166.It Sy probe_failure 167Issued when a probe fails on a vdev. 168This would occur if a vdev 169have been kicked from the system outside of ZFS (such as the kernel 170have removed the device). 171.It Sy log_replay 172Issued when the intent log cannot be replayed. 173The can occur in the case of a missing or damaged log device. 174.It Sy resilver.start 175Issued when a resilver is started. 176.It Sy resilver.finish 177Issued when the running resilver have finished. 178.It Sy scrub.start 179Issued when a scrub is started on a pool. 180.It Sy scrub.finish 181Issued when a pool has finished scrubbing. 182.It Sy scrub.abort 183Issued when a scrub is aborted on a pool. 184.It Sy scrub.resume 185Issued when a scrub is resumed on a pool. 186.It Sy scrub.paused 187Issued when a scrub is paused on a pool. 188.It Sy bootfs.vdev.attach 189.El 190. 191.Sh PAYLOADS 192This is the payload (data, information) that accompanies an 193event. 194.Pp 195For 196.Xr zed 8 , 197these are set to uppercase and prefixed with 198.Sy ZEVENT_ . 199.Pp 200.Bl -tag -compact -width "vdev_cksum_errors" 201.It Sy pool 202Pool name. 203.It Sy pool_failmode 204Failmode - 205.Sy wait , 206.Sy continue , 207or 208.Sy panic . 209See the 210.Sy failmode 211property in 212.Xr zpoolprops 7 213for more information. 214.It Sy pool_guid 215The GUID of the pool. 216.It Sy pool_context 217The load state for the pool (0=none, 1=open, 2=import, 3=tryimport, 4=recover 2185=error). 219.It Sy vdev_guid 220The GUID of the vdev in question (the vdev failing or operated upon with 221.Nm zpool Cm clear , 222etc.). 223.It Sy vdev_type 224Type of vdev - 225.Sy disk , 226.Sy file , 227.Sy mirror , 228etc. 229See the 230.Sy Virtual Devices 231section of 232.Xr zpoolconcepts 7 233for more information on possible values. 234.It Sy vdev_path 235Full path of the vdev, including any 236.Em -partX . 237.It Sy vdev_devid 238ID of vdev (if any). 239.It Sy vdev_fru 240Physical FRU location. 241.It Sy vdev_state 242State of vdev (0=uninitialized, 1=closed, 2=offline, 3=removed, 4=failed to 243open, 5=faulted, 6=degraded, 7=healthy). 244.It Sy vdev_ashift 245The ashift value of the vdev. 246.It Sy vdev_complete_ts 247The time the last I/O request completed for the specified vdev. 248.It Sy vdev_delta_ts 249The time since the last I/O request completed for the specified vdev. 250.It Sy vdev_spare_paths 251List of spares, including full path and any 252.Em -partX . 253.It Sy vdev_spare_guids 254GUID(s) of spares. 255.It Sy vdev_read_errors 256How many read errors that have been detected on the vdev. 257.It Sy vdev_write_errors 258How many write errors that have been detected on the vdev. 259.It Sy vdev_cksum_errors 260How many checksum errors that have been detected on the vdev. 261.It Sy parent_guid 262GUID of the vdev parent. 263.It Sy parent_type 264Type of parent. 265See 266.Sy vdev_type . 267.It Sy parent_path 268Path of the vdev parent (if any). 269.It Sy parent_devid 270ID of the vdev parent (if any). 271.It Sy zio_objset 272The object set number for a given I/O request. 273.It Sy zio_object 274The object number for a given I/O request. 275.It Sy zio_level 276The indirect level for the block. 277Level 0 is the lowest level and includes data blocks. 278Values > 0 indicate metadata blocks at the appropriate level. 279.It Sy zio_blkid 280The block ID for a given I/O request. 281.It Sy zio_err 282The error number for a failure when handling a given I/O request, 283compatible with 284.Xr errno 3 285with the value of 286.Sy EBADE 287used to indicate a ZFS checksum error. 288.It Sy zio_offset 289The offset in bytes of where to write the I/O request for the specified vdev. 290.It Sy zio_size 291The size in bytes of the I/O request. 292.It Sy zio_flags 293The current flags describing how the I/O request should be handled. 294See the 295.Sy I/O FLAGS 296section for the full list of I/O flags. 297.It Sy zio_stage 298The current stage of the I/O in the pipeline. 299See the 300.Sy I/O STAGES 301section for a full list of all the I/O stages. 302.It Sy zio_pipeline 303The valid pipeline stages for the I/O. 304See the 305.Sy I/O STAGES 306section for a full list of all the I/O stages. 307.It Sy zio_delay 308The time elapsed (in nanoseconds) waiting for the block layer to complete the 309I/O request. 310Unlike 311.Sy zio_delta , 312this does not include any vdev queuing time and is 313therefore solely a measure of the block layer performance. 314.It Sy zio_timestamp 315The time when a given I/O request was submitted. 316.It Sy zio_delta 317The time required to service a given I/O request. 318.It Sy prev_state 319The previous state of the vdev. 320.It Sy cksum_algorithm 321Checksum algorithm used. 322See 323.Xr zfsprops 7 324for more information on the available checksum algorithms. 325.It Sy cksum_byteswap 326Whether or not the data is byteswapped. 327.It Sy bad_ranges 328.No [\& Ns Ar start , end ) 329pairs of corruption offsets. 330Offsets are always aligned on a 64-bit boundary, 331and can include some gaps of non-corruption. 332(See 333.Sy bad_ranges_min_gap ) 334.It Sy bad_ranges_min_gap 335In order to bound the size of the 336.Sy bad_ranges 337array, gaps of non-corruption 338less than or equal to 339.Sy bad_ranges_min_gap 340bytes have been merged with 341adjacent corruption. 342Always at least 8 bytes, since corruption is detected on a 64-bit word basis. 343.It Sy bad_range_sets 344This array has one element per range in 345.Sy bad_ranges . 346Each element contains 347the count of bits in that range which were clear in the good data and set 348in the bad data. 349.It Sy bad_range_clears 350This array has one element per range in 351.Sy bad_ranges . 352Each element contains 353the count of bits for that range which were set in the good data and clear in 354the bad data. 355.It Sy bad_set_bits 356If this field exists, it is an array of 357.Pq Ar bad data No & ~( Ns Ar good data ) ; 358that is, the bits set in the bad data which are cleared in the good data. 359Each element corresponds a byte whose offset is in a range in 360.Sy bad_ranges , 361and the array is ordered by offset. 362Thus, the first element is the first byte in the first 363.Sy bad_ranges 364range, and the last element is the last byte in the last 365.Sy bad_ranges 366range. 367.It Sy bad_cleared_bits 368Like 369.Sy bad_set_bits , 370but contains 371.Pq Ar good data No & ~( Ns Ar bad data ) ; 372that is, the bits set in the good data which are cleared in the bad data. 373.El 374. 375.Sh I/O STAGES 376The ZFS I/O pipeline is comprised of various stages which are defined below. 377The individual stages are used to construct these basic I/O 378operations: Read, Write, Free, Claim, Flush and Trim. 379These stages may be 380set on an event to describe the life cycle of a given I/O request. 381.Pp 382.TS 383tab(:); 384l l l . 385Stage:Bit Mask:Operations 386_:_:_ 387ZIO_STAGE_OPEN:0x00000001:RWFCXT 388 389ZIO_STAGE_READ_BP_INIT:0x00000002:R----- 390ZIO_STAGE_WRITE_BP_INIT:0x00000004:-W---- 391ZIO_STAGE_FREE_BP_INIT:0x00000008:--F--- 392ZIO_STAGE_ISSUE_ASYNC:0x00000010:-WF--T 393ZIO_STAGE_WRITE_COMPRESS:0x00000020:-W---- 394 395ZIO_STAGE_ENCRYPT:0x00000040:-W---- 396ZIO_STAGE_CHECKSUM_GENERATE:0x00000080:-W---- 397 398ZIO_STAGE_NOP_WRITE:0x00000100:-W---- 399 400ZIO_STAGE_BRT_FREE:0x00000200:--F--- 401 402ZIO_STAGE_DDT_READ_START:0x00000400:R----- 403ZIO_STAGE_DDT_READ_DONE:0x00000800:R----- 404ZIO_STAGE_DDT_WRITE:0x00001000:-W---- 405ZIO_STAGE_DDT_FREE:0x00002000:--F--- 406 407ZIO_STAGE_GANG_ASSEMBLE:0x00004000:RWFC-- 408ZIO_STAGE_GANG_ISSUE:0x00008000:RWFC-- 409 410ZIO_STAGE_DVA_THROTTLE:0x00010000:-W---- 411ZIO_STAGE_DVA_ALLOCATE:0x00020000:-W---- 412ZIO_STAGE_DVA_FREE:0x00040000:--F--- 413ZIO_STAGE_DVA_CLAIM:0x00080000:---C-- 414 415ZIO_STAGE_READY:0x00100000:RWFCIT 416 417ZIO_STAGE_VDEV_IO_START:0x00200000:RW--XT 418ZIO_STAGE_VDEV_IO_DONE:0x00400000:RW--XT 419ZIO_STAGE_VDEV_IO_ASSESS:0x00800000:RW--XT 420 421ZIO_STAGE_CHECKSUM_VERIFY:0x01000000:R----- 422ZIO_STAGE_DIO_CHECKSUM_VERIFY:0x02000000:-W---- 423 424ZIO_STAGE_DONE:0x04000000:RWFCXT 425.TE 426. 427.Sh I/O FLAGS 428Every I/O request in the pipeline contains a set of flags which describe its 429function and are used to govern its behavior. 430These flags will be set in an event as a 431.Sy zio_flags 432payload entry. 433.Pp 434.TS 435tab(:); 436l l . 437Flag:Bit Mask 438_:_ 439ZIO_FLAG_DONT_AGGREGATE:0x00000001 440ZIO_FLAG_IO_REPAIR:0x00000002 441ZIO_FLAG_SELF_HEAL:0x00000004 442ZIO_FLAG_RESILVER:0x00000008 443ZIO_FLAG_SCRUB:0x00000010 444ZIO_FLAG_SCAN_THREAD:0x00000020 445ZIO_FLAG_PHYSICAL:0x00000040 446 447ZIO_FLAG_CANFAIL:0x00000080 448ZIO_FLAG_SPECULATIVE:0x00000100 449ZIO_FLAG_CONFIG_WRITER:0x00000200 450ZIO_FLAG_DONT_RETRY:0x00000400 451ZIO_FLAG_NODATA:0x00001000 452ZIO_FLAG_INDUCE_DAMAGE:0x00002000 453 454ZIO_FLAG_IO_ALLOCATING:0x00004000 455ZIO_FLAG_IO_RETRY:0x00008000 456ZIO_FLAG_PROBE:0x00010000 457ZIO_FLAG_TRYHARD:0x00020000 458ZIO_FLAG_OPTIONAL:0x00040000 459 460ZIO_FLAG_DONT_QUEUE:0x00080000 461ZIO_FLAG_DONT_PROPAGATE:0x00100000 462ZIO_FLAG_IO_BYPASS:0x00200000 463ZIO_FLAG_IO_REWRITE:0x00400000 464ZIO_FLAG_RAW_COMPRESS:0x00800000 465ZIO_FLAG_RAW_ENCRYPT:0x01000000 466 467ZIO_FLAG_GANG_CHILD:0x02000000 468ZIO_FLAG_DDT_CHILD:0x04000000 469ZIO_FLAG_GODFATHER:0x08000000 470ZIO_FLAG_NOPWRITE:0x10000000 471ZIO_FLAG_REEXECUTED:0x20000000 472ZIO_FLAG_DELEGATED:0x40000000 473ZIO_FLAG_FASTWRITE:0x80000000 474.TE 475. 476.Sh SEE ALSO 477.Xr zfs 4 , 478.Xr zed 8 , 479.Xr zpool-wait 8 480