1.\" 2.\" CDDL HEADER START 3.\" 4.\" The contents of this file are subject to the terms of the 5.\" Common Development and Distribution License (the "License"). 6.\" You may not use this file except in compliance with the License. 7.\" 8.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9.\" or https://opensource.org/licenses/CDDL-1.0. 10.\" See the License for the specific language governing permissions 11.\" and limitations under the License. 12.\" 13.\" When distributing Covered Code, include this CDDL HEADER in each 14.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15.\" If applicable, add the following below this CDDL HEADER, with the 16.\" fields enclosed by brackets "[]" replaced with your own identifying 17.\" information: Portions Copyright [yyyy] [name of copyright owner] 18.\" 19.\" CDDL HEADER END 20.\" 21.\" Copyright (c) 2007, Sun Microsystems, Inc. All Rights Reserved. 22.\" Copyright (c) 2012, 2018 by Delphix. All rights reserved. 23.\" Copyright (c) 2012 Cyril Plisko. All Rights Reserved. 24.\" Copyright (c) 2017 Datto Inc. 25.\" Copyright (c) 2018 George Melikov. All Rights Reserved. 26.\" Copyright 2017 Nexenta Systems, Inc. 27.\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved. 28.\" Copyright (c) 2024, Klara Inc. 29.\" 30.Dd February 28, 2024 31.Dt ZPOOL-EVENTS 8 32.Os 33. 34.Sh NAME 35.Nm zpool-events 36.Nd list recent events generated by kernel 37.Sh SYNOPSIS 38.Nm zpool 39.Cm events 40.Op Fl vHf 41.Op Ar pool 42.Nm zpool 43.Cm events 44.Fl c 45. 46.Sh DESCRIPTION 47Lists all recent events generated by the ZFS kernel modules. 48These events are consumed by the 49.Xr zed 8 50and used to automate administrative tasks such as replacing a failed device 51with a hot spare. 52For more information about the subclasses and event payloads 53that can be generated see 54.Sx EVENTS 55and the following sections. 56. 57.Sh OPTIONS 58.Bl -tag -compact -width Ds 59.It Fl c 60Clear all previous events. 61.It Fl f 62Follow mode. 63.It Fl H 64Scripted mode. 65Do not display headers, and separate fields by a 66single tab instead of arbitrary space. 67.It Fl v 68Print the entire payload for each event. 69.El 70. 71.Sh EVENTS 72These are the different event subclasses. 73The full event name would be 74.Sy ereport.fs.zfs.\& Ns Em SUBCLASS , 75but only the last part is listed here. 76.Pp 77.Bl -tag -compact -width "vdev.bad_guid_sum" 78.It Sy checksum 79Issued when a checksum error has been detected. 80.It Sy io 81Issued when there is an I/O error in a vdev in the pool. 82.It Sy data 83Issued when there have been data errors in the pool. 84.It Sy deadman 85Issued when an I/O request is determined to be "hung", this can be caused 86by lost completion events due to flaky hardware or drivers. 87See 88.Sy zfs_deadman_failmode 89in 90.Xr zfs 4 91for additional information regarding "hung" I/O detection and configuration. 92.It Sy delay 93Issued when a completed I/O request exceeds the maximum allowed time 94specified by the 95.Sy zio_slow_io_ms 96module parameter. 97This can be an indicator of problems with the underlying storage device. 98The number of delay events is ratelimited by the 99.Sy zfs_slow_io_events_per_second 100module parameter. 101.It Sy config 102Issued every time a vdev change have been done to the pool. 103.It Sy zpool 104Issued when a pool cannot be imported. 105.It Sy zpool.destroy 106Issued when a pool is destroyed. 107.It Sy zpool.export 108Issued when a pool is exported. 109.It Sy zpool.import 110Issued when a pool is imported. 111.It Sy zpool.reguid 112Issued when a REGUID (new unique identifier for the pool have been regenerated) 113have been detected. 114.It Sy vdev.unknown 115Issued when the vdev is unknown. 116Such as trying to clear device errors on a vdev that have failed/been kicked 117from the system/pool and is no longer available. 118.It Sy vdev.open_failed 119Issued when a vdev could not be opened (because it didn't exist for example). 120.It Sy vdev.corrupt_data 121Issued when corrupt data have been detected on a vdev. 122.It Sy vdev.no_replicas 123Issued when there are no more replicas to sustain the pool. 124This would lead to the pool being 125.Em DEGRADED . 126.It Sy vdev.bad_guid_sum 127Issued when a missing device in the pool have been detected. 128.It Sy vdev.too_small 129Issued when the system (kernel) have removed a device, and ZFS 130notices that the device isn't there any more. 131This is usually followed by a 132.Sy probe_failure 133event. 134.It Sy vdev.bad_label 135Issued when the label is OK but invalid. 136.It Sy vdev.bad_ashift 137Issued when the ashift alignment requirement has increased. 138.It Sy vdev.remove 139Issued when a vdev is detached from a mirror (or a spare detached from a 140vdev where it have been used to replace a failed drive - only works if 141the original drive have been re-added). 142.It Sy vdev.clear 143Issued when clearing device errors in a pool. 144Such as running 145.Nm zpool Cm clear 146on a device in the pool. 147.It Sy vdev.check 148Issued when a check to see if a given vdev could be opened is started. 149.It Sy vdev.spare 150Issued when a spare have kicked in to replace a failed device. 151.It Sy vdev.autoexpand 152Issued when a vdev can be automatically expanded. 153.It Sy io_failure 154Issued when there is an I/O failure in a vdev in the pool. 155.It Sy probe_failure 156Issued when a probe fails on a vdev. 157This would occur if a vdev 158have been kicked from the system outside of ZFS (such as the kernel 159have removed the device). 160.It Sy log_replay 161Issued when the intent log cannot be replayed. 162The can occur in the case of a missing or damaged log device. 163.It Sy resilver.start 164Issued when a resilver is started. 165.It Sy resilver.finish 166Issued when the running resilver have finished. 167.It Sy scrub.start 168Issued when a scrub is started on a pool. 169.It Sy scrub.finish 170Issued when a pool has finished scrubbing. 171.It Sy scrub.abort 172Issued when a scrub is aborted on a pool. 173.It Sy scrub.resume 174Issued when a scrub is resumed on a pool. 175.It Sy scrub.paused 176Issued when a scrub is paused on a pool. 177.It Sy bootfs.vdev.attach 178.El 179. 180.Sh PAYLOADS 181This is the payload (data, information) that accompanies an 182event. 183.Pp 184For 185.Xr zed 8 , 186these are set to uppercase and prefixed with 187.Sy ZEVENT_ . 188.Pp 189.Bl -tag -compact -width "vdev_cksum_errors" 190.It Sy pool 191Pool name. 192.It Sy pool_failmode 193Failmode - 194.Sy wait , 195.Sy continue , 196or 197.Sy panic . 198See the 199.Sy failmode 200property in 201.Xr zpoolprops 7 202for more information. 203.It Sy pool_guid 204The GUID of the pool. 205.It Sy pool_context 206The load state for the pool (0=none, 1=open, 2=import, 3=tryimport, 4=recover 2075=error). 208.It Sy vdev_guid 209The GUID of the vdev in question (the vdev failing or operated upon with 210.Nm zpool Cm clear , 211etc.). 212.It Sy vdev_type 213Type of vdev - 214.Sy disk , 215.Sy file , 216.Sy mirror , 217etc. 218See the 219.Sy Virtual Devices 220section of 221.Xr zpoolconcepts 7 222for more information on possible values. 223.It Sy vdev_path 224Full path of the vdev, including any 225.Em -partX . 226.It Sy vdev_devid 227ID of vdev (if any). 228.It Sy vdev_fru 229Physical FRU location. 230.It Sy vdev_state 231State of vdev (0=uninitialized, 1=closed, 2=offline, 3=removed, 4=failed to 232open, 5=faulted, 6=degraded, 7=healthy). 233.It Sy vdev_ashift 234The ashift value of the vdev. 235.It Sy vdev_complete_ts 236The time the last I/O request completed for the specified vdev. 237.It Sy vdev_delta_ts 238The time since the last I/O request completed for the specified vdev. 239.It Sy vdev_spare_paths 240List of spares, including full path and any 241.Em -partX . 242.It Sy vdev_spare_guids 243GUID(s) of spares. 244.It Sy vdev_read_errors 245How many read errors that have been detected on the vdev. 246.It Sy vdev_write_errors 247How many write errors that have been detected on the vdev. 248.It Sy vdev_cksum_errors 249How many checksum errors that have been detected on the vdev. 250.It Sy parent_guid 251GUID of the vdev parent. 252.It Sy parent_type 253Type of parent. 254See 255.Sy vdev_type . 256.It Sy parent_path 257Path of the vdev parent (if any). 258.It Sy parent_devid 259ID of the vdev parent (if any). 260.It Sy zio_objset 261The object set number for a given I/O request. 262.It Sy zio_object 263The object number for a given I/O request. 264.It Sy zio_level 265The indirect level for the block. 266Level 0 is the lowest level and includes data blocks. 267Values > 0 indicate metadata blocks at the appropriate level. 268.It Sy zio_blkid 269The block ID for a given I/O request. 270.It Sy zio_err 271The error number for a failure when handling a given I/O request, 272compatible with 273.Xr errno 3 274with the value of 275.Sy EBADE 276used to indicate a ZFS checksum error. 277.It Sy zio_offset 278The offset in bytes of where to write the I/O request for the specified vdev. 279.It Sy zio_size 280The size in bytes of the I/O request. 281.It Sy zio_flags 282The current flags describing how the I/O request should be handled. 283See the 284.Sy I/O FLAGS 285section for the full list of I/O flags. 286.It Sy zio_stage 287The current stage of the I/O in the pipeline. 288See the 289.Sy I/O STAGES 290section for a full list of all the I/O stages. 291.It Sy zio_pipeline 292The valid pipeline stages for the I/O. 293See the 294.Sy I/O STAGES 295section for a full list of all the I/O stages. 296.It Sy zio_delay 297The time elapsed (in nanoseconds) waiting for the block layer to complete the 298I/O request. 299Unlike 300.Sy zio_delta , 301this does not include any vdev queuing time and is 302therefore solely a measure of the block layer performance. 303.It Sy zio_timestamp 304The time when a given I/O request was submitted. 305.It Sy zio_delta 306The time required to service a given I/O request. 307.It Sy prev_state 308The previous state of the vdev. 309.It Sy cksum_algorithm 310Checksum algorithm used. 311See 312.Xr zfsprops 7 313for more information on the available checksum algorithms. 314.It Sy cksum_byteswap 315Whether or not the data is byteswapped. 316.It Sy bad_ranges 317.No [\& Ns Ar start , end ) 318pairs of corruption offsets. 319Offsets are always aligned on a 64-bit boundary, 320and can include some gaps of non-corruption. 321(See 322.Sy bad_ranges_min_gap ) 323.It Sy bad_ranges_min_gap 324In order to bound the size of the 325.Sy bad_ranges 326array, gaps of non-corruption 327less than or equal to 328.Sy bad_ranges_min_gap 329bytes have been merged with 330adjacent corruption. 331Always at least 8 bytes, since corruption is detected on a 64-bit word basis. 332.It Sy bad_range_sets 333This array has one element per range in 334.Sy bad_ranges . 335Each element contains 336the count of bits in that range which were clear in the good data and set 337in the bad data. 338.It Sy bad_range_clears 339This array has one element per range in 340.Sy bad_ranges . 341Each element contains 342the count of bits for that range which were set in the good data and clear in 343the bad data. 344.It Sy bad_set_bits 345If this field exists, it is an array of 346.Pq Ar bad data No & ~( Ns Ar good data ) ; 347that is, the bits set in the bad data which are cleared in the good data. 348Each element corresponds a byte whose offset is in a range in 349.Sy bad_ranges , 350and the array is ordered by offset. 351Thus, the first element is the first byte in the first 352.Sy bad_ranges 353range, and the last element is the last byte in the last 354.Sy bad_ranges 355range. 356.It Sy bad_cleared_bits 357Like 358.Sy bad_set_bits , 359but contains 360.Pq Ar good data No & ~( Ns Ar bad data ) ; 361that is, the bits set in the good data which are cleared in the bad data. 362.El 363. 364.Sh I/O STAGES 365The ZFS I/O pipeline is comprised of various stages which are defined below. 366The individual stages are used to construct these basic I/O 367operations: Read, Write, Free, Claim, Flush and Trim. 368These stages may be 369set on an event to describe the life cycle of a given I/O request. 370.Pp 371.TS 372tab(:); 373l l l . 374Stage:Bit Mask:Operations 375_:_:_ 376ZIO_STAGE_OPEN:0x00000001:RWFCXT 377 378ZIO_STAGE_READ_BP_INIT:0x00000002:R----- 379ZIO_STAGE_WRITE_BP_INIT:0x00000004:-W---- 380ZIO_STAGE_FREE_BP_INIT:0x00000008:--F--- 381ZIO_STAGE_ISSUE_ASYNC:0x00000010:-WF--T 382ZIO_STAGE_WRITE_COMPRESS:0x00000020:-W---- 383 384ZIO_STAGE_ENCRYPT:0x00000040:-W---- 385ZIO_STAGE_CHECKSUM_GENERATE:0x00000080:-W---- 386 387ZIO_STAGE_NOP_WRITE:0x00000100:-W---- 388 389ZIO_STAGE_BRT_FREE:0x00000200:--F--- 390 391ZIO_STAGE_DDT_READ_START:0x00000400:R----- 392ZIO_STAGE_DDT_READ_DONE:0x00000800:R----- 393ZIO_STAGE_DDT_WRITE:0x00001000:-W---- 394ZIO_STAGE_DDT_FREE:0x00002000:--F--- 395 396ZIO_STAGE_GANG_ASSEMBLE:0x00004000:RWFC-- 397ZIO_STAGE_GANG_ISSUE:0x00008000:RWFC-- 398 399ZIO_STAGE_DVA_THROTTLE:0x00010000:-W---- 400ZIO_STAGE_DVA_ALLOCATE:0x00020000:-W---- 401ZIO_STAGE_DVA_FREE:0x00040000:--F--- 402ZIO_STAGE_DVA_CLAIM:0x00080000:---C-- 403 404ZIO_STAGE_READY:0x00100000:RWFCIT 405 406ZIO_STAGE_VDEV_IO_START:0x00200000:RW--XT 407ZIO_STAGE_VDEV_IO_DONE:0x00400000:RW--XT 408ZIO_STAGE_VDEV_IO_ASSESS:0x00800000:RW--XT 409 410ZIO_STAGE_CHECKSUM_VERIFY:0x01000000:R----- 411 412ZIO_STAGE_DONE:0x02000000:RWFCXT 413.TE 414. 415.Sh I/O FLAGS 416Every I/O request in the pipeline contains a set of flags which describe its 417function and are used to govern its behavior. 418These flags will be set in an event as a 419.Sy zio_flags 420payload entry. 421.Pp 422.TS 423tab(:); 424l l . 425Flag:Bit Mask 426_:_ 427ZIO_FLAG_DONT_AGGREGATE:0x00000001 428ZIO_FLAG_IO_REPAIR:0x00000002 429ZIO_FLAG_SELF_HEAL:0x00000004 430ZIO_FLAG_RESILVER:0x00000008 431ZIO_FLAG_SCRUB:0x00000010 432ZIO_FLAG_SCAN_THREAD:0x00000020 433ZIO_FLAG_PHYSICAL:0x00000040 434 435ZIO_FLAG_CANFAIL:0x00000080 436ZIO_FLAG_SPECULATIVE:0x00000100 437ZIO_FLAG_CONFIG_WRITER:0x00000200 438ZIO_FLAG_DONT_RETRY:0x00000400 439ZIO_FLAG_NODATA:0x00001000 440ZIO_FLAG_INDUCE_DAMAGE:0x00002000 441 442ZIO_FLAG_IO_ALLOCATING:0x00004000 443ZIO_FLAG_IO_RETRY:0x00008000 444ZIO_FLAG_PROBE:0x00010000 445ZIO_FLAG_TRYHARD:0x00020000 446ZIO_FLAG_OPTIONAL:0x00040000 447 448ZIO_FLAG_DONT_QUEUE:0x00080000 449ZIO_FLAG_DONT_PROPAGATE:0x00100000 450ZIO_FLAG_IO_BYPASS:0x00200000 451ZIO_FLAG_IO_REWRITE:0x00400000 452ZIO_FLAG_RAW_COMPRESS:0x00800000 453ZIO_FLAG_RAW_ENCRYPT:0x01000000 454 455ZIO_FLAG_GANG_CHILD:0x02000000 456ZIO_FLAG_DDT_CHILD:0x04000000 457ZIO_FLAG_GODFATHER:0x08000000 458ZIO_FLAG_NOPWRITE:0x10000000 459ZIO_FLAG_REEXECUTED:0x20000000 460ZIO_FLAG_DELEGATED:0x40000000 461ZIO_FLAG_FASTWRITE:0x80000000 462.TE 463. 464.Sh SEE ALSO 465.Xr zfs 4 , 466.Xr zed 8 , 467.Xr zpool-wait 8 468