1 /* 2 * CDDL HEADER START 3 * 4 * The contents of this file are subject to the terms of the 5 * Common Development and Distribution License (the "License"). 6 * You may not use this file except in compliance with the License. 7 * 8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9 * or https://opensource.org/licenses/CDDL-1.0. 10 * See the License for the specific language governing permissions 11 * and limitations under the License. 12 * 13 * When distributing Covered Code, include this CDDL HEADER in each 14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15 * If applicable, add the following below this CDDL HEADER, with the 16 * fields enclosed by brackets "[]" replaced with your own identifying 17 * information: Portions Copyright [yyyy] [name of copyright owner] 18 * 19 * CDDL HEADER END 20 */ 21 /* 22 * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. 23 * Copyright (c) 2018, Joyent, Inc. 24 * Copyright (c) 2011, 2020, Delphix. All rights reserved. 25 * Copyright (c) 2014, Saso Kiselkov. All rights reserved. 26 * Copyright (c) 2017, Nexenta Systems, Inc. All rights reserved. 27 * Copyright (c) 2019, loli10K <ezomori.nozomu@gmail.com>. All rights reserved. 28 * Copyright (c) 2020, George Amanakis. All rights reserved. 29 * Copyright (c) 2019, Klara Inc. 30 * Copyright (c) 2019, Allan Jude 31 * Copyright (c) 2020, The FreeBSD Foundation [1] 32 * 33 * [1] Portions of this software were developed by Allan Jude 34 * under sponsorship from the FreeBSD Foundation. 35 */ 36 37 /* 38 * DVA-based Adjustable Replacement Cache 39 * 40 * While much of the theory of operation used here is 41 * based on the self-tuning, low overhead replacement cache 42 * presented by Megiddo and Modha at FAST 2003, there are some 43 * significant differences: 44 * 45 * 1. The Megiddo and Modha model assumes any page is evictable. 46 * Pages in its cache cannot be "locked" into memory. This makes 47 * the eviction algorithm simple: evict the last page in the list. 48 * This also make the performance characteristics easy to reason 49 * about. Our cache is not so simple. At any given moment, some 50 * subset of the blocks in the cache are un-evictable because we 51 * have handed out a reference to them. Blocks are only evictable 52 * when there are no external references active. This makes 53 * eviction far more problematic: we choose to evict the evictable 54 * blocks that are the "lowest" in the list. 55 * 56 * There are times when it is not possible to evict the requested 57 * space. In these circumstances we are unable to adjust the cache 58 * size. To prevent the cache growing unbounded at these times we 59 * implement a "cache throttle" that slows the flow of new data 60 * into the cache until we can make space available. 61 * 62 * 2. The Megiddo and Modha model assumes a fixed cache size. 63 * Pages are evicted when the cache is full and there is a cache 64 * miss. Our model has a variable sized cache. It grows with 65 * high use, but also tries to react to memory pressure from the 66 * operating system: decreasing its size when system memory is 67 * tight. 68 * 69 * 3. The Megiddo and Modha model assumes a fixed page size. All 70 * elements of the cache are therefore exactly the same size. So 71 * when adjusting the cache size following a cache miss, its simply 72 * a matter of choosing a single page to evict. In our model, we 73 * have variable sized cache blocks (ranging from 512 bytes to 74 * 128K bytes). We therefore choose a set of blocks to evict to make 75 * space for a cache miss that approximates as closely as possible 76 * the space used by the new block. 77 * 78 * See also: "ARC: A Self-Tuning, Low Overhead Replacement Cache" 79 * by N. Megiddo & D. Modha, FAST 2003 80 */ 81 82 /* 83 * The locking model: 84 * 85 * A new reference to a cache buffer can be obtained in two 86 * ways: 1) via a hash table lookup using the DVA as a key, 87 * or 2) via one of the ARC lists. The arc_read() interface 88 * uses method 1, while the internal ARC algorithms for 89 * adjusting the cache use method 2. We therefore provide two 90 * types of locks: 1) the hash table lock array, and 2) the 91 * ARC list locks. 92 * 93 * Buffers do not have their own mutexes, rather they rely on the 94 * hash table mutexes for the bulk of their protection (i.e. most 95 * fields in the arc_buf_hdr_t are protected by these mutexes). 96 * 97 * buf_hash_find() returns the appropriate mutex (held) when it 98 * locates the requested buffer in the hash table. It returns 99 * NULL for the mutex if the buffer was not in the table. 100 * 101 * buf_hash_remove() expects the appropriate hash mutex to be 102 * already held before it is invoked. 103 * 104 * Each ARC state also has a mutex which is used to protect the 105 * buffer list associated with the state. When attempting to 106 * obtain a hash table lock while holding an ARC list lock you 107 * must use: mutex_tryenter() to avoid deadlock. Also note that 108 * the active state mutex must be held before the ghost state mutex. 109 * 110 * It as also possible to register a callback which is run when the 111 * arc_meta_limit is reached and no buffers can be safely evicted. In 112 * this case the arc user should drop a reference on some arc buffers so 113 * they can be reclaimed and the arc_meta_limit honored. For example, 114 * when using the ZPL each dentry holds a references on a znode. These 115 * dentries must be pruned before the arc buffer holding the znode can 116 * be safely evicted. 117 * 118 * Note that the majority of the performance stats are manipulated 119 * with atomic operations. 120 * 121 * The L2ARC uses the l2ad_mtx on each vdev for the following: 122 * 123 * - L2ARC buflist creation 124 * - L2ARC buflist eviction 125 * - L2ARC write completion, which walks L2ARC buflists 126 * - ARC header destruction, as it removes from L2ARC buflists 127 * - ARC header release, as it removes from L2ARC buflists 128 */ 129 130 /* 131 * ARC operation: 132 * 133 * Every block that is in the ARC is tracked by an arc_buf_hdr_t structure. 134 * This structure can point either to a block that is still in the cache or to 135 * one that is only accessible in an L2 ARC device, or it can provide 136 * information about a block that was recently evicted. If a block is 137 * only accessible in the L2ARC, then the arc_buf_hdr_t only has enough 138 * information to retrieve it from the L2ARC device. This information is 139 * stored in the l2arc_buf_hdr_t sub-structure of the arc_buf_hdr_t. A block 140 * that is in this state cannot access the data directly. 141 * 142 * Blocks that are actively being referenced or have not been evicted 143 * are cached in the L1ARC. The L1ARC (l1arc_buf_hdr_t) is a structure within 144 * the arc_buf_hdr_t that will point to the data block in memory. A block can 145 * only be read by a consumer if it has an l1arc_buf_hdr_t. The L1ARC 146 * caches data in two ways -- in a list of ARC buffers (arc_buf_t) and 147 * also in the arc_buf_hdr_t's private physical data block pointer (b_pabd). 148 * 149 * The L1ARC's data pointer may or may not be uncompressed. The ARC has the 150 * ability to store the physical data (b_pabd) associated with the DVA of the 151 * arc_buf_hdr_t. Since the b_pabd is a copy of the on-disk physical block, 152 * it will match its on-disk compression characteristics. This behavior can be 153 * disabled by setting 'zfs_compressed_arc_enabled' to B_FALSE. When the 154 * compressed ARC functionality is disabled, the b_pabd will point to an 155 * uncompressed version of the on-disk data. 156 * 157 * Data in the L1ARC is not accessed by consumers of the ARC directly. Each 158 * arc_buf_hdr_t can have multiple ARC buffers (arc_buf_t) which reference it. 159 * Each ARC buffer (arc_buf_t) is being actively accessed by a specific ARC 160 * consumer. The ARC will provide references to this data and will keep it 161 * cached until it is no longer in use. The ARC caches only the L1ARC's physical 162 * data block and will evict any arc_buf_t that is no longer referenced. The 163 * amount of memory consumed by the arc_buf_ts' data buffers can be seen via the 164 * "overhead_size" kstat. 165 * 166 * Depending on the consumer, an arc_buf_t can be requested in uncompressed or 167 * compressed form. The typical case is that consumers will want uncompressed 168 * data, and when that happens a new data buffer is allocated where the data is 169 * decompressed for them to use. Currently the only consumer who wants 170 * compressed arc_buf_t's is "zfs send", when it streams data exactly as it 171 * exists on disk. When this happens, the arc_buf_t's data buffer is shared 172 * with the arc_buf_hdr_t. 173 * 174 * Here is a diagram showing an arc_buf_hdr_t referenced by two arc_buf_t's. The 175 * first one is owned by a compressed send consumer (and therefore references 176 * the same compressed data buffer as the arc_buf_hdr_t) and the second could be 177 * used by any other consumer (and has its own uncompressed copy of the data 178 * buffer). 179 * 180 * arc_buf_hdr_t 181 * +-----------+ 182 * | fields | 183 * | common to | 184 * | L1- and | 185 * | L2ARC | 186 * +-----------+ 187 * | l2arc_buf_hdr_t 188 * | | 189 * +-----------+ 190 * | l1arc_buf_hdr_t 191 * | | arc_buf_t 192 * | b_buf +------------>+-----------+ arc_buf_t 193 * | b_pabd +-+ |b_next +---->+-----------+ 194 * +-----------+ | |-----------| |b_next +-->NULL 195 * | |b_comp = T | +-----------+ 196 * | |b_data +-+ |b_comp = F | 197 * | +-----------+ | |b_data +-+ 198 * +->+------+ | +-----------+ | 199 * compressed | | | | 200 * data | |<--------------+ | uncompressed 201 * +------+ compressed, | data 202 * shared +-->+------+ 203 * data | | 204 * | | 205 * +------+ 206 * 207 * When a consumer reads a block, the ARC must first look to see if the 208 * arc_buf_hdr_t is cached. If the hdr is cached then the ARC allocates a new 209 * arc_buf_t and either copies uncompressed data into a new data buffer from an 210 * existing uncompressed arc_buf_t, decompresses the hdr's b_pabd buffer into a 211 * new data buffer, or shares the hdr's b_pabd buffer, depending on whether the 212 * hdr is compressed and the desired compression characteristics of the 213 * arc_buf_t consumer. If the arc_buf_t ends up sharing data with the 214 * arc_buf_hdr_t and both of them are uncompressed then the arc_buf_t must be 215 * the last buffer in the hdr's b_buf list, however a shared compressed buf can 216 * be anywhere in the hdr's list. 217 * 218 * The diagram below shows an example of an uncompressed ARC hdr that is 219 * sharing its data with an arc_buf_t (note that the shared uncompressed buf is 220 * the last element in the buf list): 221 * 222 * arc_buf_hdr_t 223 * +-----------+ 224 * | | 225 * | | 226 * | | 227 * +-----------+ 228 * l2arc_buf_hdr_t| | 229 * | | 230 * +-----------+ 231 * l1arc_buf_hdr_t| | 232 * | | arc_buf_t (shared) 233 * | b_buf +------------>+---------+ arc_buf_t 234 * | | |b_next +---->+---------+ 235 * | b_pabd +-+ |---------| |b_next +-->NULL 236 * +-----------+ | | | +---------+ 237 * | |b_data +-+ | | 238 * | +---------+ | |b_data +-+ 239 * +->+------+ | +---------+ | 240 * | | | | 241 * uncompressed | | | | 242 * data +------+ | | 243 * ^ +->+------+ | 244 * | uncompressed | | | 245 * | data | | | 246 * | +------+ | 247 * +---------------------------------+ 248 * 249 * Writing to the ARC requires that the ARC first discard the hdr's b_pabd 250 * since the physical block is about to be rewritten. The new data contents 251 * will be contained in the arc_buf_t. As the I/O pipeline performs the write, 252 * it may compress the data before writing it to disk. The ARC will be called 253 * with the transformed data and will memcpy the transformed on-disk block into 254 * a newly allocated b_pabd. Writes are always done into buffers which have 255 * either been loaned (and hence are new and don't have other readers) or 256 * buffers which have been released (and hence have their own hdr, if there 257 * were originally other readers of the buf's original hdr). This ensures that 258 * the ARC only needs to update a single buf and its hdr after a write occurs. 259 * 260 * When the L2ARC is in use, it will also take advantage of the b_pabd. The 261 * L2ARC will always write the contents of b_pabd to the L2ARC. This means 262 * that when compressed ARC is enabled that the L2ARC blocks are identical 263 * to the on-disk block in the main data pool. This provides a significant 264 * advantage since the ARC can leverage the bp's checksum when reading from the 265 * L2ARC to determine if the contents are valid. However, if the compressed 266 * ARC is disabled, then the L2ARC's block must be transformed to look 267 * like the physical block in the main data pool before comparing the 268 * checksum and determining its validity. 269 * 270 * The L1ARC has a slightly different system for storing encrypted data. 271 * Raw (encrypted + possibly compressed) data has a few subtle differences from 272 * data that is just compressed. The biggest difference is that it is not 273 * possible to decrypt encrypted data (or vice-versa) if the keys aren't loaded. 274 * The other difference is that encryption cannot be treated as a suggestion. 275 * If a caller would prefer compressed data, but they actually wind up with 276 * uncompressed data the worst thing that could happen is there might be a 277 * performance hit. If the caller requests encrypted data, however, we must be 278 * sure they actually get it or else secret information could be leaked. Raw 279 * data is stored in hdr->b_crypt_hdr.b_rabd. An encrypted header, therefore, 280 * may have both an encrypted version and a decrypted version of its data at 281 * once. When a caller needs a raw arc_buf_t, it is allocated and the data is 282 * copied out of this header. To avoid complications with b_pabd, raw buffers 283 * cannot be shared. 284 */ 285 286 #include <sys/spa.h> 287 #include <sys/zio.h> 288 #include <sys/spa_impl.h> 289 #include <sys/zio_compress.h> 290 #include <sys/zio_checksum.h> 291 #include <sys/zfs_context.h> 292 #include <sys/arc.h> 293 #include <sys/zfs_refcount.h> 294 #include <sys/vdev.h> 295 #include <sys/vdev_impl.h> 296 #include <sys/dsl_pool.h> 297 #include <sys/multilist.h> 298 #include <sys/abd.h> 299 #include <sys/zil.h> 300 #include <sys/fm/fs/zfs.h> 301 #include <sys/callb.h> 302 #include <sys/kstat.h> 303 #include <sys/zthr.h> 304 #include <zfs_fletcher.h> 305 #include <sys/arc_impl.h> 306 #include <sys/trace_zfs.h> 307 #include <sys/aggsum.h> 308 #include <sys/wmsum.h> 309 #include <cityhash.h> 310 #include <sys/vdev_trim.h> 311 #include <sys/zfs_racct.h> 312 #include <sys/zstd/zstd.h> 313 314 #ifndef _KERNEL 315 /* set with ZFS_DEBUG=watch, to enable watchpoints on frozen buffers */ 316 boolean_t arc_watch = B_FALSE; 317 #endif 318 319 /* 320 * This thread's job is to keep enough free memory in the system, by 321 * calling arc_kmem_reap_soon() plus arc_reduce_target_size(), which improves 322 * arc_available_memory(). 323 */ 324 static zthr_t *arc_reap_zthr; 325 326 /* 327 * This thread's job is to keep arc_size under arc_c, by calling 328 * arc_evict(), which improves arc_is_overflowing(). 329 */ 330 static zthr_t *arc_evict_zthr; 331 static arc_buf_hdr_t **arc_state_evict_markers; 332 static int arc_state_evict_marker_count; 333 334 static kmutex_t arc_evict_lock; 335 static boolean_t arc_evict_needed = B_FALSE; 336 static clock_t arc_last_uncached_flush; 337 338 /* 339 * Count of bytes evicted since boot. 340 */ 341 static uint64_t arc_evict_count; 342 343 /* 344 * List of arc_evict_waiter_t's, representing threads waiting for the 345 * arc_evict_count to reach specific values. 346 */ 347 static list_t arc_evict_waiters; 348 349 /* 350 * When arc_is_overflowing(), arc_get_data_impl() waits for this percent of 351 * the requested amount of data to be evicted. For example, by default for 352 * every 2KB that's evicted, 1KB of it may be "reused" by a new allocation. 353 * Since this is above 100%, it ensures that progress is made towards getting 354 * arc_size under arc_c. Since this is finite, it ensures that allocations 355 * can still happen, even during the potentially long time that arc_size is 356 * more than arc_c. 357 */ 358 static uint_t zfs_arc_eviction_pct = 200; 359 360 /* 361 * The number of headers to evict in arc_evict_state_impl() before 362 * dropping the sublist lock and evicting from another sublist. A lower 363 * value means we're more likely to evict the "correct" header (i.e. the 364 * oldest header in the arc state), but comes with higher overhead 365 * (i.e. more invocations of arc_evict_state_impl()). 366 */ 367 static uint_t zfs_arc_evict_batch_limit = 10; 368 369 /* number of seconds before growing cache again */ 370 uint_t arc_grow_retry = 5; 371 372 /* 373 * Minimum time between calls to arc_kmem_reap_soon(). 374 */ 375 static const int arc_kmem_cache_reap_retry_ms = 1000; 376 377 /* shift of arc_c for calculating overflow limit in arc_get_data_impl */ 378 static int zfs_arc_overflow_shift = 8; 379 380 /* shift of arc_c for calculating both min and max arc_p */ 381 static uint_t arc_p_min_shift = 4; 382 383 /* log2(fraction of arc to reclaim) */ 384 uint_t arc_shrink_shift = 7; 385 386 /* percent of pagecache to reclaim arc to */ 387 #ifdef _KERNEL 388 uint_t zfs_arc_pc_percent = 0; 389 #endif 390 391 /* 392 * log2(fraction of ARC which must be free to allow growing). 393 * I.e. If there is less than arc_c >> arc_no_grow_shift free memory, 394 * when reading a new block into the ARC, we will evict an equal-sized block 395 * from the ARC. 396 * 397 * This must be less than arc_shrink_shift, so that when we shrink the ARC, 398 * we will still not allow it to grow. 399 */ 400 uint_t arc_no_grow_shift = 5; 401 402 403 /* 404 * minimum lifespan of a prefetch block in clock ticks 405 * (initialized in arc_init()) 406 */ 407 static uint_t arc_min_prefetch_ms; 408 static uint_t arc_min_prescient_prefetch_ms; 409 410 /* 411 * If this percent of memory is free, don't throttle. 412 */ 413 uint_t arc_lotsfree_percent = 10; 414 415 /* 416 * The arc has filled available memory and has now warmed up. 417 */ 418 boolean_t arc_warm; 419 420 /* 421 * These tunables are for performance analysis. 422 */ 423 uint64_t zfs_arc_max = 0; 424 uint64_t zfs_arc_min = 0; 425 uint64_t zfs_arc_meta_limit = 0; 426 uint64_t zfs_arc_meta_min = 0; 427 static uint64_t zfs_arc_dnode_limit = 0; 428 static uint_t zfs_arc_dnode_reduce_percent = 10; 429 static uint_t zfs_arc_grow_retry = 0; 430 static uint_t zfs_arc_shrink_shift = 0; 431 static uint_t zfs_arc_p_min_shift = 0; 432 uint_t zfs_arc_average_blocksize = 8 * 1024; /* 8KB */ 433 434 /* 435 * ARC dirty data constraints for arc_tempreserve_space() throttle: 436 * * total dirty data limit 437 * * anon block dirty limit 438 * * each pool's anon allowance 439 */ 440 static const unsigned long zfs_arc_dirty_limit_percent = 50; 441 static const unsigned long zfs_arc_anon_limit_percent = 25; 442 static const unsigned long zfs_arc_pool_dirty_percent = 20; 443 444 /* 445 * Enable or disable compressed arc buffers. 446 */ 447 int zfs_compressed_arc_enabled = B_TRUE; 448 449 /* 450 * ARC will evict meta buffers that exceed arc_meta_limit. This 451 * tunable make arc_meta_limit adjustable for different workloads. 452 */ 453 static uint64_t zfs_arc_meta_limit_percent = 75; 454 455 /* 456 * Percentage that can be consumed by dnodes of ARC meta buffers. 457 */ 458 static uint_t zfs_arc_dnode_limit_percent = 10; 459 460 /* 461 * These tunables are Linux-specific 462 */ 463 static uint64_t zfs_arc_sys_free = 0; 464 static uint_t zfs_arc_min_prefetch_ms = 0; 465 static uint_t zfs_arc_min_prescient_prefetch_ms = 0; 466 static int zfs_arc_p_dampener_disable = 1; 467 static uint_t zfs_arc_meta_prune = 10000; 468 static uint_t zfs_arc_meta_strategy = ARC_STRATEGY_META_BALANCED; 469 static uint_t zfs_arc_meta_adjust_restarts = 4096; 470 static uint_t zfs_arc_lotsfree_percent = 10; 471 472 /* 473 * Number of arc_prune threads 474 */ 475 static int zfs_arc_prune_task_threads = 1; 476 477 /* The 7 states: */ 478 arc_state_t ARC_anon; 479 arc_state_t ARC_mru; 480 arc_state_t ARC_mru_ghost; 481 arc_state_t ARC_mfu; 482 arc_state_t ARC_mfu_ghost; 483 arc_state_t ARC_l2c_only; 484 arc_state_t ARC_uncached; 485 486 arc_stats_t arc_stats = { 487 { "hits", KSTAT_DATA_UINT64 }, 488 { "iohits", KSTAT_DATA_UINT64 }, 489 { "misses", KSTAT_DATA_UINT64 }, 490 { "demand_data_hits", KSTAT_DATA_UINT64 }, 491 { "demand_data_iohits", KSTAT_DATA_UINT64 }, 492 { "demand_data_misses", KSTAT_DATA_UINT64 }, 493 { "demand_metadata_hits", KSTAT_DATA_UINT64 }, 494 { "demand_metadata_iohits", KSTAT_DATA_UINT64 }, 495 { "demand_metadata_misses", KSTAT_DATA_UINT64 }, 496 { "prefetch_data_hits", KSTAT_DATA_UINT64 }, 497 { "prefetch_data_iohits", KSTAT_DATA_UINT64 }, 498 { "prefetch_data_misses", KSTAT_DATA_UINT64 }, 499 { "prefetch_metadata_hits", KSTAT_DATA_UINT64 }, 500 { "prefetch_metadata_iohits", KSTAT_DATA_UINT64 }, 501 { "prefetch_metadata_misses", KSTAT_DATA_UINT64 }, 502 { "mru_hits", KSTAT_DATA_UINT64 }, 503 { "mru_ghost_hits", KSTAT_DATA_UINT64 }, 504 { "mfu_hits", KSTAT_DATA_UINT64 }, 505 { "mfu_ghost_hits", KSTAT_DATA_UINT64 }, 506 { "uncached_hits", KSTAT_DATA_UINT64 }, 507 { "deleted", KSTAT_DATA_UINT64 }, 508 { "mutex_miss", KSTAT_DATA_UINT64 }, 509 { "access_skip", KSTAT_DATA_UINT64 }, 510 { "evict_skip", KSTAT_DATA_UINT64 }, 511 { "evict_not_enough", KSTAT_DATA_UINT64 }, 512 { "evict_l2_cached", KSTAT_DATA_UINT64 }, 513 { "evict_l2_eligible", KSTAT_DATA_UINT64 }, 514 { "evict_l2_eligible_mfu", KSTAT_DATA_UINT64 }, 515 { "evict_l2_eligible_mru", KSTAT_DATA_UINT64 }, 516 { "evict_l2_ineligible", KSTAT_DATA_UINT64 }, 517 { "evict_l2_skip", KSTAT_DATA_UINT64 }, 518 { "hash_elements", KSTAT_DATA_UINT64 }, 519 { "hash_elements_max", KSTAT_DATA_UINT64 }, 520 { "hash_collisions", KSTAT_DATA_UINT64 }, 521 { "hash_chains", KSTAT_DATA_UINT64 }, 522 { "hash_chain_max", KSTAT_DATA_UINT64 }, 523 { "p", KSTAT_DATA_UINT64 }, 524 { "c", KSTAT_DATA_UINT64 }, 525 { "c_min", KSTAT_DATA_UINT64 }, 526 { "c_max", KSTAT_DATA_UINT64 }, 527 { "size", KSTAT_DATA_UINT64 }, 528 { "compressed_size", KSTAT_DATA_UINT64 }, 529 { "uncompressed_size", KSTAT_DATA_UINT64 }, 530 { "overhead_size", KSTAT_DATA_UINT64 }, 531 { "hdr_size", KSTAT_DATA_UINT64 }, 532 { "data_size", KSTAT_DATA_UINT64 }, 533 { "metadata_size", KSTAT_DATA_UINT64 }, 534 { "dbuf_size", KSTAT_DATA_UINT64 }, 535 { "dnode_size", KSTAT_DATA_UINT64 }, 536 { "bonus_size", KSTAT_DATA_UINT64 }, 537 #if defined(COMPAT_FREEBSD11) 538 { "other_size", KSTAT_DATA_UINT64 }, 539 #endif 540 { "anon_size", KSTAT_DATA_UINT64 }, 541 { "anon_evictable_data", KSTAT_DATA_UINT64 }, 542 { "anon_evictable_metadata", KSTAT_DATA_UINT64 }, 543 { "mru_size", KSTAT_DATA_UINT64 }, 544 { "mru_evictable_data", KSTAT_DATA_UINT64 }, 545 { "mru_evictable_metadata", KSTAT_DATA_UINT64 }, 546 { "mru_ghost_size", KSTAT_DATA_UINT64 }, 547 { "mru_ghost_evictable_data", KSTAT_DATA_UINT64 }, 548 { "mru_ghost_evictable_metadata", KSTAT_DATA_UINT64 }, 549 { "mfu_size", KSTAT_DATA_UINT64 }, 550 { "mfu_evictable_data", KSTAT_DATA_UINT64 }, 551 { "mfu_evictable_metadata", KSTAT_DATA_UINT64 }, 552 { "mfu_ghost_size", KSTAT_DATA_UINT64 }, 553 { "mfu_ghost_evictable_data", KSTAT_DATA_UINT64 }, 554 { "mfu_ghost_evictable_metadata", KSTAT_DATA_UINT64 }, 555 { "uncached_size", KSTAT_DATA_UINT64 }, 556 { "uncached_evictable_data", KSTAT_DATA_UINT64 }, 557 { "uncached_evictable_metadata", KSTAT_DATA_UINT64 }, 558 { "l2_hits", KSTAT_DATA_UINT64 }, 559 { "l2_misses", KSTAT_DATA_UINT64 }, 560 { "l2_prefetch_asize", KSTAT_DATA_UINT64 }, 561 { "l2_mru_asize", KSTAT_DATA_UINT64 }, 562 { "l2_mfu_asize", KSTAT_DATA_UINT64 }, 563 { "l2_bufc_data_asize", KSTAT_DATA_UINT64 }, 564 { "l2_bufc_metadata_asize", KSTAT_DATA_UINT64 }, 565 { "l2_feeds", KSTAT_DATA_UINT64 }, 566 { "l2_rw_clash", KSTAT_DATA_UINT64 }, 567 { "l2_read_bytes", KSTAT_DATA_UINT64 }, 568 { "l2_write_bytes", KSTAT_DATA_UINT64 }, 569 { "l2_writes_sent", KSTAT_DATA_UINT64 }, 570 { "l2_writes_done", KSTAT_DATA_UINT64 }, 571 { "l2_writes_error", KSTAT_DATA_UINT64 }, 572 { "l2_writes_lock_retry", KSTAT_DATA_UINT64 }, 573 { "l2_evict_lock_retry", KSTAT_DATA_UINT64 }, 574 { "l2_evict_reading", KSTAT_DATA_UINT64 }, 575 { "l2_evict_l1cached", KSTAT_DATA_UINT64 }, 576 { "l2_free_on_write", KSTAT_DATA_UINT64 }, 577 { "l2_abort_lowmem", KSTAT_DATA_UINT64 }, 578 { "l2_cksum_bad", KSTAT_DATA_UINT64 }, 579 { "l2_io_error", KSTAT_DATA_UINT64 }, 580 { "l2_size", KSTAT_DATA_UINT64 }, 581 { "l2_asize", KSTAT_DATA_UINT64 }, 582 { "l2_hdr_size", KSTAT_DATA_UINT64 }, 583 { "l2_log_blk_writes", KSTAT_DATA_UINT64 }, 584 { "l2_log_blk_avg_asize", KSTAT_DATA_UINT64 }, 585 { "l2_log_blk_asize", KSTAT_DATA_UINT64 }, 586 { "l2_log_blk_count", KSTAT_DATA_UINT64 }, 587 { "l2_data_to_meta_ratio", KSTAT_DATA_UINT64 }, 588 { "l2_rebuild_success", KSTAT_DATA_UINT64 }, 589 { "l2_rebuild_unsupported", KSTAT_DATA_UINT64 }, 590 { "l2_rebuild_io_errors", KSTAT_DATA_UINT64 }, 591 { "l2_rebuild_dh_errors", KSTAT_DATA_UINT64 }, 592 { "l2_rebuild_cksum_lb_errors", KSTAT_DATA_UINT64 }, 593 { "l2_rebuild_lowmem", KSTAT_DATA_UINT64 }, 594 { "l2_rebuild_size", KSTAT_DATA_UINT64 }, 595 { "l2_rebuild_asize", KSTAT_DATA_UINT64 }, 596 { "l2_rebuild_bufs", KSTAT_DATA_UINT64 }, 597 { "l2_rebuild_bufs_precached", KSTAT_DATA_UINT64 }, 598 { "l2_rebuild_log_blks", KSTAT_DATA_UINT64 }, 599 { "memory_throttle_count", KSTAT_DATA_UINT64 }, 600 { "memory_direct_count", KSTAT_DATA_UINT64 }, 601 { "memory_indirect_count", KSTAT_DATA_UINT64 }, 602 { "memory_all_bytes", KSTAT_DATA_UINT64 }, 603 { "memory_free_bytes", KSTAT_DATA_UINT64 }, 604 { "memory_available_bytes", KSTAT_DATA_INT64 }, 605 { "arc_no_grow", KSTAT_DATA_UINT64 }, 606 { "arc_tempreserve", KSTAT_DATA_UINT64 }, 607 { "arc_loaned_bytes", KSTAT_DATA_UINT64 }, 608 { "arc_prune", KSTAT_DATA_UINT64 }, 609 { "arc_meta_used", KSTAT_DATA_UINT64 }, 610 { "arc_meta_limit", KSTAT_DATA_UINT64 }, 611 { "arc_dnode_limit", KSTAT_DATA_UINT64 }, 612 { "arc_meta_max", KSTAT_DATA_UINT64 }, 613 { "arc_meta_min", KSTAT_DATA_UINT64 }, 614 { "async_upgrade_sync", KSTAT_DATA_UINT64 }, 615 { "predictive_prefetch", KSTAT_DATA_UINT64 }, 616 { "demand_hit_predictive_prefetch", KSTAT_DATA_UINT64 }, 617 { "demand_iohit_predictive_prefetch", KSTAT_DATA_UINT64 }, 618 { "prescient_prefetch", KSTAT_DATA_UINT64 }, 619 { "demand_hit_prescient_prefetch", KSTAT_DATA_UINT64 }, 620 { "demand_iohit_prescient_prefetch", KSTAT_DATA_UINT64 }, 621 { "arc_need_free", KSTAT_DATA_UINT64 }, 622 { "arc_sys_free", KSTAT_DATA_UINT64 }, 623 { "arc_raw_size", KSTAT_DATA_UINT64 }, 624 { "cached_only_in_progress", KSTAT_DATA_UINT64 }, 625 { "abd_chunk_waste_size", KSTAT_DATA_UINT64 }, 626 }; 627 628 arc_sums_t arc_sums; 629 630 #define ARCSTAT_MAX(stat, val) { \ 631 uint64_t m; \ 632 while ((val) > (m = arc_stats.stat.value.ui64) && \ 633 (m != atomic_cas_64(&arc_stats.stat.value.ui64, m, (val)))) \ 634 continue; \ 635 } 636 637 /* 638 * We define a macro to allow ARC hits/misses to be easily broken down by 639 * two separate conditions, giving a total of four different subtypes for 640 * each of hits and misses (so eight statistics total). 641 */ 642 #define ARCSTAT_CONDSTAT(cond1, stat1, notstat1, cond2, stat2, notstat2, stat) \ 643 if (cond1) { \ 644 if (cond2) { \ 645 ARCSTAT_BUMP(arcstat_##stat1##_##stat2##_##stat); \ 646 } else { \ 647 ARCSTAT_BUMP(arcstat_##stat1##_##notstat2##_##stat); \ 648 } \ 649 } else { \ 650 if (cond2) { \ 651 ARCSTAT_BUMP(arcstat_##notstat1##_##stat2##_##stat); \ 652 } else { \ 653 ARCSTAT_BUMP(arcstat_##notstat1##_##notstat2##_##stat);\ 654 } \ 655 } 656 657 /* 658 * This macro allows us to use kstats as floating averages. Each time we 659 * update this kstat, we first factor it and the update value by 660 * ARCSTAT_AVG_FACTOR to shrink the new value's contribution to the overall 661 * average. This macro assumes that integer loads and stores are atomic, but 662 * is not safe for multiple writers updating the kstat in parallel (only the 663 * last writer's update will remain). 664 */ 665 #define ARCSTAT_F_AVG_FACTOR 3 666 #define ARCSTAT_F_AVG(stat, value) \ 667 do { \ 668 uint64_t x = ARCSTAT(stat); \ 669 x = x - x / ARCSTAT_F_AVG_FACTOR + \ 670 (value) / ARCSTAT_F_AVG_FACTOR; \ 671 ARCSTAT(stat) = x; \ 672 } while (0) 673 674 static kstat_t *arc_ksp; 675 676 /* 677 * There are several ARC variables that are critical to export as kstats -- 678 * but we don't want to have to grovel around in the kstat whenever we wish to 679 * manipulate them. For these variables, we therefore define them to be in 680 * terms of the statistic variable. This assures that we are not introducing 681 * the possibility of inconsistency by having shadow copies of the variables, 682 * while still allowing the code to be readable. 683 */ 684 #define arc_tempreserve ARCSTAT(arcstat_tempreserve) 685 #define arc_loaned_bytes ARCSTAT(arcstat_loaned_bytes) 686 #define arc_meta_limit ARCSTAT(arcstat_meta_limit) /* max size for metadata */ 687 /* max size for dnodes */ 688 #define arc_dnode_size_limit ARCSTAT(arcstat_dnode_limit) 689 #define arc_meta_min ARCSTAT(arcstat_meta_min) /* min size for metadata */ 690 #define arc_need_free ARCSTAT(arcstat_need_free) /* waiting to be evicted */ 691 692 hrtime_t arc_growtime; 693 list_t arc_prune_list; 694 kmutex_t arc_prune_mtx; 695 taskq_t *arc_prune_taskq; 696 697 #define GHOST_STATE(state) \ 698 ((state) == arc_mru_ghost || (state) == arc_mfu_ghost || \ 699 (state) == arc_l2c_only) 700 701 #define HDR_IN_HASH_TABLE(hdr) ((hdr)->b_flags & ARC_FLAG_IN_HASH_TABLE) 702 #define HDR_IO_IN_PROGRESS(hdr) ((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS) 703 #define HDR_IO_ERROR(hdr) ((hdr)->b_flags & ARC_FLAG_IO_ERROR) 704 #define HDR_PREFETCH(hdr) ((hdr)->b_flags & ARC_FLAG_PREFETCH) 705 #define HDR_PRESCIENT_PREFETCH(hdr) \ 706 ((hdr)->b_flags & ARC_FLAG_PRESCIENT_PREFETCH) 707 #define HDR_COMPRESSION_ENABLED(hdr) \ 708 ((hdr)->b_flags & ARC_FLAG_COMPRESSED_ARC) 709 710 #define HDR_L2CACHE(hdr) ((hdr)->b_flags & ARC_FLAG_L2CACHE) 711 #define HDR_UNCACHED(hdr) ((hdr)->b_flags & ARC_FLAG_UNCACHED) 712 #define HDR_L2_READING(hdr) \ 713 (((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS) && \ 714 ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR)) 715 #define HDR_L2_WRITING(hdr) ((hdr)->b_flags & ARC_FLAG_L2_WRITING) 716 #define HDR_L2_EVICTED(hdr) ((hdr)->b_flags & ARC_FLAG_L2_EVICTED) 717 #define HDR_L2_WRITE_HEAD(hdr) ((hdr)->b_flags & ARC_FLAG_L2_WRITE_HEAD) 718 #define HDR_PROTECTED(hdr) ((hdr)->b_flags & ARC_FLAG_PROTECTED) 719 #define HDR_NOAUTH(hdr) ((hdr)->b_flags & ARC_FLAG_NOAUTH) 720 #define HDR_SHARED_DATA(hdr) ((hdr)->b_flags & ARC_FLAG_SHARED_DATA) 721 722 #define HDR_ISTYPE_METADATA(hdr) \ 723 ((hdr)->b_flags & ARC_FLAG_BUFC_METADATA) 724 #define HDR_ISTYPE_DATA(hdr) (!HDR_ISTYPE_METADATA(hdr)) 725 726 #define HDR_HAS_L1HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L1HDR) 727 #define HDR_HAS_L2HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR) 728 #define HDR_HAS_RABD(hdr) \ 729 (HDR_HAS_L1HDR(hdr) && HDR_PROTECTED(hdr) && \ 730 (hdr)->b_crypt_hdr.b_rabd != NULL) 731 #define HDR_ENCRYPTED(hdr) \ 732 (HDR_PROTECTED(hdr) && DMU_OT_IS_ENCRYPTED((hdr)->b_crypt_hdr.b_ot)) 733 #define HDR_AUTHENTICATED(hdr) \ 734 (HDR_PROTECTED(hdr) && !DMU_OT_IS_ENCRYPTED((hdr)->b_crypt_hdr.b_ot)) 735 736 /* For storing compression mode in b_flags */ 737 #define HDR_COMPRESS_OFFSET (highbit64(ARC_FLAG_COMPRESS_0) - 1) 738 739 #define HDR_GET_COMPRESS(hdr) ((enum zio_compress)BF32_GET((hdr)->b_flags, \ 740 HDR_COMPRESS_OFFSET, SPA_COMPRESSBITS)) 741 #define HDR_SET_COMPRESS(hdr, cmp) BF32_SET((hdr)->b_flags, \ 742 HDR_COMPRESS_OFFSET, SPA_COMPRESSBITS, (cmp)); 743 744 #define ARC_BUF_LAST(buf) ((buf)->b_next == NULL) 745 #define ARC_BUF_SHARED(buf) ((buf)->b_flags & ARC_BUF_FLAG_SHARED) 746 #define ARC_BUF_COMPRESSED(buf) ((buf)->b_flags & ARC_BUF_FLAG_COMPRESSED) 747 #define ARC_BUF_ENCRYPTED(buf) ((buf)->b_flags & ARC_BUF_FLAG_ENCRYPTED) 748 749 /* 750 * Other sizes 751 */ 752 753 #define HDR_FULL_CRYPT_SIZE ((int64_t)sizeof (arc_buf_hdr_t)) 754 #define HDR_FULL_SIZE ((int64_t)offsetof(arc_buf_hdr_t, b_crypt_hdr)) 755 #define HDR_L2ONLY_SIZE ((int64_t)offsetof(arc_buf_hdr_t, b_l1hdr)) 756 757 /* 758 * Hash table routines 759 */ 760 761 #define BUF_LOCKS 2048 762 typedef struct buf_hash_table { 763 uint64_t ht_mask; 764 arc_buf_hdr_t **ht_table; 765 kmutex_t ht_locks[BUF_LOCKS] ____cacheline_aligned; 766 } buf_hash_table_t; 767 768 static buf_hash_table_t buf_hash_table; 769 770 #define BUF_HASH_INDEX(spa, dva, birth) \ 771 (buf_hash(spa, dva, birth) & buf_hash_table.ht_mask) 772 #define BUF_HASH_LOCK(idx) (&buf_hash_table.ht_locks[idx & (BUF_LOCKS-1)]) 773 #define HDR_LOCK(hdr) \ 774 (BUF_HASH_LOCK(BUF_HASH_INDEX(hdr->b_spa, &hdr->b_dva, hdr->b_birth))) 775 776 uint64_t zfs_crc64_table[256]; 777 778 /* 779 * Level 2 ARC 780 */ 781 782 #define L2ARC_WRITE_SIZE (8 * 1024 * 1024) /* initial write max */ 783 #define L2ARC_HEADROOM 2 /* num of writes */ 784 785 /* 786 * If we discover during ARC scan any buffers to be compressed, we boost 787 * our headroom for the next scanning cycle by this percentage multiple. 788 */ 789 #define L2ARC_HEADROOM_BOOST 200 790 #define L2ARC_FEED_SECS 1 /* caching interval secs */ 791 #define L2ARC_FEED_MIN_MS 200 /* min caching interval ms */ 792 793 /* 794 * We can feed L2ARC from two states of ARC buffers, mru and mfu, 795 * and each of the state has two types: data and metadata. 796 */ 797 #define L2ARC_FEED_TYPES 4 798 799 /* L2ARC Performance Tunables */ 800 uint64_t l2arc_write_max = L2ARC_WRITE_SIZE; /* def max write size */ 801 uint64_t l2arc_write_boost = L2ARC_WRITE_SIZE; /* extra warmup write */ 802 uint64_t l2arc_headroom = L2ARC_HEADROOM; /* # of dev writes */ 803 uint64_t l2arc_headroom_boost = L2ARC_HEADROOM_BOOST; 804 uint64_t l2arc_feed_secs = L2ARC_FEED_SECS; /* interval seconds */ 805 uint64_t l2arc_feed_min_ms = L2ARC_FEED_MIN_MS; /* min interval msecs */ 806 int l2arc_noprefetch = B_TRUE; /* don't cache prefetch bufs */ 807 int l2arc_feed_again = B_TRUE; /* turbo warmup */ 808 int l2arc_norw = B_FALSE; /* no reads during writes */ 809 static uint_t l2arc_meta_percent = 33; /* limit on headers size */ 810 811 /* 812 * L2ARC Internals 813 */ 814 static list_t L2ARC_dev_list; /* device list */ 815 static list_t *l2arc_dev_list; /* device list pointer */ 816 static kmutex_t l2arc_dev_mtx; /* device list mutex */ 817 static l2arc_dev_t *l2arc_dev_last; /* last device used */ 818 static list_t L2ARC_free_on_write; /* free after write buf list */ 819 static list_t *l2arc_free_on_write; /* free after write list ptr */ 820 static kmutex_t l2arc_free_on_write_mtx; /* mutex for list */ 821 static uint64_t l2arc_ndev; /* number of devices */ 822 823 typedef struct l2arc_read_callback { 824 arc_buf_hdr_t *l2rcb_hdr; /* read header */ 825 blkptr_t l2rcb_bp; /* original blkptr */ 826 zbookmark_phys_t l2rcb_zb; /* original bookmark */ 827 int l2rcb_flags; /* original flags */ 828 abd_t *l2rcb_abd; /* temporary buffer */ 829 } l2arc_read_callback_t; 830 831 typedef struct l2arc_data_free { 832 /* protected by l2arc_free_on_write_mtx */ 833 abd_t *l2df_abd; 834 size_t l2df_size; 835 arc_buf_contents_t l2df_type; 836 list_node_t l2df_list_node; 837 } l2arc_data_free_t; 838 839 typedef enum arc_fill_flags { 840 ARC_FILL_LOCKED = 1 << 0, /* hdr lock is held */ 841 ARC_FILL_COMPRESSED = 1 << 1, /* fill with compressed data */ 842 ARC_FILL_ENCRYPTED = 1 << 2, /* fill with encrypted data */ 843 ARC_FILL_NOAUTH = 1 << 3, /* don't attempt to authenticate */ 844 ARC_FILL_IN_PLACE = 1 << 4 /* fill in place (special case) */ 845 } arc_fill_flags_t; 846 847 typedef enum arc_ovf_level { 848 ARC_OVF_NONE, /* ARC within target size. */ 849 ARC_OVF_SOME, /* ARC is slightly overflowed. */ 850 ARC_OVF_SEVERE /* ARC is severely overflowed. */ 851 } arc_ovf_level_t; 852 853 static kmutex_t l2arc_feed_thr_lock; 854 static kcondvar_t l2arc_feed_thr_cv; 855 static uint8_t l2arc_thread_exit; 856 857 static kmutex_t l2arc_rebuild_thr_lock; 858 static kcondvar_t l2arc_rebuild_thr_cv; 859 860 enum arc_hdr_alloc_flags { 861 ARC_HDR_ALLOC_RDATA = 0x1, 862 ARC_HDR_DO_ADAPT = 0x2, 863 ARC_HDR_USE_RESERVE = 0x4, 864 ARC_HDR_ALLOC_LINEAR = 0x8, 865 }; 866 867 868 static abd_t *arc_get_data_abd(arc_buf_hdr_t *, uint64_t, const void *, int); 869 static void *arc_get_data_buf(arc_buf_hdr_t *, uint64_t, const void *); 870 static void arc_get_data_impl(arc_buf_hdr_t *, uint64_t, const void *, int); 871 static void arc_free_data_abd(arc_buf_hdr_t *, abd_t *, uint64_t, const void *); 872 static void arc_free_data_buf(arc_buf_hdr_t *, void *, uint64_t, const void *); 873 static void arc_free_data_impl(arc_buf_hdr_t *hdr, uint64_t size, 874 const void *tag); 875 static void arc_hdr_free_abd(arc_buf_hdr_t *, boolean_t); 876 static void arc_hdr_alloc_abd(arc_buf_hdr_t *, int); 877 static void arc_hdr_destroy(arc_buf_hdr_t *); 878 static void arc_access(arc_buf_hdr_t *, arc_flags_t, boolean_t); 879 static void arc_buf_watch(arc_buf_t *); 880 static void arc_change_state(arc_state_t *, arc_buf_hdr_t *); 881 882 static arc_buf_contents_t arc_buf_type(arc_buf_hdr_t *); 883 static uint32_t arc_bufc_to_flags(arc_buf_contents_t); 884 static inline void arc_hdr_set_flags(arc_buf_hdr_t *hdr, arc_flags_t flags); 885 static inline void arc_hdr_clear_flags(arc_buf_hdr_t *hdr, arc_flags_t flags); 886 887 static boolean_t l2arc_write_eligible(uint64_t, arc_buf_hdr_t *); 888 static void l2arc_read_done(zio_t *); 889 static void l2arc_do_free_on_write(void); 890 static void l2arc_hdr_arcstats_update(arc_buf_hdr_t *hdr, boolean_t incr, 891 boolean_t state_only); 892 893 #define l2arc_hdr_arcstats_increment(hdr) \ 894 l2arc_hdr_arcstats_update((hdr), B_TRUE, B_FALSE) 895 #define l2arc_hdr_arcstats_decrement(hdr) \ 896 l2arc_hdr_arcstats_update((hdr), B_FALSE, B_FALSE) 897 #define l2arc_hdr_arcstats_increment_state(hdr) \ 898 l2arc_hdr_arcstats_update((hdr), B_TRUE, B_TRUE) 899 #define l2arc_hdr_arcstats_decrement_state(hdr) \ 900 l2arc_hdr_arcstats_update((hdr), B_FALSE, B_TRUE) 901 902 /* 903 * l2arc_exclude_special : A zfs module parameter that controls whether buffers 904 * present on special vdevs are eligibile for caching in L2ARC. If 905 * set to 1, exclude dbufs on special vdevs from being cached to 906 * L2ARC. 907 */ 908 int l2arc_exclude_special = 0; 909 910 /* 911 * l2arc_mfuonly : A ZFS module parameter that controls whether only MFU 912 * metadata and data are cached from ARC into L2ARC. 913 */ 914 static int l2arc_mfuonly = 0; 915 916 /* 917 * L2ARC TRIM 918 * l2arc_trim_ahead : A ZFS module parameter that controls how much ahead of 919 * the current write size (l2arc_write_max) we should TRIM if we 920 * have filled the device. It is defined as a percentage of the 921 * write size. If set to 100 we trim twice the space required to 922 * accommodate upcoming writes. A minimum of 64MB will be trimmed. 923 * It also enables TRIM of the whole L2ARC device upon creation or 924 * addition to an existing pool or if the header of the device is 925 * invalid upon importing a pool or onlining a cache device. The 926 * default is 0, which disables TRIM on L2ARC altogether as it can 927 * put significant stress on the underlying storage devices. This 928 * will vary depending of how well the specific device handles 929 * these commands. 930 */ 931 static uint64_t l2arc_trim_ahead = 0; 932 933 /* 934 * Performance tuning of L2ARC persistence: 935 * 936 * l2arc_rebuild_enabled : A ZFS module parameter that controls whether adding 937 * an L2ARC device (either at pool import or later) will attempt 938 * to rebuild L2ARC buffer contents. 939 * l2arc_rebuild_blocks_min_l2size : A ZFS module parameter that controls 940 * whether log blocks are written to the L2ARC device. If the L2ARC 941 * device is less than 1GB, the amount of data l2arc_evict() 942 * evicts is significant compared to the amount of restored L2ARC 943 * data. In this case do not write log blocks in L2ARC in order 944 * not to waste space. 945 */ 946 static int l2arc_rebuild_enabled = B_TRUE; 947 static uint64_t l2arc_rebuild_blocks_min_l2size = 1024 * 1024 * 1024; 948 949 /* L2ARC persistence rebuild control routines. */ 950 void l2arc_rebuild_vdev(vdev_t *vd, boolean_t reopen); 951 static __attribute__((noreturn)) void l2arc_dev_rebuild_thread(void *arg); 952 static int l2arc_rebuild(l2arc_dev_t *dev); 953 954 /* L2ARC persistence read I/O routines. */ 955 static int l2arc_dev_hdr_read(l2arc_dev_t *dev); 956 static int l2arc_log_blk_read(l2arc_dev_t *dev, 957 const l2arc_log_blkptr_t *this_lp, const l2arc_log_blkptr_t *next_lp, 958 l2arc_log_blk_phys_t *this_lb, l2arc_log_blk_phys_t *next_lb, 959 zio_t *this_io, zio_t **next_io); 960 static zio_t *l2arc_log_blk_fetch(vdev_t *vd, 961 const l2arc_log_blkptr_t *lp, l2arc_log_blk_phys_t *lb); 962 static void l2arc_log_blk_fetch_abort(zio_t *zio); 963 964 /* L2ARC persistence block restoration routines. */ 965 static void l2arc_log_blk_restore(l2arc_dev_t *dev, 966 const l2arc_log_blk_phys_t *lb, uint64_t lb_asize); 967 static void l2arc_hdr_restore(const l2arc_log_ent_phys_t *le, 968 l2arc_dev_t *dev); 969 970 /* L2ARC persistence write I/O routines. */ 971 static void l2arc_log_blk_commit(l2arc_dev_t *dev, zio_t *pio, 972 l2arc_write_callback_t *cb); 973 974 /* L2ARC persistence auxiliary routines. */ 975 boolean_t l2arc_log_blkptr_valid(l2arc_dev_t *dev, 976 const l2arc_log_blkptr_t *lbp); 977 static boolean_t l2arc_log_blk_insert(l2arc_dev_t *dev, 978 const arc_buf_hdr_t *ab); 979 boolean_t l2arc_range_check_overlap(uint64_t bottom, 980 uint64_t top, uint64_t check); 981 static void l2arc_blk_fetch_done(zio_t *zio); 982 static inline uint64_t 983 l2arc_log_blk_overhead(uint64_t write_sz, l2arc_dev_t *dev); 984 985 /* 986 * We use Cityhash for this. It's fast, and has good hash properties without 987 * requiring any large static buffers. 988 */ 989 static uint64_t 990 buf_hash(uint64_t spa, const dva_t *dva, uint64_t birth) 991 { 992 return (cityhash4(spa, dva->dva_word[0], dva->dva_word[1], birth)); 993 } 994 995 #define HDR_EMPTY(hdr) \ 996 ((hdr)->b_dva.dva_word[0] == 0 && \ 997 (hdr)->b_dva.dva_word[1] == 0) 998 999 #define HDR_EMPTY_OR_LOCKED(hdr) \ 1000 (HDR_EMPTY(hdr) || MUTEX_HELD(HDR_LOCK(hdr))) 1001 1002 #define HDR_EQUAL(spa, dva, birth, hdr) \ 1003 ((hdr)->b_dva.dva_word[0] == (dva)->dva_word[0]) && \ 1004 ((hdr)->b_dva.dva_word[1] == (dva)->dva_word[1]) && \ 1005 ((hdr)->b_birth == birth) && ((hdr)->b_spa == spa) 1006 1007 static void 1008 buf_discard_identity(arc_buf_hdr_t *hdr) 1009 { 1010 hdr->b_dva.dva_word[0] = 0; 1011 hdr->b_dva.dva_word[1] = 0; 1012 hdr->b_birth = 0; 1013 } 1014 1015 static arc_buf_hdr_t * 1016 buf_hash_find(uint64_t spa, const blkptr_t *bp, kmutex_t **lockp) 1017 { 1018 const dva_t *dva = BP_IDENTITY(bp); 1019 uint64_t birth = BP_PHYSICAL_BIRTH(bp); 1020 uint64_t idx = BUF_HASH_INDEX(spa, dva, birth); 1021 kmutex_t *hash_lock = BUF_HASH_LOCK(idx); 1022 arc_buf_hdr_t *hdr; 1023 1024 mutex_enter(hash_lock); 1025 for (hdr = buf_hash_table.ht_table[idx]; hdr != NULL; 1026 hdr = hdr->b_hash_next) { 1027 if (HDR_EQUAL(spa, dva, birth, hdr)) { 1028 *lockp = hash_lock; 1029 return (hdr); 1030 } 1031 } 1032 mutex_exit(hash_lock); 1033 *lockp = NULL; 1034 return (NULL); 1035 } 1036 1037 /* 1038 * Insert an entry into the hash table. If there is already an element 1039 * equal to elem in the hash table, then the already existing element 1040 * will be returned and the new element will not be inserted. 1041 * Otherwise returns NULL. 1042 * If lockp == NULL, the caller is assumed to already hold the hash lock. 1043 */ 1044 static arc_buf_hdr_t * 1045 buf_hash_insert(arc_buf_hdr_t *hdr, kmutex_t **lockp) 1046 { 1047 uint64_t idx = BUF_HASH_INDEX(hdr->b_spa, &hdr->b_dva, hdr->b_birth); 1048 kmutex_t *hash_lock = BUF_HASH_LOCK(idx); 1049 arc_buf_hdr_t *fhdr; 1050 uint32_t i; 1051 1052 ASSERT(!DVA_IS_EMPTY(&hdr->b_dva)); 1053 ASSERT(hdr->b_birth != 0); 1054 ASSERT(!HDR_IN_HASH_TABLE(hdr)); 1055 1056 if (lockp != NULL) { 1057 *lockp = hash_lock; 1058 mutex_enter(hash_lock); 1059 } else { 1060 ASSERT(MUTEX_HELD(hash_lock)); 1061 } 1062 1063 for (fhdr = buf_hash_table.ht_table[idx], i = 0; fhdr != NULL; 1064 fhdr = fhdr->b_hash_next, i++) { 1065 if (HDR_EQUAL(hdr->b_spa, &hdr->b_dva, hdr->b_birth, fhdr)) 1066 return (fhdr); 1067 } 1068 1069 hdr->b_hash_next = buf_hash_table.ht_table[idx]; 1070 buf_hash_table.ht_table[idx] = hdr; 1071 arc_hdr_set_flags(hdr, ARC_FLAG_IN_HASH_TABLE); 1072 1073 /* collect some hash table performance data */ 1074 if (i > 0) { 1075 ARCSTAT_BUMP(arcstat_hash_collisions); 1076 if (i == 1) 1077 ARCSTAT_BUMP(arcstat_hash_chains); 1078 1079 ARCSTAT_MAX(arcstat_hash_chain_max, i); 1080 } 1081 uint64_t he = atomic_inc_64_nv( 1082 &arc_stats.arcstat_hash_elements.value.ui64); 1083 ARCSTAT_MAX(arcstat_hash_elements_max, he); 1084 1085 return (NULL); 1086 } 1087 1088 static void 1089 buf_hash_remove(arc_buf_hdr_t *hdr) 1090 { 1091 arc_buf_hdr_t *fhdr, **hdrp; 1092 uint64_t idx = BUF_HASH_INDEX(hdr->b_spa, &hdr->b_dva, hdr->b_birth); 1093 1094 ASSERT(MUTEX_HELD(BUF_HASH_LOCK(idx))); 1095 ASSERT(HDR_IN_HASH_TABLE(hdr)); 1096 1097 hdrp = &buf_hash_table.ht_table[idx]; 1098 while ((fhdr = *hdrp) != hdr) { 1099 ASSERT3P(fhdr, !=, NULL); 1100 hdrp = &fhdr->b_hash_next; 1101 } 1102 *hdrp = hdr->b_hash_next; 1103 hdr->b_hash_next = NULL; 1104 arc_hdr_clear_flags(hdr, ARC_FLAG_IN_HASH_TABLE); 1105 1106 /* collect some hash table performance data */ 1107 atomic_dec_64(&arc_stats.arcstat_hash_elements.value.ui64); 1108 1109 if (buf_hash_table.ht_table[idx] && 1110 buf_hash_table.ht_table[idx]->b_hash_next == NULL) 1111 ARCSTAT_BUMPDOWN(arcstat_hash_chains); 1112 } 1113 1114 /* 1115 * Global data structures and functions for the buf kmem cache. 1116 */ 1117 1118 static kmem_cache_t *hdr_full_cache; 1119 static kmem_cache_t *hdr_full_crypt_cache; 1120 static kmem_cache_t *hdr_l2only_cache; 1121 static kmem_cache_t *buf_cache; 1122 1123 static void 1124 buf_fini(void) 1125 { 1126 #if defined(_KERNEL) 1127 /* 1128 * Large allocations which do not require contiguous pages 1129 * should be using vmem_free() in the linux kernel\ 1130 */ 1131 vmem_free(buf_hash_table.ht_table, 1132 (buf_hash_table.ht_mask + 1) * sizeof (void *)); 1133 #else 1134 kmem_free(buf_hash_table.ht_table, 1135 (buf_hash_table.ht_mask + 1) * sizeof (void *)); 1136 #endif 1137 for (int i = 0; i < BUF_LOCKS; i++) 1138 mutex_destroy(BUF_HASH_LOCK(i)); 1139 kmem_cache_destroy(hdr_full_cache); 1140 kmem_cache_destroy(hdr_full_crypt_cache); 1141 kmem_cache_destroy(hdr_l2only_cache); 1142 kmem_cache_destroy(buf_cache); 1143 } 1144 1145 /* 1146 * Constructor callback - called when the cache is empty 1147 * and a new buf is requested. 1148 */ 1149 static int 1150 hdr_full_cons(void *vbuf, void *unused, int kmflag) 1151 { 1152 (void) unused, (void) kmflag; 1153 arc_buf_hdr_t *hdr = vbuf; 1154 1155 memset(hdr, 0, HDR_FULL_SIZE); 1156 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_NUMFUNCS; 1157 cv_init(&hdr->b_l1hdr.b_cv, NULL, CV_DEFAULT, NULL); 1158 zfs_refcount_create(&hdr->b_l1hdr.b_refcnt); 1159 #ifdef ZFS_DEBUG 1160 mutex_init(&hdr->b_l1hdr.b_freeze_lock, NULL, MUTEX_DEFAULT, NULL); 1161 #endif 1162 multilist_link_init(&hdr->b_l1hdr.b_arc_node); 1163 list_link_init(&hdr->b_l2hdr.b_l2node); 1164 arc_space_consume(HDR_FULL_SIZE, ARC_SPACE_HDRS); 1165 1166 return (0); 1167 } 1168 1169 static int 1170 hdr_full_crypt_cons(void *vbuf, void *unused, int kmflag) 1171 { 1172 (void) unused; 1173 arc_buf_hdr_t *hdr = vbuf; 1174 1175 hdr_full_cons(vbuf, unused, kmflag); 1176 memset(&hdr->b_crypt_hdr, 0, sizeof (hdr->b_crypt_hdr)); 1177 arc_space_consume(sizeof (hdr->b_crypt_hdr), ARC_SPACE_HDRS); 1178 1179 return (0); 1180 } 1181 1182 static int 1183 hdr_l2only_cons(void *vbuf, void *unused, int kmflag) 1184 { 1185 (void) unused, (void) kmflag; 1186 arc_buf_hdr_t *hdr = vbuf; 1187 1188 memset(hdr, 0, HDR_L2ONLY_SIZE); 1189 arc_space_consume(HDR_L2ONLY_SIZE, ARC_SPACE_L2HDRS); 1190 1191 return (0); 1192 } 1193 1194 static int 1195 buf_cons(void *vbuf, void *unused, int kmflag) 1196 { 1197 (void) unused, (void) kmflag; 1198 arc_buf_t *buf = vbuf; 1199 1200 memset(buf, 0, sizeof (arc_buf_t)); 1201 arc_space_consume(sizeof (arc_buf_t), ARC_SPACE_HDRS); 1202 1203 return (0); 1204 } 1205 1206 /* 1207 * Destructor callback - called when a cached buf is 1208 * no longer required. 1209 */ 1210 static void 1211 hdr_full_dest(void *vbuf, void *unused) 1212 { 1213 (void) unused; 1214 arc_buf_hdr_t *hdr = vbuf; 1215 1216 ASSERT(HDR_EMPTY(hdr)); 1217 cv_destroy(&hdr->b_l1hdr.b_cv); 1218 zfs_refcount_destroy(&hdr->b_l1hdr.b_refcnt); 1219 #ifdef ZFS_DEBUG 1220 mutex_destroy(&hdr->b_l1hdr.b_freeze_lock); 1221 #endif 1222 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 1223 arc_space_return(HDR_FULL_SIZE, ARC_SPACE_HDRS); 1224 } 1225 1226 static void 1227 hdr_full_crypt_dest(void *vbuf, void *unused) 1228 { 1229 (void) vbuf, (void) unused; 1230 1231 hdr_full_dest(vbuf, unused); 1232 arc_space_return(sizeof (((arc_buf_hdr_t *)NULL)->b_crypt_hdr), 1233 ARC_SPACE_HDRS); 1234 } 1235 1236 static void 1237 hdr_l2only_dest(void *vbuf, void *unused) 1238 { 1239 (void) unused; 1240 arc_buf_hdr_t *hdr = vbuf; 1241 1242 ASSERT(HDR_EMPTY(hdr)); 1243 arc_space_return(HDR_L2ONLY_SIZE, ARC_SPACE_L2HDRS); 1244 } 1245 1246 static void 1247 buf_dest(void *vbuf, void *unused) 1248 { 1249 (void) unused; 1250 (void) vbuf; 1251 1252 arc_space_return(sizeof (arc_buf_t), ARC_SPACE_HDRS); 1253 } 1254 1255 static void 1256 buf_init(void) 1257 { 1258 uint64_t *ct = NULL; 1259 uint64_t hsize = 1ULL << 12; 1260 int i, j; 1261 1262 /* 1263 * The hash table is big enough to fill all of physical memory 1264 * with an average block size of zfs_arc_average_blocksize (default 8K). 1265 * By default, the table will take up 1266 * totalmem * sizeof(void*) / 8K (1MB per GB with 8-byte pointers). 1267 */ 1268 while (hsize * zfs_arc_average_blocksize < arc_all_memory()) 1269 hsize <<= 1; 1270 retry: 1271 buf_hash_table.ht_mask = hsize - 1; 1272 #if defined(_KERNEL) 1273 /* 1274 * Large allocations which do not require contiguous pages 1275 * should be using vmem_alloc() in the linux kernel 1276 */ 1277 buf_hash_table.ht_table = 1278 vmem_zalloc(hsize * sizeof (void*), KM_SLEEP); 1279 #else 1280 buf_hash_table.ht_table = 1281 kmem_zalloc(hsize * sizeof (void*), KM_NOSLEEP); 1282 #endif 1283 if (buf_hash_table.ht_table == NULL) { 1284 ASSERT(hsize > (1ULL << 8)); 1285 hsize >>= 1; 1286 goto retry; 1287 } 1288 1289 hdr_full_cache = kmem_cache_create("arc_buf_hdr_t_full", HDR_FULL_SIZE, 1290 0, hdr_full_cons, hdr_full_dest, NULL, NULL, NULL, 0); 1291 hdr_full_crypt_cache = kmem_cache_create("arc_buf_hdr_t_full_crypt", 1292 HDR_FULL_CRYPT_SIZE, 0, hdr_full_crypt_cons, hdr_full_crypt_dest, 1293 NULL, NULL, NULL, 0); 1294 hdr_l2only_cache = kmem_cache_create("arc_buf_hdr_t_l2only", 1295 HDR_L2ONLY_SIZE, 0, hdr_l2only_cons, hdr_l2only_dest, NULL, 1296 NULL, NULL, 0); 1297 buf_cache = kmem_cache_create("arc_buf_t", sizeof (arc_buf_t), 1298 0, buf_cons, buf_dest, NULL, NULL, NULL, 0); 1299 1300 for (i = 0; i < 256; i++) 1301 for (ct = zfs_crc64_table + i, *ct = i, j = 8; j > 0; j--) 1302 *ct = (*ct >> 1) ^ (-(*ct & 1) & ZFS_CRC64_POLY); 1303 1304 for (i = 0; i < BUF_LOCKS; i++) 1305 mutex_init(BUF_HASH_LOCK(i), NULL, MUTEX_DEFAULT, NULL); 1306 } 1307 1308 #define ARC_MINTIME (hz>>4) /* 62 ms */ 1309 1310 /* 1311 * This is the size that the buf occupies in memory. If the buf is compressed, 1312 * it will correspond to the compressed size. You should use this method of 1313 * getting the buf size unless you explicitly need the logical size. 1314 */ 1315 uint64_t 1316 arc_buf_size(arc_buf_t *buf) 1317 { 1318 return (ARC_BUF_COMPRESSED(buf) ? 1319 HDR_GET_PSIZE(buf->b_hdr) : HDR_GET_LSIZE(buf->b_hdr)); 1320 } 1321 1322 uint64_t 1323 arc_buf_lsize(arc_buf_t *buf) 1324 { 1325 return (HDR_GET_LSIZE(buf->b_hdr)); 1326 } 1327 1328 /* 1329 * This function will return B_TRUE if the buffer is encrypted in memory. 1330 * This buffer can be decrypted by calling arc_untransform(). 1331 */ 1332 boolean_t 1333 arc_is_encrypted(arc_buf_t *buf) 1334 { 1335 return (ARC_BUF_ENCRYPTED(buf) != 0); 1336 } 1337 1338 /* 1339 * Returns B_TRUE if the buffer represents data that has not had its MAC 1340 * verified yet. 1341 */ 1342 boolean_t 1343 arc_is_unauthenticated(arc_buf_t *buf) 1344 { 1345 return (HDR_NOAUTH(buf->b_hdr) != 0); 1346 } 1347 1348 void 1349 arc_get_raw_params(arc_buf_t *buf, boolean_t *byteorder, uint8_t *salt, 1350 uint8_t *iv, uint8_t *mac) 1351 { 1352 arc_buf_hdr_t *hdr = buf->b_hdr; 1353 1354 ASSERT(HDR_PROTECTED(hdr)); 1355 1356 memcpy(salt, hdr->b_crypt_hdr.b_salt, ZIO_DATA_SALT_LEN); 1357 memcpy(iv, hdr->b_crypt_hdr.b_iv, ZIO_DATA_IV_LEN); 1358 memcpy(mac, hdr->b_crypt_hdr.b_mac, ZIO_DATA_MAC_LEN); 1359 *byteorder = (hdr->b_l1hdr.b_byteswap == DMU_BSWAP_NUMFUNCS) ? 1360 ZFS_HOST_BYTEORDER : !ZFS_HOST_BYTEORDER; 1361 } 1362 1363 /* 1364 * Indicates how this buffer is compressed in memory. If it is not compressed 1365 * the value will be ZIO_COMPRESS_OFF. It can be made normally readable with 1366 * arc_untransform() as long as it is also unencrypted. 1367 */ 1368 enum zio_compress 1369 arc_get_compression(arc_buf_t *buf) 1370 { 1371 return (ARC_BUF_COMPRESSED(buf) ? 1372 HDR_GET_COMPRESS(buf->b_hdr) : ZIO_COMPRESS_OFF); 1373 } 1374 1375 /* 1376 * Return the compression algorithm used to store this data in the ARC. If ARC 1377 * compression is enabled or this is an encrypted block, this will be the same 1378 * as what's used to store it on-disk. Otherwise, this will be ZIO_COMPRESS_OFF. 1379 */ 1380 static inline enum zio_compress 1381 arc_hdr_get_compress(arc_buf_hdr_t *hdr) 1382 { 1383 return (HDR_COMPRESSION_ENABLED(hdr) ? 1384 HDR_GET_COMPRESS(hdr) : ZIO_COMPRESS_OFF); 1385 } 1386 1387 uint8_t 1388 arc_get_complevel(arc_buf_t *buf) 1389 { 1390 return (buf->b_hdr->b_complevel); 1391 } 1392 1393 static inline boolean_t 1394 arc_buf_is_shared(arc_buf_t *buf) 1395 { 1396 boolean_t shared = (buf->b_data != NULL && 1397 buf->b_hdr->b_l1hdr.b_pabd != NULL && 1398 abd_is_linear(buf->b_hdr->b_l1hdr.b_pabd) && 1399 buf->b_data == abd_to_buf(buf->b_hdr->b_l1hdr.b_pabd)); 1400 IMPLY(shared, HDR_SHARED_DATA(buf->b_hdr)); 1401 IMPLY(shared, ARC_BUF_SHARED(buf)); 1402 IMPLY(shared, ARC_BUF_COMPRESSED(buf) || ARC_BUF_LAST(buf)); 1403 1404 /* 1405 * It would be nice to assert arc_can_share() too, but the "hdr isn't 1406 * already being shared" requirement prevents us from doing that. 1407 */ 1408 1409 return (shared); 1410 } 1411 1412 /* 1413 * Free the checksum associated with this header. If there is no checksum, this 1414 * is a no-op. 1415 */ 1416 static inline void 1417 arc_cksum_free(arc_buf_hdr_t *hdr) 1418 { 1419 #ifdef ZFS_DEBUG 1420 ASSERT(HDR_HAS_L1HDR(hdr)); 1421 1422 mutex_enter(&hdr->b_l1hdr.b_freeze_lock); 1423 if (hdr->b_l1hdr.b_freeze_cksum != NULL) { 1424 kmem_free(hdr->b_l1hdr.b_freeze_cksum, sizeof (zio_cksum_t)); 1425 hdr->b_l1hdr.b_freeze_cksum = NULL; 1426 } 1427 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1428 #endif 1429 } 1430 1431 /* 1432 * Return true iff at least one of the bufs on hdr is not compressed. 1433 * Encrypted buffers count as compressed. 1434 */ 1435 static boolean_t 1436 arc_hdr_has_uncompressed_buf(arc_buf_hdr_t *hdr) 1437 { 1438 ASSERT(hdr->b_l1hdr.b_state == arc_anon || HDR_EMPTY_OR_LOCKED(hdr)); 1439 1440 for (arc_buf_t *b = hdr->b_l1hdr.b_buf; b != NULL; b = b->b_next) { 1441 if (!ARC_BUF_COMPRESSED(b)) { 1442 return (B_TRUE); 1443 } 1444 } 1445 return (B_FALSE); 1446 } 1447 1448 1449 /* 1450 * If we've turned on the ZFS_DEBUG_MODIFY flag, verify that the buf's data 1451 * matches the checksum that is stored in the hdr. If there is no checksum, 1452 * or if the buf is compressed, this is a no-op. 1453 */ 1454 static void 1455 arc_cksum_verify(arc_buf_t *buf) 1456 { 1457 #ifdef ZFS_DEBUG 1458 arc_buf_hdr_t *hdr = buf->b_hdr; 1459 zio_cksum_t zc; 1460 1461 if (!(zfs_flags & ZFS_DEBUG_MODIFY)) 1462 return; 1463 1464 if (ARC_BUF_COMPRESSED(buf)) 1465 return; 1466 1467 ASSERT(HDR_HAS_L1HDR(hdr)); 1468 1469 mutex_enter(&hdr->b_l1hdr.b_freeze_lock); 1470 1471 if (hdr->b_l1hdr.b_freeze_cksum == NULL || HDR_IO_ERROR(hdr)) { 1472 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1473 return; 1474 } 1475 1476 fletcher_2_native(buf->b_data, arc_buf_size(buf), NULL, &zc); 1477 if (!ZIO_CHECKSUM_EQUAL(*hdr->b_l1hdr.b_freeze_cksum, zc)) 1478 panic("buffer modified while frozen!"); 1479 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1480 #endif 1481 } 1482 1483 /* 1484 * This function makes the assumption that data stored in the L2ARC 1485 * will be transformed exactly as it is in the main pool. Because of 1486 * this we can verify the checksum against the reading process's bp. 1487 */ 1488 static boolean_t 1489 arc_cksum_is_equal(arc_buf_hdr_t *hdr, zio_t *zio) 1490 { 1491 ASSERT(!BP_IS_EMBEDDED(zio->io_bp)); 1492 VERIFY3U(BP_GET_PSIZE(zio->io_bp), ==, HDR_GET_PSIZE(hdr)); 1493 1494 /* 1495 * Block pointers always store the checksum for the logical data. 1496 * If the block pointer has the gang bit set, then the checksum 1497 * it represents is for the reconstituted data and not for an 1498 * individual gang member. The zio pipeline, however, must be able to 1499 * determine the checksum of each of the gang constituents so it 1500 * treats the checksum comparison differently than what we need 1501 * for l2arc blocks. This prevents us from using the 1502 * zio_checksum_error() interface directly. Instead we must call the 1503 * zio_checksum_error_impl() so that we can ensure the checksum is 1504 * generated using the correct checksum algorithm and accounts for the 1505 * logical I/O size and not just a gang fragment. 1506 */ 1507 return (zio_checksum_error_impl(zio->io_spa, zio->io_bp, 1508 BP_GET_CHECKSUM(zio->io_bp), zio->io_abd, zio->io_size, 1509 zio->io_offset, NULL) == 0); 1510 } 1511 1512 /* 1513 * Given a buf full of data, if ZFS_DEBUG_MODIFY is enabled this computes a 1514 * checksum and attaches it to the buf's hdr so that we can ensure that the buf 1515 * isn't modified later on. If buf is compressed or there is already a checksum 1516 * on the hdr, this is a no-op (we only checksum uncompressed bufs). 1517 */ 1518 static void 1519 arc_cksum_compute(arc_buf_t *buf) 1520 { 1521 if (!(zfs_flags & ZFS_DEBUG_MODIFY)) 1522 return; 1523 1524 #ifdef ZFS_DEBUG 1525 arc_buf_hdr_t *hdr = buf->b_hdr; 1526 ASSERT(HDR_HAS_L1HDR(hdr)); 1527 mutex_enter(&hdr->b_l1hdr.b_freeze_lock); 1528 if (hdr->b_l1hdr.b_freeze_cksum != NULL || ARC_BUF_COMPRESSED(buf)) { 1529 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1530 return; 1531 } 1532 1533 ASSERT(!ARC_BUF_ENCRYPTED(buf)); 1534 ASSERT(!ARC_BUF_COMPRESSED(buf)); 1535 hdr->b_l1hdr.b_freeze_cksum = kmem_alloc(sizeof (zio_cksum_t), 1536 KM_SLEEP); 1537 fletcher_2_native(buf->b_data, arc_buf_size(buf), NULL, 1538 hdr->b_l1hdr.b_freeze_cksum); 1539 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1540 #endif 1541 arc_buf_watch(buf); 1542 } 1543 1544 #ifndef _KERNEL 1545 void 1546 arc_buf_sigsegv(int sig, siginfo_t *si, void *unused) 1547 { 1548 (void) sig, (void) unused; 1549 panic("Got SIGSEGV at address: 0x%lx\n", (long)si->si_addr); 1550 } 1551 #endif 1552 1553 static void 1554 arc_buf_unwatch(arc_buf_t *buf) 1555 { 1556 #ifndef _KERNEL 1557 if (arc_watch) { 1558 ASSERT0(mprotect(buf->b_data, arc_buf_size(buf), 1559 PROT_READ | PROT_WRITE)); 1560 } 1561 #else 1562 (void) buf; 1563 #endif 1564 } 1565 1566 static void 1567 arc_buf_watch(arc_buf_t *buf) 1568 { 1569 #ifndef _KERNEL 1570 if (arc_watch) 1571 ASSERT0(mprotect(buf->b_data, arc_buf_size(buf), 1572 PROT_READ)); 1573 #else 1574 (void) buf; 1575 #endif 1576 } 1577 1578 static arc_buf_contents_t 1579 arc_buf_type(arc_buf_hdr_t *hdr) 1580 { 1581 arc_buf_contents_t type; 1582 if (HDR_ISTYPE_METADATA(hdr)) { 1583 type = ARC_BUFC_METADATA; 1584 } else { 1585 type = ARC_BUFC_DATA; 1586 } 1587 VERIFY3U(hdr->b_type, ==, type); 1588 return (type); 1589 } 1590 1591 boolean_t 1592 arc_is_metadata(arc_buf_t *buf) 1593 { 1594 return (HDR_ISTYPE_METADATA(buf->b_hdr) != 0); 1595 } 1596 1597 static uint32_t 1598 arc_bufc_to_flags(arc_buf_contents_t type) 1599 { 1600 switch (type) { 1601 case ARC_BUFC_DATA: 1602 /* metadata field is 0 if buffer contains normal data */ 1603 return (0); 1604 case ARC_BUFC_METADATA: 1605 return (ARC_FLAG_BUFC_METADATA); 1606 default: 1607 break; 1608 } 1609 panic("undefined ARC buffer type!"); 1610 return ((uint32_t)-1); 1611 } 1612 1613 void 1614 arc_buf_thaw(arc_buf_t *buf) 1615 { 1616 arc_buf_hdr_t *hdr = buf->b_hdr; 1617 1618 ASSERT3P(hdr->b_l1hdr.b_state, ==, arc_anon); 1619 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 1620 1621 arc_cksum_verify(buf); 1622 1623 /* 1624 * Compressed buffers do not manipulate the b_freeze_cksum. 1625 */ 1626 if (ARC_BUF_COMPRESSED(buf)) 1627 return; 1628 1629 ASSERT(HDR_HAS_L1HDR(hdr)); 1630 arc_cksum_free(hdr); 1631 arc_buf_unwatch(buf); 1632 } 1633 1634 void 1635 arc_buf_freeze(arc_buf_t *buf) 1636 { 1637 if (!(zfs_flags & ZFS_DEBUG_MODIFY)) 1638 return; 1639 1640 if (ARC_BUF_COMPRESSED(buf)) 1641 return; 1642 1643 ASSERT(HDR_HAS_L1HDR(buf->b_hdr)); 1644 arc_cksum_compute(buf); 1645 } 1646 1647 /* 1648 * The arc_buf_hdr_t's b_flags should never be modified directly. Instead, 1649 * the following functions should be used to ensure that the flags are 1650 * updated in a thread-safe way. When manipulating the flags either 1651 * the hash_lock must be held or the hdr must be undiscoverable. This 1652 * ensures that we're not racing with any other threads when updating 1653 * the flags. 1654 */ 1655 static inline void 1656 arc_hdr_set_flags(arc_buf_hdr_t *hdr, arc_flags_t flags) 1657 { 1658 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1659 hdr->b_flags |= flags; 1660 } 1661 1662 static inline void 1663 arc_hdr_clear_flags(arc_buf_hdr_t *hdr, arc_flags_t flags) 1664 { 1665 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1666 hdr->b_flags &= ~flags; 1667 } 1668 1669 /* 1670 * Setting the compression bits in the arc_buf_hdr_t's b_flags is 1671 * done in a special way since we have to clear and set bits 1672 * at the same time. Consumers that wish to set the compression bits 1673 * must use this function to ensure that the flags are updated in 1674 * thread-safe manner. 1675 */ 1676 static void 1677 arc_hdr_set_compress(arc_buf_hdr_t *hdr, enum zio_compress cmp) 1678 { 1679 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1680 1681 /* 1682 * Holes and embedded blocks will always have a psize = 0 so 1683 * we ignore the compression of the blkptr and set the 1684 * want to uncompress them. Mark them as uncompressed. 1685 */ 1686 if (!zfs_compressed_arc_enabled || HDR_GET_PSIZE(hdr) == 0) { 1687 arc_hdr_clear_flags(hdr, ARC_FLAG_COMPRESSED_ARC); 1688 ASSERT(!HDR_COMPRESSION_ENABLED(hdr)); 1689 } else { 1690 arc_hdr_set_flags(hdr, ARC_FLAG_COMPRESSED_ARC); 1691 ASSERT(HDR_COMPRESSION_ENABLED(hdr)); 1692 } 1693 1694 HDR_SET_COMPRESS(hdr, cmp); 1695 ASSERT3U(HDR_GET_COMPRESS(hdr), ==, cmp); 1696 } 1697 1698 /* 1699 * Looks for another buf on the same hdr which has the data decompressed, copies 1700 * from it, and returns true. If no such buf exists, returns false. 1701 */ 1702 static boolean_t 1703 arc_buf_try_copy_decompressed_data(arc_buf_t *buf) 1704 { 1705 arc_buf_hdr_t *hdr = buf->b_hdr; 1706 boolean_t copied = B_FALSE; 1707 1708 ASSERT(HDR_HAS_L1HDR(hdr)); 1709 ASSERT3P(buf->b_data, !=, NULL); 1710 ASSERT(!ARC_BUF_COMPRESSED(buf)); 1711 1712 for (arc_buf_t *from = hdr->b_l1hdr.b_buf; from != NULL; 1713 from = from->b_next) { 1714 /* can't use our own data buffer */ 1715 if (from == buf) { 1716 continue; 1717 } 1718 1719 if (!ARC_BUF_COMPRESSED(from)) { 1720 memcpy(buf->b_data, from->b_data, arc_buf_size(buf)); 1721 copied = B_TRUE; 1722 break; 1723 } 1724 } 1725 1726 #ifdef ZFS_DEBUG 1727 /* 1728 * There were no decompressed bufs, so there should not be a 1729 * checksum on the hdr either. 1730 */ 1731 if (zfs_flags & ZFS_DEBUG_MODIFY) 1732 EQUIV(!copied, hdr->b_l1hdr.b_freeze_cksum == NULL); 1733 #endif 1734 1735 return (copied); 1736 } 1737 1738 /* 1739 * Allocates an ARC buf header that's in an evicted & L2-cached state. 1740 * This is used during l2arc reconstruction to make empty ARC buffers 1741 * which circumvent the regular disk->arc->l2arc path and instead come 1742 * into being in the reverse order, i.e. l2arc->arc. 1743 */ 1744 static arc_buf_hdr_t * 1745 arc_buf_alloc_l2only(size_t size, arc_buf_contents_t type, l2arc_dev_t *dev, 1746 dva_t dva, uint64_t daddr, int32_t psize, uint64_t birth, 1747 enum zio_compress compress, uint8_t complevel, boolean_t protected, 1748 boolean_t prefetch, arc_state_type_t arcs_state) 1749 { 1750 arc_buf_hdr_t *hdr; 1751 1752 ASSERT(size != 0); 1753 hdr = kmem_cache_alloc(hdr_l2only_cache, KM_SLEEP); 1754 hdr->b_birth = birth; 1755 hdr->b_type = type; 1756 hdr->b_flags = 0; 1757 arc_hdr_set_flags(hdr, arc_bufc_to_flags(type) | ARC_FLAG_HAS_L2HDR); 1758 HDR_SET_LSIZE(hdr, size); 1759 HDR_SET_PSIZE(hdr, psize); 1760 arc_hdr_set_compress(hdr, compress); 1761 hdr->b_complevel = complevel; 1762 if (protected) 1763 arc_hdr_set_flags(hdr, ARC_FLAG_PROTECTED); 1764 if (prefetch) 1765 arc_hdr_set_flags(hdr, ARC_FLAG_PREFETCH); 1766 hdr->b_spa = spa_load_guid(dev->l2ad_vdev->vdev_spa); 1767 1768 hdr->b_dva = dva; 1769 1770 hdr->b_l2hdr.b_dev = dev; 1771 hdr->b_l2hdr.b_daddr = daddr; 1772 hdr->b_l2hdr.b_arcs_state = arcs_state; 1773 1774 return (hdr); 1775 } 1776 1777 /* 1778 * Return the size of the block, b_pabd, that is stored in the arc_buf_hdr_t. 1779 */ 1780 static uint64_t 1781 arc_hdr_size(arc_buf_hdr_t *hdr) 1782 { 1783 uint64_t size; 1784 1785 if (arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF && 1786 HDR_GET_PSIZE(hdr) > 0) { 1787 size = HDR_GET_PSIZE(hdr); 1788 } else { 1789 ASSERT3U(HDR_GET_LSIZE(hdr), !=, 0); 1790 size = HDR_GET_LSIZE(hdr); 1791 } 1792 return (size); 1793 } 1794 1795 static int 1796 arc_hdr_authenticate(arc_buf_hdr_t *hdr, spa_t *spa, uint64_t dsobj) 1797 { 1798 int ret; 1799 uint64_t csize; 1800 uint64_t lsize = HDR_GET_LSIZE(hdr); 1801 uint64_t psize = HDR_GET_PSIZE(hdr); 1802 void *tmpbuf = NULL; 1803 abd_t *abd = hdr->b_l1hdr.b_pabd; 1804 1805 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1806 ASSERT(HDR_AUTHENTICATED(hdr)); 1807 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 1808 1809 /* 1810 * The MAC is calculated on the compressed data that is stored on disk. 1811 * However, if compressed arc is disabled we will only have the 1812 * decompressed data available to us now. Compress it into a temporary 1813 * abd so we can verify the MAC. The performance overhead of this will 1814 * be relatively low, since most objects in an encrypted objset will 1815 * be encrypted (instead of authenticated) anyway. 1816 */ 1817 if (HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 1818 !HDR_COMPRESSION_ENABLED(hdr)) { 1819 tmpbuf = zio_buf_alloc(lsize); 1820 abd = abd_get_from_buf(tmpbuf, lsize); 1821 abd_take_ownership_of_buf(abd, B_TRUE); 1822 csize = zio_compress_data(HDR_GET_COMPRESS(hdr), 1823 hdr->b_l1hdr.b_pabd, tmpbuf, lsize, hdr->b_complevel); 1824 ASSERT3U(csize, <=, psize); 1825 abd_zero_off(abd, csize, psize - csize); 1826 } 1827 1828 /* 1829 * Authentication is best effort. We authenticate whenever the key is 1830 * available. If we succeed we clear ARC_FLAG_NOAUTH. 1831 */ 1832 if (hdr->b_crypt_hdr.b_ot == DMU_OT_OBJSET) { 1833 ASSERT3U(HDR_GET_COMPRESS(hdr), ==, ZIO_COMPRESS_OFF); 1834 ASSERT3U(lsize, ==, psize); 1835 ret = spa_do_crypt_objset_mac_abd(B_FALSE, spa, dsobj, abd, 1836 psize, hdr->b_l1hdr.b_byteswap != DMU_BSWAP_NUMFUNCS); 1837 } else { 1838 ret = spa_do_crypt_mac_abd(B_FALSE, spa, dsobj, abd, psize, 1839 hdr->b_crypt_hdr.b_mac); 1840 } 1841 1842 if (ret == 0) 1843 arc_hdr_clear_flags(hdr, ARC_FLAG_NOAUTH); 1844 else if (ret != ENOENT) 1845 goto error; 1846 1847 if (tmpbuf != NULL) 1848 abd_free(abd); 1849 1850 return (0); 1851 1852 error: 1853 if (tmpbuf != NULL) 1854 abd_free(abd); 1855 1856 return (ret); 1857 } 1858 1859 /* 1860 * This function will take a header that only has raw encrypted data in 1861 * b_crypt_hdr.b_rabd and decrypt it into a new buffer which is stored in 1862 * b_l1hdr.b_pabd. If designated in the header flags, this function will 1863 * also decompress the data. 1864 */ 1865 static int 1866 arc_hdr_decrypt(arc_buf_hdr_t *hdr, spa_t *spa, const zbookmark_phys_t *zb) 1867 { 1868 int ret; 1869 abd_t *cabd = NULL; 1870 void *tmp = NULL; 1871 boolean_t no_crypt = B_FALSE; 1872 boolean_t bswap = (hdr->b_l1hdr.b_byteswap != DMU_BSWAP_NUMFUNCS); 1873 1874 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1875 ASSERT(HDR_ENCRYPTED(hdr)); 1876 1877 arc_hdr_alloc_abd(hdr, ARC_HDR_DO_ADAPT); 1878 1879 ret = spa_do_crypt_abd(B_FALSE, spa, zb, hdr->b_crypt_hdr.b_ot, 1880 B_FALSE, bswap, hdr->b_crypt_hdr.b_salt, hdr->b_crypt_hdr.b_iv, 1881 hdr->b_crypt_hdr.b_mac, HDR_GET_PSIZE(hdr), hdr->b_l1hdr.b_pabd, 1882 hdr->b_crypt_hdr.b_rabd, &no_crypt); 1883 if (ret != 0) 1884 goto error; 1885 1886 if (no_crypt) { 1887 abd_copy(hdr->b_l1hdr.b_pabd, hdr->b_crypt_hdr.b_rabd, 1888 HDR_GET_PSIZE(hdr)); 1889 } 1890 1891 /* 1892 * If this header has disabled arc compression but the b_pabd is 1893 * compressed after decrypting it, we need to decompress the newly 1894 * decrypted data. 1895 */ 1896 if (HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 1897 !HDR_COMPRESSION_ENABLED(hdr)) { 1898 /* 1899 * We want to make sure that we are correctly honoring the 1900 * zfs_abd_scatter_enabled setting, so we allocate an abd here 1901 * and then loan a buffer from it, rather than allocating a 1902 * linear buffer and wrapping it in an abd later. 1903 */ 1904 cabd = arc_get_data_abd(hdr, arc_hdr_size(hdr), hdr, 1905 ARC_HDR_DO_ADAPT); 1906 tmp = abd_borrow_buf(cabd, arc_hdr_size(hdr)); 1907 1908 ret = zio_decompress_data(HDR_GET_COMPRESS(hdr), 1909 hdr->b_l1hdr.b_pabd, tmp, HDR_GET_PSIZE(hdr), 1910 HDR_GET_LSIZE(hdr), &hdr->b_complevel); 1911 if (ret != 0) { 1912 abd_return_buf(cabd, tmp, arc_hdr_size(hdr)); 1913 goto error; 1914 } 1915 1916 abd_return_buf_copy(cabd, tmp, arc_hdr_size(hdr)); 1917 arc_free_data_abd(hdr, hdr->b_l1hdr.b_pabd, 1918 arc_hdr_size(hdr), hdr); 1919 hdr->b_l1hdr.b_pabd = cabd; 1920 } 1921 1922 return (0); 1923 1924 error: 1925 arc_hdr_free_abd(hdr, B_FALSE); 1926 if (cabd != NULL) 1927 arc_free_data_buf(hdr, cabd, arc_hdr_size(hdr), hdr); 1928 1929 return (ret); 1930 } 1931 1932 /* 1933 * This function is called during arc_buf_fill() to prepare the header's 1934 * abd plaintext pointer for use. This involves authenticated protected 1935 * data and decrypting encrypted data into the plaintext abd. 1936 */ 1937 static int 1938 arc_fill_hdr_crypt(arc_buf_hdr_t *hdr, kmutex_t *hash_lock, spa_t *spa, 1939 const zbookmark_phys_t *zb, boolean_t noauth) 1940 { 1941 int ret; 1942 1943 ASSERT(HDR_PROTECTED(hdr)); 1944 1945 if (hash_lock != NULL) 1946 mutex_enter(hash_lock); 1947 1948 if (HDR_NOAUTH(hdr) && !noauth) { 1949 /* 1950 * The caller requested authenticated data but our data has 1951 * not been authenticated yet. Verify the MAC now if we can. 1952 */ 1953 ret = arc_hdr_authenticate(hdr, spa, zb->zb_objset); 1954 if (ret != 0) 1955 goto error; 1956 } else if (HDR_HAS_RABD(hdr) && hdr->b_l1hdr.b_pabd == NULL) { 1957 /* 1958 * If we only have the encrypted version of the data, but the 1959 * unencrypted version was requested we take this opportunity 1960 * to store the decrypted version in the header for future use. 1961 */ 1962 ret = arc_hdr_decrypt(hdr, spa, zb); 1963 if (ret != 0) 1964 goto error; 1965 } 1966 1967 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 1968 1969 if (hash_lock != NULL) 1970 mutex_exit(hash_lock); 1971 1972 return (0); 1973 1974 error: 1975 if (hash_lock != NULL) 1976 mutex_exit(hash_lock); 1977 1978 return (ret); 1979 } 1980 1981 /* 1982 * This function is used by the dbuf code to decrypt bonus buffers in place. 1983 * The dbuf code itself doesn't have any locking for decrypting a shared dnode 1984 * block, so we use the hash lock here to protect against concurrent calls to 1985 * arc_buf_fill(). 1986 */ 1987 static void 1988 arc_buf_untransform_in_place(arc_buf_t *buf) 1989 { 1990 arc_buf_hdr_t *hdr = buf->b_hdr; 1991 1992 ASSERT(HDR_ENCRYPTED(hdr)); 1993 ASSERT3U(hdr->b_crypt_hdr.b_ot, ==, DMU_OT_DNODE); 1994 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1995 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 1996 1997 zio_crypt_copy_dnode_bonus(hdr->b_l1hdr.b_pabd, buf->b_data, 1998 arc_buf_size(buf)); 1999 buf->b_flags &= ~ARC_BUF_FLAG_ENCRYPTED; 2000 buf->b_flags &= ~ARC_BUF_FLAG_COMPRESSED; 2001 hdr->b_crypt_hdr.b_ebufcnt -= 1; 2002 } 2003 2004 /* 2005 * Given a buf that has a data buffer attached to it, this function will 2006 * efficiently fill the buf with data of the specified compression setting from 2007 * the hdr and update the hdr's b_freeze_cksum if necessary. If the buf and hdr 2008 * are already sharing a data buf, no copy is performed. 2009 * 2010 * If the buf is marked as compressed but uncompressed data was requested, this 2011 * will allocate a new data buffer for the buf, remove that flag, and fill the 2012 * buf with uncompressed data. You can't request a compressed buf on a hdr with 2013 * uncompressed data, and (since we haven't added support for it yet) if you 2014 * want compressed data your buf must already be marked as compressed and have 2015 * the correct-sized data buffer. 2016 */ 2017 static int 2018 arc_buf_fill(arc_buf_t *buf, spa_t *spa, const zbookmark_phys_t *zb, 2019 arc_fill_flags_t flags) 2020 { 2021 int error = 0; 2022 arc_buf_hdr_t *hdr = buf->b_hdr; 2023 boolean_t hdr_compressed = 2024 (arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF); 2025 boolean_t compressed = (flags & ARC_FILL_COMPRESSED) != 0; 2026 boolean_t encrypted = (flags & ARC_FILL_ENCRYPTED) != 0; 2027 dmu_object_byteswap_t bswap = hdr->b_l1hdr.b_byteswap; 2028 kmutex_t *hash_lock = (flags & ARC_FILL_LOCKED) ? NULL : HDR_LOCK(hdr); 2029 2030 ASSERT3P(buf->b_data, !=, NULL); 2031 IMPLY(compressed, hdr_compressed || ARC_BUF_ENCRYPTED(buf)); 2032 IMPLY(compressed, ARC_BUF_COMPRESSED(buf)); 2033 IMPLY(encrypted, HDR_ENCRYPTED(hdr)); 2034 IMPLY(encrypted, ARC_BUF_ENCRYPTED(buf)); 2035 IMPLY(encrypted, ARC_BUF_COMPRESSED(buf)); 2036 IMPLY(encrypted, !ARC_BUF_SHARED(buf)); 2037 2038 /* 2039 * If the caller wanted encrypted data we just need to copy it from 2040 * b_rabd and potentially byteswap it. We won't be able to do any 2041 * further transforms on it. 2042 */ 2043 if (encrypted) { 2044 ASSERT(HDR_HAS_RABD(hdr)); 2045 abd_copy_to_buf(buf->b_data, hdr->b_crypt_hdr.b_rabd, 2046 HDR_GET_PSIZE(hdr)); 2047 goto byteswap; 2048 } 2049 2050 /* 2051 * Adjust encrypted and authenticated headers to accommodate 2052 * the request if needed. Dnode blocks (ARC_FILL_IN_PLACE) are 2053 * allowed to fail decryption due to keys not being loaded 2054 * without being marked as an IO error. 2055 */ 2056 if (HDR_PROTECTED(hdr)) { 2057 error = arc_fill_hdr_crypt(hdr, hash_lock, spa, 2058 zb, !!(flags & ARC_FILL_NOAUTH)); 2059 if (error == EACCES && (flags & ARC_FILL_IN_PLACE) != 0) { 2060 return (error); 2061 } else if (error != 0) { 2062 if (hash_lock != NULL) 2063 mutex_enter(hash_lock); 2064 arc_hdr_set_flags(hdr, ARC_FLAG_IO_ERROR); 2065 if (hash_lock != NULL) 2066 mutex_exit(hash_lock); 2067 return (error); 2068 } 2069 } 2070 2071 /* 2072 * There is a special case here for dnode blocks which are 2073 * decrypting their bonus buffers. These blocks may request to 2074 * be decrypted in-place. This is necessary because there may 2075 * be many dnodes pointing into this buffer and there is 2076 * currently no method to synchronize replacing the backing 2077 * b_data buffer and updating all of the pointers. Here we use 2078 * the hash lock to ensure there are no races. If the need 2079 * arises for other types to be decrypted in-place, they must 2080 * add handling here as well. 2081 */ 2082 if ((flags & ARC_FILL_IN_PLACE) != 0) { 2083 ASSERT(!hdr_compressed); 2084 ASSERT(!compressed); 2085 ASSERT(!encrypted); 2086 2087 if (HDR_ENCRYPTED(hdr) && ARC_BUF_ENCRYPTED(buf)) { 2088 ASSERT3U(hdr->b_crypt_hdr.b_ot, ==, DMU_OT_DNODE); 2089 2090 if (hash_lock != NULL) 2091 mutex_enter(hash_lock); 2092 arc_buf_untransform_in_place(buf); 2093 if (hash_lock != NULL) 2094 mutex_exit(hash_lock); 2095 2096 /* Compute the hdr's checksum if necessary */ 2097 arc_cksum_compute(buf); 2098 } 2099 2100 return (0); 2101 } 2102 2103 if (hdr_compressed == compressed) { 2104 if (!arc_buf_is_shared(buf)) { 2105 abd_copy_to_buf(buf->b_data, hdr->b_l1hdr.b_pabd, 2106 arc_buf_size(buf)); 2107 } 2108 } else { 2109 ASSERT(hdr_compressed); 2110 ASSERT(!compressed); 2111 2112 /* 2113 * If the buf is sharing its data with the hdr, unlink it and 2114 * allocate a new data buffer for the buf. 2115 */ 2116 if (arc_buf_is_shared(buf)) { 2117 ASSERT(ARC_BUF_COMPRESSED(buf)); 2118 2119 /* We need to give the buf its own b_data */ 2120 buf->b_flags &= ~ARC_BUF_FLAG_SHARED; 2121 buf->b_data = 2122 arc_get_data_buf(hdr, HDR_GET_LSIZE(hdr), buf); 2123 arc_hdr_clear_flags(hdr, ARC_FLAG_SHARED_DATA); 2124 2125 /* Previously overhead was 0; just add new overhead */ 2126 ARCSTAT_INCR(arcstat_overhead_size, HDR_GET_LSIZE(hdr)); 2127 } else if (ARC_BUF_COMPRESSED(buf)) { 2128 /* We need to reallocate the buf's b_data */ 2129 arc_free_data_buf(hdr, buf->b_data, HDR_GET_PSIZE(hdr), 2130 buf); 2131 buf->b_data = 2132 arc_get_data_buf(hdr, HDR_GET_LSIZE(hdr), buf); 2133 2134 /* We increased the size of b_data; update overhead */ 2135 ARCSTAT_INCR(arcstat_overhead_size, 2136 HDR_GET_LSIZE(hdr) - HDR_GET_PSIZE(hdr)); 2137 } 2138 2139 /* 2140 * Regardless of the buf's previous compression settings, it 2141 * should not be compressed at the end of this function. 2142 */ 2143 buf->b_flags &= ~ARC_BUF_FLAG_COMPRESSED; 2144 2145 /* 2146 * Try copying the data from another buf which already has a 2147 * decompressed version. If that's not possible, it's time to 2148 * bite the bullet and decompress the data from the hdr. 2149 */ 2150 if (arc_buf_try_copy_decompressed_data(buf)) { 2151 /* Skip byteswapping and checksumming (already done) */ 2152 return (0); 2153 } else { 2154 error = zio_decompress_data(HDR_GET_COMPRESS(hdr), 2155 hdr->b_l1hdr.b_pabd, buf->b_data, 2156 HDR_GET_PSIZE(hdr), HDR_GET_LSIZE(hdr), 2157 &hdr->b_complevel); 2158 2159 /* 2160 * Absent hardware errors or software bugs, this should 2161 * be impossible, but log it anyway so we can debug it. 2162 */ 2163 if (error != 0) { 2164 zfs_dbgmsg( 2165 "hdr %px, compress %d, psize %d, lsize %d", 2166 hdr, arc_hdr_get_compress(hdr), 2167 HDR_GET_PSIZE(hdr), HDR_GET_LSIZE(hdr)); 2168 if (hash_lock != NULL) 2169 mutex_enter(hash_lock); 2170 arc_hdr_set_flags(hdr, ARC_FLAG_IO_ERROR); 2171 if (hash_lock != NULL) 2172 mutex_exit(hash_lock); 2173 return (SET_ERROR(EIO)); 2174 } 2175 } 2176 } 2177 2178 byteswap: 2179 /* Byteswap the buf's data if necessary */ 2180 if (bswap != DMU_BSWAP_NUMFUNCS) { 2181 ASSERT(!HDR_SHARED_DATA(hdr)); 2182 ASSERT3U(bswap, <, DMU_BSWAP_NUMFUNCS); 2183 dmu_ot_byteswap[bswap].ob_func(buf->b_data, HDR_GET_LSIZE(hdr)); 2184 } 2185 2186 /* Compute the hdr's checksum if necessary */ 2187 arc_cksum_compute(buf); 2188 2189 return (0); 2190 } 2191 2192 /* 2193 * If this function is being called to decrypt an encrypted buffer or verify an 2194 * authenticated one, the key must be loaded and a mapping must be made 2195 * available in the keystore via spa_keystore_create_mapping() or one of its 2196 * callers. 2197 */ 2198 int 2199 arc_untransform(arc_buf_t *buf, spa_t *spa, const zbookmark_phys_t *zb, 2200 boolean_t in_place) 2201 { 2202 int ret; 2203 arc_fill_flags_t flags = 0; 2204 2205 if (in_place) 2206 flags |= ARC_FILL_IN_PLACE; 2207 2208 ret = arc_buf_fill(buf, spa, zb, flags); 2209 if (ret == ECKSUM) { 2210 /* 2211 * Convert authentication and decryption errors to EIO 2212 * (and generate an ereport) before leaving the ARC. 2213 */ 2214 ret = SET_ERROR(EIO); 2215 spa_log_error(spa, zb); 2216 (void) zfs_ereport_post(FM_EREPORT_ZFS_AUTHENTICATION, 2217 spa, NULL, zb, NULL, 0); 2218 } 2219 2220 return (ret); 2221 } 2222 2223 /* 2224 * Increment the amount of evictable space in the arc_state_t's refcount. 2225 * We account for the space used by the hdr and the arc buf individually 2226 * so that we can add and remove them from the refcount individually. 2227 */ 2228 static void 2229 arc_evictable_space_increment(arc_buf_hdr_t *hdr, arc_state_t *state) 2230 { 2231 arc_buf_contents_t type = arc_buf_type(hdr); 2232 2233 ASSERT(HDR_HAS_L1HDR(hdr)); 2234 2235 if (GHOST_STATE(state)) { 2236 ASSERT0(hdr->b_l1hdr.b_bufcnt); 2237 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 2238 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2239 ASSERT(!HDR_HAS_RABD(hdr)); 2240 (void) zfs_refcount_add_many(&state->arcs_esize[type], 2241 HDR_GET_LSIZE(hdr), hdr); 2242 return; 2243 } 2244 2245 if (hdr->b_l1hdr.b_pabd != NULL) { 2246 (void) zfs_refcount_add_many(&state->arcs_esize[type], 2247 arc_hdr_size(hdr), hdr); 2248 } 2249 if (HDR_HAS_RABD(hdr)) { 2250 (void) zfs_refcount_add_many(&state->arcs_esize[type], 2251 HDR_GET_PSIZE(hdr), hdr); 2252 } 2253 2254 for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; 2255 buf = buf->b_next) { 2256 if (arc_buf_is_shared(buf)) 2257 continue; 2258 (void) zfs_refcount_add_many(&state->arcs_esize[type], 2259 arc_buf_size(buf), buf); 2260 } 2261 } 2262 2263 /* 2264 * Decrement the amount of evictable space in the arc_state_t's refcount. 2265 * We account for the space used by the hdr and the arc buf individually 2266 * so that we can add and remove them from the refcount individually. 2267 */ 2268 static void 2269 arc_evictable_space_decrement(arc_buf_hdr_t *hdr, arc_state_t *state) 2270 { 2271 arc_buf_contents_t type = arc_buf_type(hdr); 2272 2273 ASSERT(HDR_HAS_L1HDR(hdr)); 2274 2275 if (GHOST_STATE(state)) { 2276 ASSERT0(hdr->b_l1hdr.b_bufcnt); 2277 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 2278 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2279 ASSERT(!HDR_HAS_RABD(hdr)); 2280 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2281 HDR_GET_LSIZE(hdr), hdr); 2282 return; 2283 } 2284 2285 if (hdr->b_l1hdr.b_pabd != NULL) { 2286 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2287 arc_hdr_size(hdr), hdr); 2288 } 2289 if (HDR_HAS_RABD(hdr)) { 2290 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2291 HDR_GET_PSIZE(hdr), hdr); 2292 } 2293 2294 for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; 2295 buf = buf->b_next) { 2296 if (arc_buf_is_shared(buf)) 2297 continue; 2298 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2299 arc_buf_size(buf), buf); 2300 } 2301 } 2302 2303 /* 2304 * Add a reference to this hdr indicating that someone is actively 2305 * referencing that memory. When the refcount transitions from 0 to 1, 2306 * we remove it from the respective arc_state_t list to indicate that 2307 * it is not evictable. 2308 */ 2309 static void 2310 add_reference(arc_buf_hdr_t *hdr, const void *tag) 2311 { 2312 arc_state_t *state = hdr->b_l1hdr.b_state; 2313 2314 ASSERT(HDR_HAS_L1HDR(hdr)); 2315 if (!HDR_EMPTY(hdr) && !MUTEX_HELD(HDR_LOCK(hdr))) { 2316 ASSERT(state == arc_anon); 2317 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 2318 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 2319 } 2320 2321 if ((zfs_refcount_add(&hdr->b_l1hdr.b_refcnt, tag) == 1) && 2322 state != arc_anon && state != arc_l2c_only) { 2323 /* We don't use the L2-only state list. */ 2324 multilist_remove(&state->arcs_list[arc_buf_type(hdr)], hdr); 2325 arc_evictable_space_decrement(hdr, state); 2326 } 2327 } 2328 2329 /* 2330 * Remove a reference from this hdr. When the reference transitions from 2331 * 1 to 0 and we're not anonymous, then we add this hdr to the arc_state_t's 2332 * list making it eligible for eviction. 2333 */ 2334 static int 2335 remove_reference(arc_buf_hdr_t *hdr, const void *tag) 2336 { 2337 int cnt; 2338 arc_state_t *state = hdr->b_l1hdr.b_state; 2339 2340 ASSERT(HDR_HAS_L1HDR(hdr)); 2341 ASSERT(state == arc_anon || MUTEX_HELD(HDR_LOCK(hdr))); 2342 ASSERT(!GHOST_STATE(state)); /* arc_l2c_only counts as a ghost. */ 2343 2344 if ((cnt = zfs_refcount_remove(&hdr->b_l1hdr.b_refcnt, tag)) != 0) 2345 return (cnt); 2346 2347 if (state == arc_anon) { 2348 arc_hdr_destroy(hdr); 2349 return (0); 2350 } 2351 if (state == arc_uncached && !HDR_PREFETCH(hdr)) { 2352 arc_change_state(arc_anon, hdr); 2353 arc_hdr_destroy(hdr); 2354 return (0); 2355 } 2356 multilist_insert(&state->arcs_list[arc_buf_type(hdr)], hdr); 2357 arc_evictable_space_increment(hdr, state); 2358 return (0); 2359 } 2360 2361 /* 2362 * Returns detailed information about a specific arc buffer. When the 2363 * state_index argument is set the function will calculate the arc header 2364 * list position for its arc state. Since this requires a linear traversal 2365 * callers are strongly encourage not to do this. However, it can be helpful 2366 * for targeted analysis so the functionality is provided. 2367 */ 2368 void 2369 arc_buf_info(arc_buf_t *ab, arc_buf_info_t *abi, int state_index) 2370 { 2371 (void) state_index; 2372 arc_buf_hdr_t *hdr = ab->b_hdr; 2373 l1arc_buf_hdr_t *l1hdr = NULL; 2374 l2arc_buf_hdr_t *l2hdr = NULL; 2375 arc_state_t *state = NULL; 2376 2377 memset(abi, 0, sizeof (arc_buf_info_t)); 2378 2379 if (hdr == NULL) 2380 return; 2381 2382 abi->abi_flags = hdr->b_flags; 2383 2384 if (HDR_HAS_L1HDR(hdr)) { 2385 l1hdr = &hdr->b_l1hdr; 2386 state = l1hdr->b_state; 2387 } 2388 if (HDR_HAS_L2HDR(hdr)) 2389 l2hdr = &hdr->b_l2hdr; 2390 2391 if (l1hdr) { 2392 abi->abi_bufcnt = l1hdr->b_bufcnt; 2393 abi->abi_access = l1hdr->b_arc_access; 2394 abi->abi_mru_hits = l1hdr->b_mru_hits; 2395 abi->abi_mru_ghost_hits = l1hdr->b_mru_ghost_hits; 2396 abi->abi_mfu_hits = l1hdr->b_mfu_hits; 2397 abi->abi_mfu_ghost_hits = l1hdr->b_mfu_ghost_hits; 2398 abi->abi_holds = zfs_refcount_count(&l1hdr->b_refcnt); 2399 } 2400 2401 if (l2hdr) { 2402 abi->abi_l2arc_dattr = l2hdr->b_daddr; 2403 abi->abi_l2arc_hits = l2hdr->b_hits; 2404 } 2405 2406 abi->abi_state_type = state ? state->arcs_state : ARC_STATE_ANON; 2407 abi->abi_state_contents = arc_buf_type(hdr); 2408 abi->abi_size = arc_hdr_size(hdr); 2409 } 2410 2411 /* 2412 * Move the supplied buffer to the indicated state. The hash lock 2413 * for the buffer must be held by the caller. 2414 */ 2415 static void 2416 arc_change_state(arc_state_t *new_state, arc_buf_hdr_t *hdr) 2417 { 2418 arc_state_t *old_state; 2419 int64_t refcnt; 2420 uint32_t bufcnt; 2421 boolean_t update_old, update_new; 2422 arc_buf_contents_t buftype = arc_buf_type(hdr); 2423 2424 /* 2425 * We almost always have an L1 hdr here, since we call arc_hdr_realloc() 2426 * in arc_read() when bringing a buffer out of the L2ARC. However, the 2427 * L1 hdr doesn't always exist when we change state to arc_anon before 2428 * destroying a header, in which case reallocating to add the L1 hdr is 2429 * pointless. 2430 */ 2431 if (HDR_HAS_L1HDR(hdr)) { 2432 old_state = hdr->b_l1hdr.b_state; 2433 refcnt = zfs_refcount_count(&hdr->b_l1hdr.b_refcnt); 2434 bufcnt = hdr->b_l1hdr.b_bufcnt; 2435 update_old = (bufcnt > 0 || hdr->b_l1hdr.b_pabd != NULL || 2436 HDR_HAS_RABD(hdr)); 2437 2438 IMPLY(GHOST_STATE(old_state), bufcnt == 0); 2439 IMPLY(GHOST_STATE(new_state), bufcnt == 0); 2440 IMPLY(GHOST_STATE(old_state), hdr->b_l1hdr.b_buf == NULL); 2441 IMPLY(GHOST_STATE(new_state), hdr->b_l1hdr.b_buf == NULL); 2442 IMPLY(old_state == arc_anon, bufcnt <= 1); 2443 } else { 2444 old_state = arc_l2c_only; 2445 refcnt = 0; 2446 bufcnt = 0; 2447 update_old = B_FALSE; 2448 } 2449 update_new = update_old; 2450 if (GHOST_STATE(old_state)) 2451 update_old = B_TRUE; 2452 if (GHOST_STATE(new_state)) 2453 update_new = B_TRUE; 2454 2455 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 2456 ASSERT3P(new_state, !=, old_state); 2457 2458 /* 2459 * If this buffer is evictable, transfer it from the 2460 * old state list to the new state list. 2461 */ 2462 if (refcnt == 0) { 2463 if (old_state != arc_anon && old_state != arc_l2c_only) { 2464 ASSERT(HDR_HAS_L1HDR(hdr)); 2465 /* remove_reference() saves on insert. */ 2466 if (multilist_link_active(&hdr->b_l1hdr.b_arc_node)) { 2467 multilist_remove(&old_state->arcs_list[buftype], 2468 hdr); 2469 arc_evictable_space_decrement(hdr, old_state); 2470 } 2471 } 2472 if (new_state != arc_anon && new_state != arc_l2c_only) { 2473 /* 2474 * An L1 header always exists here, since if we're 2475 * moving to some L1-cached state (i.e. not l2c_only or 2476 * anonymous), we realloc the header to add an L1hdr 2477 * beforehand. 2478 */ 2479 ASSERT(HDR_HAS_L1HDR(hdr)); 2480 multilist_insert(&new_state->arcs_list[buftype], hdr); 2481 arc_evictable_space_increment(hdr, new_state); 2482 } 2483 } 2484 2485 ASSERT(!HDR_EMPTY(hdr)); 2486 if (new_state == arc_anon && HDR_IN_HASH_TABLE(hdr)) 2487 buf_hash_remove(hdr); 2488 2489 /* adjust state sizes (ignore arc_l2c_only) */ 2490 2491 if (update_new && new_state != arc_l2c_only) { 2492 ASSERT(HDR_HAS_L1HDR(hdr)); 2493 if (GHOST_STATE(new_state)) { 2494 ASSERT0(bufcnt); 2495 2496 /* 2497 * When moving a header to a ghost state, we first 2498 * remove all arc buffers. Thus, we'll have a 2499 * bufcnt of zero, and no arc buffer to use for 2500 * the reference. As a result, we use the arc 2501 * header pointer for the reference. 2502 */ 2503 (void) zfs_refcount_add_many(&new_state->arcs_size, 2504 HDR_GET_LSIZE(hdr), hdr); 2505 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2506 ASSERT(!HDR_HAS_RABD(hdr)); 2507 } else { 2508 uint32_t buffers = 0; 2509 2510 /* 2511 * Each individual buffer holds a unique reference, 2512 * thus we must remove each of these references one 2513 * at a time. 2514 */ 2515 for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; 2516 buf = buf->b_next) { 2517 ASSERT3U(bufcnt, !=, 0); 2518 buffers++; 2519 2520 /* 2521 * When the arc_buf_t is sharing the data 2522 * block with the hdr, the owner of the 2523 * reference belongs to the hdr. Only 2524 * add to the refcount if the arc_buf_t is 2525 * not shared. 2526 */ 2527 if (arc_buf_is_shared(buf)) 2528 continue; 2529 2530 (void) zfs_refcount_add_many( 2531 &new_state->arcs_size, 2532 arc_buf_size(buf), buf); 2533 } 2534 ASSERT3U(bufcnt, ==, buffers); 2535 2536 if (hdr->b_l1hdr.b_pabd != NULL) { 2537 (void) zfs_refcount_add_many( 2538 &new_state->arcs_size, 2539 arc_hdr_size(hdr), hdr); 2540 } 2541 2542 if (HDR_HAS_RABD(hdr)) { 2543 (void) zfs_refcount_add_many( 2544 &new_state->arcs_size, 2545 HDR_GET_PSIZE(hdr), hdr); 2546 } 2547 } 2548 } 2549 2550 if (update_old && old_state != arc_l2c_only) { 2551 ASSERT(HDR_HAS_L1HDR(hdr)); 2552 if (GHOST_STATE(old_state)) { 2553 ASSERT0(bufcnt); 2554 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2555 ASSERT(!HDR_HAS_RABD(hdr)); 2556 2557 /* 2558 * When moving a header off of a ghost state, 2559 * the header will not contain any arc buffers. 2560 * We use the arc header pointer for the reference 2561 * which is exactly what we did when we put the 2562 * header on the ghost state. 2563 */ 2564 2565 (void) zfs_refcount_remove_many(&old_state->arcs_size, 2566 HDR_GET_LSIZE(hdr), hdr); 2567 } else { 2568 uint32_t buffers = 0; 2569 2570 /* 2571 * Each individual buffer holds a unique reference, 2572 * thus we must remove each of these references one 2573 * at a time. 2574 */ 2575 for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; 2576 buf = buf->b_next) { 2577 ASSERT3U(bufcnt, !=, 0); 2578 buffers++; 2579 2580 /* 2581 * When the arc_buf_t is sharing the data 2582 * block with the hdr, the owner of the 2583 * reference belongs to the hdr. Only 2584 * add to the refcount if the arc_buf_t is 2585 * not shared. 2586 */ 2587 if (arc_buf_is_shared(buf)) 2588 continue; 2589 2590 (void) zfs_refcount_remove_many( 2591 &old_state->arcs_size, arc_buf_size(buf), 2592 buf); 2593 } 2594 ASSERT3U(bufcnt, ==, buffers); 2595 ASSERT(hdr->b_l1hdr.b_pabd != NULL || 2596 HDR_HAS_RABD(hdr)); 2597 2598 if (hdr->b_l1hdr.b_pabd != NULL) { 2599 (void) zfs_refcount_remove_many( 2600 &old_state->arcs_size, arc_hdr_size(hdr), 2601 hdr); 2602 } 2603 2604 if (HDR_HAS_RABD(hdr)) { 2605 (void) zfs_refcount_remove_many( 2606 &old_state->arcs_size, HDR_GET_PSIZE(hdr), 2607 hdr); 2608 } 2609 } 2610 } 2611 2612 if (HDR_HAS_L1HDR(hdr)) { 2613 hdr->b_l1hdr.b_state = new_state; 2614 2615 if (HDR_HAS_L2HDR(hdr) && new_state != arc_l2c_only) { 2616 l2arc_hdr_arcstats_decrement_state(hdr); 2617 hdr->b_l2hdr.b_arcs_state = new_state->arcs_state; 2618 l2arc_hdr_arcstats_increment_state(hdr); 2619 } 2620 } 2621 } 2622 2623 void 2624 arc_space_consume(uint64_t space, arc_space_type_t type) 2625 { 2626 ASSERT(type >= 0 && type < ARC_SPACE_NUMTYPES); 2627 2628 switch (type) { 2629 default: 2630 break; 2631 case ARC_SPACE_DATA: 2632 ARCSTAT_INCR(arcstat_data_size, space); 2633 break; 2634 case ARC_SPACE_META: 2635 ARCSTAT_INCR(arcstat_metadata_size, space); 2636 break; 2637 case ARC_SPACE_BONUS: 2638 ARCSTAT_INCR(arcstat_bonus_size, space); 2639 break; 2640 case ARC_SPACE_DNODE: 2641 aggsum_add(&arc_sums.arcstat_dnode_size, space); 2642 break; 2643 case ARC_SPACE_DBUF: 2644 ARCSTAT_INCR(arcstat_dbuf_size, space); 2645 break; 2646 case ARC_SPACE_HDRS: 2647 ARCSTAT_INCR(arcstat_hdr_size, space); 2648 break; 2649 case ARC_SPACE_L2HDRS: 2650 aggsum_add(&arc_sums.arcstat_l2_hdr_size, space); 2651 break; 2652 case ARC_SPACE_ABD_CHUNK_WASTE: 2653 /* 2654 * Note: this includes space wasted by all scatter ABD's, not 2655 * just those allocated by the ARC. But the vast majority of 2656 * scatter ABD's come from the ARC, because other users are 2657 * very short-lived. 2658 */ 2659 ARCSTAT_INCR(arcstat_abd_chunk_waste_size, space); 2660 break; 2661 } 2662 2663 if (type != ARC_SPACE_DATA && type != ARC_SPACE_ABD_CHUNK_WASTE) 2664 aggsum_add(&arc_sums.arcstat_meta_used, space); 2665 2666 aggsum_add(&arc_sums.arcstat_size, space); 2667 } 2668 2669 void 2670 arc_space_return(uint64_t space, arc_space_type_t type) 2671 { 2672 ASSERT(type >= 0 && type < ARC_SPACE_NUMTYPES); 2673 2674 switch (type) { 2675 default: 2676 break; 2677 case ARC_SPACE_DATA: 2678 ARCSTAT_INCR(arcstat_data_size, -space); 2679 break; 2680 case ARC_SPACE_META: 2681 ARCSTAT_INCR(arcstat_metadata_size, -space); 2682 break; 2683 case ARC_SPACE_BONUS: 2684 ARCSTAT_INCR(arcstat_bonus_size, -space); 2685 break; 2686 case ARC_SPACE_DNODE: 2687 aggsum_add(&arc_sums.arcstat_dnode_size, -space); 2688 break; 2689 case ARC_SPACE_DBUF: 2690 ARCSTAT_INCR(arcstat_dbuf_size, -space); 2691 break; 2692 case ARC_SPACE_HDRS: 2693 ARCSTAT_INCR(arcstat_hdr_size, -space); 2694 break; 2695 case ARC_SPACE_L2HDRS: 2696 aggsum_add(&arc_sums.arcstat_l2_hdr_size, -space); 2697 break; 2698 case ARC_SPACE_ABD_CHUNK_WASTE: 2699 ARCSTAT_INCR(arcstat_abd_chunk_waste_size, -space); 2700 break; 2701 } 2702 2703 if (type != ARC_SPACE_DATA && type != ARC_SPACE_ABD_CHUNK_WASTE) { 2704 ASSERT(aggsum_compare(&arc_sums.arcstat_meta_used, 2705 space) >= 0); 2706 ARCSTAT_MAX(arcstat_meta_max, 2707 aggsum_upper_bound(&arc_sums.arcstat_meta_used)); 2708 aggsum_add(&arc_sums.arcstat_meta_used, -space); 2709 } 2710 2711 ASSERT(aggsum_compare(&arc_sums.arcstat_size, space) >= 0); 2712 aggsum_add(&arc_sums.arcstat_size, -space); 2713 } 2714 2715 /* 2716 * Given a hdr and a buf, returns whether that buf can share its b_data buffer 2717 * with the hdr's b_pabd. 2718 */ 2719 static boolean_t 2720 arc_can_share(arc_buf_hdr_t *hdr, arc_buf_t *buf) 2721 { 2722 /* 2723 * The criteria for sharing a hdr's data are: 2724 * 1. the buffer is not encrypted 2725 * 2. the hdr's compression matches the buf's compression 2726 * 3. the hdr doesn't need to be byteswapped 2727 * 4. the hdr isn't already being shared 2728 * 5. the buf is either compressed or it is the last buf in the hdr list 2729 * 2730 * Criterion #5 maintains the invariant that shared uncompressed 2731 * bufs must be the final buf in the hdr's b_buf list. Reading this, you 2732 * might ask, "if a compressed buf is allocated first, won't that be the 2733 * last thing in the list?", but in that case it's impossible to create 2734 * a shared uncompressed buf anyway (because the hdr must be compressed 2735 * to have the compressed buf). You might also think that #3 is 2736 * sufficient to make this guarantee, however it's possible 2737 * (specifically in the rare L2ARC write race mentioned in 2738 * arc_buf_alloc_impl()) there will be an existing uncompressed buf that 2739 * is shareable, but wasn't at the time of its allocation. Rather than 2740 * allow a new shared uncompressed buf to be created and then shuffle 2741 * the list around to make it the last element, this simply disallows 2742 * sharing if the new buf isn't the first to be added. 2743 */ 2744 ASSERT3P(buf->b_hdr, ==, hdr); 2745 boolean_t hdr_compressed = 2746 arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF; 2747 boolean_t buf_compressed = ARC_BUF_COMPRESSED(buf) != 0; 2748 return (!ARC_BUF_ENCRYPTED(buf) && 2749 buf_compressed == hdr_compressed && 2750 hdr->b_l1hdr.b_byteswap == DMU_BSWAP_NUMFUNCS && 2751 !HDR_SHARED_DATA(hdr) && 2752 (ARC_BUF_LAST(buf) || ARC_BUF_COMPRESSED(buf))); 2753 } 2754 2755 /* 2756 * Allocate a buf for this hdr. If you care about the data that's in the hdr, 2757 * or if you want a compressed buffer, pass those flags in. Returns 0 if the 2758 * copy was made successfully, or an error code otherwise. 2759 */ 2760 static int 2761 arc_buf_alloc_impl(arc_buf_hdr_t *hdr, spa_t *spa, const zbookmark_phys_t *zb, 2762 const void *tag, boolean_t encrypted, boolean_t compressed, 2763 boolean_t noauth, boolean_t fill, arc_buf_t **ret) 2764 { 2765 arc_buf_t *buf; 2766 arc_fill_flags_t flags = ARC_FILL_LOCKED; 2767 2768 ASSERT(HDR_HAS_L1HDR(hdr)); 2769 ASSERT3U(HDR_GET_LSIZE(hdr), >, 0); 2770 VERIFY(hdr->b_type == ARC_BUFC_DATA || 2771 hdr->b_type == ARC_BUFC_METADATA); 2772 ASSERT3P(ret, !=, NULL); 2773 ASSERT3P(*ret, ==, NULL); 2774 IMPLY(encrypted, compressed); 2775 2776 buf = *ret = kmem_cache_alloc(buf_cache, KM_PUSHPAGE); 2777 buf->b_hdr = hdr; 2778 buf->b_data = NULL; 2779 buf->b_next = hdr->b_l1hdr.b_buf; 2780 buf->b_flags = 0; 2781 2782 add_reference(hdr, tag); 2783 2784 /* 2785 * We're about to change the hdr's b_flags. We must either 2786 * hold the hash_lock or be undiscoverable. 2787 */ 2788 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 2789 2790 /* 2791 * Only honor requests for compressed bufs if the hdr is actually 2792 * compressed. This must be overridden if the buffer is encrypted since 2793 * encrypted buffers cannot be decompressed. 2794 */ 2795 if (encrypted) { 2796 buf->b_flags |= ARC_BUF_FLAG_COMPRESSED; 2797 buf->b_flags |= ARC_BUF_FLAG_ENCRYPTED; 2798 flags |= ARC_FILL_COMPRESSED | ARC_FILL_ENCRYPTED; 2799 } else if (compressed && 2800 arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF) { 2801 buf->b_flags |= ARC_BUF_FLAG_COMPRESSED; 2802 flags |= ARC_FILL_COMPRESSED; 2803 } 2804 2805 if (noauth) { 2806 ASSERT0(encrypted); 2807 flags |= ARC_FILL_NOAUTH; 2808 } 2809 2810 /* 2811 * If the hdr's data can be shared then we share the data buffer and 2812 * set the appropriate bit in the hdr's b_flags to indicate the hdr is 2813 * sharing it's b_pabd with the arc_buf_t. Otherwise, we allocate a new 2814 * buffer to store the buf's data. 2815 * 2816 * There are two additional restrictions here because we're sharing 2817 * hdr -> buf instead of the usual buf -> hdr. First, the hdr can't be 2818 * actively involved in an L2ARC write, because if this buf is used by 2819 * an arc_write() then the hdr's data buffer will be released when the 2820 * write completes, even though the L2ARC write might still be using it. 2821 * Second, the hdr's ABD must be linear so that the buf's user doesn't 2822 * need to be ABD-aware. It must be allocated via 2823 * zio_[data_]buf_alloc(), not as a page, because we need to be able 2824 * to abd_release_ownership_of_buf(), which isn't allowed on "linear 2825 * page" buffers because the ABD code needs to handle freeing them 2826 * specially. 2827 */ 2828 boolean_t can_share = arc_can_share(hdr, buf) && 2829 !HDR_L2_WRITING(hdr) && 2830 hdr->b_l1hdr.b_pabd != NULL && 2831 abd_is_linear(hdr->b_l1hdr.b_pabd) && 2832 !abd_is_linear_page(hdr->b_l1hdr.b_pabd); 2833 2834 /* Set up b_data and sharing */ 2835 if (can_share) { 2836 buf->b_data = abd_to_buf(hdr->b_l1hdr.b_pabd); 2837 buf->b_flags |= ARC_BUF_FLAG_SHARED; 2838 arc_hdr_set_flags(hdr, ARC_FLAG_SHARED_DATA); 2839 } else { 2840 buf->b_data = 2841 arc_get_data_buf(hdr, arc_buf_size(buf), buf); 2842 ARCSTAT_INCR(arcstat_overhead_size, arc_buf_size(buf)); 2843 } 2844 VERIFY3P(buf->b_data, !=, NULL); 2845 2846 hdr->b_l1hdr.b_buf = buf; 2847 hdr->b_l1hdr.b_bufcnt += 1; 2848 if (encrypted) 2849 hdr->b_crypt_hdr.b_ebufcnt += 1; 2850 2851 /* 2852 * If the user wants the data from the hdr, we need to either copy or 2853 * decompress the data. 2854 */ 2855 if (fill) { 2856 ASSERT3P(zb, !=, NULL); 2857 return (arc_buf_fill(buf, spa, zb, flags)); 2858 } 2859 2860 return (0); 2861 } 2862 2863 static const char *arc_onloan_tag = "onloan"; 2864 2865 static inline void 2866 arc_loaned_bytes_update(int64_t delta) 2867 { 2868 atomic_add_64(&arc_loaned_bytes, delta); 2869 2870 /* assert that it did not wrap around */ 2871 ASSERT3S(atomic_add_64_nv(&arc_loaned_bytes, 0), >=, 0); 2872 } 2873 2874 /* 2875 * Loan out an anonymous arc buffer. Loaned buffers are not counted as in 2876 * flight data by arc_tempreserve_space() until they are "returned". Loaned 2877 * buffers must be returned to the arc before they can be used by the DMU or 2878 * freed. 2879 */ 2880 arc_buf_t * 2881 arc_loan_buf(spa_t *spa, boolean_t is_metadata, int size) 2882 { 2883 arc_buf_t *buf = arc_alloc_buf(spa, arc_onloan_tag, 2884 is_metadata ? ARC_BUFC_METADATA : ARC_BUFC_DATA, size); 2885 2886 arc_loaned_bytes_update(arc_buf_size(buf)); 2887 2888 return (buf); 2889 } 2890 2891 arc_buf_t * 2892 arc_loan_compressed_buf(spa_t *spa, uint64_t psize, uint64_t lsize, 2893 enum zio_compress compression_type, uint8_t complevel) 2894 { 2895 arc_buf_t *buf = arc_alloc_compressed_buf(spa, arc_onloan_tag, 2896 psize, lsize, compression_type, complevel); 2897 2898 arc_loaned_bytes_update(arc_buf_size(buf)); 2899 2900 return (buf); 2901 } 2902 2903 arc_buf_t * 2904 arc_loan_raw_buf(spa_t *spa, uint64_t dsobj, boolean_t byteorder, 2905 const uint8_t *salt, const uint8_t *iv, const uint8_t *mac, 2906 dmu_object_type_t ot, uint64_t psize, uint64_t lsize, 2907 enum zio_compress compression_type, uint8_t complevel) 2908 { 2909 arc_buf_t *buf = arc_alloc_raw_buf(spa, arc_onloan_tag, dsobj, 2910 byteorder, salt, iv, mac, ot, psize, lsize, compression_type, 2911 complevel); 2912 2913 atomic_add_64(&arc_loaned_bytes, psize); 2914 return (buf); 2915 } 2916 2917 2918 /* 2919 * Return a loaned arc buffer to the arc. 2920 */ 2921 void 2922 arc_return_buf(arc_buf_t *buf, const void *tag) 2923 { 2924 arc_buf_hdr_t *hdr = buf->b_hdr; 2925 2926 ASSERT3P(buf->b_data, !=, NULL); 2927 ASSERT(HDR_HAS_L1HDR(hdr)); 2928 (void) zfs_refcount_add(&hdr->b_l1hdr.b_refcnt, tag); 2929 (void) zfs_refcount_remove(&hdr->b_l1hdr.b_refcnt, arc_onloan_tag); 2930 2931 arc_loaned_bytes_update(-arc_buf_size(buf)); 2932 } 2933 2934 /* Detach an arc_buf from a dbuf (tag) */ 2935 void 2936 arc_loan_inuse_buf(arc_buf_t *buf, const void *tag) 2937 { 2938 arc_buf_hdr_t *hdr = buf->b_hdr; 2939 2940 ASSERT3P(buf->b_data, !=, NULL); 2941 ASSERT(HDR_HAS_L1HDR(hdr)); 2942 (void) zfs_refcount_add(&hdr->b_l1hdr.b_refcnt, arc_onloan_tag); 2943 (void) zfs_refcount_remove(&hdr->b_l1hdr.b_refcnt, tag); 2944 2945 arc_loaned_bytes_update(arc_buf_size(buf)); 2946 } 2947 2948 static void 2949 l2arc_free_abd_on_write(abd_t *abd, size_t size, arc_buf_contents_t type) 2950 { 2951 l2arc_data_free_t *df = kmem_alloc(sizeof (*df), KM_SLEEP); 2952 2953 df->l2df_abd = abd; 2954 df->l2df_size = size; 2955 df->l2df_type = type; 2956 mutex_enter(&l2arc_free_on_write_mtx); 2957 list_insert_head(l2arc_free_on_write, df); 2958 mutex_exit(&l2arc_free_on_write_mtx); 2959 } 2960 2961 static void 2962 arc_hdr_free_on_write(arc_buf_hdr_t *hdr, boolean_t free_rdata) 2963 { 2964 arc_state_t *state = hdr->b_l1hdr.b_state; 2965 arc_buf_contents_t type = arc_buf_type(hdr); 2966 uint64_t size = (free_rdata) ? HDR_GET_PSIZE(hdr) : arc_hdr_size(hdr); 2967 2968 /* protected by hash lock, if in the hash table */ 2969 if (multilist_link_active(&hdr->b_l1hdr.b_arc_node)) { 2970 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 2971 ASSERT(state != arc_anon && state != arc_l2c_only); 2972 2973 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2974 size, hdr); 2975 } 2976 (void) zfs_refcount_remove_many(&state->arcs_size, size, hdr); 2977 if (type == ARC_BUFC_METADATA) { 2978 arc_space_return(size, ARC_SPACE_META); 2979 } else { 2980 ASSERT(type == ARC_BUFC_DATA); 2981 arc_space_return(size, ARC_SPACE_DATA); 2982 } 2983 2984 if (free_rdata) { 2985 l2arc_free_abd_on_write(hdr->b_crypt_hdr.b_rabd, size, type); 2986 } else { 2987 l2arc_free_abd_on_write(hdr->b_l1hdr.b_pabd, size, type); 2988 } 2989 } 2990 2991 /* 2992 * Share the arc_buf_t's data with the hdr. Whenever we are sharing the 2993 * data buffer, we transfer the refcount ownership to the hdr and update 2994 * the appropriate kstats. 2995 */ 2996 static void 2997 arc_share_buf(arc_buf_hdr_t *hdr, arc_buf_t *buf) 2998 { 2999 ASSERT(arc_can_share(hdr, buf)); 3000 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 3001 ASSERT(!ARC_BUF_ENCRYPTED(buf)); 3002 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 3003 3004 /* 3005 * Start sharing the data buffer. We transfer the 3006 * refcount ownership to the hdr since it always owns 3007 * the refcount whenever an arc_buf_t is shared. 3008 */ 3009 zfs_refcount_transfer_ownership_many(&hdr->b_l1hdr.b_state->arcs_size, 3010 arc_hdr_size(hdr), buf, hdr); 3011 hdr->b_l1hdr.b_pabd = abd_get_from_buf(buf->b_data, arc_buf_size(buf)); 3012 abd_take_ownership_of_buf(hdr->b_l1hdr.b_pabd, 3013 HDR_ISTYPE_METADATA(hdr)); 3014 arc_hdr_set_flags(hdr, ARC_FLAG_SHARED_DATA); 3015 buf->b_flags |= ARC_BUF_FLAG_SHARED; 3016 3017 /* 3018 * Since we've transferred ownership to the hdr we need 3019 * to increment its compressed and uncompressed kstats and 3020 * decrement the overhead size. 3021 */ 3022 ARCSTAT_INCR(arcstat_compressed_size, arc_hdr_size(hdr)); 3023 ARCSTAT_INCR(arcstat_uncompressed_size, HDR_GET_LSIZE(hdr)); 3024 ARCSTAT_INCR(arcstat_overhead_size, -arc_buf_size(buf)); 3025 } 3026 3027 static void 3028 arc_unshare_buf(arc_buf_hdr_t *hdr, arc_buf_t *buf) 3029 { 3030 ASSERT(arc_buf_is_shared(buf)); 3031 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 3032 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 3033 3034 /* 3035 * We are no longer sharing this buffer so we need 3036 * to transfer its ownership to the rightful owner. 3037 */ 3038 zfs_refcount_transfer_ownership_many(&hdr->b_l1hdr.b_state->arcs_size, 3039 arc_hdr_size(hdr), hdr, buf); 3040 arc_hdr_clear_flags(hdr, ARC_FLAG_SHARED_DATA); 3041 abd_release_ownership_of_buf(hdr->b_l1hdr.b_pabd); 3042 abd_free(hdr->b_l1hdr.b_pabd); 3043 hdr->b_l1hdr.b_pabd = NULL; 3044 buf->b_flags &= ~ARC_BUF_FLAG_SHARED; 3045 3046 /* 3047 * Since the buffer is no longer shared between 3048 * the arc buf and the hdr, count it as overhead. 3049 */ 3050 ARCSTAT_INCR(arcstat_compressed_size, -arc_hdr_size(hdr)); 3051 ARCSTAT_INCR(arcstat_uncompressed_size, -HDR_GET_LSIZE(hdr)); 3052 ARCSTAT_INCR(arcstat_overhead_size, arc_buf_size(buf)); 3053 } 3054 3055 /* 3056 * Remove an arc_buf_t from the hdr's buf list and return the last 3057 * arc_buf_t on the list. If no buffers remain on the list then return 3058 * NULL. 3059 */ 3060 static arc_buf_t * 3061 arc_buf_remove(arc_buf_hdr_t *hdr, arc_buf_t *buf) 3062 { 3063 ASSERT(HDR_HAS_L1HDR(hdr)); 3064 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 3065 3066 arc_buf_t **bufp = &hdr->b_l1hdr.b_buf; 3067 arc_buf_t *lastbuf = NULL; 3068 3069 /* 3070 * Remove the buf from the hdr list and locate the last 3071 * remaining buffer on the list. 3072 */ 3073 while (*bufp != NULL) { 3074 if (*bufp == buf) 3075 *bufp = buf->b_next; 3076 3077 /* 3078 * If we've removed a buffer in the middle of 3079 * the list then update the lastbuf and update 3080 * bufp. 3081 */ 3082 if (*bufp != NULL) { 3083 lastbuf = *bufp; 3084 bufp = &(*bufp)->b_next; 3085 } 3086 } 3087 buf->b_next = NULL; 3088 ASSERT3P(lastbuf, !=, buf); 3089 IMPLY(hdr->b_l1hdr.b_bufcnt > 0, lastbuf != NULL); 3090 IMPLY(hdr->b_l1hdr.b_bufcnt > 0, hdr->b_l1hdr.b_buf != NULL); 3091 IMPLY(lastbuf != NULL, ARC_BUF_LAST(lastbuf)); 3092 3093 return (lastbuf); 3094 } 3095 3096 /* 3097 * Free up buf->b_data and pull the arc_buf_t off of the arc_buf_hdr_t's 3098 * list and free it. 3099 */ 3100 static void 3101 arc_buf_destroy_impl(arc_buf_t *buf) 3102 { 3103 arc_buf_hdr_t *hdr = buf->b_hdr; 3104 3105 /* 3106 * Free up the data associated with the buf but only if we're not 3107 * sharing this with the hdr. If we are sharing it with the hdr, the 3108 * hdr is responsible for doing the free. 3109 */ 3110 if (buf->b_data != NULL) { 3111 /* 3112 * We're about to change the hdr's b_flags. We must either 3113 * hold the hash_lock or be undiscoverable. 3114 */ 3115 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 3116 3117 arc_cksum_verify(buf); 3118 arc_buf_unwatch(buf); 3119 3120 if (arc_buf_is_shared(buf)) { 3121 arc_hdr_clear_flags(hdr, ARC_FLAG_SHARED_DATA); 3122 } else { 3123 uint64_t size = arc_buf_size(buf); 3124 arc_free_data_buf(hdr, buf->b_data, size, buf); 3125 ARCSTAT_INCR(arcstat_overhead_size, -size); 3126 } 3127 buf->b_data = NULL; 3128 3129 ASSERT(hdr->b_l1hdr.b_bufcnt > 0); 3130 hdr->b_l1hdr.b_bufcnt -= 1; 3131 3132 if (ARC_BUF_ENCRYPTED(buf)) { 3133 hdr->b_crypt_hdr.b_ebufcnt -= 1; 3134 3135 /* 3136 * If we have no more encrypted buffers and we've 3137 * already gotten a copy of the decrypted data we can 3138 * free b_rabd to save some space. 3139 */ 3140 if (hdr->b_crypt_hdr.b_ebufcnt == 0 && 3141 HDR_HAS_RABD(hdr) && hdr->b_l1hdr.b_pabd != NULL && 3142 !HDR_IO_IN_PROGRESS(hdr)) { 3143 arc_hdr_free_abd(hdr, B_TRUE); 3144 } 3145 } 3146 } 3147 3148 arc_buf_t *lastbuf = arc_buf_remove(hdr, buf); 3149 3150 if (ARC_BUF_SHARED(buf) && !ARC_BUF_COMPRESSED(buf)) { 3151 /* 3152 * If the current arc_buf_t is sharing its data buffer with the 3153 * hdr, then reassign the hdr's b_pabd to share it with the new 3154 * buffer at the end of the list. The shared buffer is always 3155 * the last one on the hdr's buffer list. 3156 * 3157 * There is an equivalent case for compressed bufs, but since 3158 * they aren't guaranteed to be the last buf in the list and 3159 * that is an exceedingly rare case, we just allow that space be 3160 * wasted temporarily. We must also be careful not to share 3161 * encrypted buffers, since they cannot be shared. 3162 */ 3163 if (lastbuf != NULL && !ARC_BUF_ENCRYPTED(lastbuf)) { 3164 /* Only one buf can be shared at once */ 3165 VERIFY(!arc_buf_is_shared(lastbuf)); 3166 /* hdr is uncompressed so can't have compressed buf */ 3167 VERIFY(!ARC_BUF_COMPRESSED(lastbuf)); 3168 3169 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 3170 arc_hdr_free_abd(hdr, B_FALSE); 3171 3172 /* 3173 * We must setup a new shared block between the 3174 * last buffer and the hdr. The data would have 3175 * been allocated by the arc buf so we need to transfer 3176 * ownership to the hdr since it's now being shared. 3177 */ 3178 arc_share_buf(hdr, lastbuf); 3179 } 3180 } else if (HDR_SHARED_DATA(hdr)) { 3181 /* 3182 * Uncompressed shared buffers are always at the end 3183 * of the list. Compressed buffers don't have the 3184 * same requirements. This makes it hard to 3185 * simply assert that the lastbuf is shared so 3186 * we rely on the hdr's compression flags to determine 3187 * if we have a compressed, shared buffer. 3188 */ 3189 ASSERT3P(lastbuf, !=, NULL); 3190 ASSERT(arc_buf_is_shared(lastbuf) || 3191 arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF); 3192 } 3193 3194 /* 3195 * Free the checksum if we're removing the last uncompressed buf from 3196 * this hdr. 3197 */ 3198 if (!arc_hdr_has_uncompressed_buf(hdr)) { 3199 arc_cksum_free(hdr); 3200 } 3201 3202 /* clean up the buf */ 3203 buf->b_hdr = NULL; 3204 kmem_cache_free(buf_cache, buf); 3205 } 3206 3207 static void 3208 arc_hdr_alloc_abd(arc_buf_hdr_t *hdr, int alloc_flags) 3209 { 3210 uint64_t size; 3211 boolean_t alloc_rdata = ((alloc_flags & ARC_HDR_ALLOC_RDATA) != 0); 3212 3213 ASSERT3U(HDR_GET_LSIZE(hdr), >, 0); 3214 ASSERT(HDR_HAS_L1HDR(hdr)); 3215 ASSERT(!HDR_SHARED_DATA(hdr) || alloc_rdata); 3216 IMPLY(alloc_rdata, HDR_PROTECTED(hdr)); 3217 3218 if (alloc_rdata) { 3219 size = HDR_GET_PSIZE(hdr); 3220 ASSERT3P(hdr->b_crypt_hdr.b_rabd, ==, NULL); 3221 hdr->b_crypt_hdr.b_rabd = arc_get_data_abd(hdr, size, hdr, 3222 alloc_flags); 3223 ASSERT3P(hdr->b_crypt_hdr.b_rabd, !=, NULL); 3224 ARCSTAT_INCR(arcstat_raw_size, size); 3225 } else { 3226 size = arc_hdr_size(hdr); 3227 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 3228 hdr->b_l1hdr.b_pabd = arc_get_data_abd(hdr, size, hdr, 3229 alloc_flags); 3230 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 3231 } 3232 3233 ARCSTAT_INCR(arcstat_compressed_size, size); 3234 ARCSTAT_INCR(arcstat_uncompressed_size, HDR_GET_LSIZE(hdr)); 3235 } 3236 3237 static void 3238 arc_hdr_free_abd(arc_buf_hdr_t *hdr, boolean_t free_rdata) 3239 { 3240 uint64_t size = (free_rdata) ? HDR_GET_PSIZE(hdr) : arc_hdr_size(hdr); 3241 3242 ASSERT(HDR_HAS_L1HDR(hdr)); 3243 ASSERT(hdr->b_l1hdr.b_pabd != NULL || HDR_HAS_RABD(hdr)); 3244 IMPLY(free_rdata, HDR_HAS_RABD(hdr)); 3245 3246 /* 3247 * If the hdr is currently being written to the l2arc then 3248 * we defer freeing the data by adding it to the l2arc_free_on_write 3249 * list. The l2arc will free the data once it's finished 3250 * writing it to the l2arc device. 3251 */ 3252 if (HDR_L2_WRITING(hdr)) { 3253 arc_hdr_free_on_write(hdr, free_rdata); 3254 ARCSTAT_BUMP(arcstat_l2_free_on_write); 3255 } else if (free_rdata) { 3256 arc_free_data_abd(hdr, hdr->b_crypt_hdr.b_rabd, size, hdr); 3257 } else { 3258 arc_free_data_abd(hdr, hdr->b_l1hdr.b_pabd, size, hdr); 3259 } 3260 3261 if (free_rdata) { 3262 hdr->b_crypt_hdr.b_rabd = NULL; 3263 ARCSTAT_INCR(arcstat_raw_size, -size); 3264 } else { 3265 hdr->b_l1hdr.b_pabd = NULL; 3266 } 3267 3268 if (hdr->b_l1hdr.b_pabd == NULL && !HDR_HAS_RABD(hdr)) 3269 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_NUMFUNCS; 3270 3271 ARCSTAT_INCR(arcstat_compressed_size, -size); 3272 ARCSTAT_INCR(arcstat_uncompressed_size, -HDR_GET_LSIZE(hdr)); 3273 } 3274 3275 /* 3276 * Allocate empty anonymous ARC header. The header will get its identity 3277 * assigned and buffers attached later as part of read or write operations. 3278 * 3279 * In case of read arc_read() assigns header its identify (b_dva + b_birth), 3280 * inserts it into ARC hash to become globally visible and allocates physical 3281 * (b_pabd) or raw (b_rabd) ABD buffer to read into from disk. On disk read 3282 * completion arc_read_done() allocates ARC buffer(s) as needed, potentially 3283 * sharing one of them with the physical ABD buffer. 3284 * 3285 * In case of write arc_alloc_buf() allocates ARC buffer to be filled with 3286 * data. Then after compression and/or encryption arc_write_ready() allocates 3287 * and fills (or potentially shares) physical (b_pabd) or raw (b_rabd) ABD 3288 * buffer. On disk write completion arc_write_done() assigns the header its 3289 * new identity (b_dva + b_birth) and inserts into ARC hash. 3290 * 3291 * In case of partial overwrite the old data is read first as described. Then 3292 * arc_release() either allocates new anonymous ARC header and moves the ARC 3293 * buffer to it, or reuses the old ARC header by discarding its identity and 3294 * removing it from ARC hash. After buffer modification normal write process 3295 * follows as described. 3296 */ 3297 static arc_buf_hdr_t * 3298 arc_hdr_alloc(uint64_t spa, int32_t psize, int32_t lsize, 3299 boolean_t protected, enum zio_compress compression_type, uint8_t complevel, 3300 arc_buf_contents_t type) 3301 { 3302 arc_buf_hdr_t *hdr; 3303 3304 VERIFY(type == ARC_BUFC_DATA || type == ARC_BUFC_METADATA); 3305 if (protected) { 3306 hdr = kmem_cache_alloc(hdr_full_crypt_cache, KM_PUSHPAGE); 3307 } else { 3308 hdr = kmem_cache_alloc(hdr_full_cache, KM_PUSHPAGE); 3309 } 3310 3311 ASSERT(HDR_EMPTY(hdr)); 3312 #ifdef ZFS_DEBUG 3313 ASSERT3P(hdr->b_l1hdr.b_freeze_cksum, ==, NULL); 3314 #endif 3315 HDR_SET_PSIZE(hdr, psize); 3316 HDR_SET_LSIZE(hdr, lsize); 3317 hdr->b_spa = spa; 3318 hdr->b_type = type; 3319 hdr->b_flags = 0; 3320 arc_hdr_set_flags(hdr, arc_bufc_to_flags(type) | ARC_FLAG_HAS_L1HDR); 3321 arc_hdr_set_compress(hdr, compression_type); 3322 hdr->b_complevel = complevel; 3323 if (protected) 3324 arc_hdr_set_flags(hdr, ARC_FLAG_PROTECTED); 3325 3326 hdr->b_l1hdr.b_state = arc_anon; 3327 hdr->b_l1hdr.b_arc_access = 0; 3328 hdr->b_l1hdr.b_mru_hits = 0; 3329 hdr->b_l1hdr.b_mru_ghost_hits = 0; 3330 hdr->b_l1hdr.b_mfu_hits = 0; 3331 hdr->b_l1hdr.b_mfu_ghost_hits = 0; 3332 hdr->b_l1hdr.b_bufcnt = 0; 3333 hdr->b_l1hdr.b_buf = NULL; 3334 3335 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 3336 3337 return (hdr); 3338 } 3339 3340 /* 3341 * Transition between the two allocation states for the arc_buf_hdr struct. 3342 * The arc_buf_hdr struct can be allocated with (hdr_full_cache) or without 3343 * (hdr_l2only_cache) the fields necessary for the L1 cache - the smaller 3344 * version is used when a cache buffer is only in the L2ARC in order to reduce 3345 * memory usage. 3346 */ 3347 static arc_buf_hdr_t * 3348 arc_hdr_realloc(arc_buf_hdr_t *hdr, kmem_cache_t *old, kmem_cache_t *new) 3349 { 3350 ASSERT(HDR_HAS_L2HDR(hdr)); 3351 3352 arc_buf_hdr_t *nhdr; 3353 l2arc_dev_t *dev = hdr->b_l2hdr.b_dev; 3354 3355 ASSERT((old == hdr_full_cache && new == hdr_l2only_cache) || 3356 (old == hdr_l2only_cache && new == hdr_full_cache)); 3357 3358 /* 3359 * if the caller wanted a new full header and the header is to be 3360 * encrypted we will actually allocate the header from the full crypt 3361 * cache instead. The same applies to freeing from the old cache. 3362 */ 3363 if (HDR_PROTECTED(hdr) && new == hdr_full_cache) 3364 new = hdr_full_crypt_cache; 3365 if (HDR_PROTECTED(hdr) && old == hdr_full_cache) 3366 old = hdr_full_crypt_cache; 3367 3368 nhdr = kmem_cache_alloc(new, KM_PUSHPAGE); 3369 3370 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 3371 buf_hash_remove(hdr); 3372 3373 memcpy(nhdr, hdr, HDR_L2ONLY_SIZE); 3374 3375 if (new == hdr_full_cache || new == hdr_full_crypt_cache) { 3376 arc_hdr_set_flags(nhdr, ARC_FLAG_HAS_L1HDR); 3377 /* 3378 * arc_access and arc_change_state need to be aware that a 3379 * header has just come out of L2ARC, so we set its state to 3380 * l2c_only even though it's about to change. 3381 */ 3382 nhdr->b_l1hdr.b_state = arc_l2c_only; 3383 3384 /* Verify previous threads set to NULL before freeing */ 3385 ASSERT3P(nhdr->b_l1hdr.b_pabd, ==, NULL); 3386 ASSERT(!HDR_HAS_RABD(hdr)); 3387 } else { 3388 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 3389 ASSERT0(hdr->b_l1hdr.b_bufcnt); 3390 #ifdef ZFS_DEBUG 3391 ASSERT3P(hdr->b_l1hdr.b_freeze_cksum, ==, NULL); 3392 #endif 3393 3394 /* 3395 * If we've reached here, We must have been called from 3396 * arc_evict_hdr(), as such we should have already been 3397 * removed from any ghost list we were previously on 3398 * (which protects us from racing with arc_evict_state), 3399 * thus no locking is needed during this check. 3400 */ 3401 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 3402 3403 /* 3404 * A buffer must not be moved into the arc_l2c_only 3405 * state if it's not finished being written out to the 3406 * l2arc device. Otherwise, the b_l1hdr.b_pabd field 3407 * might try to be accessed, even though it was removed. 3408 */ 3409 VERIFY(!HDR_L2_WRITING(hdr)); 3410 VERIFY3P(hdr->b_l1hdr.b_pabd, ==, NULL); 3411 ASSERT(!HDR_HAS_RABD(hdr)); 3412 3413 arc_hdr_clear_flags(nhdr, ARC_FLAG_HAS_L1HDR); 3414 } 3415 /* 3416 * The header has been reallocated so we need to re-insert it into any 3417 * lists it was on. 3418 */ 3419 (void) buf_hash_insert(nhdr, NULL); 3420 3421 ASSERT(list_link_active(&hdr->b_l2hdr.b_l2node)); 3422 3423 mutex_enter(&dev->l2ad_mtx); 3424 3425 /* 3426 * We must place the realloc'ed header back into the list at 3427 * the same spot. Otherwise, if it's placed earlier in the list, 3428 * l2arc_write_buffers() could find it during the function's 3429 * write phase, and try to write it out to the l2arc. 3430 */ 3431 list_insert_after(&dev->l2ad_buflist, hdr, nhdr); 3432 list_remove(&dev->l2ad_buflist, hdr); 3433 3434 mutex_exit(&dev->l2ad_mtx); 3435 3436 /* 3437 * Since we're using the pointer address as the tag when 3438 * incrementing and decrementing the l2ad_alloc refcount, we 3439 * must remove the old pointer (that we're about to destroy) and 3440 * add the new pointer to the refcount. Otherwise we'd remove 3441 * the wrong pointer address when calling arc_hdr_destroy() later. 3442 */ 3443 3444 (void) zfs_refcount_remove_many(&dev->l2ad_alloc, 3445 arc_hdr_size(hdr), hdr); 3446 (void) zfs_refcount_add_many(&dev->l2ad_alloc, 3447 arc_hdr_size(nhdr), nhdr); 3448 3449 buf_discard_identity(hdr); 3450 kmem_cache_free(old, hdr); 3451 3452 return (nhdr); 3453 } 3454 3455 /* 3456 * This function allows an L1 header to be reallocated as a crypt 3457 * header and vice versa. If we are going to a crypt header, the 3458 * new fields will be zeroed out. 3459 */ 3460 static arc_buf_hdr_t * 3461 arc_hdr_realloc_crypt(arc_buf_hdr_t *hdr, boolean_t need_crypt) 3462 { 3463 arc_buf_hdr_t *nhdr; 3464 arc_buf_t *buf; 3465 kmem_cache_t *ncache, *ocache; 3466 3467 /* 3468 * This function requires that hdr is in the arc_anon state. 3469 * Therefore it won't have any L2ARC data for us to worry 3470 * about copying. 3471 */ 3472 ASSERT(HDR_HAS_L1HDR(hdr)); 3473 ASSERT(!HDR_HAS_L2HDR(hdr)); 3474 ASSERT3U(!!HDR_PROTECTED(hdr), !=, need_crypt); 3475 ASSERT3P(hdr->b_l1hdr.b_state, ==, arc_anon); 3476 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 3477 ASSERT(!list_link_active(&hdr->b_l2hdr.b_l2node)); 3478 ASSERT3P(hdr->b_hash_next, ==, NULL); 3479 3480 if (need_crypt) { 3481 ncache = hdr_full_crypt_cache; 3482 ocache = hdr_full_cache; 3483 } else { 3484 ncache = hdr_full_cache; 3485 ocache = hdr_full_crypt_cache; 3486 } 3487 3488 nhdr = kmem_cache_alloc(ncache, KM_PUSHPAGE); 3489 3490 /* 3491 * Copy all members that aren't locks or condvars to the new header. 3492 * No lists are pointing to us (as we asserted above), so we don't 3493 * need to worry about the list nodes. 3494 */ 3495 nhdr->b_dva = hdr->b_dva; 3496 nhdr->b_birth = hdr->b_birth; 3497 nhdr->b_type = hdr->b_type; 3498 nhdr->b_flags = hdr->b_flags; 3499 nhdr->b_psize = hdr->b_psize; 3500 nhdr->b_lsize = hdr->b_lsize; 3501 nhdr->b_spa = hdr->b_spa; 3502 #ifdef ZFS_DEBUG 3503 nhdr->b_l1hdr.b_freeze_cksum = hdr->b_l1hdr.b_freeze_cksum; 3504 #endif 3505 nhdr->b_l1hdr.b_bufcnt = hdr->b_l1hdr.b_bufcnt; 3506 nhdr->b_l1hdr.b_byteswap = hdr->b_l1hdr.b_byteswap; 3507 nhdr->b_l1hdr.b_state = hdr->b_l1hdr.b_state; 3508 nhdr->b_l1hdr.b_arc_access = hdr->b_l1hdr.b_arc_access; 3509 nhdr->b_l1hdr.b_mru_hits = hdr->b_l1hdr.b_mru_hits; 3510 nhdr->b_l1hdr.b_mru_ghost_hits = hdr->b_l1hdr.b_mru_ghost_hits; 3511 nhdr->b_l1hdr.b_mfu_hits = hdr->b_l1hdr.b_mfu_hits; 3512 nhdr->b_l1hdr.b_mfu_ghost_hits = hdr->b_l1hdr.b_mfu_ghost_hits; 3513 nhdr->b_l1hdr.b_acb = hdr->b_l1hdr.b_acb; 3514 nhdr->b_l1hdr.b_pabd = hdr->b_l1hdr.b_pabd; 3515 3516 /* 3517 * This zfs_refcount_add() exists only to ensure that the individual 3518 * arc buffers always point to a header that is referenced, avoiding 3519 * a small race condition that could trigger ASSERTs. 3520 */ 3521 (void) zfs_refcount_add(&nhdr->b_l1hdr.b_refcnt, FTAG); 3522 nhdr->b_l1hdr.b_buf = hdr->b_l1hdr.b_buf; 3523 for (buf = nhdr->b_l1hdr.b_buf; buf != NULL; buf = buf->b_next) 3524 buf->b_hdr = nhdr; 3525 3526 zfs_refcount_transfer(&nhdr->b_l1hdr.b_refcnt, &hdr->b_l1hdr.b_refcnt); 3527 (void) zfs_refcount_remove(&nhdr->b_l1hdr.b_refcnt, FTAG); 3528 ASSERT0(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt)); 3529 3530 if (need_crypt) { 3531 arc_hdr_set_flags(nhdr, ARC_FLAG_PROTECTED); 3532 } else { 3533 arc_hdr_clear_flags(nhdr, ARC_FLAG_PROTECTED); 3534 } 3535 3536 /* unset all members of the original hdr */ 3537 memset(&hdr->b_dva, 0, sizeof (dva_t)); 3538 hdr->b_birth = 0; 3539 hdr->b_type = ARC_BUFC_INVALID; 3540 hdr->b_flags = 0; 3541 hdr->b_psize = 0; 3542 hdr->b_lsize = 0; 3543 hdr->b_spa = 0; 3544 #ifdef ZFS_DEBUG 3545 hdr->b_l1hdr.b_freeze_cksum = NULL; 3546 #endif 3547 hdr->b_l1hdr.b_buf = NULL; 3548 hdr->b_l1hdr.b_bufcnt = 0; 3549 hdr->b_l1hdr.b_byteswap = 0; 3550 hdr->b_l1hdr.b_state = NULL; 3551 hdr->b_l1hdr.b_arc_access = 0; 3552 hdr->b_l1hdr.b_mru_hits = 0; 3553 hdr->b_l1hdr.b_mru_ghost_hits = 0; 3554 hdr->b_l1hdr.b_mfu_hits = 0; 3555 hdr->b_l1hdr.b_mfu_ghost_hits = 0; 3556 hdr->b_l1hdr.b_acb = NULL; 3557 hdr->b_l1hdr.b_pabd = NULL; 3558 3559 if (ocache == hdr_full_crypt_cache) { 3560 ASSERT(!HDR_HAS_RABD(hdr)); 3561 hdr->b_crypt_hdr.b_ot = DMU_OT_NONE; 3562 hdr->b_crypt_hdr.b_ebufcnt = 0; 3563 hdr->b_crypt_hdr.b_dsobj = 0; 3564 memset(hdr->b_crypt_hdr.b_salt, 0, ZIO_DATA_SALT_LEN); 3565 memset(hdr->b_crypt_hdr.b_iv, 0, ZIO_DATA_IV_LEN); 3566 memset(hdr->b_crypt_hdr.b_mac, 0, ZIO_DATA_MAC_LEN); 3567 } 3568 3569 buf_discard_identity(hdr); 3570 kmem_cache_free(ocache, hdr); 3571 3572 return (nhdr); 3573 } 3574 3575 /* 3576 * This function is used by the send / receive code to convert a newly 3577 * allocated arc_buf_t to one that is suitable for a raw encrypted write. It 3578 * is also used to allow the root objset block to be updated without altering 3579 * its embedded MACs. Both block types will always be uncompressed so we do not 3580 * have to worry about compression type or psize. 3581 */ 3582 void 3583 arc_convert_to_raw(arc_buf_t *buf, uint64_t dsobj, boolean_t byteorder, 3584 dmu_object_type_t ot, const uint8_t *salt, const uint8_t *iv, 3585 const uint8_t *mac) 3586 { 3587 arc_buf_hdr_t *hdr = buf->b_hdr; 3588 3589 ASSERT(ot == DMU_OT_DNODE || ot == DMU_OT_OBJSET); 3590 ASSERT(HDR_HAS_L1HDR(hdr)); 3591 ASSERT3P(hdr->b_l1hdr.b_state, ==, arc_anon); 3592 3593 buf->b_flags |= (ARC_BUF_FLAG_COMPRESSED | ARC_BUF_FLAG_ENCRYPTED); 3594 if (!HDR_PROTECTED(hdr)) 3595 hdr = arc_hdr_realloc_crypt(hdr, B_TRUE); 3596 hdr->b_crypt_hdr.b_dsobj = dsobj; 3597 hdr->b_crypt_hdr.b_ot = ot; 3598 hdr->b_l1hdr.b_byteswap = (byteorder == ZFS_HOST_BYTEORDER) ? 3599 DMU_BSWAP_NUMFUNCS : DMU_OT_BYTESWAP(ot); 3600 if (!arc_hdr_has_uncompressed_buf(hdr)) 3601 arc_cksum_free(hdr); 3602 3603 if (salt != NULL) 3604 memcpy(hdr->b_crypt_hdr.b_salt, salt, ZIO_DATA_SALT_LEN); 3605 if (iv != NULL) 3606 memcpy(hdr->b_crypt_hdr.b_iv, iv, ZIO_DATA_IV_LEN); 3607 if (mac != NULL) 3608 memcpy(hdr->b_crypt_hdr.b_mac, mac, ZIO_DATA_MAC_LEN); 3609 } 3610 3611 /* 3612 * Allocate a new arc_buf_hdr_t and arc_buf_t and return the buf to the caller. 3613 * The buf is returned thawed since we expect the consumer to modify it. 3614 */ 3615 arc_buf_t * 3616 arc_alloc_buf(spa_t *spa, const void *tag, arc_buf_contents_t type, 3617 int32_t size) 3618 { 3619 arc_buf_hdr_t *hdr = arc_hdr_alloc(spa_load_guid(spa), size, size, 3620 B_FALSE, ZIO_COMPRESS_OFF, 0, type); 3621 3622 arc_buf_t *buf = NULL; 3623 VERIFY0(arc_buf_alloc_impl(hdr, spa, NULL, tag, B_FALSE, B_FALSE, 3624 B_FALSE, B_FALSE, &buf)); 3625 arc_buf_thaw(buf); 3626 3627 return (buf); 3628 } 3629 3630 /* 3631 * Allocate a compressed buf in the same manner as arc_alloc_buf. Don't use this 3632 * for bufs containing metadata. 3633 */ 3634 arc_buf_t * 3635 arc_alloc_compressed_buf(spa_t *spa, const void *tag, uint64_t psize, 3636 uint64_t lsize, enum zio_compress compression_type, uint8_t complevel) 3637 { 3638 ASSERT3U(lsize, >, 0); 3639 ASSERT3U(lsize, >=, psize); 3640 ASSERT3U(compression_type, >, ZIO_COMPRESS_OFF); 3641 ASSERT3U(compression_type, <, ZIO_COMPRESS_FUNCTIONS); 3642 3643 arc_buf_hdr_t *hdr = arc_hdr_alloc(spa_load_guid(spa), psize, lsize, 3644 B_FALSE, compression_type, complevel, ARC_BUFC_DATA); 3645 3646 arc_buf_t *buf = NULL; 3647 VERIFY0(arc_buf_alloc_impl(hdr, spa, NULL, tag, B_FALSE, 3648 B_TRUE, B_FALSE, B_FALSE, &buf)); 3649 arc_buf_thaw(buf); 3650 3651 /* 3652 * To ensure that the hdr has the correct data in it if we call 3653 * arc_untransform() on this buf before it's been written to disk, 3654 * it's easiest if we just set up sharing between the buf and the hdr. 3655 */ 3656 arc_share_buf(hdr, buf); 3657 3658 return (buf); 3659 } 3660 3661 arc_buf_t * 3662 arc_alloc_raw_buf(spa_t *spa, const void *tag, uint64_t dsobj, 3663 boolean_t byteorder, const uint8_t *salt, const uint8_t *iv, 3664 const uint8_t *mac, dmu_object_type_t ot, uint64_t psize, uint64_t lsize, 3665 enum zio_compress compression_type, uint8_t complevel) 3666 { 3667 arc_buf_hdr_t *hdr; 3668 arc_buf_t *buf; 3669 arc_buf_contents_t type = DMU_OT_IS_METADATA(ot) ? 3670 ARC_BUFC_METADATA : ARC_BUFC_DATA; 3671 3672 ASSERT3U(lsize, >, 0); 3673 ASSERT3U(lsize, >=, psize); 3674 ASSERT3U(compression_type, >=, ZIO_COMPRESS_OFF); 3675 ASSERT3U(compression_type, <, ZIO_COMPRESS_FUNCTIONS); 3676 3677 hdr = arc_hdr_alloc(spa_load_guid(spa), psize, lsize, B_TRUE, 3678 compression_type, complevel, type); 3679 3680 hdr->b_crypt_hdr.b_dsobj = dsobj; 3681 hdr->b_crypt_hdr.b_ot = ot; 3682 hdr->b_l1hdr.b_byteswap = (byteorder == ZFS_HOST_BYTEORDER) ? 3683 DMU_BSWAP_NUMFUNCS : DMU_OT_BYTESWAP(ot); 3684 memcpy(hdr->b_crypt_hdr.b_salt, salt, ZIO_DATA_SALT_LEN); 3685 memcpy(hdr->b_crypt_hdr.b_iv, iv, ZIO_DATA_IV_LEN); 3686 memcpy(hdr->b_crypt_hdr.b_mac, mac, ZIO_DATA_MAC_LEN); 3687 3688 /* 3689 * This buffer will be considered encrypted even if the ot is not an 3690 * encrypted type. It will become authenticated instead in 3691 * arc_write_ready(). 3692 */ 3693 buf = NULL; 3694 VERIFY0(arc_buf_alloc_impl(hdr, spa, NULL, tag, B_TRUE, B_TRUE, 3695 B_FALSE, B_FALSE, &buf)); 3696 arc_buf_thaw(buf); 3697 3698 return (buf); 3699 } 3700 3701 static void 3702 l2arc_hdr_arcstats_update(arc_buf_hdr_t *hdr, boolean_t incr, 3703 boolean_t state_only) 3704 { 3705 l2arc_buf_hdr_t *l2hdr = &hdr->b_l2hdr; 3706 l2arc_dev_t *dev = l2hdr->b_dev; 3707 uint64_t lsize = HDR_GET_LSIZE(hdr); 3708 uint64_t psize = HDR_GET_PSIZE(hdr); 3709 uint64_t asize = vdev_psize_to_asize(dev->l2ad_vdev, psize); 3710 arc_buf_contents_t type = hdr->b_type; 3711 int64_t lsize_s; 3712 int64_t psize_s; 3713 int64_t asize_s; 3714 3715 if (incr) { 3716 lsize_s = lsize; 3717 psize_s = psize; 3718 asize_s = asize; 3719 } else { 3720 lsize_s = -lsize; 3721 psize_s = -psize; 3722 asize_s = -asize; 3723 } 3724 3725 /* If the buffer is a prefetch, count it as such. */ 3726 if (HDR_PREFETCH(hdr)) { 3727 ARCSTAT_INCR(arcstat_l2_prefetch_asize, asize_s); 3728 } else { 3729 /* 3730 * We use the value stored in the L2 header upon initial 3731 * caching in L2ARC. This value will be updated in case 3732 * an MRU/MRU_ghost buffer transitions to MFU but the L2ARC 3733 * metadata (log entry) cannot currently be updated. Having 3734 * the ARC state in the L2 header solves the problem of a 3735 * possibly absent L1 header (apparent in buffers restored 3736 * from persistent L2ARC). 3737 */ 3738 switch (hdr->b_l2hdr.b_arcs_state) { 3739 case ARC_STATE_MRU_GHOST: 3740 case ARC_STATE_MRU: 3741 ARCSTAT_INCR(arcstat_l2_mru_asize, asize_s); 3742 break; 3743 case ARC_STATE_MFU_GHOST: 3744 case ARC_STATE_MFU: 3745 ARCSTAT_INCR(arcstat_l2_mfu_asize, asize_s); 3746 break; 3747 default: 3748 break; 3749 } 3750 } 3751 3752 if (state_only) 3753 return; 3754 3755 ARCSTAT_INCR(arcstat_l2_psize, psize_s); 3756 ARCSTAT_INCR(arcstat_l2_lsize, lsize_s); 3757 3758 switch (type) { 3759 case ARC_BUFC_DATA: 3760 ARCSTAT_INCR(arcstat_l2_bufc_data_asize, asize_s); 3761 break; 3762 case ARC_BUFC_METADATA: 3763 ARCSTAT_INCR(arcstat_l2_bufc_metadata_asize, asize_s); 3764 break; 3765 default: 3766 break; 3767 } 3768 } 3769 3770 3771 static void 3772 arc_hdr_l2hdr_destroy(arc_buf_hdr_t *hdr) 3773 { 3774 l2arc_buf_hdr_t *l2hdr = &hdr->b_l2hdr; 3775 l2arc_dev_t *dev = l2hdr->b_dev; 3776 uint64_t psize = HDR_GET_PSIZE(hdr); 3777 uint64_t asize = vdev_psize_to_asize(dev->l2ad_vdev, psize); 3778 3779 ASSERT(MUTEX_HELD(&dev->l2ad_mtx)); 3780 ASSERT(HDR_HAS_L2HDR(hdr)); 3781 3782 list_remove(&dev->l2ad_buflist, hdr); 3783 3784 l2arc_hdr_arcstats_decrement(hdr); 3785 vdev_space_update(dev->l2ad_vdev, -asize, 0, 0); 3786 3787 (void) zfs_refcount_remove_many(&dev->l2ad_alloc, arc_hdr_size(hdr), 3788 hdr); 3789 arc_hdr_clear_flags(hdr, ARC_FLAG_HAS_L2HDR); 3790 } 3791 3792 static void 3793 arc_hdr_destroy(arc_buf_hdr_t *hdr) 3794 { 3795 if (HDR_HAS_L1HDR(hdr)) { 3796 ASSERT(hdr->b_l1hdr.b_buf == NULL || 3797 hdr->b_l1hdr.b_bufcnt > 0); 3798 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 3799 ASSERT3P(hdr->b_l1hdr.b_state, ==, arc_anon); 3800 } 3801 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 3802 ASSERT(!HDR_IN_HASH_TABLE(hdr)); 3803 3804 if (HDR_HAS_L2HDR(hdr)) { 3805 l2arc_dev_t *dev = hdr->b_l2hdr.b_dev; 3806 boolean_t buflist_held = MUTEX_HELD(&dev->l2ad_mtx); 3807 3808 if (!buflist_held) 3809 mutex_enter(&dev->l2ad_mtx); 3810 3811 /* 3812 * Even though we checked this conditional above, we 3813 * need to check this again now that we have the 3814 * l2ad_mtx. This is because we could be racing with 3815 * another thread calling l2arc_evict() which might have 3816 * destroyed this header's L2 portion as we were waiting 3817 * to acquire the l2ad_mtx. If that happens, we don't 3818 * want to re-destroy the header's L2 portion. 3819 */ 3820 if (HDR_HAS_L2HDR(hdr)) { 3821 3822 if (!HDR_EMPTY(hdr)) 3823 buf_discard_identity(hdr); 3824 3825 arc_hdr_l2hdr_destroy(hdr); 3826 } 3827 3828 if (!buflist_held) 3829 mutex_exit(&dev->l2ad_mtx); 3830 } 3831 3832 /* 3833 * The header's identify can only be safely discarded once it is no 3834 * longer discoverable. This requires removing it from the hash table 3835 * and the l2arc header list. After this point the hash lock can not 3836 * be used to protect the header. 3837 */ 3838 if (!HDR_EMPTY(hdr)) 3839 buf_discard_identity(hdr); 3840 3841 if (HDR_HAS_L1HDR(hdr)) { 3842 arc_cksum_free(hdr); 3843 3844 while (hdr->b_l1hdr.b_buf != NULL) 3845 arc_buf_destroy_impl(hdr->b_l1hdr.b_buf); 3846 3847 if (hdr->b_l1hdr.b_pabd != NULL) 3848 arc_hdr_free_abd(hdr, B_FALSE); 3849 3850 if (HDR_HAS_RABD(hdr)) 3851 arc_hdr_free_abd(hdr, B_TRUE); 3852 } 3853 3854 ASSERT3P(hdr->b_hash_next, ==, NULL); 3855 if (HDR_HAS_L1HDR(hdr)) { 3856 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 3857 ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); 3858 #ifdef ZFS_DEBUG 3859 ASSERT3P(hdr->b_l1hdr.b_freeze_cksum, ==, NULL); 3860 #endif 3861 3862 if (!HDR_PROTECTED(hdr)) { 3863 kmem_cache_free(hdr_full_cache, hdr); 3864 } else { 3865 kmem_cache_free(hdr_full_crypt_cache, hdr); 3866 } 3867 } else { 3868 kmem_cache_free(hdr_l2only_cache, hdr); 3869 } 3870 } 3871 3872 void 3873 arc_buf_destroy(arc_buf_t *buf, const void *tag) 3874 { 3875 arc_buf_hdr_t *hdr = buf->b_hdr; 3876 3877 if (hdr->b_l1hdr.b_state == arc_anon) { 3878 ASSERT3U(hdr->b_l1hdr.b_bufcnt, ==, 1); 3879 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 3880 VERIFY0(remove_reference(hdr, tag)); 3881 return; 3882 } 3883 3884 kmutex_t *hash_lock = HDR_LOCK(hdr); 3885 mutex_enter(hash_lock); 3886 3887 ASSERT3P(hdr, ==, buf->b_hdr); 3888 ASSERT(hdr->b_l1hdr.b_bufcnt > 0); 3889 ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); 3890 ASSERT3P(hdr->b_l1hdr.b_state, !=, arc_anon); 3891 ASSERT3P(buf->b_data, !=, NULL); 3892 3893 arc_buf_destroy_impl(buf); 3894 (void) remove_reference(hdr, tag); 3895 mutex_exit(hash_lock); 3896 } 3897 3898 /* 3899 * Evict the arc_buf_hdr that is provided as a parameter. The resultant 3900 * state of the header is dependent on its state prior to entering this 3901 * function. The following transitions are possible: 3902 * 3903 * - arc_mru -> arc_mru_ghost 3904 * - arc_mfu -> arc_mfu_ghost 3905 * - arc_mru_ghost -> arc_l2c_only 3906 * - arc_mru_ghost -> deleted 3907 * - arc_mfu_ghost -> arc_l2c_only 3908 * - arc_mfu_ghost -> deleted 3909 * - arc_uncached -> deleted 3910 * 3911 * Return total size of evicted data buffers for eviction progress tracking. 3912 * When evicting from ghost states return logical buffer size to make eviction 3913 * progress at the same (or at least comparable) rate as from non-ghost states. 3914 * 3915 * Return *real_evicted for actual ARC size reduction to wake up threads 3916 * waiting for it. For non-ghost states it includes size of evicted data 3917 * buffers (the headers are not freed there). For ghost states it includes 3918 * only the evicted headers size. 3919 */ 3920 static int64_t 3921 arc_evict_hdr(arc_buf_hdr_t *hdr, uint64_t *real_evicted) 3922 { 3923 arc_state_t *evicted_state, *state; 3924 int64_t bytes_evicted = 0; 3925 uint_t min_lifetime = HDR_PRESCIENT_PREFETCH(hdr) ? 3926 arc_min_prescient_prefetch_ms : arc_min_prefetch_ms; 3927 3928 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 3929 ASSERT(HDR_HAS_L1HDR(hdr)); 3930 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 3931 ASSERT0(hdr->b_l1hdr.b_bufcnt); 3932 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 3933 ASSERT0(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt)); 3934 3935 *real_evicted = 0; 3936 state = hdr->b_l1hdr.b_state; 3937 if (GHOST_STATE(state)) { 3938 3939 /* 3940 * l2arc_write_buffers() relies on a header's L1 portion 3941 * (i.e. its b_pabd field) during it's write phase. 3942 * Thus, we cannot push a header onto the arc_l2c_only 3943 * state (removing its L1 piece) until the header is 3944 * done being written to the l2arc. 3945 */ 3946 if (HDR_HAS_L2HDR(hdr) && HDR_L2_WRITING(hdr)) { 3947 ARCSTAT_BUMP(arcstat_evict_l2_skip); 3948 return (bytes_evicted); 3949 } 3950 3951 ARCSTAT_BUMP(arcstat_deleted); 3952 bytes_evicted += HDR_GET_LSIZE(hdr); 3953 3954 DTRACE_PROBE1(arc__delete, arc_buf_hdr_t *, hdr); 3955 3956 if (HDR_HAS_L2HDR(hdr)) { 3957 ASSERT(hdr->b_l1hdr.b_pabd == NULL); 3958 ASSERT(!HDR_HAS_RABD(hdr)); 3959 /* 3960 * This buffer is cached on the 2nd Level ARC; 3961 * don't destroy the header. 3962 */ 3963 arc_change_state(arc_l2c_only, hdr); 3964 /* 3965 * dropping from L1+L2 cached to L2-only, 3966 * realloc to remove the L1 header. 3967 */ 3968 (void) arc_hdr_realloc(hdr, hdr_full_cache, 3969 hdr_l2only_cache); 3970 *real_evicted += HDR_FULL_SIZE - HDR_L2ONLY_SIZE; 3971 } else { 3972 arc_change_state(arc_anon, hdr); 3973 arc_hdr_destroy(hdr); 3974 *real_evicted += HDR_FULL_SIZE; 3975 } 3976 return (bytes_evicted); 3977 } 3978 3979 ASSERT(state == arc_mru || state == arc_mfu || state == arc_uncached); 3980 evicted_state = (state == arc_uncached) ? arc_anon : 3981 ((state == arc_mru) ? arc_mru_ghost : arc_mfu_ghost); 3982 3983 /* prefetch buffers have a minimum lifespan */ 3984 if ((hdr->b_flags & (ARC_FLAG_PREFETCH | ARC_FLAG_INDIRECT)) && 3985 ddi_get_lbolt() - hdr->b_l1hdr.b_arc_access < 3986 MSEC_TO_TICK(min_lifetime)) { 3987 ARCSTAT_BUMP(arcstat_evict_skip); 3988 return (bytes_evicted); 3989 } 3990 3991 if (HDR_HAS_L2HDR(hdr)) { 3992 ARCSTAT_INCR(arcstat_evict_l2_cached, HDR_GET_LSIZE(hdr)); 3993 } else { 3994 if (l2arc_write_eligible(hdr->b_spa, hdr)) { 3995 ARCSTAT_INCR(arcstat_evict_l2_eligible, 3996 HDR_GET_LSIZE(hdr)); 3997 3998 switch (state->arcs_state) { 3999 case ARC_STATE_MRU: 4000 ARCSTAT_INCR( 4001 arcstat_evict_l2_eligible_mru, 4002 HDR_GET_LSIZE(hdr)); 4003 break; 4004 case ARC_STATE_MFU: 4005 ARCSTAT_INCR( 4006 arcstat_evict_l2_eligible_mfu, 4007 HDR_GET_LSIZE(hdr)); 4008 break; 4009 default: 4010 break; 4011 } 4012 } else { 4013 ARCSTAT_INCR(arcstat_evict_l2_ineligible, 4014 HDR_GET_LSIZE(hdr)); 4015 } 4016 } 4017 4018 bytes_evicted += arc_hdr_size(hdr); 4019 *real_evicted += arc_hdr_size(hdr); 4020 4021 /* 4022 * If this hdr is being evicted and has a compressed buffer then we 4023 * discard it here before we change states. This ensures that the 4024 * accounting is updated correctly in arc_free_data_impl(). 4025 */ 4026 if (hdr->b_l1hdr.b_pabd != NULL) 4027 arc_hdr_free_abd(hdr, B_FALSE); 4028 4029 if (HDR_HAS_RABD(hdr)) 4030 arc_hdr_free_abd(hdr, B_TRUE); 4031 4032 arc_change_state(evicted_state, hdr); 4033 DTRACE_PROBE1(arc__evict, arc_buf_hdr_t *, hdr); 4034 if (evicted_state == arc_anon) { 4035 arc_hdr_destroy(hdr); 4036 *real_evicted += HDR_FULL_SIZE; 4037 } else { 4038 ASSERT(HDR_IN_HASH_TABLE(hdr)); 4039 } 4040 4041 return (bytes_evicted); 4042 } 4043 4044 static void 4045 arc_set_need_free(void) 4046 { 4047 ASSERT(MUTEX_HELD(&arc_evict_lock)); 4048 int64_t remaining = arc_free_memory() - arc_sys_free / 2; 4049 arc_evict_waiter_t *aw = list_tail(&arc_evict_waiters); 4050 if (aw == NULL) { 4051 arc_need_free = MAX(-remaining, 0); 4052 } else { 4053 arc_need_free = 4054 MAX(-remaining, (int64_t)(aw->aew_count - arc_evict_count)); 4055 } 4056 } 4057 4058 static uint64_t 4059 arc_evict_state_impl(multilist_t *ml, int idx, arc_buf_hdr_t *marker, 4060 uint64_t spa, uint64_t bytes) 4061 { 4062 multilist_sublist_t *mls; 4063 uint64_t bytes_evicted = 0, real_evicted = 0; 4064 arc_buf_hdr_t *hdr; 4065 kmutex_t *hash_lock; 4066 uint_t evict_count = zfs_arc_evict_batch_limit; 4067 4068 ASSERT3P(marker, !=, NULL); 4069 4070 mls = multilist_sublist_lock(ml, idx); 4071 4072 for (hdr = multilist_sublist_prev(mls, marker); likely(hdr != NULL); 4073 hdr = multilist_sublist_prev(mls, marker)) { 4074 if ((evict_count == 0) || (bytes_evicted >= bytes)) 4075 break; 4076 4077 /* 4078 * To keep our iteration location, move the marker 4079 * forward. Since we're not holding hdr's hash lock, we 4080 * must be very careful and not remove 'hdr' from the 4081 * sublist. Otherwise, other consumers might mistake the 4082 * 'hdr' as not being on a sublist when they call the 4083 * multilist_link_active() function (they all rely on 4084 * the hash lock protecting concurrent insertions and 4085 * removals). multilist_sublist_move_forward() was 4086 * specifically implemented to ensure this is the case 4087 * (only 'marker' will be removed and re-inserted). 4088 */ 4089 multilist_sublist_move_forward(mls, marker); 4090 4091 /* 4092 * The only case where the b_spa field should ever be 4093 * zero, is the marker headers inserted by 4094 * arc_evict_state(). It's possible for multiple threads 4095 * to be calling arc_evict_state() concurrently (e.g. 4096 * dsl_pool_close() and zio_inject_fault()), so we must 4097 * skip any markers we see from these other threads. 4098 */ 4099 if (hdr->b_spa == 0) 4100 continue; 4101 4102 /* we're only interested in evicting buffers of a certain spa */ 4103 if (spa != 0 && hdr->b_spa != spa) { 4104 ARCSTAT_BUMP(arcstat_evict_skip); 4105 continue; 4106 } 4107 4108 hash_lock = HDR_LOCK(hdr); 4109 4110 /* 4111 * We aren't calling this function from any code path 4112 * that would already be holding a hash lock, so we're 4113 * asserting on this assumption to be defensive in case 4114 * this ever changes. Without this check, it would be 4115 * possible to incorrectly increment arcstat_mutex_miss 4116 * below (e.g. if the code changed such that we called 4117 * this function with a hash lock held). 4118 */ 4119 ASSERT(!MUTEX_HELD(hash_lock)); 4120 4121 if (mutex_tryenter(hash_lock)) { 4122 uint64_t revicted; 4123 uint64_t evicted = arc_evict_hdr(hdr, &revicted); 4124 mutex_exit(hash_lock); 4125 4126 bytes_evicted += evicted; 4127 real_evicted += revicted; 4128 4129 /* 4130 * If evicted is zero, arc_evict_hdr() must have 4131 * decided to skip this header, don't increment 4132 * evict_count in this case. 4133 */ 4134 if (evicted != 0) 4135 evict_count--; 4136 4137 } else { 4138 ARCSTAT_BUMP(arcstat_mutex_miss); 4139 } 4140 } 4141 4142 multilist_sublist_unlock(mls); 4143 4144 /* 4145 * Increment the count of evicted bytes, and wake up any threads that 4146 * are waiting for the count to reach this value. Since the list is 4147 * ordered by ascending aew_count, we pop off the beginning of the 4148 * list until we reach the end, or a waiter that's past the current 4149 * "count". Doing this outside the loop reduces the number of times 4150 * we need to acquire the global arc_evict_lock. 4151 * 4152 * Only wake when there's sufficient free memory in the system 4153 * (specifically, arc_sys_free/2, which by default is a bit more than 4154 * 1/64th of RAM). See the comments in arc_wait_for_eviction(). 4155 */ 4156 mutex_enter(&arc_evict_lock); 4157 arc_evict_count += real_evicted; 4158 4159 if (arc_free_memory() > arc_sys_free / 2) { 4160 arc_evict_waiter_t *aw; 4161 while ((aw = list_head(&arc_evict_waiters)) != NULL && 4162 aw->aew_count <= arc_evict_count) { 4163 list_remove(&arc_evict_waiters, aw); 4164 cv_broadcast(&aw->aew_cv); 4165 } 4166 } 4167 arc_set_need_free(); 4168 mutex_exit(&arc_evict_lock); 4169 4170 /* 4171 * If the ARC size is reduced from arc_c_max to arc_c_min (especially 4172 * if the average cached block is small), eviction can be on-CPU for 4173 * many seconds. To ensure that other threads that may be bound to 4174 * this CPU are able to make progress, make a voluntary preemption 4175 * call here. 4176 */ 4177 kpreempt(KPREEMPT_SYNC); 4178 4179 return (bytes_evicted); 4180 } 4181 4182 /* 4183 * Allocate an array of buffer headers used as placeholders during arc state 4184 * eviction. 4185 */ 4186 static arc_buf_hdr_t ** 4187 arc_state_alloc_markers(int count) 4188 { 4189 arc_buf_hdr_t **markers; 4190 4191 markers = kmem_zalloc(sizeof (*markers) * count, KM_SLEEP); 4192 for (int i = 0; i < count; i++) { 4193 markers[i] = kmem_cache_alloc(hdr_full_cache, KM_SLEEP); 4194 4195 /* 4196 * A b_spa of 0 is used to indicate that this header is 4197 * a marker. This fact is used in arc_evict_type() and 4198 * arc_evict_state_impl(). 4199 */ 4200 markers[i]->b_spa = 0; 4201 4202 } 4203 return (markers); 4204 } 4205 4206 static void 4207 arc_state_free_markers(arc_buf_hdr_t **markers, int count) 4208 { 4209 for (int i = 0; i < count; i++) 4210 kmem_cache_free(hdr_full_cache, markers[i]); 4211 kmem_free(markers, sizeof (*markers) * count); 4212 } 4213 4214 /* 4215 * Evict buffers from the given arc state, until we've removed the 4216 * specified number of bytes. Move the removed buffers to the 4217 * appropriate evict state. 4218 * 4219 * This function makes a "best effort". It skips over any buffers 4220 * it can't get a hash_lock on, and so, may not catch all candidates. 4221 * It may also return without evicting as much space as requested. 4222 * 4223 * If bytes is specified using the special value ARC_EVICT_ALL, this 4224 * will evict all available (i.e. unlocked and evictable) buffers from 4225 * the given arc state; which is used by arc_flush(). 4226 */ 4227 static uint64_t 4228 arc_evict_state(arc_state_t *state, uint64_t spa, uint64_t bytes, 4229 arc_buf_contents_t type) 4230 { 4231 uint64_t total_evicted = 0; 4232 multilist_t *ml = &state->arcs_list[type]; 4233 int num_sublists; 4234 arc_buf_hdr_t **markers; 4235 4236 num_sublists = multilist_get_num_sublists(ml); 4237 4238 /* 4239 * If we've tried to evict from each sublist, made some 4240 * progress, but still have not hit the target number of bytes 4241 * to evict, we want to keep trying. The markers allow us to 4242 * pick up where we left off for each individual sublist, rather 4243 * than starting from the tail each time. 4244 */ 4245 if (zthr_iscurthread(arc_evict_zthr)) { 4246 markers = arc_state_evict_markers; 4247 ASSERT3S(num_sublists, <=, arc_state_evict_marker_count); 4248 } else { 4249 markers = arc_state_alloc_markers(num_sublists); 4250 } 4251 for (int i = 0; i < num_sublists; i++) { 4252 multilist_sublist_t *mls; 4253 4254 mls = multilist_sublist_lock(ml, i); 4255 multilist_sublist_insert_tail(mls, markers[i]); 4256 multilist_sublist_unlock(mls); 4257 } 4258 4259 /* 4260 * While we haven't hit our target number of bytes to evict, or 4261 * we're evicting all available buffers. 4262 */ 4263 while (total_evicted < bytes) { 4264 int sublist_idx = multilist_get_random_index(ml); 4265 uint64_t scan_evicted = 0; 4266 4267 /* 4268 * Try to reduce pinned dnodes with a floor of arc_dnode_limit. 4269 * Request that 10% of the LRUs be scanned by the superblock 4270 * shrinker. 4271 */ 4272 if (type == ARC_BUFC_DATA && aggsum_compare( 4273 &arc_sums.arcstat_dnode_size, arc_dnode_size_limit) > 0) { 4274 arc_prune_async((aggsum_upper_bound( 4275 &arc_sums.arcstat_dnode_size) - 4276 arc_dnode_size_limit) / sizeof (dnode_t) / 4277 zfs_arc_dnode_reduce_percent); 4278 } 4279 4280 /* 4281 * Start eviction using a randomly selected sublist, 4282 * this is to try and evenly balance eviction across all 4283 * sublists. Always starting at the same sublist 4284 * (e.g. index 0) would cause evictions to favor certain 4285 * sublists over others. 4286 */ 4287 for (int i = 0; i < num_sublists; i++) { 4288 uint64_t bytes_remaining; 4289 uint64_t bytes_evicted; 4290 4291 if (total_evicted < bytes) 4292 bytes_remaining = bytes - total_evicted; 4293 else 4294 break; 4295 4296 bytes_evicted = arc_evict_state_impl(ml, sublist_idx, 4297 markers[sublist_idx], spa, bytes_remaining); 4298 4299 scan_evicted += bytes_evicted; 4300 total_evicted += bytes_evicted; 4301 4302 /* we've reached the end, wrap to the beginning */ 4303 if (++sublist_idx >= num_sublists) 4304 sublist_idx = 0; 4305 } 4306 4307 /* 4308 * If we didn't evict anything during this scan, we have 4309 * no reason to believe we'll evict more during another 4310 * scan, so break the loop. 4311 */ 4312 if (scan_evicted == 0) { 4313 /* This isn't possible, let's make that obvious */ 4314 ASSERT3S(bytes, !=, 0); 4315 4316 /* 4317 * When bytes is ARC_EVICT_ALL, the only way to 4318 * break the loop is when scan_evicted is zero. 4319 * In that case, we actually have evicted enough, 4320 * so we don't want to increment the kstat. 4321 */ 4322 if (bytes != ARC_EVICT_ALL) { 4323 ASSERT3S(total_evicted, <, bytes); 4324 ARCSTAT_BUMP(arcstat_evict_not_enough); 4325 } 4326 4327 break; 4328 } 4329 } 4330 4331 for (int i = 0; i < num_sublists; i++) { 4332 multilist_sublist_t *mls = multilist_sublist_lock(ml, i); 4333 multilist_sublist_remove(mls, markers[i]); 4334 multilist_sublist_unlock(mls); 4335 } 4336 if (markers != arc_state_evict_markers) 4337 arc_state_free_markers(markers, num_sublists); 4338 4339 return (total_evicted); 4340 } 4341 4342 /* 4343 * Flush all "evictable" data of the given type from the arc state 4344 * specified. This will not evict any "active" buffers (i.e. referenced). 4345 * 4346 * When 'retry' is set to B_FALSE, the function will make a single pass 4347 * over the state and evict any buffers that it can. Since it doesn't 4348 * continually retry the eviction, it might end up leaving some buffers 4349 * in the ARC due to lock misses. 4350 * 4351 * When 'retry' is set to B_TRUE, the function will continually retry the 4352 * eviction until *all* evictable buffers have been removed from the 4353 * state. As a result, if concurrent insertions into the state are 4354 * allowed (e.g. if the ARC isn't shutting down), this function might 4355 * wind up in an infinite loop, continually trying to evict buffers. 4356 */ 4357 static uint64_t 4358 arc_flush_state(arc_state_t *state, uint64_t spa, arc_buf_contents_t type, 4359 boolean_t retry) 4360 { 4361 uint64_t evicted = 0; 4362 4363 while (zfs_refcount_count(&state->arcs_esize[type]) != 0) { 4364 evicted += arc_evict_state(state, spa, ARC_EVICT_ALL, type); 4365 4366 if (!retry) 4367 break; 4368 } 4369 4370 return (evicted); 4371 } 4372 4373 /* 4374 * Evict the specified number of bytes from the state specified, 4375 * restricting eviction to the spa and type given. This function 4376 * prevents us from trying to evict more from a state's list than 4377 * is "evictable", and to skip evicting altogether when passed a 4378 * negative value for "bytes". In contrast, arc_evict_state() will 4379 * evict everything it can, when passed a negative value for "bytes". 4380 */ 4381 static uint64_t 4382 arc_evict_impl(arc_state_t *state, uint64_t spa, int64_t bytes, 4383 arc_buf_contents_t type) 4384 { 4385 uint64_t delta; 4386 4387 if (bytes > 0 && zfs_refcount_count(&state->arcs_esize[type]) > 0) { 4388 delta = MIN(zfs_refcount_count(&state->arcs_esize[type]), 4389 bytes); 4390 return (arc_evict_state(state, spa, delta, type)); 4391 } 4392 4393 return (0); 4394 } 4395 4396 /* 4397 * The goal of this function is to evict enough meta data buffers from the 4398 * ARC in order to enforce the arc_meta_limit. Achieving this is slightly 4399 * more complicated than it appears because it is common for data buffers 4400 * to have holds on meta data buffers. In addition, dnode meta data buffers 4401 * will be held by the dnodes in the block preventing them from being freed. 4402 * This means we can't simply traverse the ARC and expect to always find 4403 * enough unheld meta data buffer to release. 4404 * 4405 * Therefore, this function has been updated to make alternating passes 4406 * over the ARC releasing data buffers and then newly unheld meta data 4407 * buffers. This ensures forward progress is maintained and meta_used 4408 * will decrease. Normally this is sufficient, but if required the ARC 4409 * will call the registered prune callbacks causing dentry and inodes to 4410 * be dropped from the VFS cache. This will make dnode meta data buffers 4411 * available for reclaim. 4412 */ 4413 static uint64_t 4414 arc_evict_meta_balanced(uint64_t meta_used) 4415 { 4416 int64_t delta, adjustmnt; 4417 uint64_t total_evicted = 0, prune = 0; 4418 arc_buf_contents_t type = ARC_BUFC_DATA; 4419 uint_t restarts = zfs_arc_meta_adjust_restarts; 4420 4421 restart: 4422 /* 4423 * This slightly differs than the way we evict from the mru in 4424 * arc_evict because we don't have a "target" value (i.e. no 4425 * "meta" arc_p). As a result, I think we can completely 4426 * cannibalize the metadata in the MRU before we evict the 4427 * metadata from the MFU. I think we probably need to implement a 4428 * "metadata arc_p" value to do this properly. 4429 */ 4430 adjustmnt = meta_used - arc_meta_limit; 4431 4432 if (adjustmnt > 0 && 4433 zfs_refcount_count(&arc_mru->arcs_esize[type]) > 0) { 4434 delta = MIN(zfs_refcount_count(&arc_mru->arcs_esize[type]), 4435 adjustmnt); 4436 total_evicted += arc_evict_impl(arc_mru, 0, delta, type); 4437 adjustmnt -= delta; 4438 } 4439 4440 /* 4441 * We can't afford to recalculate adjustmnt here. If we do, 4442 * new metadata buffers can sneak into the MRU or ANON lists, 4443 * thus penalize the MFU metadata. Although the fudge factor is 4444 * small, it has been empirically shown to be significant for 4445 * certain workloads (e.g. creating many empty directories). As 4446 * such, we use the original calculation for adjustmnt, and 4447 * simply decrement the amount of data evicted from the MRU. 4448 */ 4449 4450 if (adjustmnt > 0 && 4451 zfs_refcount_count(&arc_mfu->arcs_esize[type]) > 0) { 4452 delta = MIN(zfs_refcount_count(&arc_mfu->arcs_esize[type]), 4453 adjustmnt); 4454 total_evicted += arc_evict_impl(arc_mfu, 0, delta, type); 4455 } 4456 4457 adjustmnt = meta_used - arc_meta_limit; 4458 4459 if (adjustmnt > 0 && 4460 zfs_refcount_count(&arc_mru_ghost->arcs_esize[type]) > 0) { 4461 delta = MIN(adjustmnt, 4462 zfs_refcount_count(&arc_mru_ghost->arcs_esize[type])); 4463 total_evicted += arc_evict_impl(arc_mru_ghost, 0, delta, type); 4464 adjustmnt -= delta; 4465 } 4466 4467 if (adjustmnt > 0 && 4468 zfs_refcount_count(&arc_mfu_ghost->arcs_esize[type]) > 0) { 4469 delta = MIN(adjustmnt, 4470 zfs_refcount_count(&arc_mfu_ghost->arcs_esize[type])); 4471 total_evicted += arc_evict_impl(arc_mfu_ghost, 0, delta, type); 4472 } 4473 4474 /* 4475 * If after attempting to make the requested adjustment to the ARC 4476 * the meta limit is still being exceeded then request that the 4477 * higher layers drop some cached objects which have holds on ARC 4478 * meta buffers. Requests to the upper layers will be made with 4479 * increasingly large scan sizes until the ARC is below the limit. 4480 */ 4481 if (meta_used > arc_meta_limit || arc_available_memory() < 0) { 4482 if (type == ARC_BUFC_DATA) { 4483 type = ARC_BUFC_METADATA; 4484 } else { 4485 type = ARC_BUFC_DATA; 4486 4487 if (zfs_arc_meta_prune) { 4488 prune += zfs_arc_meta_prune; 4489 arc_prune_async(prune); 4490 } 4491 } 4492 4493 if (restarts > 0) { 4494 restarts--; 4495 goto restart; 4496 } 4497 } 4498 return (total_evicted); 4499 } 4500 4501 /* 4502 * Evict metadata buffers from the cache, such that arcstat_meta_used is 4503 * capped by the arc_meta_limit tunable. 4504 */ 4505 static uint64_t 4506 arc_evict_meta_only(uint64_t meta_used) 4507 { 4508 uint64_t total_evicted = 0; 4509 int64_t target; 4510 4511 /* 4512 * If we're over the meta limit, we want to evict enough 4513 * metadata to get back under the meta limit. We don't want to 4514 * evict so much that we drop the MRU below arc_p, though. If 4515 * we're over the meta limit more than we're over arc_p, we 4516 * evict some from the MRU here, and some from the MFU below. 4517 */ 4518 target = MIN((int64_t)(meta_used - arc_meta_limit), 4519 (int64_t)(zfs_refcount_count(&arc_anon->arcs_size) + 4520 zfs_refcount_count(&arc_mru->arcs_size) - arc_p)); 4521 4522 total_evicted += arc_evict_impl(arc_mru, 0, target, ARC_BUFC_METADATA); 4523 4524 /* 4525 * Similar to the above, we want to evict enough bytes to get us 4526 * below the meta limit, but not so much as to drop us below the 4527 * space allotted to the MFU (which is defined as arc_c - arc_p). 4528 */ 4529 target = MIN((int64_t)(meta_used - arc_meta_limit), 4530 (int64_t)(zfs_refcount_count(&arc_mfu->arcs_size) - 4531 (arc_c - arc_p))); 4532 4533 total_evicted += arc_evict_impl(arc_mfu, 0, target, ARC_BUFC_METADATA); 4534 4535 return (total_evicted); 4536 } 4537 4538 static uint64_t 4539 arc_evict_meta(uint64_t meta_used) 4540 { 4541 if (zfs_arc_meta_strategy == ARC_STRATEGY_META_ONLY) 4542 return (arc_evict_meta_only(meta_used)); 4543 else 4544 return (arc_evict_meta_balanced(meta_used)); 4545 } 4546 4547 /* 4548 * Return the type of the oldest buffer in the given arc state 4549 * 4550 * This function will select a random sublist of type ARC_BUFC_DATA and 4551 * a random sublist of type ARC_BUFC_METADATA. The tail of each sublist 4552 * is compared, and the type which contains the "older" buffer will be 4553 * returned. 4554 */ 4555 static arc_buf_contents_t 4556 arc_evict_type(arc_state_t *state) 4557 { 4558 multilist_t *data_ml = &state->arcs_list[ARC_BUFC_DATA]; 4559 multilist_t *meta_ml = &state->arcs_list[ARC_BUFC_METADATA]; 4560 int data_idx = multilist_get_random_index(data_ml); 4561 int meta_idx = multilist_get_random_index(meta_ml); 4562 multilist_sublist_t *data_mls; 4563 multilist_sublist_t *meta_mls; 4564 arc_buf_contents_t type; 4565 arc_buf_hdr_t *data_hdr; 4566 arc_buf_hdr_t *meta_hdr; 4567 4568 /* 4569 * We keep the sublist lock until we're finished, to prevent 4570 * the headers from being destroyed via arc_evict_state(). 4571 */ 4572 data_mls = multilist_sublist_lock(data_ml, data_idx); 4573 meta_mls = multilist_sublist_lock(meta_ml, meta_idx); 4574 4575 /* 4576 * These two loops are to ensure we skip any markers that 4577 * might be at the tail of the lists due to arc_evict_state(). 4578 */ 4579 4580 for (data_hdr = multilist_sublist_tail(data_mls); data_hdr != NULL; 4581 data_hdr = multilist_sublist_prev(data_mls, data_hdr)) { 4582 if (data_hdr->b_spa != 0) 4583 break; 4584 } 4585 4586 for (meta_hdr = multilist_sublist_tail(meta_mls); meta_hdr != NULL; 4587 meta_hdr = multilist_sublist_prev(meta_mls, meta_hdr)) { 4588 if (meta_hdr->b_spa != 0) 4589 break; 4590 } 4591 4592 if (data_hdr == NULL && meta_hdr == NULL) { 4593 type = ARC_BUFC_DATA; 4594 } else if (data_hdr == NULL) { 4595 ASSERT3P(meta_hdr, !=, NULL); 4596 type = ARC_BUFC_METADATA; 4597 } else if (meta_hdr == NULL) { 4598 ASSERT3P(data_hdr, !=, NULL); 4599 type = ARC_BUFC_DATA; 4600 } else { 4601 ASSERT3P(data_hdr, !=, NULL); 4602 ASSERT3P(meta_hdr, !=, NULL); 4603 4604 /* The headers can't be on the sublist without an L1 header */ 4605 ASSERT(HDR_HAS_L1HDR(data_hdr)); 4606 ASSERT(HDR_HAS_L1HDR(meta_hdr)); 4607 4608 if (data_hdr->b_l1hdr.b_arc_access < 4609 meta_hdr->b_l1hdr.b_arc_access) { 4610 type = ARC_BUFC_DATA; 4611 } else { 4612 type = ARC_BUFC_METADATA; 4613 } 4614 } 4615 4616 multilist_sublist_unlock(meta_mls); 4617 multilist_sublist_unlock(data_mls); 4618 4619 return (type); 4620 } 4621 4622 /* 4623 * Evict buffers from the cache, such that arcstat_size is capped by arc_c. 4624 */ 4625 static uint64_t 4626 arc_evict(void) 4627 { 4628 uint64_t total_evicted = 0; 4629 uint64_t bytes; 4630 int64_t target; 4631 uint64_t asize = aggsum_value(&arc_sums.arcstat_size); 4632 uint64_t ameta = aggsum_value(&arc_sums.arcstat_meta_used); 4633 4634 /* 4635 * If we're over arc_meta_limit, we want to correct that before 4636 * potentially evicting data buffers below. 4637 */ 4638 total_evicted += arc_evict_meta(ameta); 4639 4640 /* 4641 * Adjust MRU size 4642 * 4643 * If we're over the target cache size, we want to evict enough 4644 * from the list to get back to our target size. We don't want 4645 * to evict too much from the MRU, such that it drops below 4646 * arc_p. So, if we're over our target cache size more than 4647 * the MRU is over arc_p, we'll evict enough to get back to 4648 * arc_p here, and then evict more from the MFU below. 4649 */ 4650 target = MIN((int64_t)(asize - arc_c), 4651 (int64_t)(zfs_refcount_count(&arc_anon->arcs_size) + 4652 zfs_refcount_count(&arc_mru->arcs_size) + ameta - arc_p)); 4653 4654 /* 4655 * If we're below arc_meta_min, always prefer to evict data. 4656 * Otherwise, try to satisfy the requested number of bytes to 4657 * evict from the type which contains older buffers; in an 4658 * effort to keep newer buffers in the cache regardless of their 4659 * type. If we cannot satisfy the number of bytes from this 4660 * type, spill over into the next type. 4661 */ 4662 if (arc_evict_type(arc_mru) == ARC_BUFC_METADATA && 4663 ameta > arc_meta_min) { 4664 bytes = arc_evict_impl(arc_mru, 0, target, ARC_BUFC_METADATA); 4665 total_evicted += bytes; 4666 4667 /* 4668 * If we couldn't evict our target number of bytes from 4669 * metadata, we try to get the rest from data. 4670 */ 4671 target -= bytes; 4672 4673 total_evicted += 4674 arc_evict_impl(arc_mru, 0, target, ARC_BUFC_DATA); 4675 } else { 4676 bytes = arc_evict_impl(arc_mru, 0, target, ARC_BUFC_DATA); 4677 total_evicted += bytes; 4678 4679 /* 4680 * If we couldn't evict our target number of bytes from 4681 * data, we try to get the rest from metadata. 4682 */ 4683 target -= bytes; 4684 4685 total_evicted += 4686 arc_evict_impl(arc_mru, 0, target, ARC_BUFC_METADATA); 4687 } 4688 4689 /* 4690 * Re-sum ARC stats after the first round of evictions. 4691 */ 4692 asize = aggsum_value(&arc_sums.arcstat_size); 4693 ameta = aggsum_value(&arc_sums.arcstat_meta_used); 4694 4695 4696 /* 4697 * Adjust MFU size 4698 * 4699 * Now that we've tried to evict enough from the MRU to get its 4700 * size back to arc_p, if we're still above the target cache 4701 * size, we evict the rest from the MFU. 4702 */ 4703 target = asize - arc_c; 4704 4705 if (arc_evict_type(arc_mfu) == ARC_BUFC_METADATA && 4706 ameta > arc_meta_min) { 4707 bytes = arc_evict_impl(arc_mfu, 0, target, ARC_BUFC_METADATA); 4708 total_evicted += bytes; 4709 4710 /* 4711 * If we couldn't evict our target number of bytes from 4712 * metadata, we try to get the rest from data. 4713 */ 4714 target -= bytes; 4715 4716 total_evicted += 4717 arc_evict_impl(arc_mfu, 0, target, ARC_BUFC_DATA); 4718 } else { 4719 bytes = arc_evict_impl(arc_mfu, 0, target, ARC_BUFC_DATA); 4720 total_evicted += bytes; 4721 4722 /* 4723 * If we couldn't evict our target number of bytes from 4724 * data, we try to get the rest from data. 4725 */ 4726 target -= bytes; 4727 4728 total_evicted += 4729 arc_evict_impl(arc_mfu, 0, target, ARC_BUFC_METADATA); 4730 } 4731 4732 /* 4733 * Adjust ghost lists 4734 * 4735 * In addition to the above, the ARC also defines target values 4736 * for the ghost lists. The sum of the mru list and mru ghost 4737 * list should never exceed the target size of the cache, and 4738 * the sum of the mru list, mfu list, mru ghost list, and mfu 4739 * ghost list should never exceed twice the target size of the 4740 * cache. The following logic enforces these limits on the ghost 4741 * caches, and evicts from them as needed. 4742 */ 4743 target = zfs_refcount_count(&arc_mru->arcs_size) + 4744 zfs_refcount_count(&arc_mru_ghost->arcs_size) - arc_c; 4745 4746 bytes = arc_evict_impl(arc_mru_ghost, 0, target, ARC_BUFC_DATA); 4747 total_evicted += bytes; 4748 4749 target -= bytes; 4750 4751 total_evicted += 4752 arc_evict_impl(arc_mru_ghost, 0, target, ARC_BUFC_METADATA); 4753 4754 /* 4755 * We assume the sum of the mru list and mfu list is less than 4756 * or equal to arc_c (we enforced this above), which means we 4757 * can use the simpler of the two equations below: 4758 * 4759 * mru + mfu + mru ghost + mfu ghost <= 2 * arc_c 4760 * mru ghost + mfu ghost <= arc_c 4761 */ 4762 target = zfs_refcount_count(&arc_mru_ghost->arcs_size) + 4763 zfs_refcount_count(&arc_mfu_ghost->arcs_size) - arc_c; 4764 4765 bytes = arc_evict_impl(arc_mfu_ghost, 0, target, ARC_BUFC_DATA); 4766 total_evicted += bytes; 4767 4768 target -= bytes; 4769 4770 total_evicted += 4771 arc_evict_impl(arc_mfu_ghost, 0, target, ARC_BUFC_METADATA); 4772 4773 return (total_evicted); 4774 } 4775 4776 void 4777 arc_flush(spa_t *spa, boolean_t retry) 4778 { 4779 uint64_t guid = 0; 4780 4781 /* 4782 * If retry is B_TRUE, a spa must not be specified since we have 4783 * no good way to determine if all of a spa's buffers have been 4784 * evicted from an arc state. 4785 */ 4786 ASSERT(!retry || spa == NULL); 4787 4788 if (spa != NULL) 4789 guid = spa_load_guid(spa); 4790 4791 (void) arc_flush_state(arc_mru, guid, ARC_BUFC_DATA, retry); 4792 (void) arc_flush_state(arc_mru, guid, ARC_BUFC_METADATA, retry); 4793 4794 (void) arc_flush_state(arc_mfu, guid, ARC_BUFC_DATA, retry); 4795 (void) arc_flush_state(arc_mfu, guid, ARC_BUFC_METADATA, retry); 4796 4797 (void) arc_flush_state(arc_mru_ghost, guid, ARC_BUFC_DATA, retry); 4798 (void) arc_flush_state(arc_mru_ghost, guid, ARC_BUFC_METADATA, retry); 4799 4800 (void) arc_flush_state(arc_mfu_ghost, guid, ARC_BUFC_DATA, retry); 4801 (void) arc_flush_state(arc_mfu_ghost, guid, ARC_BUFC_METADATA, retry); 4802 4803 (void) arc_flush_state(arc_uncached, guid, ARC_BUFC_DATA, retry); 4804 (void) arc_flush_state(arc_uncached, guid, ARC_BUFC_METADATA, retry); 4805 } 4806 4807 void 4808 arc_reduce_target_size(int64_t to_free) 4809 { 4810 uint64_t asize = aggsum_value(&arc_sums.arcstat_size); 4811 4812 /* 4813 * All callers want the ARC to actually evict (at least) this much 4814 * memory. Therefore we reduce from the lower of the current size and 4815 * the target size. This way, even if arc_c is much higher than 4816 * arc_size (as can be the case after many calls to arc_freed(), we will 4817 * immediately have arc_c < arc_size and therefore the arc_evict_zthr 4818 * will evict. 4819 */ 4820 uint64_t c = MIN(arc_c, asize); 4821 4822 if (c > to_free && c - to_free > arc_c_min) { 4823 arc_c = c - to_free; 4824 atomic_add_64(&arc_p, -(arc_p >> arc_shrink_shift)); 4825 if (arc_p > arc_c) 4826 arc_p = (arc_c >> 1); 4827 ASSERT(arc_c >= arc_c_min); 4828 ASSERT((int64_t)arc_p >= 0); 4829 } else { 4830 arc_c = arc_c_min; 4831 } 4832 4833 if (asize > arc_c) { 4834 /* See comment in arc_evict_cb_check() on why lock+flag */ 4835 mutex_enter(&arc_evict_lock); 4836 arc_evict_needed = B_TRUE; 4837 mutex_exit(&arc_evict_lock); 4838 zthr_wakeup(arc_evict_zthr); 4839 } 4840 } 4841 4842 /* 4843 * Determine if the system is under memory pressure and is asking 4844 * to reclaim memory. A return value of B_TRUE indicates that the system 4845 * is under memory pressure and that the arc should adjust accordingly. 4846 */ 4847 boolean_t 4848 arc_reclaim_needed(void) 4849 { 4850 return (arc_available_memory() < 0); 4851 } 4852 4853 void 4854 arc_kmem_reap_soon(void) 4855 { 4856 size_t i; 4857 kmem_cache_t *prev_cache = NULL; 4858 kmem_cache_t *prev_data_cache = NULL; 4859 4860 #ifdef _KERNEL 4861 if ((aggsum_compare(&arc_sums.arcstat_meta_used, 4862 arc_meta_limit) >= 0) && zfs_arc_meta_prune) { 4863 /* 4864 * We are exceeding our meta-data cache limit. 4865 * Prune some entries to release holds on meta-data. 4866 */ 4867 arc_prune_async(zfs_arc_meta_prune); 4868 } 4869 #if defined(_ILP32) 4870 /* 4871 * Reclaim unused memory from all kmem caches. 4872 */ 4873 kmem_reap(); 4874 #endif 4875 #endif 4876 4877 for (i = 0; i < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT; i++) { 4878 #if defined(_ILP32) 4879 /* reach upper limit of cache size on 32-bit */ 4880 if (zio_buf_cache[i] == NULL) 4881 break; 4882 #endif 4883 if (zio_buf_cache[i] != prev_cache) { 4884 prev_cache = zio_buf_cache[i]; 4885 kmem_cache_reap_now(zio_buf_cache[i]); 4886 } 4887 if (zio_data_buf_cache[i] != prev_data_cache) { 4888 prev_data_cache = zio_data_buf_cache[i]; 4889 kmem_cache_reap_now(zio_data_buf_cache[i]); 4890 } 4891 } 4892 kmem_cache_reap_now(buf_cache); 4893 kmem_cache_reap_now(hdr_full_cache); 4894 kmem_cache_reap_now(hdr_l2only_cache); 4895 kmem_cache_reap_now(zfs_btree_leaf_cache); 4896 abd_cache_reap_now(); 4897 } 4898 4899 static boolean_t 4900 arc_evict_cb_check(void *arg, zthr_t *zthr) 4901 { 4902 (void) arg, (void) zthr; 4903 4904 #ifdef ZFS_DEBUG 4905 /* 4906 * This is necessary in order to keep the kstat information 4907 * up to date for tools that display kstat data such as the 4908 * mdb ::arc dcmd and the Linux crash utility. These tools 4909 * typically do not call kstat's update function, but simply 4910 * dump out stats from the most recent update. Without 4911 * this call, these commands may show stale stats for the 4912 * anon, mru, mru_ghost, mfu, and mfu_ghost lists. Even 4913 * with this call, the data might be out of date if the 4914 * evict thread hasn't been woken recently; but that should 4915 * suffice. The arc_state_t structures can be queried 4916 * directly if more accurate information is needed. 4917 */ 4918 if (arc_ksp != NULL) 4919 arc_ksp->ks_update(arc_ksp, KSTAT_READ); 4920 #endif 4921 4922 /* 4923 * We have to rely on arc_wait_for_eviction() to tell us when to 4924 * evict, rather than checking if we are overflowing here, so that we 4925 * are sure to not leave arc_wait_for_eviction() waiting on aew_cv. 4926 * If we have become "not overflowing" since arc_wait_for_eviction() 4927 * checked, we need to wake it up. We could broadcast the CV here, 4928 * but arc_wait_for_eviction() may have not yet gone to sleep. We 4929 * would need to use a mutex to ensure that this function doesn't 4930 * broadcast until arc_wait_for_eviction() has gone to sleep (e.g. 4931 * the arc_evict_lock). However, the lock ordering of such a lock 4932 * would necessarily be incorrect with respect to the zthr_lock, 4933 * which is held before this function is called, and is held by 4934 * arc_wait_for_eviction() when it calls zthr_wakeup(). 4935 */ 4936 if (arc_evict_needed) 4937 return (B_TRUE); 4938 4939 /* 4940 * If we have buffers in uncached state, evict them periodically. 4941 */ 4942 return ((zfs_refcount_count(&arc_uncached->arcs_esize[ARC_BUFC_DATA]) + 4943 zfs_refcount_count(&arc_uncached->arcs_esize[ARC_BUFC_METADATA]) && 4944 ddi_get_lbolt() - arc_last_uncached_flush > 4945 MSEC_TO_TICK(arc_min_prefetch_ms / 2))); 4946 } 4947 4948 /* 4949 * Keep arc_size under arc_c by running arc_evict which evicts data 4950 * from the ARC. 4951 */ 4952 static void 4953 arc_evict_cb(void *arg, zthr_t *zthr) 4954 { 4955 (void) arg, (void) zthr; 4956 4957 uint64_t evicted = 0; 4958 fstrans_cookie_t cookie = spl_fstrans_mark(); 4959 4960 /* Always try to evict from uncached state. */ 4961 arc_last_uncached_flush = ddi_get_lbolt(); 4962 evicted += arc_flush_state(arc_uncached, 0, ARC_BUFC_DATA, B_FALSE); 4963 evicted += arc_flush_state(arc_uncached, 0, ARC_BUFC_METADATA, B_FALSE); 4964 4965 /* Evict from other states only if told to. */ 4966 if (arc_evict_needed) 4967 evicted += arc_evict(); 4968 4969 /* 4970 * If evicted is zero, we couldn't evict anything 4971 * via arc_evict(). This could be due to hash lock 4972 * collisions, but more likely due to the majority of 4973 * arc buffers being unevictable. Therefore, even if 4974 * arc_size is above arc_c, another pass is unlikely to 4975 * be helpful and could potentially cause us to enter an 4976 * infinite loop. Additionally, zthr_iscancelled() is 4977 * checked here so that if the arc is shutting down, the 4978 * broadcast will wake any remaining arc evict waiters. 4979 */ 4980 mutex_enter(&arc_evict_lock); 4981 arc_evict_needed = !zthr_iscancelled(arc_evict_zthr) && 4982 evicted > 0 && aggsum_compare(&arc_sums.arcstat_size, arc_c) > 0; 4983 if (!arc_evict_needed) { 4984 /* 4985 * We're either no longer overflowing, or we 4986 * can't evict anything more, so we should wake 4987 * arc_get_data_impl() sooner. 4988 */ 4989 arc_evict_waiter_t *aw; 4990 while ((aw = list_remove_head(&arc_evict_waiters)) != NULL) { 4991 cv_broadcast(&aw->aew_cv); 4992 } 4993 arc_set_need_free(); 4994 } 4995 mutex_exit(&arc_evict_lock); 4996 spl_fstrans_unmark(cookie); 4997 } 4998 4999 static boolean_t 5000 arc_reap_cb_check(void *arg, zthr_t *zthr) 5001 { 5002 (void) arg, (void) zthr; 5003 5004 int64_t free_memory = arc_available_memory(); 5005 static int reap_cb_check_counter = 0; 5006 5007 /* 5008 * If a kmem reap is already active, don't schedule more. We must 5009 * check for this because kmem_cache_reap_soon() won't actually 5010 * block on the cache being reaped (this is to prevent callers from 5011 * becoming implicitly blocked by a system-wide kmem reap -- which, 5012 * on a system with many, many full magazines, can take minutes). 5013 */ 5014 if (!kmem_cache_reap_active() && free_memory < 0) { 5015 5016 arc_no_grow = B_TRUE; 5017 arc_warm = B_TRUE; 5018 /* 5019 * Wait at least zfs_grow_retry (default 5) seconds 5020 * before considering growing. 5021 */ 5022 arc_growtime = gethrtime() + SEC2NSEC(arc_grow_retry); 5023 return (B_TRUE); 5024 } else if (free_memory < arc_c >> arc_no_grow_shift) { 5025 arc_no_grow = B_TRUE; 5026 } else if (gethrtime() >= arc_growtime) { 5027 arc_no_grow = B_FALSE; 5028 } 5029 5030 /* 5031 * Called unconditionally every 60 seconds to reclaim unused 5032 * zstd compression and decompression context. This is done 5033 * here to avoid the need for an independent thread. 5034 */ 5035 if (!((reap_cb_check_counter++) % 60)) 5036 zfs_zstd_cache_reap_now(); 5037 5038 return (B_FALSE); 5039 } 5040 5041 /* 5042 * Keep enough free memory in the system by reaping the ARC's kmem 5043 * caches. To cause more slabs to be reapable, we may reduce the 5044 * target size of the cache (arc_c), causing the arc_evict_cb() 5045 * to free more buffers. 5046 */ 5047 static void 5048 arc_reap_cb(void *arg, zthr_t *zthr) 5049 { 5050 (void) arg, (void) zthr; 5051 5052 int64_t free_memory; 5053 fstrans_cookie_t cookie = spl_fstrans_mark(); 5054 5055 /* 5056 * Kick off asynchronous kmem_reap()'s of all our caches. 5057 */ 5058 arc_kmem_reap_soon(); 5059 5060 /* 5061 * Wait at least arc_kmem_cache_reap_retry_ms between 5062 * arc_kmem_reap_soon() calls. Without this check it is possible to 5063 * end up in a situation where we spend lots of time reaping 5064 * caches, while we're near arc_c_min. Waiting here also gives the 5065 * subsequent free memory check a chance of finding that the 5066 * asynchronous reap has already freed enough memory, and we don't 5067 * need to call arc_reduce_target_size(). 5068 */ 5069 delay((hz * arc_kmem_cache_reap_retry_ms + 999) / 1000); 5070 5071 /* 5072 * Reduce the target size as needed to maintain the amount of free 5073 * memory in the system at a fraction of the arc_size (1/128th by 5074 * default). If oversubscribed (free_memory < 0) then reduce the 5075 * target arc_size by the deficit amount plus the fractional 5076 * amount. If free memory is positive but less than the fractional 5077 * amount, reduce by what is needed to hit the fractional amount. 5078 */ 5079 free_memory = arc_available_memory(); 5080 5081 int64_t can_free = arc_c - arc_c_min; 5082 if (can_free > 0) { 5083 int64_t to_free = (can_free >> arc_shrink_shift) - free_memory; 5084 if (to_free > 0) 5085 arc_reduce_target_size(to_free); 5086 } 5087 spl_fstrans_unmark(cookie); 5088 } 5089 5090 #ifdef _KERNEL 5091 /* 5092 * Determine the amount of memory eligible for eviction contained in the 5093 * ARC. All clean data reported by the ghost lists can always be safely 5094 * evicted. Due to arc_c_min, the same does not hold for all clean data 5095 * contained by the regular mru and mfu lists. 5096 * 5097 * In the case of the regular mru and mfu lists, we need to report as 5098 * much clean data as possible, such that evicting that same reported 5099 * data will not bring arc_size below arc_c_min. Thus, in certain 5100 * circumstances, the total amount of clean data in the mru and mfu 5101 * lists might not actually be evictable. 5102 * 5103 * The following two distinct cases are accounted for: 5104 * 5105 * 1. The sum of the amount of dirty data contained by both the mru and 5106 * mfu lists, plus the ARC's other accounting (e.g. the anon list), 5107 * is greater than or equal to arc_c_min. 5108 * (i.e. amount of dirty data >= arc_c_min) 5109 * 5110 * This is the easy case; all clean data contained by the mru and mfu 5111 * lists is evictable. Evicting all clean data can only drop arc_size 5112 * to the amount of dirty data, which is greater than arc_c_min. 5113 * 5114 * 2. The sum of the amount of dirty data contained by both the mru and 5115 * mfu lists, plus the ARC's other accounting (e.g. the anon list), 5116 * is less than arc_c_min. 5117 * (i.e. arc_c_min > amount of dirty data) 5118 * 5119 * 2.1. arc_size is greater than or equal arc_c_min. 5120 * (i.e. arc_size >= arc_c_min > amount of dirty data) 5121 * 5122 * In this case, not all clean data from the regular mru and mfu 5123 * lists is actually evictable; we must leave enough clean data 5124 * to keep arc_size above arc_c_min. Thus, the maximum amount of 5125 * evictable data from the two lists combined, is exactly the 5126 * difference between arc_size and arc_c_min. 5127 * 5128 * 2.2. arc_size is less than arc_c_min 5129 * (i.e. arc_c_min > arc_size > amount of dirty data) 5130 * 5131 * In this case, none of the data contained in the mru and mfu 5132 * lists is evictable, even if it's clean. Since arc_size is 5133 * already below arc_c_min, evicting any more would only 5134 * increase this negative difference. 5135 */ 5136 5137 #endif /* _KERNEL */ 5138 5139 /* 5140 * Adapt arc info given the number of bytes we are trying to add and 5141 * the state that we are coming from. This function is only called 5142 * when we are adding new content to the cache. 5143 */ 5144 static void 5145 arc_adapt(int bytes, arc_state_t *state) 5146 { 5147 int mult; 5148 uint64_t arc_p_min = (arc_c >> arc_p_min_shift); 5149 int64_t mrug_size = zfs_refcount_count(&arc_mru_ghost->arcs_size); 5150 int64_t mfug_size = zfs_refcount_count(&arc_mfu_ghost->arcs_size); 5151 5152 ASSERT(bytes > 0); 5153 /* 5154 * Adapt the target size of the MRU list: 5155 * - if we just hit in the MRU ghost list, then increase 5156 * the target size of the MRU list. 5157 * - if we just hit in the MFU ghost list, then increase 5158 * the target size of the MFU list by decreasing the 5159 * target size of the MRU list. 5160 */ 5161 if (state == arc_mru_ghost) { 5162 mult = (mrug_size >= mfug_size) ? 1 : (mfug_size / mrug_size); 5163 if (!zfs_arc_p_dampener_disable) 5164 mult = MIN(mult, 10); /* avoid wild arc_p adjustment */ 5165 5166 arc_p = MIN(arc_c - arc_p_min, arc_p + (uint64_t)bytes * mult); 5167 } else if (state == arc_mfu_ghost) { 5168 uint64_t delta; 5169 5170 mult = (mfug_size >= mrug_size) ? 1 : (mrug_size / mfug_size); 5171 if (!zfs_arc_p_dampener_disable) 5172 mult = MIN(mult, 10); 5173 5174 delta = MIN(bytes * mult, arc_p); 5175 arc_p = MAX(arc_p_min, arc_p - delta); 5176 } 5177 ASSERT((int64_t)arc_p >= 0); 5178 5179 /* 5180 * Wake reap thread if we do not have any available memory 5181 */ 5182 if (arc_reclaim_needed()) { 5183 zthr_wakeup(arc_reap_zthr); 5184 return; 5185 } 5186 5187 if (arc_no_grow) 5188 return; 5189 5190 if (arc_c >= arc_c_max) 5191 return; 5192 5193 /* 5194 * If we're within (2 * maxblocksize) bytes of the target 5195 * cache size, increment the target cache size 5196 */ 5197 ASSERT3U(arc_c, >=, 2ULL << SPA_MAXBLOCKSHIFT); 5198 if (aggsum_upper_bound(&arc_sums.arcstat_size) >= 5199 arc_c - (2ULL << SPA_MAXBLOCKSHIFT)) { 5200 atomic_add_64(&arc_c, (int64_t)bytes); 5201 if (arc_c > arc_c_max) 5202 arc_c = arc_c_max; 5203 else if (state == arc_anon && arc_p < arc_c >> 1) 5204 atomic_add_64(&arc_p, (int64_t)bytes); 5205 if (arc_p > arc_c) 5206 arc_p = arc_c; 5207 } 5208 ASSERT((int64_t)arc_p >= 0); 5209 } 5210 5211 /* 5212 * Check if arc_size has grown past our upper threshold, determined by 5213 * zfs_arc_overflow_shift. 5214 */ 5215 static arc_ovf_level_t 5216 arc_is_overflowing(boolean_t use_reserve) 5217 { 5218 /* Always allow at least one block of overflow */ 5219 int64_t overflow = MAX(SPA_MAXBLOCKSIZE, 5220 arc_c >> zfs_arc_overflow_shift); 5221 5222 /* 5223 * We just compare the lower bound here for performance reasons. Our 5224 * primary goals are to make sure that the arc never grows without 5225 * bound, and that it can reach its maximum size. This check 5226 * accomplishes both goals. The maximum amount we could run over by is 5227 * 2 * aggsum_borrow_multiplier * NUM_CPUS * the average size of a block 5228 * in the ARC. In practice, that's in the tens of MB, which is low 5229 * enough to be safe. 5230 */ 5231 int64_t over = aggsum_lower_bound(&arc_sums.arcstat_size) - 5232 arc_c - overflow / 2; 5233 if (!use_reserve) 5234 overflow /= 2; 5235 return (over < 0 ? ARC_OVF_NONE : 5236 over < overflow ? ARC_OVF_SOME : ARC_OVF_SEVERE); 5237 } 5238 5239 static abd_t * 5240 arc_get_data_abd(arc_buf_hdr_t *hdr, uint64_t size, const void *tag, 5241 int alloc_flags) 5242 { 5243 arc_buf_contents_t type = arc_buf_type(hdr); 5244 5245 arc_get_data_impl(hdr, size, tag, alloc_flags); 5246 if (alloc_flags & ARC_HDR_ALLOC_LINEAR) 5247 return (abd_alloc_linear(size, type == ARC_BUFC_METADATA)); 5248 else 5249 return (abd_alloc(size, type == ARC_BUFC_METADATA)); 5250 } 5251 5252 static void * 5253 arc_get_data_buf(arc_buf_hdr_t *hdr, uint64_t size, const void *tag) 5254 { 5255 arc_buf_contents_t type = arc_buf_type(hdr); 5256 5257 arc_get_data_impl(hdr, size, tag, ARC_HDR_DO_ADAPT); 5258 if (type == ARC_BUFC_METADATA) { 5259 return (zio_buf_alloc(size)); 5260 } else { 5261 ASSERT(type == ARC_BUFC_DATA); 5262 return (zio_data_buf_alloc(size)); 5263 } 5264 } 5265 5266 /* 5267 * Wait for the specified amount of data (in bytes) to be evicted from the 5268 * ARC, and for there to be sufficient free memory in the system. Waiting for 5269 * eviction ensures that the memory used by the ARC decreases. Waiting for 5270 * free memory ensures that the system won't run out of free pages, regardless 5271 * of ARC behavior and settings. See arc_lowmem_init(). 5272 */ 5273 void 5274 arc_wait_for_eviction(uint64_t amount, boolean_t use_reserve) 5275 { 5276 switch (arc_is_overflowing(use_reserve)) { 5277 case ARC_OVF_NONE: 5278 return; 5279 case ARC_OVF_SOME: 5280 /* 5281 * This is a bit racy without taking arc_evict_lock, but the 5282 * worst that can happen is we either call zthr_wakeup() extra 5283 * time due to race with other thread here, or the set flag 5284 * get cleared by arc_evict_cb(), which is unlikely due to 5285 * big hysteresis, but also not important since at this level 5286 * of overflow the eviction is purely advisory. Same time 5287 * taking the global lock here every time without waiting for 5288 * the actual eviction creates a significant lock contention. 5289 */ 5290 if (!arc_evict_needed) { 5291 arc_evict_needed = B_TRUE; 5292 zthr_wakeup(arc_evict_zthr); 5293 } 5294 return; 5295 case ARC_OVF_SEVERE: 5296 default: 5297 { 5298 arc_evict_waiter_t aw; 5299 list_link_init(&aw.aew_node); 5300 cv_init(&aw.aew_cv, NULL, CV_DEFAULT, NULL); 5301 5302 uint64_t last_count = 0; 5303 mutex_enter(&arc_evict_lock); 5304 if (!list_is_empty(&arc_evict_waiters)) { 5305 arc_evict_waiter_t *last = 5306 list_tail(&arc_evict_waiters); 5307 last_count = last->aew_count; 5308 } else if (!arc_evict_needed) { 5309 arc_evict_needed = B_TRUE; 5310 zthr_wakeup(arc_evict_zthr); 5311 } 5312 /* 5313 * Note, the last waiter's count may be less than 5314 * arc_evict_count if we are low on memory in which 5315 * case arc_evict_state_impl() may have deferred 5316 * wakeups (but still incremented arc_evict_count). 5317 */ 5318 aw.aew_count = MAX(last_count, arc_evict_count) + amount; 5319 5320 list_insert_tail(&arc_evict_waiters, &aw); 5321 5322 arc_set_need_free(); 5323 5324 DTRACE_PROBE3(arc__wait__for__eviction, 5325 uint64_t, amount, 5326 uint64_t, arc_evict_count, 5327 uint64_t, aw.aew_count); 5328 5329 /* 5330 * We will be woken up either when arc_evict_count reaches 5331 * aew_count, or when the ARC is no longer overflowing and 5332 * eviction completes. 5333 * In case of "false" wakeup, we will still be on the list. 5334 */ 5335 do { 5336 cv_wait(&aw.aew_cv, &arc_evict_lock); 5337 } while (list_link_active(&aw.aew_node)); 5338 mutex_exit(&arc_evict_lock); 5339 5340 cv_destroy(&aw.aew_cv); 5341 } 5342 } 5343 } 5344 5345 /* 5346 * Allocate a block and return it to the caller. If we are hitting the 5347 * hard limit for the cache size, we must sleep, waiting for the eviction 5348 * thread to catch up. If we're past the target size but below the hard 5349 * limit, we'll only signal the reclaim thread and continue on. 5350 */ 5351 static void 5352 arc_get_data_impl(arc_buf_hdr_t *hdr, uint64_t size, const void *tag, 5353 int alloc_flags) 5354 { 5355 arc_state_t *state = hdr->b_l1hdr.b_state; 5356 arc_buf_contents_t type = arc_buf_type(hdr); 5357 5358 if (alloc_flags & ARC_HDR_DO_ADAPT) 5359 arc_adapt(size, state); 5360 5361 /* 5362 * If arc_size is currently overflowing, we must be adding data 5363 * faster than we are evicting. To ensure we don't compound the 5364 * problem by adding more data and forcing arc_size to grow even 5365 * further past it's target size, we wait for the eviction thread to 5366 * make some progress. We also wait for there to be sufficient free 5367 * memory in the system, as measured by arc_free_memory(). 5368 * 5369 * Specifically, we wait for zfs_arc_eviction_pct percent of the 5370 * requested size to be evicted. This should be more than 100%, to 5371 * ensure that that progress is also made towards getting arc_size 5372 * under arc_c. See the comment above zfs_arc_eviction_pct. 5373 */ 5374 arc_wait_for_eviction(size * zfs_arc_eviction_pct / 100, 5375 alloc_flags & ARC_HDR_USE_RESERVE); 5376 5377 VERIFY3U(hdr->b_type, ==, type); 5378 if (type == ARC_BUFC_METADATA) { 5379 arc_space_consume(size, ARC_SPACE_META); 5380 } else { 5381 arc_space_consume(size, ARC_SPACE_DATA); 5382 } 5383 5384 /* 5385 * Update the state size. Note that ghost states have a 5386 * "ghost size" and so don't need to be updated. 5387 */ 5388 if (!GHOST_STATE(state)) { 5389 5390 (void) zfs_refcount_add_many(&state->arcs_size, size, tag); 5391 5392 /* 5393 * If this is reached via arc_read, the link is 5394 * protected by the hash lock. If reached via 5395 * arc_buf_alloc, the header should not be accessed by 5396 * any other thread. And, if reached via arc_read_done, 5397 * the hash lock will protect it if it's found in the 5398 * hash table; otherwise no other thread should be 5399 * trying to [add|remove]_reference it. 5400 */ 5401 if (multilist_link_active(&hdr->b_l1hdr.b_arc_node)) { 5402 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 5403 (void) zfs_refcount_add_many(&state->arcs_esize[type], 5404 size, tag); 5405 } 5406 5407 /* 5408 * If we are growing the cache, and we are adding anonymous 5409 * data, and we have outgrown arc_p, update arc_p 5410 */ 5411 if (aggsum_upper_bound(&arc_sums.arcstat_size) < arc_c && 5412 hdr->b_l1hdr.b_state == arc_anon && 5413 (zfs_refcount_count(&arc_anon->arcs_size) + 5414 zfs_refcount_count(&arc_mru->arcs_size) > arc_p && 5415 arc_p < arc_c >> 1)) 5416 arc_p = MIN(arc_c, arc_p + size); 5417 } 5418 } 5419 5420 static void 5421 arc_free_data_abd(arc_buf_hdr_t *hdr, abd_t *abd, uint64_t size, 5422 const void *tag) 5423 { 5424 arc_free_data_impl(hdr, size, tag); 5425 abd_free(abd); 5426 } 5427 5428 static void 5429 arc_free_data_buf(arc_buf_hdr_t *hdr, void *buf, uint64_t size, const void *tag) 5430 { 5431 arc_buf_contents_t type = arc_buf_type(hdr); 5432 5433 arc_free_data_impl(hdr, size, tag); 5434 if (type == ARC_BUFC_METADATA) { 5435 zio_buf_free(buf, size); 5436 } else { 5437 ASSERT(type == ARC_BUFC_DATA); 5438 zio_data_buf_free(buf, size); 5439 } 5440 } 5441 5442 /* 5443 * Free the arc data buffer. 5444 */ 5445 static void 5446 arc_free_data_impl(arc_buf_hdr_t *hdr, uint64_t size, const void *tag) 5447 { 5448 arc_state_t *state = hdr->b_l1hdr.b_state; 5449 arc_buf_contents_t type = arc_buf_type(hdr); 5450 5451 /* protected by hash lock, if in the hash table */ 5452 if (multilist_link_active(&hdr->b_l1hdr.b_arc_node)) { 5453 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 5454 ASSERT(state != arc_anon && state != arc_l2c_only); 5455 5456 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 5457 size, tag); 5458 } 5459 (void) zfs_refcount_remove_many(&state->arcs_size, size, tag); 5460 5461 VERIFY3U(hdr->b_type, ==, type); 5462 if (type == ARC_BUFC_METADATA) { 5463 arc_space_return(size, ARC_SPACE_META); 5464 } else { 5465 ASSERT(type == ARC_BUFC_DATA); 5466 arc_space_return(size, ARC_SPACE_DATA); 5467 } 5468 } 5469 5470 /* 5471 * This routine is called whenever a buffer is accessed. 5472 */ 5473 static void 5474 arc_access(arc_buf_hdr_t *hdr, arc_flags_t arc_flags, boolean_t hit) 5475 { 5476 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 5477 ASSERT(HDR_HAS_L1HDR(hdr)); 5478 5479 /* 5480 * Update buffer prefetch status. 5481 */ 5482 boolean_t was_prefetch = HDR_PREFETCH(hdr); 5483 boolean_t now_prefetch = arc_flags & ARC_FLAG_PREFETCH; 5484 if (was_prefetch != now_prefetch) { 5485 if (was_prefetch) { 5486 ARCSTAT_CONDSTAT(hit, demand_hit, demand_iohit, 5487 HDR_PRESCIENT_PREFETCH(hdr), prescient, predictive, 5488 prefetch); 5489 } 5490 if (HDR_HAS_L2HDR(hdr)) 5491 l2arc_hdr_arcstats_decrement_state(hdr); 5492 if (was_prefetch) { 5493 arc_hdr_clear_flags(hdr, 5494 ARC_FLAG_PREFETCH | ARC_FLAG_PRESCIENT_PREFETCH); 5495 } else { 5496 arc_hdr_set_flags(hdr, ARC_FLAG_PREFETCH); 5497 } 5498 if (HDR_HAS_L2HDR(hdr)) 5499 l2arc_hdr_arcstats_increment_state(hdr); 5500 } 5501 if (now_prefetch) { 5502 if (arc_flags & ARC_FLAG_PRESCIENT_PREFETCH) { 5503 arc_hdr_set_flags(hdr, ARC_FLAG_PRESCIENT_PREFETCH); 5504 ARCSTAT_BUMP(arcstat_prescient_prefetch); 5505 } else { 5506 ARCSTAT_BUMP(arcstat_predictive_prefetch); 5507 } 5508 } 5509 if (arc_flags & ARC_FLAG_L2CACHE) 5510 arc_hdr_set_flags(hdr, ARC_FLAG_L2CACHE); 5511 5512 clock_t now = ddi_get_lbolt(); 5513 if (hdr->b_l1hdr.b_state == arc_anon) { 5514 arc_state_t *new_state; 5515 /* 5516 * This buffer is not in the cache, and does not appear in 5517 * our "ghost" lists. Add it to the MRU or uncached state. 5518 */ 5519 ASSERT0(hdr->b_l1hdr.b_arc_access); 5520 hdr->b_l1hdr.b_arc_access = now; 5521 if (HDR_UNCACHED(hdr)) { 5522 new_state = arc_uncached; 5523 DTRACE_PROBE1(new_state__uncached, arc_buf_hdr_t *, 5524 hdr); 5525 } else { 5526 new_state = arc_mru; 5527 DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, hdr); 5528 } 5529 arc_change_state(new_state, hdr); 5530 } else if (hdr->b_l1hdr.b_state == arc_mru) { 5531 /* 5532 * This buffer has been accessed once recently and either 5533 * its read is still in progress or it is in the cache. 5534 */ 5535 if (HDR_IO_IN_PROGRESS(hdr)) { 5536 hdr->b_l1hdr.b_arc_access = now; 5537 return; 5538 } 5539 hdr->b_l1hdr.b_mru_hits++; 5540 ARCSTAT_BUMP(arcstat_mru_hits); 5541 5542 /* 5543 * If the previous access was a prefetch, then it already 5544 * handled possible promotion, so nothing more to do for now. 5545 */ 5546 if (was_prefetch) { 5547 hdr->b_l1hdr.b_arc_access = now; 5548 return; 5549 } 5550 5551 /* 5552 * If more than ARC_MINTIME have passed from the previous 5553 * hit, promote the buffer to the MFU state. 5554 */ 5555 if (ddi_time_after(now, hdr->b_l1hdr.b_arc_access + 5556 ARC_MINTIME)) { 5557 hdr->b_l1hdr.b_arc_access = now; 5558 DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, hdr); 5559 arc_change_state(arc_mfu, hdr); 5560 } 5561 } else if (hdr->b_l1hdr.b_state == arc_mru_ghost) { 5562 arc_state_t *new_state; 5563 /* 5564 * This buffer has been accessed once recently, but was 5565 * evicted from the cache. Would we have bigger MRU, it 5566 * would be an MRU hit, so handle it the same way, except 5567 * we don't need to check the previous access time. 5568 */ 5569 hdr->b_l1hdr.b_mru_ghost_hits++; 5570 ARCSTAT_BUMP(arcstat_mru_ghost_hits); 5571 hdr->b_l1hdr.b_arc_access = now; 5572 if (was_prefetch) { 5573 new_state = arc_mru; 5574 DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, hdr); 5575 } else { 5576 new_state = arc_mfu; 5577 DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, hdr); 5578 } 5579 arc_change_state(new_state, hdr); 5580 } else if (hdr->b_l1hdr.b_state == arc_mfu) { 5581 /* 5582 * This buffer has been accessed more than once and either 5583 * still in the cache or being restored from one of ghosts. 5584 */ 5585 if (!HDR_IO_IN_PROGRESS(hdr)) { 5586 hdr->b_l1hdr.b_mfu_hits++; 5587 ARCSTAT_BUMP(arcstat_mfu_hits); 5588 } 5589 hdr->b_l1hdr.b_arc_access = now; 5590 } else if (hdr->b_l1hdr.b_state == arc_mfu_ghost) { 5591 /* 5592 * This buffer has been accessed more than once recently, but 5593 * has been evicted from the cache. Would we have bigger MFU 5594 * it would stay in cache, so move it back to MFU state. 5595 */ 5596 hdr->b_l1hdr.b_mfu_ghost_hits++; 5597 ARCSTAT_BUMP(arcstat_mfu_ghost_hits); 5598 hdr->b_l1hdr.b_arc_access = now; 5599 DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, hdr); 5600 arc_change_state(arc_mfu, hdr); 5601 } else if (hdr->b_l1hdr.b_state == arc_uncached) { 5602 /* 5603 * This buffer is uncacheable, but we got a hit. Probably 5604 * a demand read after prefetch. Nothing more to do here. 5605 */ 5606 if (!HDR_IO_IN_PROGRESS(hdr)) 5607 ARCSTAT_BUMP(arcstat_uncached_hits); 5608 hdr->b_l1hdr.b_arc_access = now; 5609 } else if (hdr->b_l1hdr.b_state == arc_l2c_only) { 5610 /* 5611 * This buffer is on the 2nd Level ARC and was not accessed 5612 * for a long time, so treat it as new and put into MRU. 5613 */ 5614 hdr->b_l1hdr.b_arc_access = now; 5615 DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, hdr); 5616 arc_change_state(arc_mru, hdr); 5617 } else { 5618 cmn_err(CE_PANIC, "invalid arc state 0x%p", 5619 hdr->b_l1hdr.b_state); 5620 } 5621 } 5622 5623 /* 5624 * This routine is called by dbuf_hold() to update the arc_access() state 5625 * which otherwise would be skipped for entries in the dbuf cache. 5626 */ 5627 void 5628 arc_buf_access(arc_buf_t *buf) 5629 { 5630 arc_buf_hdr_t *hdr = buf->b_hdr; 5631 5632 /* 5633 * Avoid taking the hash_lock when possible as an optimization. 5634 * The header must be checked again under the hash_lock in order 5635 * to handle the case where it is concurrently being released. 5636 */ 5637 if (hdr->b_l1hdr.b_state == arc_anon || HDR_EMPTY(hdr)) 5638 return; 5639 5640 kmutex_t *hash_lock = HDR_LOCK(hdr); 5641 mutex_enter(hash_lock); 5642 5643 if (hdr->b_l1hdr.b_state == arc_anon || HDR_EMPTY(hdr)) { 5644 mutex_exit(hash_lock); 5645 ARCSTAT_BUMP(arcstat_access_skip); 5646 return; 5647 } 5648 5649 ASSERT(hdr->b_l1hdr.b_state == arc_mru || 5650 hdr->b_l1hdr.b_state == arc_mfu || 5651 hdr->b_l1hdr.b_state == arc_uncached); 5652 5653 DTRACE_PROBE1(arc__hit, arc_buf_hdr_t *, hdr); 5654 arc_access(hdr, 0, B_TRUE); 5655 mutex_exit(hash_lock); 5656 5657 ARCSTAT_BUMP(arcstat_hits); 5658 ARCSTAT_CONDSTAT(B_TRUE /* demand */, demand, prefetch, 5659 !HDR_ISTYPE_METADATA(hdr), data, metadata, hits); 5660 } 5661 5662 /* a generic arc_read_done_func_t which you can use */ 5663 void 5664 arc_bcopy_func(zio_t *zio, const zbookmark_phys_t *zb, const blkptr_t *bp, 5665 arc_buf_t *buf, void *arg) 5666 { 5667 (void) zio, (void) zb, (void) bp; 5668 5669 if (buf == NULL) 5670 return; 5671 5672 memcpy(arg, buf->b_data, arc_buf_size(buf)); 5673 arc_buf_destroy(buf, arg); 5674 } 5675 5676 /* a generic arc_read_done_func_t */ 5677 void 5678 arc_getbuf_func(zio_t *zio, const zbookmark_phys_t *zb, const blkptr_t *bp, 5679 arc_buf_t *buf, void *arg) 5680 { 5681 (void) zb, (void) bp; 5682 arc_buf_t **bufp = arg; 5683 5684 if (buf == NULL) { 5685 ASSERT(zio == NULL || zio->io_error != 0); 5686 *bufp = NULL; 5687 } else { 5688 ASSERT(zio == NULL || zio->io_error == 0); 5689 *bufp = buf; 5690 ASSERT(buf->b_data != NULL); 5691 } 5692 } 5693 5694 static void 5695 arc_hdr_verify(arc_buf_hdr_t *hdr, blkptr_t *bp) 5696 { 5697 if (BP_IS_HOLE(bp) || BP_IS_EMBEDDED(bp)) { 5698 ASSERT3U(HDR_GET_PSIZE(hdr), ==, 0); 5699 ASSERT3U(arc_hdr_get_compress(hdr), ==, ZIO_COMPRESS_OFF); 5700 } else { 5701 if (HDR_COMPRESSION_ENABLED(hdr)) { 5702 ASSERT3U(arc_hdr_get_compress(hdr), ==, 5703 BP_GET_COMPRESS(bp)); 5704 } 5705 ASSERT3U(HDR_GET_LSIZE(hdr), ==, BP_GET_LSIZE(bp)); 5706 ASSERT3U(HDR_GET_PSIZE(hdr), ==, BP_GET_PSIZE(bp)); 5707 ASSERT3U(!!HDR_PROTECTED(hdr), ==, BP_IS_PROTECTED(bp)); 5708 } 5709 } 5710 5711 static void 5712 arc_read_done(zio_t *zio) 5713 { 5714 blkptr_t *bp = zio->io_bp; 5715 arc_buf_hdr_t *hdr = zio->io_private; 5716 kmutex_t *hash_lock = NULL; 5717 arc_callback_t *callback_list; 5718 arc_callback_t *acb; 5719 5720 /* 5721 * The hdr was inserted into hash-table and removed from lists 5722 * prior to starting I/O. We should find this header, since 5723 * it's in the hash table, and it should be legit since it's 5724 * not possible to evict it during the I/O. The only possible 5725 * reason for it not to be found is if we were freed during the 5726 * read. 5727 */ 5728 if (HDR_IN_HASH_TABLE(hdr)) { 5729 arc_buf_hdr_t *found; 5730 5731 ASSERT3U(hdr->b_birth, ==, BP_PHYSICAL_BIRTH(zio->io_bp)); 5732 ASSERT3U(hdr->b_dva.dva_word[0], ==, 5733 BP_IDENTITY(zio->io_bp)->dva_word[0]); 5734 ASSERT3U(hdr->b_dva.dva_word[1], ==, 5735 BP_IDENTITY(zio->io_bp)->dva_word[1]); 5736 5737 found = buf_hash_find(hdr->b_spa, zio->io_bp, &hash_lock); 5738 5739 ASSERT((found == hdr && 5740 DVA_EQUAL(&hdr->b_dva, BP_IDENTITY(zio->io_bp))) || 5741 (found == hdr && HDR_L2_READING(hdr))); 5742 ASSERT3P(hash_lock, !=, NULL); 5743 } 5744 5745 if (BP_IS_PROTECTED(bp)) { 5746 hdr->b_crypt_hdr.b_ot = BP_GET_TYPE(bp); 5747 hdr->b_crypt_hdr.b_dsobj = zio->io_bookmark.zb_objset; 5748 zio_crypt_decode_params_bp(bp, hdr->b_crypt_hdr.b_salt, 5749 hdr->b_crypt_hdr.b_iv); 5750 5751 if (zio->io_error == 0) { 5752 if (BP_GET_TYPE(bp) == DMU_OT_INTENT_LOG) { 5753 void *tmpbuf; 5754 5755 tmpbuf = abd_borrow_buf_copy(zio->io_abd, 5756 sizeof (zil_chain_t)); 5757 zio_crypt_decode_mac_zil(tmpbuf, 5758 hdr->b_crypt_hdr.b_mac); 5759 abd_return_buf(zio->io_abd, tmpbuf, 5760 sizeof (zil_chain_t)); 5761 } else { 5762 zio_crypt_decode_mac_bp(bp, 5763 hdr->b_crypt_hdr.b_mac); 5764 } 5765 } 5766 } 5767 5768 if (zio->io_error == 0) { 5769 /* byteswap if necessary */ 5770 if (BP_SHOULD_BYTESWAP(zio->io_bp)) { 5771 if (BP_GET_LEVEL(zio->io_bp) > 0) { 5772 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_UINT64; 5773 } else { 5774 hdr->b_l1hdr.b_byteswap = 5775 DMU_OT_BYTESWAP(BP_GET_TYPE(zio->io_bp)); 5776 } 5777 } else { 5778 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_NUMFUNCS; 5779 } 5780 if (!HDR_L2_READING(hdr)) { 5781 hdr->b_complevel = zio->io_prop.zp_complevel; 5782 } 5783 } 5784 5785 arc_hdr_clear_flags(hdr, ARC_FLAG_L2_EVICTED); 5786 if (l2arc_noprefetch && HDR_PREFETCH(hdr)) 5787 arc_hdr_clear_flags(hdr, ARC_FLAG_L2CACHE); 5788 5789 callback_list = hdr->b_l1hdr.b_acb; 5790 ASSERT3P(callback_list, !=, NULL); 5791 hdr->b_l1hdr.b_acb = NULL; 5792 5793 /* 5794 * If a read request has a callback (i.e. acb_done is not NULL), then we 5795 * make a buf containing the data according to the parameters which were 5796 * passed in. The implementation of arc_buf_alloc_impl() ensures that we 5797 * aren't needlessly decompressing the data multiple times. 5798 */ 5799 int callback_cnt = 0; 5800 for (acb = callback_list; acb != NULL; acb = acb->acb_next) { 5801 5802 /* We need the last one to call below in original order. */ 5803 callback_list = acb; 5804 5805 if (!acb->acb_done || acb->acb_nobuf) 5806 continue; 5807 5808 callback_cnt++; 5809 5810 if (zio->io_error != 0) 5811 continue; 5812 5813 int error = arc_buf_alloc_impl(hdr, zio->io_spa, 5814 &acb->acb_zb, acb->acb_private, acb->acb_encrypted, 5815 acb->acb_compressed, acb->acb_noauth, B_TRUE, 5816 &acb->acb_buf); 5817 5818 /* 5819 * Assert non-speculative zios didn't fail because an 5820 * encryption key wasn't loaded 5821 */ 5822 ASSERT((zio->io_flags & ZIO_FLAG_SPECULATIVE) || 5823 error != EACCES); 5824 5825 /* 5826 * If we failed to decrypt, report an error now (as the zio 5827 * layer would have done if it had done the transforms). 5828 */ 5829 if (error == ECKSUM) { 5830 ASSERT(BP_IS_PROTECTED(bp)); 5831 error = SET_ERROR(EIO); 5832 if ((zio->io_flags & ZIO_FLAG_SPECULATIVE) == 0) { 5833 spa_log_error(zio->io_spa, &acb->acb_zb); 5834 (void) zfs_ereport_post( 5835 FM_EREPORT_ZFS_AUTHENTICATION, 5836 zio->io_spa, NULL, &acb->acb_zb, zio, 0); 5837 } 5838 } 5839 5840 if (error != 0) { 5841 /* 5842 * Decompression or decryption failed. Set 5843 * io_error so that when we call acb_done 5844 * (below), we will indicate that the read 5845 * failed. Note that in the unusual case 5846 * where one callback is compressed and another 5847 * uncompressed, we will mark all of them 5848 * as failed, even though the uncompressed 5849 * one can't actually fail. In this case, 5850 * the hdr will not be anonymous, because 5851 * if there are multiple callbacks, it's 5852 * because multiple threads found the same 5853 * arc buf in the hash table. 5854 */ 5855 zio->io_error = error; 5856 } 5857 } 5858 5859 /* 5860 * If there are multiple callbacks, we must have the hash lock, 5861 * because the only way for multiple threads to find this hdr is 5862 * in the hash table. This ensures that if there are multiple 5863 * callbacks, the hdr is not anonymous. If it were anonymous, 5864 * we couldn't use arc_buf_destroy() in the error case below. 5865 */ 5866 ASSERT(callback_cnt < 2 || hash_lock != NULL); 5867 5868 if (zio->io_error == 0) { 5869 arc_hdr_verify(hdr, zio->io_bp); 5870 } else { 5871 arc_hdr_set_flags(hdr, ARC_FLAG_IO_ERROR); 5872 if (hdr->b_l1hdr.b_state != arc_anon) 5873 arc_change_state(arc_anon, hdr); 5874 if (HDR_IN_HASH_TABLE(hdr)) 5875 buf_hash_remove(hdr); 5876 } 5877 5878 /* 5879 * Broadcast before we drop the hash_lock to avoid the possibility 5880 * that the hdr (and hence the cv) might be freed before we get to 5881 * the cv_broadcast(). 5882 */ 5883 cv_broadcast(&hdr->b_l1hdr.b_cv); 5884 5885 arc_hdr_clear_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 5886 (void) remove_reference(hdr, hdr); 5887 5888 if (hash_lock != NULL) 5889 mutex_exit(hash_lock); 5890 5891 /* execute each callback and free its structure */ 5892 while ((acb = callback_list) != NULL) { 5893 if (acb->acb_done != NULL) { 5894 if (zio->io_error != 0 && acb->acb_buf != NULL) { 5895 /* 5896 * If arc_buf_alloc_impl() fails during 5897 * decompression, the buf will still be 5898 * allocated, and needs to be freed here. 5899 */ 5900 arc_buf_destroy(acb->acb_buf, 5901 acb->acb_private); 5902 acb->acb_buf = NULL; 5903 } 5904 acb->acb_done(zio, &zio->io_bookmark, zio->io_bp, 5905 acb->acb_buf, acb->acb_private); 5906 } 5907 5908 if (acb->acb_zio_dummy != NULL) { 5909 acb->acb_zio_dummy->io_error = zio->io_error; 5910 zio_nowait(acb->acb_zio_dummy); 5911 } 5912 5913 callback_list = acb->acb_prev; 5914 if (acb->acb_wait) { 5915 mutex_enter(&acb->acb_wait_lock); 5916 acb->acb_wait_error = zio->io_error; 5917 acb->acb_wait = B_FALSE; 5918 cv_signal(&acb->acb_wait_cv); 5919 mutex_exit(&acb->acb_wait_lock); 5920 /* acb will be freed by the waiting thread. */ 5921 } else { 5922 kmem_free(acb, sizeof (arc_callback_t)); 5923 } 5924 } 5925 } 5926 5927 /* 5928 * "Read" the block at the specified DVA (in bp) via the 5929 * cache. If the block is found in the cache, invoke the provided 5930 * callback immediately and return. Note that the `zio' parameter 5931 * in the callback will be NULL in this case, since no IO was 5932 * required. If the block is not in the cache pass the read request 5933 * on to the spa with a substitute callback function, so that the 5934 * requested block will be added to the cache. 5935 * 5936 * If a read request arrives for a block that has a read in-progress, 5937 * either wait for the in-progress read to complete (and return the 5938 * results); or, if this is a read with a "done" func, add a record 5939 * to the read to invoke the "done" func when the read completes, 5940 * and return; or just return. 5941 * 5942 * arc_read_done() will invoke all the requested "done" functions 5943 * for readers of this block. 5944 */ 5945 int 5946 arc_read(zio_t *pio, spa_t *spa, const blkptr_t *bp, 5947 arc_read_done_func_t *done, void *private, zio_priority_t priority, 5948 int zio_flags, arc_flags_t *arc_flags, const zbookmark_phys_t *zb) 5949 { 5950 arc_buf_hdr_t *hdr = NULL; 5951 kmutex_t *hash_lock = NULL; 5952 zio_t *rzio; 5953 uint64_t guid = spa_load_guid(spa); 5954 boolean_t compressed_read = (zio_flags & ZIO_FLAG_RAW_COMPRESS) != 0; 5955 boolean_t encrypted_read = BP_IS_ENCRYPTED(bp) && 5956 (zio_flags & ZIO_FLAG_RAW_ENCRYPT) != 0; 5957 boolean_t noauth_read = BP_IS_AUTHENTICATED(bp) && 5958 (zio_flags & ZIO_FLAG_RAW_ENCRYPT) != 0; 5959 boolean_t embedded_bp = !!BP_IS_EMBEDDED(bp); 5960 boolean_t no_buf = *arc_flags & ARC_FLAG_NO_BUF; 5961 arc_buf_t *buf = NULL; 5962 int rc = 0; 5963 5964 ASSERT(!embedded_bp || 5965 BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA); 5966 ASSERT(!BP_IS_HOLE(bp)); 5967 ASSERT(!BP_IS_REDACTED(bp)); 5968 5969 /* 5970 * Normally SPL_FSTRANS will already be set since kernel threads which 5971 * expect to call the DMU interfaces will set it when created. System 5972 * calls are similarly handled by setting/cleaning the bit in the 5973 * registered callback (module/os/.../zfs/zpl_*). 5974 * 5975 * External consumers such as Lustre which call the exported DMU 5976 * interfaces may not have set SPL_FSTRANS. To avoid a deadlock 5977 * on the hash_lock always set and clear the bit. 5978 */ 5979 fstrans_cookie_t cookie = spl_fstrans_mark(); 5980 top: 5981 /* 5982 * Verify the block pointer contents are reasonable. This should 5983 * always be the case since the blkptr is protected by a checksum. 5984 * However, if there is damage it's desirable to detect this early 5985 * and treat it as a checksum error. This allows an alternate blkptr 5986 * to be tried when one is available (e.g. ditto blocks). 5987 */ 5988 if (!zfs_blkptr_verify(spa, bp, zio_flags & ZIO_FLAG_CONFIG_WRITER, 5989 BLK_VERIFY_LOG)) { 5990 rc = SET_ERROR(ECKSUM); 5991 goto done; 5992 } 5993 5994 if (!embedded_bp) { 5995 /* 5996 * Embedded BP's have no DVA and require no I/O to "read". 5997 * Create an anonymous arc buf to back it. 5998 */ 5999 hdr = buf_hash_find(guid, bp, &hash_lock); 6000 } 6001 6002 /* 6003 * Determine if we have an L1 cache hit or a cache miss. For simplicity 6004 * we maintain encrypted data separately from compressed / uncompressed 6005 * data. If the user is requesting raw encrypted data and we don't have 6006 * that in the header we will read from disk to guarantee that we can 6007 * get it even if the encryption keys aren't loaded. 6008 */ 6009 if (hdr != NULL && HDR_HAS_L1HDR(hdr) && (HDR_HAS_RABD(hdr) || 6010 (hdr->b_l1hdr.b_pabd != NULL && !encrypted_read))) { 6011 boolean_t is_data = !HDR_ISTYPE_METADATA(hdr); 6012 6013 if (HDR_IO_IN_PROGRESS(hdr)) { 6014 if (*arc_flags & ARC_FLAG_CACHED_ONLY) { 6015 mutex_exit(hash_lock); 6016 ARCSTAT_BUMP(arcstat_cached_only_in_progress); 6017 rc = SET_ERROR(ENOENT); 6018 goto done; 6019 } 6020 6021 zio_t *head_zio = hdr->b_l1hdr.b_acb->acb_zio_head; 6022 ASSERT3P(head_zio, !=, NULL); 6023 if ((hdr->b_flags & ARC_FLAG_PRIO_ASYNC_READ) && 6024 priority == ZIO_PRIORITY_SYNC_READ) { 6025 /* 6026 * This is a sync read that needs to wait for 6027 * an in-flight async read. Request that the 6028 * zio have its priority upgraded. 6029 */ 6030 zio_change_priority(head_zio, priority); 6031 DTRACE_PROBE1(arc__async__upgrade__sync, 6032 arc_buf_hdr_t *, hdr); 6033 ARCSTAT_BUMP(arcstat_async_upgrade_sync); 6034 } 6035 6036 DTRACE_PROBE1(arc__iohit, arc_buf_hdr_t *, hdr); 6037 arc_access(hdr, *arc_flags, B_FALSE); 6038 6039 /* 6040 * If there are multiple threads reading the same block 6041 * and that block is not yet in the ARC, then only one 6042 * thread will do the physical I/O and all other 6043 * threads will wait until that I/O completes. 6044 * Synchronous reads use the acb_wait_cv whereas nowait 6045 * reads register a callback. Both are signalled/called 6046 * in arc_read_done. 6047 * 6048 * Errors of the physical I/O may need to be propagated. 6049 * Synchronous read errors are returned here from 6050 * arc_read_done via acb_wait_error. Nowait reads 6051 * attach the acb_zio_dummy zio to pio and 6052 * arc_read_done propagates the physical I/O's io_error 6053 * to acb_zio_dummy, and thereby to pio. 6054 */ 6055 arc_callback_t *acb = NULL; 6056 if (done || pio || *arc_flags & ARC_FLAG_WAIT) { 6057 acb = kmem_zalloc(sizeof (arc_callback_t), 6058 KM_SLEEP); 6059 acb->acb_done = done; 6060 acb->acb_private = private; 6061 acb->acb_compressed = compressed_read; 6062 acb->acb_encrypted = encrypted_read; 6063 acb->acb_noauth = noauth_read; 6064 acb->acb_nobuf = no_buf; 6065 if (*arc_flags & ARC_FLAG_WAIT) { 6066 acb->acb_wait = B_TRUE; 6067 mutex_init(&acb->acb_wait_lock, NULL, 6068 MUTEX_DEFAULT, NULL); 6069 cv_init(&acb->acb_wait_cv, NULL, 6070 CV_DEFAULT, NULL); 6071 } 6072 acb->acb_zb = *zb; 6073 if (pio != NULL) { 6074 acb->acb_zio_dummy = zio_null(pio, 6075 spa, NULL, NULL, NULL, zio_flags); 6076 } 6077 acb->acb_zio_head = head_zio; 6078 acb->acb_next = hdr->b_l1hdr.b_acb; 6079 if (hdr->b_l1hdr.b_acb) 6080 hdr->b_l1hdr.b_acb->acb_prev = acb; 6081 hdr->b_l1hdr.b_acb = acb; 6082 } 6083 mutex_exit(hash_lock); 6084 6085 ARCSTAT_BUMP(arcstat_iohits); 6086 ARCSTAT_CONDSTAT(!(*arc_flags & ARC_FLAG_PREFETCH), 6087 demand, prefetch, is_data, data, metadata, iohits); 6088 6089 if (*arc_flags & ARC_FLAG_WAIT) { 6090 mutex_enter(&acb->acb_wait_lock); 6091 while (acb->acb_wait) { 6092 cv_wait(&acb->acb_wait_cv, 6093 &acb->acb_wait_lock); 6094 } 6095 rc = acb->acb_wait_error; 6096 mutex_exit(&acb->acb_wait_lock); 6097 mutex_destroy(&acb->acb_wait_lock); 6098 cv_destroy(&acb->acb_wait_cv); 6099 kmem_free(acb, sizeof (arc_callback_t)); 6100 } 6101 goto out; 6102 } 6103 6104 ASSERT(hdr->b_l1hdr.b_state == arc_mru || 6105 hdr->b_l1hdr.b_state == arc_mfu || 6106 hdr->b_l1hdr.b_state == arc_uncached); 6107 6108 DTRACE_PROBE1(arc__hit, arc_buf_hdr_t *, hdr); 6109 arc_access(hdr, *arc_flags, B_TRUE); 6110 6111 if (done && !no_buf) { 6112 ASSERT(!embedded_bp || !BP_IS_HOLE(bp)); 6113 6114 /* Get a buf with the desired data in it. */ 6115 rc = arc_buf_alloc_impl(hdr, spa, zb, private, 6116 encrypted_read, compressed_read, noauth_read, 6117 B_TRUE, &buf); 6118 if (rc == ECKSUM) { 6119 /* 6120 * Convert authentication and decryption errors 6121 * to EIO (and generate an ereport if needed) 6122 * before leaving the ARC. 6123 */ 6124 rc = SET_ERROR(EIO); 6125 if ((zio_flags & ZIO_FLAG_SPECULATIVE) == 0) { 6126 spa_log_error(spa, zb); 6127 (void) zfs_ereport_post( 6128 FM_EREPORT_ZFS_AUTHENTICATION, 6129 spa, NULL, zb, NULL, 0); 6130 } 6131 } 6132 if (rc != 0) { 6133 arc_buf_destroy_impl(buf); 6134 buf = NULL; 6135 (void) remove_reference(hdr, private); 6136 } 6137 6138 /* assert any errors weren't due to unloaded keys */ 6139 ASSERT((zio_flags & ZIO_FLAG_SPECULATIVE) || 6140 rc != EACCES); 6141 } 6142 mutex_exit(hash_lock); 6143 ARCSTAT_BUMP(arcstat_hits); 6144 ARCSTAT_CONDSTAT(!(*arc_flags & ARC_FLAG_PREFETCH), 6145 demand, prefetch, is_data, data, metadata, hits); 6146 *arc_flags |= ARC_FLAG_CACHED; 6147 goto done; 6148 } else { 6149 uint64_t lsize = BP_GET_LSIZE(bp); 6150 uint64_t psize = BP_GET_PSIZE(bp); 6151 arc_callback_t *acb; 6152 vdev_t *vd = NULL; 6153 uint64_t addr = 0; 6154 boolean_t devw = B_FALSE; 6155 uint64_t size; 6156 abd_t *hdr_abd; 6157 int alloc_flags = encrypted_read ? ARC_HDR_ALLOC_RDATA : 0; 6158 6159 if (*arc_flags & ARC_FLAG_CACHED_ONLY) { 6160 if (hash_lock != NULL) 6161 mutex_exit(hash_lock); 6162 rc = SET_ERROR(ENOENT); 6163 goto done; 6164 } 6165 6166 if (hdr == NULL) { 6167 /* 6168 * This block is not in the cache or it has 6169 * embedded data. 6170 */ 6171 arc_buf_hdr_t *exists = NULL; 6172 arc_buf_contents_t type = BP_GET_BUFC_TYPE(bp); 6173 hdr = arc_hdr_alloc(spa_load_guid(spa), psize, lsize, 6174 BP_IS_PROTECTED(bp), BP_GET_COMPRESS(bp), 0, type); 6175 6176 if (!embedded_bp) { 6177 hdr->b_dva = *BP_IDENTITY(bp); 6178 hdr->b_birth = BP_PHYSICAL_BIRTH(bp); 6179 exists = buf_hash_insert(hdr, &hash_lock); 6180 } 6181 if (exists != NULL) { 6182 /* somebody beat us to the hash insert */ 6183 mutex_exit(hash_lock); 6184 buf_discard_identity(hdr); 6185 arc_hdr_destroy(hdr); 6186 goto top; /* restart the IO request */ 6187 } 6188 } else { 6189 /* 6190 * This block is in the ghost cache or encrypted data 6191 * was requested and we didn't have it. If it was 6192 * L2-only (and thus didn't have an L1 hdr), 6193 * we realloc the header to add an L1 hdr. 6194 */ 6195 if (!HDR_HAS_L1HDR(hdr)) { 6196 hdr = arc_hdr_realloc(hdr, hdr_l2only_cache, 6197 hdr_full_cache); 6198 } 6199 6200 if (GHOST_STATE(hdr->b_l1hdr.b_state)) { 6201 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 6202 ASSERT(!HDR_HAS_RABD(hdr)); 6203 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 6204 ASSERT0(zfs_refcount_count( 6205 &hdr->b_l1hdr.b_refcnt)); 6206 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 6207 #ifdef ZFS_DEBUG 6208 ASSERT3P(hdr->b_l1hdr.b_freeze_cksum, ==, NULL); 6209 #endif 6210 } else if (HDR_IO_IN_PROGRESS(hdr)) { 6211 /* 6212 * If this header already had an IO in progress 6213 * and we are performing another IO to fetch 6214 * encrypted data we must wait until the first 6215 * IO completes so as not to confuse 6216 * arc_read_done(). This should be very rare 6217 * and so the performance impact shouldn't 6218 * matter. 6219 */ 6220 cv_wait(&hdr->b_l1hdr.b_cv, hash_lock); 6221 mutex_exit(hash_lock); 6222 goto top; 6223 } 6224 } 6225 if (*arc_flags & ARC_FLAG_UNCACHED) { 6226 arc_hdr_set_flags(hdr, ARC_FLAG_UNCACHED); 6227 if (!encrypted_read) 6228 alloc_flags |= ARC_HDR_ALLOC_LINEAR; 6229 } 6230 6231 /* 6232 * Call arc_adapt() explicitly before arc_access() to allow 6233 * its logic to balance MRU/MFU based on the original state. 6234 */ 6235 arc_adapt(arc_hdr_size(hdr), hdr->b_l1hdr.b_state); 6236 /* 6237 * Take additional reference for IO_IN_PROGRESS. It stops 6238 * arc_access() from putting this header without any buffers 6239 * and so other references but obviously nonevictable onto 6240 * the evictable list of MRU or MFU state. 6241 */ 6242 add_reference(hdr, hdr); 6243 if (!embedded_bp) 6244 arc_access(hdr, *arc_flags, B_FALSE); 6245 arc_hdr_set_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 6246 arc_hdr_alloc_abd(hdr, alloc_flags); 6247 if (encrypted_read) { 6248 ASSERT(HDR_HAS_RABD(hdr)); 6249 size = HDR_GET_PSIZE(hdr); 6250 hdr_abd = hdr->b_crypt_hdr.b_rabd; 6251 zio_flags |= ZIO_FLAG_RAW; 6252 } else { 6253 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 6254 size = arc_hdr_size(hdr); 6255 hdr_abd = hdr->b_l1hdr.b_pabd; 6256 6257 if (arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF) { 6258 zio_flags |= ZIO_FLAG_RAW_COMPRESS; 6259 } 6260 6261 /* 6262 * For authenticated bp's, we do not ask the ZIO layer 6263 * to authenticate them since this will cause the entire 6264 * IO to fail if the key isn't loaded. Instead, we 6265 * defer authentication until arc_buf_fill(), which will 6266 * verify the data when the key is available. 6267 */ 6268 if (BP_IS_AUTHENTICATED(bp)) 6269 zio_flags |= ZIO_FLAG_RAW_ENCRYPT; 6270 } 6271 6272 if (BP_IS_AUTHENTICATED(bp)) 6273 arc_hdr_set_flags(hdr, ARC_FLAG_NOAUTH); 6274 if (BP_GET_LEVEL(bp) > 0) 6275 arc_hdr_set_flags(hdr, ARC_FLAG_INDIRECT); 6276 ASSERT(!GHOST_STATE(hdr->b_l1hdr.b_state)); 6277 6278 acb = kmem_zalloc(sizeof (arc_callback_t), KM_SLEEP); 6279 acb->acb_done = done; 6280 acb->acb_private = private; 6281 acb->acb_compressed = compressed_read; 6282 acb->acb_encrypted = encrypted_read; 6283 acb->acb_noauth = noauth_read; 6284 acb->acb_zb = *zb; 6285 6286 ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); 6287 hdr->b_l1hdr.b_acb = acb; 6288 6289 if (HDR_HAS_L2HDR(hdr) && 6290 (vd = hdr->b_l2hdr.b_dev->l2ad_vdev) != NULL) { 6291 devw = hdr->b_l2hdr.b_dev->l2ad_writing; 6292 addr = hdr->b_l2hdr.b_daddr; 6293 /* 6294 * Lock out L2ARC device removal. 6295 */ 6296 if (vdev_is_dead(vd) || 6297 !spa_config_tryenter(spa, SCL_L2ARC, vd, RW_READER)) 6298 vd = NULL; 6299 } 6300 6301 /* 6302 * We count both async reads and scrub IOs as asynchronous so 6303 * that both can be upgraded in the event of a cache hit while 6304 * the read IO is still in-flight. 6305 */ 6306 if (priority == ZIO_PRIORITY_ASYNC_READ || 6307 priority == ZIO_PRIORITY_SCRUB) 6308 arc_hdr_set_flags(hdr, ARC_FLAG_PRIO_ASYNC_READ); 6309 else 6310 arc_hdr_clear_flags(hdr, ARC_FLAG_PRIO_ASYNC_READ); 6311 6312 /* 6313 * At this point, we have a level 1 cache miss or a blkptr 6314 * with embedded data. Try again in L2ARC if possible. 6315 */ 6316 ASSERT3U(HDR_GET_LSIZE(hdr), ==, lsize); 6317 6318 /* 6319 * Skip ARC stat bump for block pointers with embedded 6320 * data. The data are read from the blkptr itself via 6321 * decode_embedded_bp_compressed(). 6322 */ 6323 if (!embedded_bp) { 6324 DTRACE_PROBE4(arc__miss, arc_buf_hdr_t *, hdr, 6325 blkptr_t *, bp, uint64_t, lsize, 6326 zbookmark_phys_t *, zb); 6327 ARCSTAT_BUMP(arcstat_misses); 6328 ARCSTAT_CONDSTAT(!(*arc_flags & ARC_FLAG_PREFETCH), 6329 demand, prefetch, !HDR_ISTYPE_METADATA(hdr), data, 6330 metadata, misses); 6331 zfs_racct_read(size, 1); 6332 } 6333 6334 /* Check if the spa even has l2 configured */ 6335 const boolean_t spa_has_l2 = l2arc_ndev != 0 && 6336 spa->spa_l2cache.sav_count > 0; 6337 6338 if (vd != NULL && spa_has_l2 && !(l2arc_norw && devw)) { 6339 /* 6340 * Read from the L2ARC if the following are true: 6341 * 1. The L2ARC vdev was previously cached. 6342 * 2. This buffer still has L2ARC metadata. 6343 * 3. This buffer isn't currently writing to the L2ARC. 6344 * 4. The L2ARC entry wasn't evicted, which may 6345 * also have invalidated the vdev. 6346 * 5. This isn't prefetch or l2arc_noprefetch is 0. 6347 */ 6348 if (HDR_HAS_L2HDR(hdr) && 6349 !HDR_L2_WRITING(hdr) && !HDR_L2_EVICTED(hdr) && 6350 !(l2arc_noprefetch && 6351 (*arc_flags & ARC_FLAG_PREFETCH))) { 6352 l2arc_read_callback_t *cb; 6353 abd_t *abd; 6354 uint64_t asize; 6355 6356 DTRACE_PROBE1(l2arc__hit, arc_buf_hdr_t *, hdr); 6357 ARCSTAT_BUMP(arcstat_l2_hits); 6358 hdr->b_l2hdr.b_hits++; 6359 6360 cb = kmem_zalloc(sizeof (l2arc_read_callback_t), 6361 KM_SLEEP); 6362 cb->l2rcb_hdr = hdr; 6363 cb->l2rcb_bp = *bp; 6364 cb->l2rcb_zb = *zb; 6365 cb->l2rcb_flags = zio_flags; 6366 6367 /* 6368 * When Compressed ARC is disabled, but the 6369 * L2ARC block is compressed, arc_hdr_size() 6370 * will have returned LSIZE rather than PSIZE. 6371 */ 6372 if (HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 6373 !HDR_COMPRESSION_ENABLED(hdr) && 6374 HDR_GET_PSIZE(hdr) != 0) { 6375 size = HDR_GET_PSIZE(hdr); 6376 } 6377 6378 asize = vdev_psize_to_asize(vd, size); 6379 if (asize != size) { 6380 abd = abd_alloc_for_io(asize, 6381 HDR_ISTYPE_METADATA(hdr)); 6382 cb->l2rcb_abd = abd; 6383 } else { 6384 abd = hdr_abd; 6385 } 6386 6387 ASSERT(addr >= VDEV_LABEL_START_SIZE && 6388 addr + asize <= vd->vdev_psize - 6389 VDEV_LABEL_END_SIZE); 6390 6391 /* 6392 * l2arc read. The SCL_L2ARC lock will be 6393 * released by l2arc_read_done(). 6394 * Issue a null zio if the underlying buffer 6395 * was squashed to zero size by compression. 6396 */ 6397 ASSERT3U(arc_hdr_get_compress(hdr), !=, 6398 ZIO_COMPRESS_EMPTY); 6399 rzio = zio_read_phys(pio, vd, addr, 6400 asize, abd, 6401 ZIO_CHECKSUM_OFF, 6402 l2arc_read_done, cb, priority, 6403 zio_flags | ZIO_FLAG_DONT_CACHE | 6404 ZIO_FLAG_CANFAIL | 6405 ZIO_FLAG_DONT_PROPAGATE | 6406 ZIO_FLAG_DONT_RETRY, B_FALSE); 6407 acb->acb_zio_head = rzio; 6408 6409 if (hash_lock != NULL) 6410 mutex_exit(hash_lock); 6411 6412 DTRACE_PROBE2(l2arc__read, vdev_t *, vd, 6413 zio_t *, rzio); 6414 ARCSTAT_INCR(arcstat_l2_read_bytes, 6415 HDR_GET_PSIZE(hdr)); 6416 6417 if (*arc_flags & ARC_FLAG_NOWAIT) { 6418 zio_nowait(rzio); 6419 goto out; 6420 } 6421 6422 ASSERT(*arc_flags & ARC_FLAG_WAIT); 6423 if (zio_wait(rzio) == 0) 6424 goto out; 6425 6426 /* l2arc read error; goto zio_read() */ 6427 if (hash_lock != NULL) 6428 mutex_enter(hash_lock); 6429 } else { 6430 DTRACE_PROBE1(l2arc__miss, 6431 arc_buf_hdr_t *, hdr); 6432 ARCSTAT_BUMP(arcstat_l2_misses); 6433 if (HDR_L2_WRITING(hdr)) 6434 ARCSTAT_BUMP(arcstat_l2_rw_clash); 6435 spa_config_exit(spa, SCL_L2ARC, vd); 6436 } 6437 } else { 6438 if (vd != NULL) 6439 spa_config_exit(spa, SCL_L2ARC, vd); 6440 6441 /* 6442 * Only a spa with l2 should contribute to l2 6443 * miss stats. (Including the case of having a 6444 * faulted cache device - that's also a miss.) 6445 */ 6446 if (spa_has_l2) { 6447 /* 6448 * Skip ARC stat bump for block pointers with 6449 * embedded data. The data are read from the 6450 * blkptr itself via 6451 * decode_embedded_bp_compressed(). 6452 */ 6453 if (!embedded_bp) { 6454 DTRACE_PROBE1(l2arc__miss, 6455 arc_buf_hdr_t *, hdr); 6456 ARCSTAT_BUMP(arcstat_l2_misses); 6457 } 6458 } 6459 } 6460 6461 rzio = zio_read(pio, spa, bp, hdr_abd, size, 6462 arc_read_done, hdr, priority, zio_flags, zb); 6463 acb->acb_zio_head = rzio; 6464 6465 if (hash_lock != NULL) 6466 mutex_exit(hash_lock); 6467 6468 if (*arc_flags & ARC_FLAG_WAIT) { 6469 rc = zio_wait(rzio); 6470 goto out; 6471 } 6472 6473 ASSERT(*arc_flags & ARC_FLAG_NOWAIT); 6474 zio_nowait(rzio); 6475 } 6476 6477 out: 6478 /* embedded bps don't actually go to disk */ 6479 if (!embedded_bp) 6480 spa_read_history_add(spa, zb, *arc_flags); 6481 spl_fstrans_unmark(cookie); 6482 return (rc); 6483 6484 done: 6485 if (done) 6486 done(NULL, zb, bp, buf, private); 6487 if (pio && rc != 0) { 6488 zio_t *zio = zio_null(pio, spa, NULL, NULL, NULL, zio_flags); 6489 zio->io_error = rc; 6490 zio_nowait(zio); 6491 } 6492 goto out; 6493 } 6494 6495 arc_prune_t * 6496 arc_add_prune_callback(arc_prune_func_t *func, void *private) 6497 { 6498 arc_prune_t *p; 6499 6500 p = kmem_alloc(sizeof (*p), KM_SLEEP); 6501 p->p_pfunc = func; 6502 p->p_private = private; 6503 list_link_init(&p->p_node); 6504 zfs_refcount_create(&p->p_refcnt); 6505 6506 mutex_enter(&arc_prune_mtx); 6507 zfs_refcount_add(&p->p_refcnt, &arc_prune_list); 6508 list_insert_head(&arc_prune_list, p); 6509 mutex_exit(&arc_prune_mtx); 6510 6511 return (p); 6512 } 6513 6514 void 6515 arc_remove_prune_callback(arc_prune_t *p) 6516 { 6517 boolean_t wait = B_FALSE; 6518 mutex_enter(&arc_prune_mtx); 6519 list_remove(&arc_prune_list, p); 6520 if (zfs_refcount_remove(&p->p_refcnt, &arc_prune_list) > 0) 6521 wait = B_TRUE; 6522 mutex_exit(&arc_prune_mtx); 6523 6524 /* wait for arc_prune_task to finish */ 6525 if (wait) 6526 taskq_wait_outstanding(arc_prune_taskq, 0); 6527 ASSERT0(zfs_refcount_count(&p->p_refcnt)); 6528 zfs_refcount_destroy(&p->p_refcnt); 6529 kmem_free(p, sizeof (*p)); 6530 } 6531 6532 /* 6533 * Notify the arc that a block was freed, and thus will never be used again. 6534 */ 6535 void 6536 arc_freed(spa_t *spa, const blkptr_t *bp) 6537 { 6538 arc_buf_hdr_t *hdr; 6539 kmutex_t *hash_lock; 6540 uint64_t guid = spa_load_guid(spa); 6541 6542 ASSERT(!BP_IS_EMBEDDED(bp)); 6543 6544 hdr = buf_hash_find(guid, bp, &hash_lock); 6545 if (hdr == NULL) 6546 return; 6547 6548 /* 6549 * We might be trying to free a block that is still doing I/O 6550 * (i.e. prefetch) or has some other reference (i.e. a dedup-ed, 6551 * dmu_sync-ed block). A block may also have a reference if it is 6552 * part of a dedup-ed, dmu_synced write. The dmu_sync() function would 6553 * have written the new block to its final resting place on disk but 6554 * without the dedup flag set. This would have left the hdr in the MRU 6555 * state and discoverable. When the txg finally syncs it detects that 6556 * the block was overridden in open context and issues an override I/O. 6557 * Since this is a dedup block, the override I/O will determine if the 6558 * block is already in the DDT. If so, then it will replace the io_bp 6559 * with the bp from the DDT and allow the I/O to finish. When the I/O 6560 * reaches the done callback, dbuf_write_override_done, it will 6561 * check to see if the io_bp and io_bp_override are identical. 6562 * If they are not, then it indicates that the bp was replaced with 6563 * the bp in the DDT and the override bp is freed. This allows 6564 * us to arrive here with a reference on a block that is being 6565 * freed. So if we have an I/O in progress, or a reference to 6566 * this hdr, then we don't destroy the hdr. 6567 */ 6568 if (!HDR_HAS_L1HDR(hdr) || 6569 zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)) { 6570 arc_change_state(arc_anon, hdr); 6571 arc_hdr_destroy(hdr); 6572 mutex_exit(hash_lock); 6573 } else { 6574 mutex_exit(hash_lock); 6575 } 6576 6577 } 6578 6579 /* 6580 * Release this buffer from the cache, making it an anonymous buffer. This 6581 * must be done after a read and prior to modifying the buffer contents. 6582 * If the buffer has more than one reference, we must make 6583 * a new hdr for the buffer. 6584 */ 6585 void 6586 arc_release(arc_buf_t *buf, const void *tag) 6587 { 6588 arc_buf_hdr_t *hdr = buf->b_hdr; 6589 6590 /* 6591 * It would be nice to assert that if its DMU metadata (level > 6592 * 0 || it's the dnode file), then it must be syncing context. 6593 * But we don't know that information at this level. 6594 */ 6595 6596 ASSERT(HDR_HAS_L1HDR(hdr)); 6597 6598 /* 6599 * We don't grab the hash lock prior to this check, because if 6600 * the buffer's header is in the arc_anon state, it won't be 6601 * linked into the hash table. 6602 */ 6603 if (hdr->b_l1hdr.b_state == arc_anon) { 6604 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 6605 ASSERT(!HDR_IN_HASH_TABLE(hdr)); 6606 ASSERT(!HDR_HAS_L2HDR(hdr)); 6607 6608 ASSERT3U(hdr->b_l1hdr.b_bufcnt, ==, 1); 6609 ASSERT3S(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt), ==, 1); 6610 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 6611 6612 hdr->b_l1hdr.b_arc_access = 0; 6613 6614 /* 6615 * If the buf is being overridden then it may already 6616 * have a hdr that is not empty. 6617 */ 6618 buf_discard_identity(hdr); 6619 arc_buf_thaw(buf); 6620 6621 return; 6622 } 6623 6624 kmutex_t *hash_lock = HDR_LOCK(hdr); 6625 mutex_enter(hash_lock); 6626 6627 /* 6628 * This assignment is only valid as long as the hash_lock is 6629 * held, we must be careful not to reference state or the 6630 * b_state field after dropping the lock. 6631 */ 6632 arc_state_t *state = hdr->b_l1hdr.b_state; 6633 ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); 6634 ASSERT3P(state, !=, arc_anon); 6635 6636 /* this buffer is not on any list */ 6637 ASSERT3S(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt), >, 0); 6638 6639 if (HDR_HAS_L2HDR(hdr)) { 6640 mutex_enter(&hdr->b_l2hdr.b_dev->l2ad_mtx); 6641 6642 /* 6643 * We have to recheck this conditional again now that 6644 * we're holding the l2ad_mtx to prevent a race with 6645 * another thread which might be concurrently calling 6646 * l2arc_evict(). In that case, l2arc_evict() might have 6647 * destroyed the header's L2 portion as we were waiting 6648 * to acquire the l2ad_mtx. 6649 */ 6650 if (HDR_HAS_L2HDR(hdr)) 6651 arc_hdr_l2hdr_destroy(hdr); 6652 6653 mutex_exit(&hdr->b_l2hdr.b_dev->l2ad_mtx); 6654 } 6655 6656 /* 6657 * Do we have more than one buf? 6658 */ 6659 if (hdr->b_l1hdr.b_bufcnt > 1) { 6660 arc_buf_hdr_t *nhdr; 6661 uint64_t spa = hdr->b_spa; 6662 uint64_t psize = HDR_GET_PSIZE(hdr); 6663 uint64_t lsize = HDR_GET_LSIZE(hdr); 6664 boolean_t protected = HDR_PROTECTED(hdr); 6665 enum zio_compress compress = arc_hdr_get_compress(hdr); 6666 arc_buf_contents_t type = arc_buf_type(hdr); 6667 VERIFY3U(hdr->b_type, ==, type); 6668 6669 ASSERT(hdr->b_l1hdr.b_buf != buf || buf->b_next != NULL); 6670 VERIFY3S(remove_reference(hdr, tag), >, 0); 6671 6672 if (arc_buf_is_shared(buf) && !ARC_BUF_COMPRESSED(buf)) { 6673 ASSERT3P(hdr->b_l1hdr.b_buf, !=, buf); 6674 ASSERT(ARC_BUF_LAST(buf)); 6675 } 6676 6677 /* 6678 * Pull the data off of this hdr and attach it to 6679 * a new anonymous hdr. Also find the last buffer 6680 * in the hdr's buffer list. 6681 */ 6682 arc_buf_t *lastbuf = arc_buf_remove(hdr, buf); 6683 ASSERT3P(lastbuf, !=, NULL); 6684 6685 /* 6686 * If the current arc_buf_t and the hdr are sharing their data 6687 * buffer, then we must stop sharing that block. 6688 */ 6689 if (arc_buf_is_shared(buf)) { 6690 ASSERT3P(hdr->b_l1hdr.b_buf, !=, buf); 6691 VERIFY(!arc_buf_is_shared(lastbuf)); 6692 6693 /* 6694 * First, sever the block sharing relationship between 6695 * buf and the arc_buf_hdr_t. 6696 */ 6697 arc_unshare_buf(hdr, buf); 6698 6699 /* 6700 * Now we need to recreate the hdr's b_pabd. Since we 6701 * have lastbuf handy, we try to share with it, but if 6702 * we can't then we allocate a new b_pabd and copy the 6703 * data from buf into it. 6704 */ 6705 if (arc_can_share(hdr, lastbuf)) { 6706 arc_share_buf(hdr, lastbuf); 6707 } else { 6708 arc_hdr_alloc_abd(hdr, ARC_HDR_DO_ADAPT); 6709 abd_copy_from_buf(hdr->b_l1hdr.b_pabd, 6710 buf->b_data, psize); 6711 } 6712 VERIFY3P(lastbuf->b_data, !=, NULL); 6713 } else if (HDR_SHARED_DATA(hdr)) { 6714 /* 6715 * Uncompressed shared buffers are always at the end 6716 * of the list. Compressed buffers don't have the 6717 * same requirements. This makes it hard to 6718 * simply assert that the lastbuf is shared so 6719 * we rely on the hdr's compression flags to determine 6720 * if we have a compressed, shared buffer. 6721 */ 6722 ASSERT(arc_buf_is_shared(lastbuf) || 6723 arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF); 6724 ASSERT(!ARC_BUF_SHARED(buf)); 6725 } 6726 6727 ASSERT(hdr->b_l1hdr.b_pabd != NULL || HDR_HAS_RABD(hdr)); 6728 ASSERT3P(state, !=, arc_l2c_only); 6729 6730 (void) zfs_refcount_remove_many(&state->arcs_size, 6731 arc_buf_size(buf), buf); 6732 6733 if (zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)) { 6734 ASSERT3P(state, !=, arc_l2c_only); 6735 (void) zfs_refcount_remove_many( 6736 &state->arcs_esize[type], 6737 arc_buf_size(buf), buf); 6738 } 6739 6740 hdr->b_l1hdr.b_bufcnt -= 1; 6741 if (ARC_BUF_ENCRYPTED(buf)) 6742 hdr->b_crypt_hdr.b_ebufcnt -= 1; 6743 6744 arc_cksum_verify(buf); 6745 arc_buf_unwatch(buf); 6746 6747 /* if this is the last uncompressed buf free the checksum */ 6748 if (!arc_hdr_has_uncompressed_buf(hdr)) 6749 arc_cksum_free(hdr); 6750 6751 mutex_exit(hash_lock); 6752 6753 nhdr = arc_hdr_alloc(spa, psize, lsize, protected, 6754 compress, hdr->b_complevel, type); 6755 ASSERT3P(nhdr->b_l1hdr.b_buf, ==, NULL); 6756 ASSERT0(nhdr->b_l1hdr.b_bufcnt); 6757 ASSERT0(zfs_refcount_count(&nhdr->b_l1hdr.b_refcnt)); 6758 VERIFY3U(nhdr->b_type, ==, type); 6759 ASSERT(!HDR_SHARED_DATA(nhdr)); 6760 6761 nhdr->b_l1hdr.b_buf = buf; 6762 nhdr->b_l1hdr.b_bufcnt = 1; 6763 if (ARC_BUF_ENCRYPTED(buf)) 6764 nhdr->b_crypt_hdr.b_ebufcnt = 1; 6765 (void) zfs_refcount_add(&nhdr->b_l1hdr.b_refcnt, tag); 6766 buf->b_hdr = nhdr; 6767 6768 (void) zfs_refcount_add_many(&arc_anon->arcs_size, 6769 arc_buf_size(buf), buf); 6770 } else { 6771 ASSERT(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt) == 1); 6772 /* protected by hash lock, or hdr is on arc_anon */ 6773 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 6774 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 6775 hdr->b_l1hdr.b_mru_hits = 0; 6776 hdr->b_l1hdr.b_mru_ghost_hits = 0; 6777 hdr->b_l1hdr.b_mfu_hits = 0; 6778 hdr->b_l1hdr.b_mfu_ghost_hits = 0; 6779 arc_change_state(arc_anon, hdr); 6780 hdr->b_l1hdr.b_arc_access = 0; 6781 6782 mutex_exit(hash_lock); 6783 buf_discard_identity(hdr); 6784 arc_buf_thaw(buf); 6785 } 6786 } 6787 6788 int 6789 arc_released(arc_buf_t *buf) 6790 { 6791 return (buf->b_data != NULL && 6792 buf->b_hdr->b_l1hdr.b_state == arc_anon); 6793 } 6794 6795 #ifdef ZFS_DEBUG 6796 int 6797 arc_referenced(arc_buf_t *buf) 6798 { 6799 return (zfs_refcount_count(&buf->b_hdr->b_l1hdr.b_refcnt)); 6800 } 6801 #endif 6802 6803 static void 6804 arc_write_ready(zio_t *zio) 6805 { 6806 arc_write_callback_t *callback = zio->io_private; 6807 arc_buf_t *buf = callback->awcb_buf; 6808 arc_buf_hdr_t *hdr = buf->b_hdr; 6809 blkptr_t *bp = zio->io_bp; 6810 uint64_t psize = BP_IS_HOLE(bp) ? 0 : BP_GET_PSIZE(bp); 6811 fstrans_cookie_t cookie = spl_fstrans_mark(); 6812 6813 ASSERT(HDR_HAS_L1HDR(hdr)); 6814 ASSERT(!zfs_refcount_is_zero(&buf->b_hdr->b_l1hdr.b_refcnt)); 6815 ASSERT(hdr->b_l1hdr.b_bufcnt > 0); 6816 6817 /* 6818 * If we're reexecuting this zio because the pool suspended, then 6819 * cleanup any state that was previously set the first time the 6820 * callback was invoked. 6821 */ 6822 if (zio->io_flags & ZIO_FLAG_REEXECUTED) { 6823 arc_cksum_free(hdr); 6824 arc_buf_unwatch(buf); 6825 if (hdr->b_l1hdr.b_pabd != NULL) { 6826 if (arc_buf_is_shared(buf)) { 6827 arc_unshare_buf(hdr, buf); 6828 } else { 6829 arc_hdr_free_abd(hdr, B_FALSE); 6830 } 6831 } 6832 6833 if (HDR_HAS_RABD(hdr)) 6834 arc_hdr_free_abd(hdr, B_TRUE); 6835 } 6836 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 6837 ASSERT(!HDR_HAS_RABD(hdr)); 6838 ASSERT(!HDR_SHARED_DATA(hdr)); 6839 ASSERT(!arc_buf_is_shared(buf)); 6840 6841 callback->awcb_ready(zio, buf, callback->awcb_private); 6842 6843 if (HDR_IO_IN_PROGRESS(hdr)) { 6844 ASSERT(zio->io_flags & ZIO_FLAG_REEXECUTED); 6845 } else { 6846 arc_hdr_set_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 6847 add_reference(hdr, hdr); /* For IO_IN_PROGRESS. */ 6848 } 6849 6850 if (BP_IS_PROTECTED(bp) != !!HDR_PROTECTED(hdr)) 6851 hdr = arc_hdr_realloc_crypt(hdr, BP_IS_PROTECTED(bp)); 6852 6853 if (BP_IS_PROTECTED(bp)) { 6854 /* ZIL blocks are written through zio_rewrite */ 6855 ASSERT3U(BP_GET_TYPE(bp), !=, DMU_OT_INTENT_LOG); 6856 ASSERT(HDR_PROTECTED(hdr)); 6857 6858 if (BP_SHOULD_BYTESWAP(bp)) { 6859 if (BP_GET_LEVEL(bp) > 0) { 6860 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_UINT64; 6861 } else { 6862 hdr->b_l1hdr.b_byteswap = 6863 DMU_OT_BYTESWAP(BP_GET_TYPE(bp)); 6864 } 6865 } else { 6866 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_NUMFUNCS; 6867 } 6868 6869 hdr->b_crypt_hdr.b_ot = BP_GET_TYPE(bp); 6870 hdr->b_crypt_hdr.b_dsobj = zio->io_bookmark.zb_objset; 6871 zio_crypt_decode_params_bp(bp, hdr->b_crypt_hdr.b_salt, 6872 hdr->b_crypt_hdr.b_iv); 6873 zio_crypt_decode_mac_bp(bp, hdr->b_crypt_hdr.b_mac); 6874 } 6875 6876 /* 6877 * If this block was written for raw encryption but the zio layer 6878 * ended up only authenticating it, adjust the buffer flags now. 6879 */ 6880 if (BP_IS_AUTHENTICATED(bp) && ARC_BUF_ENCRYPTED(buf)) { 6881 arc_hdr_set_flags(hdr, ARC_FLAG_NOAUTH); 6882 buf->b_flags &= ~ARC_BUF_FLAG_ENCRYPTED; 6883 if (BP_GET_COMPRESS(bp) == ZIO_COMPRESS_OFF) 6884 buf->b_flags &= ~ARC_BUF_FLAG_COMPRESSED; 6885 } else if (BP_IS_HOLE(bp) && ARC_BUF_ENCRYPTED(buf)) { 6886 buf->b_flags &= ~ARC_BUF_FLAG_ENCRYPTED; 6887 buf->b_flags &= ~ARC_BUF_FLAG_COMPRESSED; 6888 } 6889 6890 /* this must be done after the buffer flags are adjusted */ 6891 arc_cksum_compute(buf); 6892 6893 enum zio_compress compress; 6894 if (BP_IS_HOLE(bp) || BP_IS_EMBEDDED(bp)) { 6895 compress = ZIO_COMPRESS_OFF; 6896 } else { 6897 ASSERT3U(HDR_GET_LSIZE(hdr), ==, BP_GET_LSIZE(bp)); 6898 compress = BP_GET_COMPRESS(bp); 6899 } 6900 HDR_SET_PSIZE(hdr, psize); 6901 arc_hdr_set_compress(hdr, compress); 6902 hdr->b_complevel = zio->io_prop.zp_complevel; 6903 6904 if (zio->io_error != 0 || psize == 0) 6905 goto out; 6906 6907 /* 6908 * Fill the hdr with data. If the buffer is encrypted we have no choice 6909 * but to copy the data into b_radb. If the hdr is compressed, the data 6910 * we want is available from the zio, otherwise we can take it from 6911 * the buf. 6912 * 6913 * We might be able to share the buf's data with the hdr here. However, 6914 * doing so would cause the ARC to be full of linear ABDs if we write a 6915 * lot of shareable data. As a compromise, we check whether scattered 6916 * ABDs are allowed, and assume that if they are then the user wants 6917 * the ARC to be primarily filled with them regardless of the data being 6918 * written. Therefore, if they're allowed then we allocate one and copy 6919 * the data into it; otherwise, we share the data directly if we can. 6920 */ 6921 if (ARC_BUF_ENCRYPTED(buf)) { 6922 ASSERT3U(psize, >, 0); 6923 ASSERT(ARC_BUF_COMPRESSED(buf)); 6924 arc_hdr_alloc_abd(hdr, ARC_HDR_DO_ADAPT | ARC_HDR_ALLOC_RDATA | 6925 ARC_HDR_USE_RESERVE); 6926 abd_copy(hdr->b_crypt_hdr.b_rabd, zio->io_abd, psize); 6927 } else if (!(HDR_UNCACHED(hdr) || 6928 abd_size_alloc_linear(arc_buf_size(buf))) || 6929 !arc_can_share(hdr, buf)) { 6930 /* 6931 * Ideally, we would always copy the io_abd into b_pabd, but the 6932 * user may have disabled compressed ARC, thus we must check the 6933 * hdr's compression setting rather than the io_bp's. 6934 */ 6935 if (BP_IS_ENCRYPTED(bp)) { 6936 ASSERT3U(psize, >, 0); 6937 arc_hdr_alloc_abd(hdr, ARC_HDR_DO_ADAPT | 6938 ARC_HDR_ALLOC_RDATA | ARC_HDR_USE_RESERVE); 6939 abd_copy(hdr->b_crypt_hdr.b_rabd, zio->io_abd, psize); 6940 } else if (arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF && 6941 !ARC_BUF_COMPRESSED(buf)) { 6942 ASSERT3U(psize, >, 0); 6943 arc_hdr_alloc_abd(hdr, ARC_HDR_DO_ADAPT | 6944 ARC_HDR_USE_RESERVE); 6945 abd_copy(hdr->b_l1hdr.b_pabd, zio->io_abd, psize); 6946 } else { 6947 ASSERT3U(zio->io_orig_size, ==, arc_hdr_size(hdr)); 6948 arc_hdr_alloc_abd(hdr, ARC_HDR_DO_ADAPT | 6949 ARC_HDR_USE_RESERVE); 6950 abd_copy_from_buf(hdr->b_l1hdr.b_pabd, buf->b_data, 6951 arc_buf_size(buf)); 6952 } 6953 } else { 6954 ASSERT3P(buf->b_data, ==, abd_to_buf(zio->io_orig_abd)); 6955 ASSERT3U(zio->io_orig_size, ==, arc_buf_size(buf)); 6956 ASSERT3U(hdr->b_l1hdr.b_bufcnt, ==, 1); 6957 6958 arc_share_buf(hdr, buf); 6959 } 6960 6961 out: 6962 arc_hdr_verify(hdr, bp); 6963 spl_fstrans_unmark(cookie); 6964 } 6965 6966 static void 6967 arc_write_children_ready(zio_t *zio) 6968 { 6969 arc_write_callback_t *callback = zio->io_private; 6970 arc_buf_t *buf = callback->awcb_buf; 6971 6972 callback->awcb_children_ready(zio, buf, callback->awcb_private); 6973 } 6974 6975 /* 6976 * The SPA calls this callback for each physical write that happens on behalf 6977 * of a logical write. See the comment in dbuf_write_physdone() for details. 6978 */ 6979 static void 6980 arc_write_physdone(zio_t *zio) 6981 { 6982 arc_write_callback_t *cb = zio->io_private; 6983 if (cb->awcb_physdone != NULL) 6984 cb->awcb_physdone(zio, cb->awcb_buf, cb->awcb_private); 6985 } 6986 6987 static void 6988 arc_write_done(zio_t *zio) 6989 { 6990 arc_write_callback_t *callback = zio->io_private; 6991 arc_buf_t *buf = callback->awcb_buf; 6992 arc_buf_hdr_t *hdr = buf->b_hdr; 6993 6994 ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); 6995 6996 if (zio->io_error == 0) { 6997 arc_hdr_verify(hdr, zio->io_bp); 6998 6999 if (BP_IS_HOLE(zio->io_bp) || BP_IS_EMBEDDED(zio->io_bp)) { 7000 buf_discard_identity(hdr); 7001 } else { 7002 hdr->b_dva = *BP_IDENTITY(zio->io_bp); 7003 hdr->b_birth = BP_PHYSICAL_BIRTH(zio->io_bp); 7004 } 7005 } else { 7006 ASSERT(HDR_EMPTY(hdr)); 7007 } 7008 7009 /* 7010 * If the block to be written was all-zero or compressed enough to be 7011 * embedded in the BP, no write was performed so there will be no 7012 * dva/birth/checksum. The buffer must therefore remain anonymous 7013 * (and uncached). 7014 */ 7015 if (!HDR_EMPTY(hdr)) { 7016 arc_buf_hdr_t *exists; 7017 kmutex_t *hash_lock; 7018 7019 ASSERT3U(zio->io_error, ==, 0); 7020 7021 arc_cksum_verify(buf); 7022 7023 exists = buf_hash_insert(hdr, &hash_lock); 7024 if (exists != NULL) { 7025 /* 7026 * This can only happen if we overwrite for 7027 * sync-to-convergence, because we remove 7028 * buffers from the hash table when we arc_free(). 7029 */ 7030 if (zio->io_flags & ZIO_FLAG_IO_REWRITE) { 7031 if (!BP_EQUAL(&zio->io_bp_orig, zio->io_bp)) 7032 panic("bad overwrite, hdr=%p exists=%p", 7033 (void *)hdr, (void *)exists); 7034 ASSERT(zfs_refcount_is_zero( 7035 &exists->b_l1hdr.b_refcnt)); 7036 arc_change_state(arc_anon, exists); 7037 arc_hdr_destroy(exists); 7038 mutex_exit(hash_lock); 7039 exists = buf_hash_insert(hdr, &hash_lock); 7040 ASSERT3P(exists, ==, NULL); 7041 } else if (zio->io_flags & ZIO_FLAG_NOPWRITE) { 7042 /* nopwrite */ 7043 ASSERT(zio->io_prop.zp_nopwrite); 7044 if (!BP_EQUAL(&zio->io_bp_orig, zio->io_bp)) 7045 panic("bad nopwrite, hdr=%p exists=%p", 7046 (void *)hdr, (void *)exists); 7047 } else { 7048 /* Dedup */ 7049 ASSERT(hdr->b_l1hdr.b_bufcnt == 1); 7050 ASSERT(hdr->b_l1hdr.b_state == arc_anon); 7051 ASSERT(BP_GET_DEDUP(zio->io_bp)); 7052 ASSERT(BP_GET_LEVEL(zio->io_bp) == 0); 7053 } 7054 } 7055 arc_hdr_clear_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 7056 VERIFY3S(remove_reference(hdr, hdr), >, 0); 7057 /* if it's not anon, we are doing a scrub */ 7058 if (exists == NULL && hdr->b_l1hdr.b_state == arc_anon) 7059 arc_access(hdr, 0, B_FALSE); 7060 mutex_exit(hash_lock); 7061 } else { 7062 arc_hdr_clear_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 7063 VERIFY3S(remove_reference(hdr, hdr), >, 0); 7064 } 7065 7066 callback->awcb_done(zio, buf, callback->awcb_private); 7067 7068 abd_free(zio->io_abd); 7069 kmem_free(callback, sizeof (arc_write_callback_t)); 7070 } 7071 7072 zio_t * 7073 arc_write(zio_t *pio, spa_t *spa, uint64_t txg, 7074 blkptr_t *bp, arc_buf_t *buf, boolean_t uncached, boolean_t l2arc, 7075 const zio_prop_t *zp, arc_write_done_func_t *ready, 7076 arc_write_done_func_t *children_ready, arc_write_done_func_t *physdone, 7077 arc_write_done_func_t *done, void *private, zio_priority_t priority, 7078 int zio_flags, const zbookmark_phys_t *zb) 7079 { 7080 arc_buf_hdr_t *hdr = buf->b_hdr; 7081 arc_write_callback_t *callback; 7082 zio_t *zio; 7083 zio_prop_t localprop = *zp; 7084 7085 ASSERT3P(ready, !=, NULL); 7086 ASSERT3P(done, !=, NULL); 7087 ASSERT(!HDR_IO_ERROR(hdr)); 7088 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 7089 ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); 7090 ASSERT3U(hdr->b_l1hdr.b_bufcnt, >, 0); 7091 if (uncached) 7092 arc_hdr_set_flags(hdr, ARC_FLAG_UNCACHED); 7093 else if (l2arc) 7094 arc_hdr_set_flags(hdr, ARC_FLAG_L2CACHE); 7095 7096 if (ARC_BUF_ENCRYPTED(buf)) { 7097 ASSERT(ARC_BUF_COMPRESSED(buf)); 7098 localprop.zp_encrypt = B_TRUE; 7099 localprop.zp_compress = HDR_GET_COMPRESS(hdr); 7100 localprop.zp_complevel = hdr->b_complevel; 7101 localprop.zp_byteorder = 7102 (hdr->b_l1hdr.b_byteswap == DMU_BSWAP_NUMFUNCS) ? 7103 ZFS_HOST_BYTEORDER : !ZFS_HOST_BYTEORDER; 7104 memcpy(localprop.zp_salt, hdr->b_crypt_hdr.b_salt, 7105 ZIO_DATA_SALT_LEN); 7106 memcpy(localprop.zp_iv, hdr->b_crypt_hdr.b_iv, 7107 ZIO_DATA_IV_LEN); 7108 memcpy(localprop.zp_mac, hdr->b_crypt_hdr.b_mac, 7109 ZIO_DATA_MAC_LEN); 7110 if (DMU_OT_IS_ENCRYPTED(localprop.zp_type)) { 7111 localprop.zp_nopwrite = B_FALSE; 7112 localprop.zp_copies = 7113 MIN(localprop.zp_copies, SPA_DVAS_PER_BP - 1); 7114 } 7115 zio_flags |= ZIO_FLAG_RAW; 7116 } else if (ARC_BUF_COMPRESSED(buf)) { 7117 ASSERT3U(HDR_GET_LSIZE(hdr), !=, arc_buf_size(buf)); 7118 localprop.zp_compress = HDR_GET_COMPRESS(hdr); 7119 localprop.zp_complevel = hdr->b_complevel; 7120 zio_flags |= ZIO_FLAG_RAW_COMPRESS; 7121 } 7122 callback = kmem_zalloc(sizeof (arc_write_callback_t), KM_SLEEP); 7123 callback->awcb_ready = ready; 7124 callback->awcb_children_ready = children_ready; 7125 callback->awcb_physdone = physdone; 7126 callback->awcb_done = done; 7127 callback->awcb_private = private; 7128 callback->awcb_buf = buf; 7129 7130 /* 7131 * The hdr's b_pabd is now stale, free it now. A new data block 7132 * will be allocated when the zio pipeline calls arc_write_ready(). 7133 */ 7134 if (hdr->b_l1hdr.b_pabd != NULL) { 7135 /* 7136 * If the buf is currently sharing the data block with 7137 * the hdr then we need to break that relationship here. 7138 * The hdr will remain with a NULL data pointer and the 7139 * buf will take sole ownership of the block. 7140 */ 7141 if (arc_buf_is_shared(buf)) { 7142 arc_unshare_buf(hdr, buf); 7143 } else { 7144 arc_hdr_free_abd(hdr, B_FALSE); 7145 } 7146 VERIFY3P(buf->b_data, !=, NULL); 7147 } 7148 7149 if (HDR_HAS_RABD(hdr)) 7150 arc_hdr_free_abd(hdr, B_TRUE); 7151 7152 if (!(zio_flags & ZIO_FLAG_RAW)) 7153 arc_hdr_set_compress(hdr, ZIO_COMPRESS_OFF); 7154 7155 ASSERT(!arc_buf_is_shared(buf)); 7156 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 7157 7158 zio = zio_write(pio, spa, txg, bp, 7159 abd_get_from_buf(buf->b_data, HDR_GET_LSIZE(hdr)), 7160 HDR_GET_LSIZE(hdr), arc_buf_size(buf), &localprop, arc_write_ready, 7161 (children_ready != NULL) ? arc_write_children_ready : NULL, 7162 arc_write_physdone, arc_write_done, callback, 7163 priority, zio_flags, zb); 7164 7165 return (zio); 7166 } 7167 7168 void 7169 arc_tempreserve_clear(uint64_t reserve) 7170 { 7171 atomic_add_64(&arc_tempreserve, -reserve); 7172 ASSERT((int64_t)arc_tempreserve >= 0); 7173 } 7174 7175 int 7176 arc_tempreserve_space(spa_t *spa, uint64_t reserve, uint64_t txg) 7177 { 7178 int error; 7179 uint64_t anon_size; 7180 7181 if (!arc_no_grow && 7182 reserve > arc_c/4 && 7183 reserve * 4 > (2ULL << SPA_MAXBLOCKSHIFT)) 7184 arc_c = MIN(arc_c_max, reserve * 4); 7185 7186 /* 7187 * Throttle when the calculated memory footprint for the TXG 7188 * exceeds the target ARC size. 7189 */ 7190 if (reserve > arc_c) { 7191 DMU_TX_STAT_BUMP(dmu_tx_memory_reserve); 7192 return (SET_ERROR(ERESTART)); 7193 } 7194 7195 /* 7196 * Don't count loaned bufs as in flight dirty data to prevent long 7197 * network delays from blocking transactions that are ready to be 7198 * assigned to a txg. 7199 */ 7200 7201 /* assert that it has not wrapped around */ 7202 ASSERT3S(atomic_add_64_nv(&arc_loaned_bytes, 0), >=, 0); 7203 7204 anon_size = MAX((int64_t)(zfs_refcount_count(&arc_anon->arcs_size) - 7205 arc_loaned_bytes), 0); 7206 7207 /* 7208 * Writes will, almost always, require additional memory allocations 7209 * in order to compress/encrypt/etc the data. We therefore need to 7210 * make sure that there is sufficient available memory for this. 7211 */ 7212 error = arc_memory_throttle(spa, reserve, txg); 7213 if (error != 0) 7214 return (error); 7215 7216 /* 7217 * Throttle writes when the amount of dirty data in the cache 7218 * gets too large. We try to keep the cache less than half full 7219 * of dirty blocks so that our sync times don't grow too large. 7220 * 7221 * In the case of one pool being built on another pool, we want 7222 * to make sure we don't end up throttling the lower (backing) 7223 * pool when the upper pool is the majority contributor to dirty 7224 * data. To insure we make forward progress during throttling, we 7225 * also check the current pool's net dirty data and only throttle 7226 * if it exceeds zfs_arc_pool_dirty_percent of the anonymous dirty 7227 * data in the cache. 7228 * 7229 * Note: if two requests come in concurrently, we might let them 7230 * both succeed, when one of them should fail. Not a huge deal. 7231 */ 7232 uint64_t total_dirty = reserve + arc_tempreserve + anon_size; 7233 uint64_t spa_dirty_anon = spa_dirty_data(spa); 7234 uint64_t rarc_c = arc_warm ? arc_c : arc_c_max; 7235 if (total_dirty > rarc_c * zfs_arc_dirty_limit_percent / 100 && 7236 anon_size > rarc_c * zfs_arc_anon_limit_percent / 100 && 7237 spa_dirty_anon > anon_size * zfs_arc_pool_dirty_percent / 100) { 7238 #ifdef ZFS_DEBUG 7239 uint64_t meta_esize = zfs_refcount_count( 7240 &arc_anon->arcs_esize[ARC_BUFC_METADATA]); 7241 uint64_t data_esize = 7242 zfs_refcount_count(&arc_anon->arcs_esize[ARC_BUFC_DATA]); 7243 dprintf("failing, arc_tempreserve=%lluK anon_meta=%lluK " 7244 "anon_data=%lluK tempreserve=%lluK rarc_c=%lluK\n", 7245 (u_longlong_t)arc_tempreserve >> 10, 7246 (u_longlong_t)meta_esize >> 10, 7247 (u_longlong_t)data_esize >> 10, 7248 (u_longlong_t)reserve >> 10, 7249 (u_longlong_t)rarc_c >> 10); 7250 #endif 7251 DMU_TX_STAT_BUMP(dmu_tx_dirty_throttle); 7252 return (SET_ERROR(ERESTART)); 7253 } 7254 atomic_add_64(&arc_tempreserve, reserve); 7255 return (0); 7256 } 7257 7258 static void 7259 arc_kstat_update_state(arc_state_t *state, kstat_named_t *size, 7260 kstat_named_t *evict_data, kstat_named_t *evict_metadata) 7261 { 7262 size->value.ui64 = zfs_refcount_count(&state->arcs_size); 7263 evict_data->value.ui64 = 7264 zfs_refcount_count(&state->arcs_esize[ARC_BUFC_DATA]); 7265 evict_metadata->value.ui64 = 7266 zfs_refcount_count(&state->arcs_esize[ARC_BUFC_METADATA]); 7267 } 7268 7269 static int 7270 arc_kstat_update(kstat_t *ksp, int rw) 7271 { 7272 arc_stats_t *as = ksp->ks_data; 7273 7274 if (rw == KSTAT_WRITE) 7275 return (SET_ERROR(EACCES)); 7276 7277 as->arcstat_hits.value.ui64 = 7278 wmsum_value(&arc_sums.arcstat_hits); 7279 as->arcstat_iohits.value.ui64 = 7280 wmsum_value(&arc_sums.arcstat_iohits); 7281 as->arcstat_misses.value.ui64 = 7282 wmsum_value(&arc_sums.arcstat_misses); 7283 as->arcstat_demand_data_hits.value.ui64 = 7284 wmsum_value(&arc_sums.arcstat_demand_data_hits); 7285 as->arcstat_demand_data_iohits.value.ui64 = 7286 wmsum_value(&arc_sums.arcstat_demand_data_iohits); 7287 as->arcstat_demand_data_misses.value.ui64 = 7288 wmsum_value(&arc_sums.arcstat_demand_data_misses); 7289 as->arcstat_demand_metadata_hits.value.ui64 = 7290 wmsum_value(&arc_sums.arcstat_demand_metadata_hits); 7291 as->arcstat_demand_metadata_iohits.value.ui64 = 7292 wmsum_value(&arc_sums.arcstat_demand_metadata_iohits); 7293 as->arcstat_demand_metadata_misses.value.ui64 = 7294 wmsum_value(&arc_sums.arcstat_demand_metadata_misses); 7295 as->arcstat_prefetch_data_hits.value.ui64 = 7296 wmsum_value(&arc_sums.arcstat_prefetch_data_hits); 7297 as->arcstat_prefetch_data_iohits.value.ui64 = 7298 wmsum_value(&arc_sums.arcstat_prefetch_data_iohits); 7299 as->arcstat_prefetch_data_misses.value.ui64 = 7300 wmsum_value(&arc_sums.arcstat_prefetch_data_misses); 7301 as->arcstat_prefetch_metadata_hits.value.ui64 = 7302 wmsum_value(&arc_sums.arcstat_prefetch_metadata_hits); 7303 as->arcstat_prefetch_metadata_iohits.value.ui64 = 7304 wmsum_value(&arc_sums.arcstat_prefetch_metadata_iohits); 7305 as->arcstat_prefetch_metadata_misses.value.ui64 = 7306 wmsum_value(&arc_sums.arcstat_prefetch_metadata_misses); 7307 as->arcstat_mru_hits.value.ui64 = 7308 wmsum_value(&arc_sums.arcstat_mru_hits); 7309 as->arcstat_mru_ghost_hits.value.ui64 = 7310 wmsum_value(&arc_sums.arcstat_mru_ghost_hits); 7311 as->arcstat_mfu_hits.value.ui64 = 7312 wmsum_value(&arc_sums.arcstat_mfu_hits); 7313 as->arcstat_mfu_ghost_hits.value.ui64 = 7314 wmsum_value(&arc_sums.arcstat_mfu_ghost_hits); 7315 as->arcstat_uncached_hits.value.ui64 = 7316 wmsum_value(&arc_sums.arcstat_uncached_hits); 7317 as->arcstat_deleted.value.ui64 = 7318 wmsum_value(&arc_sums.arcstat_deleted); 7319 as->arcstat_mutex_miss.value.ui64 = 7320 wmsum_value(&arc_sums.arcstat_mutex_miss); 7321 as->arcstat_access_skip.value.ui64 = 7322 wmsum_value(&arc_sums.arcstat_access_skip); 7323 as->arcstat_evict_skip.value.ui64 = 7324 wmsum_value(&arc_sums.arcstat_evict_skip); 7325 as->arcstat_evict_not_enough.value.ui64 = 7326 wmsum_value(&arc_sums.arcstat_evict_not_enough); 7327 as->arcstat_evict_l2_cached.value.ui64 = 7328 wmsum_value(&arc_sums.arcstat_evict_l2_cached); 7329 as->arcstat_evict_l2_eligible.value.ui64 = 7330 wmsum_value(&arc_sums.arcstat_evict_l2_eligible); 7331 as->arcstat_evict_l2_eligible_mfu.value.ui64 = 7332 wmsum_value(&arc_sums.arcstat_evict_l2_eligible_mfu); 7333 as->arcstat_evict_l2_eligible_mru.value.ui64 = 7334 wmsum_value(&arc_sums.arcstat_evict_l2_eligible_mru); 7335 as->arcstat_evict_l2_ineligible.value.ui64 = 7336 wmsum_value(&arc_sums.arcstat_evict_l2_ineligible); 7337 as->arcstat_evict_l2_skip.value.ui64 = 7338 wmsum_value(&arc_sums.arcstat_evict_l2_skip); 7339 as->arcstat_hash_collisions.value.ui64 = 7340 wmsum_value(&arc_sums.arcstat_hash_collisions); 7341 as->arcstat_hash_chains.value.ui64 = 7342 wmsum_value(&arc_sums.arcstat_hash_chains); 7343 as->arcstat_size.value.ui64 = 7344 aggsum_value(&arc_sums.arcstat_size); 7345 as->arcstat_compressed_size.value.ui64 = 7346 wmsum_value(&arc_sums.arcstat_compressed_size); 7347 as->arcstat_uncompressed_size.value.ui64 = 7348 wmsum_value(&arc_sums.arcstat_uncompressed_size); 7349 as->arcstat_overhead_size.value.ui64 = 7350 wmsum_value(&arc_sums.arcstat_overhead_size); 7351 as->arcstat_hdr_size.value.ui64 = 7352 wmsum_value(&arc_sums.arcstat_hdr_size); 7353 as->arcstat_data_size.value.ui64 = 7354 wmsum_value(&arc_sums.arcstat_data_size); 7355 as->arcstat_metadata_size.value.ui64 = 7356 wmsum_value(&arc_sums.arcstat_metadata_size); 7357 as->arcstat_dbuf_size.value.ui64 = 7358 wmsum_value(&arc_sums.arcstat_dbuf_size); 7359 #if defined(COMPAT_FREEBSD11) 7360 as->arcstat_other_size.value.ui64 = 7361 wmsum_value(&arc_sums.arcstat_bonus_size) + 7362 aggsum_value(&arc_sums.arcstat_dnode_size) + 7363 wmsum_value(&arc_sums.arcstat_dbuf_size); 7364 #endif 7365 7366 arc_kstat_update_state(arc_anon, 7367 &as->arcstat_anon_size, 7368 &as->arcstat_anon_evictable_data, 7369 &as->arcstat_anon_evictable_metadata); 7370 arc_kstat_update_state(arc_mru, 7371 &as->arcstat_mru_size, 7372 &as->arcstat_mru_evictable_data, 7373 &as->arcstat_mru_evictable_metadata); 7374 arc_kstat_update_state(arc_mru_ghost, 7375 &as->arcstat_mru_ghost_size, 7376 &as->arcstat_mru_ghost_evictable_data, 7377 &as->arcstat_mru_ghost_evictable_metadata); 7378 arc_kstat_update_state(arc_mfu, 7379 &as->arcstat_mfu_size, 7380 &as->arcstat_mfu_evictable_data, 7381 &as->arcstat_mfu_evictable_metadata); 7382 arc_kstat_update_state(arc_mfu_ghost, 7383 &as->arcstat_mfu_ghost_size, 7384 &as->arcstat_mfu_ghost_evictable_data, 7385 &as->arcstat_mfu_ghost_evictable_metadata); 7386 arc_kstat_update_state(arc_uncached, 7387 &as->arcstat_uncached_size, 7388 &as->arcstat_uncached_evictable_data, 7389 &as->arcstat_uncached_evictable_metadata); 7390 7391 as->arcstat_dnode_size.value.ui64 = 7392 aggsum_value(&arc_sums.arcstat_dnode_size); 7393 as->arcstat_bonus_size.value.ui64 = 7394 wmsum_value(&arc_sums.arcstat_bonus_size); 7395 as->arcstat_l2_hits.value.ui64 = 7396 wmsum_value(&arc_sums.arcstat_l2_hits); 7397 as->arcstat_l2_misses.value.ui64 = 7398 wmsum_value(&arc_sums.arcstat_l2_misses); 7399 as->arcstat_l2_prefetch_asize.value.ui64 = 7400 wmsum_value(&arc_sums.arcstat_l2_prefetch_asize); 7401 as->arcstat_l2_mru_asize.value.ui64 = 7402 wmsum_value(&arc_sums.arcstat_l2_mru_asize); 7403 as->arcstat_l2_mfu_asize.value.ui64 = 7404 wmsum_value(&arc_sums.arcstat_l2_mfu_asize); 7405 as->arcstat_l2_bufc_data_asize.value.ui64 = 7406 wmsum_value(&arc_sums.arcstat_l2_bufc_data_asize); 7407 as->arcstat_l2_bufc_metadata_asize.value.ui64 = 7408 wmsum_value(&arc_sums.arcstat_l2_bufc_metadata_asize); 7409 as->arcstat_l2_feeds.value.ui64 = 7410 wmsum_value(&arc_sums.arcstat_l2_feeds); 7411 as->arcstat_l2_rw_clash.value.ui64 = 7412 wmsum_value(&arc_sums.arcstat_l2_rw_clash); 7413 as->arcstat_l2_read_bytes.value.ui64 = 7414 wmsum_value(&arc_sums.arcstat_l2_read_bytes); 7415 as->arcstat_l2_write_bytes.value.ui64 = 7416 wmsum_value(&arc_sums.arcstat_l2_write_bytes); 7417 as->arcstat_l2_writes_sent.value.ui64 = 7418 wmsum_value(&arc_sums.arcstat_l2_writes_sent); 7419 as->arcstat_l2_writes_done.value.ui64 = 7420 wmsum_value(&arc_sums.arcstat_l2_writes_done); 7421 as->arcstat_l2_writes_error.value.ui64 = 7422 wmsum_value(&arc_sums.arcstat_l2_writes_error); 7423 as->arcstat_l2_writes_lock_retry.value.ui64 = 7424 wmsum_value(&arc_sums.arcstat_l2_writes_lock_retry); 7425 as->arcstat_l2_evict_lock_retry.value.ui64 = 7426 wmsum_value(&arc_sums.arcstat_l2_evict_lock_retry); 7427 as->arcstat_l2_evict_reading.value.ui64 = 7428 wmsum_value(&arc_sums.arcstat_l2_evict_reading); 7429 as->arcstat_l2_evict_l1cached.value.ui64 = 7430 wmsum_value(&arc_sums.arcstat_l2_evict_l1cached); 7431 as->arcstat_l2_free_on_write.value.ui64 = 7432 wmsum_value(&arc_sums.arcstat_l2_free_on_write); 7433 as->arcstat_l2_abort_lowmem.value.ui64 = 7434 wmsum_value(&arc_sums.arcstat_l2_abort_lowmem); 7435 as->arcstat_l2_cksum_bad.value.ui64 = 7436 wmsum_value(&arc_sums.arcstat_l2_cksum_bad); 7437 as->arcstat_l2_io_error.value.ui64 = 7438 wmsum_value(&arc_sums.arcstat_l2_io_error); 7439 as->arcstat_l2_lsize.value.ui64 = 7440 wmsum_value(&arc_sums.arcstat_l2_lsize); 7441 as->arcstat_l2_psize.value.ui64 = 7442 wmsum_value(&arc_sums.arcstat_l2_psize); 7443 as->arcstat_l2_hdr_size.value.ui64 = 7444 aggsum_value(&arc_sums.arcstat_l2_hdr_size); 7445 as->arcstat_l2_log_blk_writes.value.ui64 = 7446 wmsum_value(&arc_sums.arcstat_l2_log_blk_writes); 7447 as->arcstat_l2_log_blk_asize.value.ui64 = 7448 wmsum_value(&arc_sums.arcstat_l2_log_blk_asize); 7449 as->arcstat_l2_log_blk_count.value.ui64 = 7450 wmsum_value(&arc_sums.arcstat_l2_log_blk_count); 7451 as->arcstat_l2_rebuild_success.value.ui64 = 7452 wmsum_value(&arc_sums.arcstat_l2_rebuild_success); 7453 as->arcstat_l2_rebuild_abort_unsupported.value.ui64 = 7454 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_unsupported); 7455 as->arcstat_l2_rebuild_abort_io_errors.value.ui64 = 7456 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_io_errors); 7457 as->arcstat_l2_rebuild_abort_dh_errors.value.ui64 = 7458 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_dh_errors); 7459 as->arcstat_l2_rebuild_abort_cksum_lb_errors.value.ui64 = 7460 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_cksum_lb_errors); 7461 as->arcstat_l2_rebuild_abort_lowmem.value.ui64 = 7462 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_lowmem); 7463 as->arcstat_l2_rebuild_size.value.ui64 = 7464 wmsum_value(&arc_sums.arcstat_l2_rebuild_size); 7465 as->arcstat_l2_rebuild_asize.value.ui64 = 7466 wmsum_value(&arc_sums.arcstat_l2_rebuild_asize); 7467 as->arcstat_l2_rebuild_bufs.value.ui64 = 7468 wmsum_value(&arc_sums.arcstat_l2_rebuild_bufs); 7469 as->arcstat_l2_rebuild_bufs_precached.value.ui64 = 7470 wmsum_value(&arc_sums.arcstat_l2_rebuild_bufs_precached); 7471 as->arcstat_l2_rebuild_log_blks.value.ui64 = 7472 wmsum_value(&arc_sums.arcstat_l2_rebuild_log_blks); 7473 as->arcstat_memory_throttle_count.value.ui64 = 7474 wmsum_value(&arc_sums.arcstat_memory_throttle_count); 7475 as->arcstat_memory_direct_count.value.ui64 = 7476 wmsum_value(&arc_sums.arcstat_memory_direct_count); 7477 as->arcstat_memory_indirect_count.value.ui64 = 7478 wmsum_value(&arc_sums.arcstat_memory_indirect_count); 7479 7480 as->arcstat_memory_all_bytes.value.ui64 = 7481 arc_all_memory(); 7482 as->arcstat_memory_free_bytes.value.ui64 = 7483 arc_free_memory(); 7484 as->arcstat_memory_available_bytes.value.i64 = 7485 arc_available_memory(); 7486 7487 as->arcstat_prune.value.ui64 = 7488 wmsum_value(&arc_sums.arcstat_prune); 7489 as->arcstat_meta_used.value.ui64 = 7490 aggsum_value(&arc_sums.arcstat_meta_used); 7491 as->arcstat_async_upgrade_sync.value.ui64 = 7492 wmsum_value(&arc_sums.arcstat_async_upgrade_sync); 7493 as->arcstat_predictive_prefetch.value.ui64 = 7494 wmsum_value(&arc_sums.arcstat_predictive_prefetch); 7495 as->arcstat_demand_hit_predictive_prefetch.value.ui64 = 7496 wmsum_value(&arc_sums.arcstat_demand_hit_predictive_prefetch); 7497 as->arcstat_demand_iohit_predictive_prefetch.value.ui64 = 7498 wmsum_value(&arc_sums.arcstat_demand_iohit_predictive_prefetch); 7499 as->arcstat_prescient_prefetch.value.ui64 = 7500 wmsum_value(&arc_sums.arcstat_prescient_prefetch); 7501 as->arcstat_demand_hit_prescient_prefetch.value.ui64 = 7502 wmsum_value(&arc_sums.arcstat_demand_hit_prescient_prefetch); 7503 as->arcstat_demand_iohit_prescient_prefetch.value.ui64 = 7504 wmsum_value(&arc_sums.arcstat_demand_iohit_prescient_prefetch); 7505 as->arcstat_raw_size.value.ui64 = 7506 wmsum_value(&arc_sums.arcstat_raw_size); 7507 as->arcstat_cached_only_in_progress.value.ui64 = 7508 wmsum_value(&arc_sums.arcstat_cached_only_in_progress); 7509 as->arcstat_abd_chunk_waste_size.value.ui64 = 7510 wmsum_value(&arc_sums.arcstat_abd_chunk_waste_size); 7511 7512 return (0); 7513 } 7514 7515 /* 7516 * This function *must* return indices evenly distributed between all 7517 * sublists of the multilist. This is needed due to how the ARC eviction 7518 * code is laid out; arc_evict_state() assumes ARC buffers are evenly 7519 * distributed between all sublists and uses this assumption when 7520 * deciding which sublist to evict from and how much to evict from it. 7521 */ 7522 static unsigned int 7523 arc_state_multilist_index_func(multilist_t *ml, void *obj) 7524 { 7525 arc_buf_hdr_t *hdr = obj; 7526 7527 /* 7528 * We rely on b_dva to generate evenly distributed index 7529 * numbers using buf_hash below. So, as an added precaution, 7530 * let's make sure we never add empty buffers to the arc lists. 7531 */ 7532 ASSERT(!HDR_EMPTY(hdr)); 7533 7534 /* 7535 * The assumption here, is the hash value for a given 7536 * arc_buf_hdr_t will remain constant throughout its lifetime 7537 * (i.e. its b_spa, b_dva, and b_birth fields don't change). 7538 * Thus, we don't need to store the header's sublist index 7539 * on insertion, as this index can be recalculated on removal. 7540 * 7541 * Also, the low order bits of the hash value are thought to be 7542 * distributed evenly. Otherwise, in the case that the multilist 7543 * has a power of two number of sublists, each sublists' usage 7544 * would not be evenly distributed. In this context full 64bit 7545 * division would be a waste of time, so limit it to 32 bits. 7546 */ 7547 return ((unsigned int)buf_hash(hdr->b_spa, &hdr->b_dva, hdr->b_birth) % 7548 multilist_get_num_sublists(ml)); 7549 } 7550 7551 static unsigned int 7552 arc_state_l2c_multilist_index_func(multilist_t *ml, void *obj) 7553 { 7554 panic("Header %p insert into arc_l2c_only %p", obj, ml); 7555 } 7556 7557 #define WARN_IF_TUNING_IGNORED(tuning, value, do_warn) do { \ 7558 if ((do_warn) && (tuning) && ((tuning) != (value))) { \ 7559 cmn_err(CE_WARN, \ 7560 "ignoring tunable %s (using %llu instead)", \ 7561 (#tuning), (u_longlong_t)(value)); \ 7562 } \ 7563 } while (0) 7564 7565 /* 7566 * Called during module initialization and periodically thereafter to 7567 * apply reasonable changes to the exposed performance tunings. Can also be 7568 * called explicitly by param_set_arc_*() functions when ARC tunables are 7569 * updated manually. Non-zero zfs_* values which differ from the currently set 7570 * values will be applied. 7571 */ 7572 void 7573 arc_tuning_update(boolean_t verbose) 7574 { 7575 uint64_t allmem = arc_all_memory(); 7576 unsigned long limit; 7577 7578 /* Valid range: 32M - <arc_c_max> */ 7579 if ((zfs_arc_min) && (zfs_arc_min != arc_c_min) && 7580 (zfs_arc_min >= 2ULL << SPA_MAXBLOCKSHIFT) && 7581 (zfs_arc_min <= arc_c_max)) { 7582 arc_c_min = zfs_arc_min; 7583 arc_c = MAX(arc_c, arc_c_min); 7584 } 7585 WARN_IF_TUNING_IGNORED(zfs_arc_min, arc_c_min, verbose); 7586 7587 /* Valid range: 64M - <all physical memory> */ 7588 if ((zfs_arc_max) && (zfs_arc_max != arc_c_max) && 7589 (zfs_arc_max >= MIN_ARC_MAX) && (zfs_arc_max < allmem) && 7590 (zfs_arc_max > arc_c_min)) { 7591 arc_c_max = zfs_arc_max; 7592 arc_c = MIN(arc_c, arc_c_max); 7593 arc_p = (arc_c >> 1); 7594 if (arc_meta_limit > arc_c_max) 7595 arc_meta_limit = arc_c_max; 7596 if (arc_dnode_size_limit > arc_meta_limit) 7597 arc_dnode_size_limit = arc_meta_limit; 7598 } 7599 WARN_IF_TUNING_IGNORED(zfs_arc_max, arc_c_max, verbose); 7600 7601 /* Valid range: 16M - <arc_c_max> */ 7602 if ((zfs_arc_meta_min) && (zfs_arc_meta_min != arc_meta_min) && 7603 (zfs_arc_meta_min >= 1ULL << SPA_MAXBLOCKSHIFT) && 7604 (zfs_arc_meta_min <= arc_c_max)) { 7605 arc_meta_min = zfs_arc_meta_min; 7606 if (arc_meta_limit < arc_meta_min) 7607 arc_meta_limit = arc_meta_min; 7608 if (arc_dnode_size_limit < arc_meta_min) 7609 arc_dnode_size_limit = arc_meta_min; 7610 } 7611 WARN_IF_TUNING_IGNORED(zfs_arc_meta_min, arc_meta_min, verbose); 7612 7613 /* Valid range: <arc_meta_min> - <arc_c_max> */ 7614 limit = zfs_arc_meta_limit ? zfs_arc_meta_limit : 7615 MIN(zfs_arc_meta_limit_percent, 100) * arc_c_max / 100; 7616 if ((limit != arc_meta_limit) && 7617 (limit >= arc_meta_min) && 7618 (limit <= arc_c_max)) 7619 arc_meta_limit = limit; 7620 WARN_IF_TUNING_IGNORED(zfs_arc_meta_limit, arc_meta_limit, verbose); 7621 7622 /* Valid range: <arc_meta_min> - <arc_meta_limit> */ 7623 limit = zfs_arc_dnode_limit ? zfs_arc_dnode_limit : 7624 MIN(zfs_arc_dnode_limit_percent, 100) * arc_meta_limit / 100; 7625 if ((limit != arc_dnode_size_limit) && 7626 (limit >= arc_meta_min) && 7627 (limit <= arc_meta_limit)) 7628 arc_dnode_size_limit = limit; 7629 WARN_IF_TUNING_IGNORED(zfs_arc_dnode_limit, arc_dnode_size_limit, 7630 verbose); 7631 7632 /* Valid range: 1 - N */ 7633 if (zfs_arc_grow_retry) 7634 arc_grow_retry = zfs_arc_grow_retry; 7635 7636 /* Valid range: 1 - N */ 7637 if (zfs_arc_shrink_shift) { 7638 arc_shrink_shift = zfs_arc_shrink_shift; 7639 arc_no_grow_shift = MIN(arc_no_grow_shift, arc_shrink_shift -1); 7640 } 7641 7642 /* Valid range: 1 - N */ 7643 if (zfs_arc_p_min_shift) 7644 arc_p_min_shift = zfs_arc_p_min_shift; 7645 7646 /* Valid range: 1 - N ms */ 7647 if (zfs_arc_min_prefetch_ms) 7648 arc_min_prefetch_ms = zfs_arc_min_prefetch_ms; 7649 7650 /* Valid range: 1 - N ms */ 7651 if (zfs_arc_min_prescient_prefetch_ms) { 7652 arc_min_prescient_prefetch_ms = 7653 zfs_arc_min_prescient_prefetch_ms; 7654 } 7655 7656 /* Valid range: 0 - 100 */ 7657 if (zfs_arc_lotsfree_percent <= 100) 7658 arc_lotsfree_percent = zfs_arc_lotsfree_percent; 7659 WARN_IF_TUNING_IGNORED(zfs_arc_lotsfree_percent, arc_lotsfree_percent, 7660 verbose); 7661 7662 /* Valid range: 0 - <all physical memory> */ 7663 if ((zfs_arc_sys_free) && (zfs_arc_sys_free != arc_sys_free)) 7664 arc_sys_free = MIN(zfs_arc_sys_free, allmem); 7665 WARN_IF_TUNING_IGNORED(zfs_arc_sys_free, arc_sys_free, verbose); 7666 } 7667 7668 static void 7669 arc_state_multilist_init(multilist_t *ml, 7670 multilist_sublist_index_func_t *index_func, int *maxcountp) 7671 { 7672 multilist_create(ml, sizeof (arc_buf_hdr_t), 7673 offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node), index_func); 7674 *maxcountp = MAX(*maxcountp, multilist_get_num_sublists(ml)); 7675 } 7676 7677 static void 7678 arc_state_init(void) 7679 { 7680 int num_sublists = 0; 7681 7682 arc_state_multilist_init(&arc_mru->arcs_list[ARC_BUFC_METADATA], 7683 arc_state_multilist_index_func, &num_sublists); 7684 arc_state_multilist_init(&arc_mru->arcs_list[ARC_BUFC_DATA], 7685 arc_state_multilist_index_func, &num_sublists); 7686 arc_state_multilist_init(&arc_mru_ghost->arcs_list[ARC_BUFC_METADATA], 7687 arc_state_multilist_index_func, &num_sublists); 7688 arc_state_multilist_init(&arc_mru_ghost->arcs_list[ARC_BUFC_DATA], 7689 arc_state_multilist_index_func, &num_sublists); 7690 arc_state_multilist_init(&arc_mfu->arcs_list[ARC_BUFC_METADATA], 7691 arc_state_multilist_index_func, &num_sublists); 7692 arc_state_multilist_init(&arc_mfu->arcs_list[ARC_BUFC_DATA], 7693 arc_state_multilist_index_func, &num_sublists); 7694 arc_state_multilist_init(&arc_mfu_ghost->arcs_list[ARC_BUFC_METADATA], 7695 arc_state_multilist_index_func, &num_sublists); 7696 arc_state_multilist_init(&arc_mfu_ghost->arcs_list[ARC_BUFC_DATA], 7697 arc_state_multilist_index_func, &num_sublists); 7698 arc_state_multilist_init(&arc_uncached->arcs_list[ARC_BUFC_METADATA], 7699 arc_state_multilist_index_func, &num_sublists); 7700 arc_state_multilist_init(&arc_uncached->arcs_list[ARC_BUFC_DATA], 7701 arc_state_multilist_index_func, &num_sublists); 7702 7703 /* 7704 * L2 headers should never be on the L2 state list since they don't 7705 * have L1 headers allocated. Special index function asserts that. 7706 */ 7707 arc_state_multilist_init(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA], 7708 arc_state_l2c_multilist_index_func, &num_sublists); 7709 arc_state_multilist_init(&arc_l2c_only->arcs_list[ARC_BUFC_DATA], 7710 arc_state_l2c_multilist_index_func, &num_sublists); 7711 7712 /* 7713 * Keep track of the number of markers needed to reclaim buffers from 7714 * any ARC state. The markers will be pre-allocated so as to minimize 7715 * the number of memory allocations performed by the eviction thread. 7716 */ 7717 arc_state_evict_marker_count = num_sublists; 7718 7719 zfs_refcount_create(&arc_anon->arcs_esize[ARC_BUFC_METADATA]); 7720 zfs_refcount_create(&arc_anon->arcs_esize[ARC_BUFC_DATA]); 7721 zfs_refcount_create(&arc_mru->arcs_esize[ARC_BUFC_METADATA]); 7722 zfs_refcount_create(&arc_mru->arcs_esize[ARC_BUFC_DATA]); 7723 zfs_refcount_create(&arc_mru_ghost->arcs_esize[ARC_BUFC_METADATA]); 7724 zfs_refcount_create(&arc_mru_ghost->arcs_esize[ARC_BUFC_DATA]); 7725 zfs_refcount_create(&arc_mfu->arcs_esize[ARC_BUFC_METADATA]); 7726 zfs_refcount_create(&arc_mfu->arcs_esize[ARC_BUFC_DATA]); 7727 zfs_refcount_create(&arc_mfu_ghost->arcs_esize[ARC_BUFC_METADATA]); 7728 zfs_refcount_create(&arc_mfu_ghost->arcs_esize[ARC_BUFC_DATA]); 7729 zfs_refcount_create(&arc_l2c_only->arcs_esize[ARC_BUFC_METADATA]); 7730 zfs_refcount_create(&arc_l2c_only->arcs_esize[ARC_BUFC_DATA]); 7731 zfs_refcount_create(&arc_uncached->arcs_esize[ARC_BUFC_METADATA]); 7732 zfs_refcount_create(&arc_uncached->arcs_esize[ARC_BUFC_DATA]); 7733 7734 zfs_refcount_create(&arc_anon->arcs_size); 7735 zfs_refcount_create(&arc_mru->arcs_size); 7736 zfs_refcount_create(&arc_mru_ghost->arcs_size); 7737 zfs_refcount_create(&arc_mfu->arcs_size); 7738 zfs_refcount_create(&arc_mfu_ghost->arcs_size); 7739 zfs_refcount_create(&arc_l2c_only->arcs_size); 7740 zfs_refcount_create(&arc_uncached->arcs_size); 7741 7742 wmsum_init(&arc_sums.arcstat_hits, 0); 7743 wmsum_init(&arc_sums.arcstat_iohits, 0); 7744 wmsum_init(&arc_sums.arcstat_misses, 0); 7745 wmsum_init(&arc_sums.arcstat_demand_data_hits, 0); 7746 wmsum_init(&arc_sums.arcstat_demand_data_iohits, 0); 7747 wmsum_init(&arc_sums.arcstat_demand_data_misses, 0); 7748 wmsum_init(&arc_sums.arcstat_demand_metadata_hits, 0); 7749 wmsum_init(&arc_sums.arcstat_demand_metadata_iohits, 0); 7750 wmsum_init(&arc_sums.arcstat_demand_metadata_misses, 0); 7751 wmsum_init(&arc_sums.arcstat_prefetch_data_hits, 0); 7752 wmsum_init(&arc_sums.arcstat_prefetch_data_iohits, 0); 7753 wmsum_init(&arc_sums.arcstat_prefetch_data_misses, 0); 7754 wmsum_init(&arc_sums.arcstat_prefetch_metadata_hits, 0); 7755 wmsum_init(&arc_sums.arcstat_prefetch_metadata_iohits, 0); 7756 wmsum_init(&arc_sums.arcstat_prefetch_metadata_misses, 0); 7757 wmsum_init(&arc_sums.arcstat_mru_hits, 0); 7758 wmsum_init(&arc_sums.arcstat_mru_ghost_hits, 0); 7759 wmsum_init(&arc_sums.arcstat_mfu_hits, 0); 7760 wmsum_init(&arc_sums.arcstat_mfu_ghost_hits, 0); 7761 wmsum_init(&arc_sums.arcstat_uncached_hits, 0); 7762 wmsum_init(&arc_sums.arcstat_deleted, 0); 7763 wmsum_init(&arc_sums.arcstat_mutex_miss, 0); 7764 wmsum_init(&arc_sums.arcstat_access_skip, 0); 7765 wmsum_init(&arc_sums.arcstat_evict_skip, 0); 7766 wmsum_init(&arc_sums.arcstat_evict_not_enough, 0); 7767 wmsum_init(&arc_sums.arcstat_evict_l2_cached, 0); 7768 wmsum_init(&arc_sums.arcstat_evict_l2_eligible, 0); 7769 wmsum_init(&arc_sums.arcstat_evict_l2_eligible_mfu, 0); 7770 wmsum_init(&arc_sums.arcstat_evict_l2_eligible_mru, 0); 7771 wmsum_init(&arc_sums.arcstat_evict_l2_ineligible, 0); 7772 wmsum_init(&arc_sums.arcstat_evict_l2_skip, 0); 7773 wmsum_init(&arc_sums.arcstat_hash_collisions, 0); 7774 wmsum_init(&arc_sums.arcstat_hash_chains, 0); 7775 aggsum_init(&arc_sums.arcstat_size, 0); 7776 wmsum_init(&arc_sums.arcstat_compressed_size, 0); 7777 wmsum_init(&arc_sums.arcstat_uncompressed_size, 0); 7778 wmsum_init(&arc_sums.arcstat_overhead_size, 0); 7779 wmsum_init(&arc_sums.arcstat_hdr_size, 0); 7780 wmsum_init(&arc_sums.arcstat_data_size, 0); 7781 wmsum_init(&arc_sums.arcstat_metadata_size, 0); 7782 wmsum_init(&arc_sums.arcstat_dbuf_size, 0); 7783 aggsum_init(&arc_sums.arcstat_dnode_size, 0); 7784 wmsum_init(&arc_sums.arcstat_bonus_size, 0); 7785 wmsum_init(&arc_sums.arcstat_l2_hits, 0); 7786 wmsum_init(&arc_sums.arcstat_l2_misses, 0); 7787 wmsum_init(&arc_sums.arcstat_l2_prefetch_asize, 0); 7788 wmsum_init(&arc_sums.arcstat_l2_mru_asize, 0); 7789 wmsum_init(&arc_sums.arcstat_l2_mfu_asize, 0); 7790 wmsum_init(&arc_sums.arcstat_l2_bufc_data_asize, 0); 7791 wmsum_init(&arc_sums.arcstat_l2_bufc_metadata_asize, 0); 7792 wmsum_init(&arc_sums.arcstat_l2_feeds, 0); 7793 wmsum_init(&arc_sums.arcstat_l2_rw_clash, 0); 7794 wmsum_init(&arc_sums.arcstat_l2_read_bytes, 0); 7795 wmsum_init(&arc_sums.arcstat_l2_write_bytes, 0); 7796 wmsum_init(&arc_sums.arcstat_l2_writes_sent, 0); 7797 wmsum_init(&arc_sums.arcstat_l2_writes_done, 0); 7798 wmsum_init(&arc_sums.arcstat_l2_writes_error, 0); 7799 wmsum_init(&arc_sums.arcstat_l2_writes_lock_retry, 0); 7800 wmsum_init(&arc_sums.arcstat_l2_evict_lock_retry, 0); 7801 wmsum_init(&arc_sums.arcstat_l2_evict_reading, 0); 7802 wmsum_init(&arc_sums.arcstat_l2_evict_l1cached, 0); 7803 wmsum_init(&arc_sums.arcstat_l2_free_on_write, 0); 7804 wmsum_init(&arc_sums.arcstat_l2_abort_lowmem, 0); 7805 wmsum_init(&arc_sums.arcstat_l2_cksum_bad, 0); 7806 wmsum_init(&arc_sums.arcstat_l2_io_error, 0); 7807 wmsum_init(&arc_sums.arcstat_l2_lsize, 0); 7808 wmsum_init(&arc_sums.arcstat_l2_psize, 0); 7809 aggsum_init(&arc_sums.arcstat_l2_hdr_size, 0); 7810 wmsum_init(&arc_sums.arcstat_l2_log_blk_writes, 0); 7811 wmsum_init(&arc_sums.arcstat_l2_log_blk_asize, 0); 7812 wmsum_init(&arc_sums.arcstat_l2_log_blk_count, 0); 7813 wmsum_init(&arc_sums.arcstat_l2_rebuild_success, 0); 7814 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_unsupported, 0); 7815 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_io_errors, 0); 7816 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_dh_errors, 0); 7817 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_cksum_lb_errors, 0); 7818 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_lowmem, 0); 7819 wmsum_init(&arc_sums.arcstat_l2_rebuild_size, 0); 7820 wmsum_init(&arc_sums.arcstat_l2_rebuild_asize, 0); 7821 wmsum_init(&arc_sums.arcstat_l2_rebuild_bufs, 0); 7822 wmsum_init(&arc_sums.arcstat_l2_rebuild_bufs_precached, 0); 7823 wmsum_init(&arc_sums.arcstat_l2_rebuild_log_blks, 0); 7824 wmsum_init(&arc_sums.arcstat_memory_throttle_count, 0); 7825 wmsum_init(&arc_sums.arcstat_memory_direct_count, 0); 7826 wmsum_init(&arc_sums.arcstat_memory_indirect_count, 0); 7827 wmsum_init(&arc_sums.arcstat_prune, 0); 7828 aggsum_init(&arc_sums.arcstat_meta_used, 0); 7829 wmsum_init(&arc_sums.arcstat_async_upgrade_sync, 0); 7830 wmsum_init(&arc_sums.arcstat_predictive_prefetch, 0); 7831 wmsum_init(&arc_sums.arcstat_demand_hit_predictive_prefetch, 0); 7832 wmsum_init(&arc_sums.arcstat_demand_iohit_predictive_prefetch, 0); 7833 wmsum_init(&arc_sums.arcstat_prescient_prefetch, 0); 7834 wmsum_init(&arc_sums.arcstat_demand_hit_prescient_prefetch, 0); 7835 wmsum_init(&arc_sums.arcstat_demand_iohit_prescient_prefetch, 0); 7836 wmsum_init(&arc_sums.arcstat_raw_size, 0); 7837 wmsum_init(&arc_sums.arcstat_cached_only_in_progress, 0); 7838 wmsum_init(&arc_sums.arcstat_abd_chunk_waste_size, 0); 7839 7840 arc_anon->arcs_state = ARC_STATE_ANON; 7841 arc_mru->arcs_state = ARC_STATE_MRU; 7842 arc_mru_ghost->arcs_state = ARC_STATE_MRU_GHOST; 7843 arc_mfu->arcs_state = ARC_STATE_MFU; 7844 arc_mfu_ghost->arcs_state = ARC_STATE_MFU_GHOST; 7845 arc_l2c_only->arcs_state = ARC_STATE_L2C_ONLY; 7846 arc_uncached->arcs_state = ARC_STATE_UNCACHED; 7847 } 7848 7849 static void 7850 arc_state_fini(void) 7851 { 7852 zfs_refcount_destroy(&arc_anon->arcs_esize[ARC_BUFC_METADATA]); 7853 zfs_refcount_destroy(&arc_anon->arcs_esize[ARC_BUFC_DATA]); 7854 zfs_refcount_destroy(&arc_mru->arcs_esize[ARC_BUFC_METADATA]); 7855 zfs_refcount_destroy(&arc_mru->arcs_esize[ARC_BUFC_DATA]); 7856 zfs_refcount_destroy(&arc_mru_ghost->arcs_esize[ARC_BUFC_METADATA]); 7857 zfs_refcount_destroy(&arc_mru_ghost->arcs_esize[ARC_BUFC_DATA]); 7858 zfs_refcount_destroy(&arc_mfu->arcs_esize[ARC_BUFC_METADATA]); 7859 zfs_refcount_destroy(&arc_mfu->arcs_esize[ARC_BUFC_DATA]); 7860 zfs_refcount_destroy(&arc_mfu_ghost->arcs_esize[ARC_BUFC_METADATA]); 7861 zfs_refcount_destroy(&arc_mfu_ghost->arcs_esize[ARC_BUFC_DATA]); 7862 zfs_refcount_destroy(&arc_l2c_only->arcs_esize[ARC_BUFC_METADATA]); 7863 zfs_refcount_destroy(&arc_l2c_only->arcs_esize[ARC_BUFC_DATA]); 7864 zfs_refcount_destroy(&arc_uncached->arcs_esize[ARC_BUFC_METADATA]); 7865 zfs_refcount_destroy(&arc_uncached->arcs_esize[ARC_BUFC_DATA]); 7866 7867 zfs_refcount_destroy(&arc_anon->arcs_size); 7868 zfs_refcount_destroy(&arc_mru->arcs_size); 7869 zfs_refcount_destroy(&arc_mru_ghost->arcs_size); 7870 zfs_refcount_destroy(&arc_mfu->arcs_size); 7871 zfs_refcount_destroy(&arc_mfu_ghost->arcs_size); 7872 zfs_refcount_destroy(&arc_l2c_only->arcs_size); 7873 zfs_refcount_destroy(&arc_uncached->arcs_size); 7874 7875 multilist_destroy(&arc_mru->arcs_list[ARC_BUFC_METADATA]); 7876 multilist_destroy(&arc_mru_ghost->arcs_list[ARC_BUFC_METADATA]); 7877 multilist_destroy(&arc_mfu->arcs_list[ARC_BUFC_METADATA]); 7878 multilist_destroy(&arc_mfu_ghost->arcs_list[ARC_BUFC_METADATA]); 7879 multilist_destroy(&arc_mru->arcs_list[ARC_BUFC_DATA]); 7880 multilist_destroy(&arc_mru_ghost->arcs_list[ARC_BUFC_DATA]); 7881 multilist_destroy(&arc_mfu->arcs_list[ARC_BUFC_DATA]); 7882 multilist_destroy(&arc_mfu_ghost->arcs_list[ARC_BUFC_DATA]); 7883 multilist_destroy(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA]); 7884 multilist_destroy(&arc_l2c_only->arcs_list[ARC_BUFC_DATA]); 7885 multilist_destroy(&arc_uncached->arcs_list[ARC_BUFC_METADATA]); 7886 multilist_destroy(&arc_uncached->arcs_list[ARC_BUFC_DATA]); 7887 7888 wmsum_fini(&arc_sums.arcstat_hits); 7889 wmsum_fini(&arc_sums.arcstat_iohits); 7890 wmsum_fini(&arc_sums.arcstat_misses); 7891 wmsum_fini(&arc_sums.arcstat_demand_data_hits); 7892 wmsum_fini(&arc_sums.arcstat_demand_data_iohits); 7893 wmsum_fini(&arc_sums.arcstat_demand_data_misses); 7894 wmsum_fini(&arc_sums.arcstat_demand_metadata_hits); 7895 wmsum_fini(&arc_sums.arcstat_demand_metadata_iohits); 7896 wmsum_fini(&arc_sums.arcstat_demand_metadata_misses); 7897 wmsum_fini(&arc_sums.arcstat_prefetch_data_hits); 7898 wmsum_fini(&arc_sums.arcstat_prefetch_data_iohits); 7899 wmsum_fini(&arc_sums.arcstat_prefetch_data_misses); 7900 wmsum_fini(&arc_sums.arcstat_prefetch_metadata_hits); 7901 wmsum_fini(&arc_sums.arcstat_prefetch_metadata_iohits); 7902 wmsum_fini(&arc_sums.arcstat_prefetch_metadata_misses); 7903 wmsum_fini(&arc_sums.arcstat_mru_hits); 7904 wmsum_fini(&arc_sums.arcstat_mru_ghost_hits); 7905 wmsum_fini(&arc_sums.arcstat_mfu_hits); 7906 wmsum_fini(&arc_sums.arcstat_mfu_ghost_hits); 7907 wmsum_fini(&arc_sums.arcstat_uncached_hits); 7908 wmsum_fini(&arc_sums.arcstat_deleted); 7909 wmsum_fini(&arc_sums.arcstat_mutex_miss); 7910 wmsum_fini(&arc_sums.arcstat_access_skip); 7911 wmsum_fini(&arc_sums.arcstat_evict_skip); 7912 wmsum_fini(&arc_sums.arcstat_evict_not_enough); 7913 wmsum_fini(&arc_sums.arcstat_evict_l2_cached); 7914 wmsum_fini(&arc_sums.arcstat_evict_l2_eligible); 7915 wmsum_fini(&arc_sums.arcstat_evict_l2_eligible_mfu); 7916 wmsum_fini(&arc_sums.arcstat_evict_l2_eligible_mru); 7917 wmsum_fini(&arc_sums.arcstat_evict_l2_ineligible); 7918 wmsum_fini(&arc_sums.arcstat_evict_l2_skip); 7919 wmsum_fini(&arc_sums.arcstat_hash_collisions); 7920 wmsum_fini(&arc_sums.arcstat_hash_chains); 7921 aggsum_fini(&arc_sums.arcstat_size); 7922 wmsum_fini(&arc_sums.arcstat_compressed_size); 7923 wmsum_fini(&arc_sums.arcstat_uncompressed_size); 7924 wmsum_fini(&arc_sums.arcstat_overhead_size); 7925 wmsum_fini(&arc_sums.arcstat_hdr_size); 7926 wmsum_fini(&arc_sums.arcstat_data_size); 7927 wmsum_fini(&arc_sums.arcstat_metadata_size); 7928 wmsum_fini(&arc_sums.arcstat_dbuf_size); 7929 aggsum_fini(&arc_sums.arcstat_dnode_size); 7930 wmsum_fini(&arc_sums.arcstat_bonus_size); 7931 wmsum_fini(&arc_sums.arcstat_l2_hits); 7932 wmsum_fini(&arc_sums.arcstat_l2_misses); 7933 wmsum_fini(&arc_sums.arcstat_l2_prefetch_asize); 7934 wmsum_fini(&arc_sums.arcstat_l2_mru_asize); 7935 wmsum_fini(&arc_sums.arcstat_l2_mfu_asize); 7936 wmsum_fini(&arc_sums.arcstat_l2_bufc_data_asize); 7937 wmsum_fini(&arc_sums.arcstat_l2_bufc_metadata_asize); 7938 wmsum_fini(&arc_sums.arcstat_l2_feeds); 7939 wmsum_fini(&arc_sums.arcstat_l2_rw_clash); 7940 wmsum_fini(&arc_sums.arcstat_l2_read_bytes); 7941 wmsum_fini(&arc_sums.arcstat_l2_write_bytes); 7942 wmsum_fini(&arc_sums.arcstat_l2_writes_sent); 7943 wmsum_fini(&arc_sums.arcstat_l2_writes_done); 7944 wmsum_fini(&arc_sums.arcstat_l2_writes_error); 7945 wmsum_fini(&arc_sums.arcstat_l2_writes_lock_retry); 7946 wmsum_fini(&arc_sums.arcstat_l2_evict_lock_retry); 7947 wmsum_fini(&arc_sums.arcstat_l2_evict_reading); 7948 wmsum_fini(&arc_sums.arcstat_l2_evict_l1cached); 7949 wmsum_fini(&arc_sums.arcstat_l2_free_on_write); 7950 wmsum_fini(&arc_sums.arcstat_l2_abort_lowmem); 7951 wmsum_fini(&arc_sums.arcstat_l2_cksum_bad); 7952 wmsum_fini(&arc_sums.arcstat_l2_io_error); 7953 wmsum_fini(&arc_sums.arcstat_l2_lsize); 7954 wmsum_fini(&arc_sums.arcstat_l2_psize); 7955 aggsum_fini(&arc_sums.arcstat_l2_hdr_size); 7956 wmsum_fini(&arc_sums.arcstat_l2_log_blk_writes); 7957 wmsum_fini(&arc_sums.arcstat_l2_log_blk_asize); 7958 wmsum_fini(&arc_sums.arcstat_l2_log_blk_count); 7959 wmsum_fini(&arc_sums.arcstat_l2_rebuild_success); 7960 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_unsupported); 7961 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_io_errors); 7962 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_dh_errors); 7963 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_cksum_lb_errors); 7964 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_lowmem); 7965 wmsum_fini(&arc_sums.arcstat_l2_rebuild_size); 7966 wmsum_fini(&arc_sums.arcstat_l2_rebuild_asize); 7967 wmsum_fini(&arc_sums.arcstat_l2_rebuild_bufs); 7968 wmsum_fini(&arc_sums.arcstat_l2_rebuild_bufs_precached); 7969 wmsum_fini(&arc_sums.arcstat_l2_rebuild_log_blks); 7970 wmsum_fini(&arc_sums.arcstat_memory_throttle_count); 7971 wmsum_fini(&arc_sums.arcstat_memory_direct_count); 7972 wmsum_fini(&arc_sums.arcstat_memory_indirect_count); 7973 wmsum_fini(&arc_sums.arcstat_prune); 7974 aggsum_fini(&arc_sums.arcstat_meta_used); 7975 wmsum_fini(&arc_sums.arcstat_async_upgrade_sync); 7976 wmsum_fini(&arc_sums.arcstat_predictive_prefetch); 7977 wmsum_fini(&arc_sums.arcstat_demand_hit_predictive_prefetch); 7978 wmsum_fini(&arc_sums.arcstat_demand_iohit_predictive_prefetch); 7979 wmsum_fini(&arc_sums.arcstat_prescient_prefetch); 7980 wmsum_fini(&arc_sums.arcstat_demand_hit_prescient_prefetch); 7981 wmsum_fini(&arc_sums.arcstat_demand_iohit_prescient_prefetch); 7982 wmsum_fini(&arc_sums.arcstat_raw_size); 7983 wmsum_fini(&arc_sums.arcstat_cached_only_in_progress); 7984 wmsum_fini(&arc_sums.arcstat_abd_chunk_waste_size); 7985 } 7986 7987 uint64_t 7988 arc_target_bytes(void) 7989 { 7990 return (arc_c); 7991 } 7992 7993 void 7994 arc_set_limits(uint64_t allmem) 7995 { 7996 /* Set min cache to 1/32 of all memory, or 32MB, whichever is more. */ 7997 arc_c_min = MAX(allmem / 32, 2ULL << SPA_MAXBLOCKSHIFT); 7998 7999 /* How to set default max varies by platform. */ 8000 arc_c_max = arc_default_max(arc_c_min, allmem); 8001 } 8002 void 8003 arc_init(void) 8004 { 8005 uint64_t percent, allmem = arc_all_memory(); 8006 mutex_init(&arc_evict_lock, NULL, MUTEX_DEFAULT, NULL); 8007 list_create(&arc_evict_waiters, sizeof (arc_evict_waiter_t), 8008 offsetof(arc_evict_waiter_t, aew_node)); 8009 8010 arc_min_prefetch_ms = 1000; 8011 arc_min_prescient_prefetch_ms = 6000; 8012 8013 #if defined(_KERNEL) 8014 arc_lowmem_init(); 8015 #endif 8016 8017 arc_set_limits(allmem); 8018 8019 #ifdef _KERNEL 8020 /* 8021 * If zfs_arc_max is non-zero at init, meaning it was set in the kernel 8022 * environment before the module was loaded, don't block setting the 8023 * maximum because it is less than arc_c_min, instead, reset arc_c_min 8024 * to a lower value. 8025 * zfs_arc_min will be handled by arc_tuning_update(). 8026 */ 8027 if (zfs_arc_max != 0 && zfs_arc_max >= MIN_ARC_MAX && 8028 zfs_arc_max < allmem) { 8029 arc_c_max = zfs_arc_max; 8030 if (arc_c_min >= arc_c_max) { 8031 arc_c_min = MAX(zfs_arc_max / 2, 8032 2ULL << SPA_MAXBLOCKSHIFT); 8033 } 8034 } 8035 #else 8036 /* 8037 * In userland, there's only the memory pressure that we artificially 8038 * create (see arc_available_memory()). Don't let arc_c get too 8039 * small, because it can cause transactions to be larger than 8040 * arc_c, causing arc_tempreserve_space() to fail. 8041 */ 8042 arc_c_min = MAX(arc_c_max / 2, 2ULL << SPA_MAXBLOCKSHIFT); 8043 #endif 8044 8045 arc_c = arc_c_min; 8046 arc_p = (arc_c >> 1); 8047 8048 /* Set min to 1/2 of arc_c_min */ 8049 arc_meta_min = 1ULL << SPA_MAXBLOCKSHIFT; 8050 /* 8051 * Set arc_meta_limit to a percent of arc_c_max with a floor of 8052 * arc_meta_min, and a ceiling of arc_c_max. 8053 */ 8054 percent = MIN(zfs_arc_meta_limit_percent, 100); 8055 arc_meta_limit = MAX(arc_meta_min, (percent * arc_c_max) / 100); 8056 percent = MIN(zfs_arc_dnode_limit_percent, 100); 8057 arc_dnode_size_limit = (percent * arc_meta_limit) / 100; 8058 8059 /* Apply user specified tunings */ 8060 arc_tuning_update(B_TRUE); 8061 8062 /* if kmem_flags are set, lets try to use less memory */ 8063 if (kmem_debugging()) 8064 arc_c = arc_c / 2; 8065 if (arc_c < arc_c_min) 8066 arc_c = arc_c_min; 8067 8068 arc_register_hotplug(); 8069 8070 arc_state_init(); 8071 8072 buf_init(); 8073 8074 list_create(&arc_prune_list, sizeof (arc_prune_t), 8075 offsetof(arc_prune_t, p_node)); 8076 mutex_init(&arc_prune_mtx, NULL, MUTEX_DEFAULT, NULL); 8077 8078 arc_prune_taskq = taskq_create("arc_prune", zfs_arc_prune_task_threads, 8079 defclsyspri, 100, INT_MAX, TASKQ_PREPOPULATE | TASKQ_DYNAMIC); 8080 8081 arc_ksp = kstat_create("zfs", 0, "arcstats", "misc", KSTAT_TYPE_NAMED, 8082 sizeof (arc_stats) / sizeof (kstat_named_t), KSTAT_FLAG_VIRTUAL); 8083 8084 if (arc_ksp != NULL) { 8085 arc_ksp->ks_data = &arc_stats; 8086 arc_ksp->ks_update = arc_kstat_update; 8087 kstat_install(arc_ksp); 8088 } 8089 8090 arc_state_evict_markers = 8091 arc_state_alloc_markers(arc_state_evict_marker_count); 8092 arc_evict_zthr = zthr_create_timer("arc_evict", 8093 arc_evict_cb_check, arc_evict_cb, NULL, SEC2NSEC(1), defclsyspri); 8094 arc_reap_zthr = zthr_create_timer("arc_reap", 8095 arc_reap_cb_check, arc_reap_cb, NULL, SEC2NSEC(1), minclsyspri); 8096 8097 arc_warm = B_FALSE; 8098 8099 /* 8100 * Calculate maximum amount of dirty data per pool. 8101 * 8102 * If it has been set by a module parameter, take that. 8103 * Otherwise, use a percentage of physical memory defined by 8104 * zfs_dirty_data_max_percent (default 10%) with a cap at 8105 * zfs_dirty_data_max_max (default 4G or 25% of physical memory). 8106 */ 8107 #ifdef __LP64__ 8108 if (zfs_dirty_data_max_max == 0) 8109 zfs_dirty_data_max_max = MIN(4ULL * 1024 * 1024 * 1024, 8110 allmem * zfs_dirty_data_max_max_percent / 100); 8111 #else 8112 if (zfs_dirty_data_max_max == 0) 8113 zfs_dirty_data_max_max = MIN(1ULL * 1024 * 1024 * 1024, 8114 allmem * zfs_dirty_data_max_max_percent / 100); 8115 #endif 8116 8117 if (zfs_dirty_data_max == 0) { 8118 zfs_dirty_data_max = allmem * 8119 zfs_dirty_data_max_percent / 100; 8120 zfs_dirty_data_max = MIN(zfs_dirty_data_max, 8121 zfs_dirty_data_max_max); 8122 } 8123 8124 if (zfs_wrlog_data_max == 0) { 8125 8126 /* 8127 * dp_wrlog_total is reduced for each txg at the end of 8128 * spa_sync(). However, dp_dirty_total is reduced every time 8129 * a block is written out. Thus under normal operation, 8130 * dp_wrlog_total could grow 2 times as big as 8131 * zfs_dirty_data_max. 8132 */ 8133 zfs_wrlog_data_max = zfs_dirty_data_max * 2; 8134 } 8135 } 8136 8137 void 8138 arc_fini(void) 8139 { 8140 arc_prune_t *p; 8141 8142 #ifdef _KERNEL 8143 arc_lowmem_fini(); 8144 #endif /* _KERNEL */ 8145 8146 /* Use B_TRUE to ensure *all* buffers are evicted */ 8147 arc_flush(NULL, B_TRUE); 8148 8149 if (arc_ksp != NULL) { 8150 kstat_delete(arc_ksp); 8151 arc_ksp = NULL; 8152 } 8153 8154 taskq_wait(arc_prune_taskq); 8155 taskq_destroy(arc_prune_taskq); 8156 8157 mutex_enter(&arc_prune_mtx); 8158 while ((p = list_head(&arc_prune_list)) != NULL) { 8159 list_remove(&arc_prune_list, p); 8160 zfs_refcount_remove(&p->p_refcnt, &arc_prune_list); 8161 zfs_refcount_destroy(&p->p_refcnt); 8162 kmem_free(p, sizeof (*p)); 8163 } 8164 mutex_exit(&arc_prune_mtx); 8165 8166 list_destroy(&arc_prune_list); 8167 mutex_destroy(&arc_prune_mtx); 8168 8169 (void) zthr_cancel(arc_evict_zthr); 8170 (void) zthr_cancel(arc_reap_zthr); 8171 arc_state_free_markers(arc_state_evict_markers, 8172 arc_state_evict_marker_count); 8173 8174 mutex_destroy(&arc_evict_lock); 8175 list_destroy(&arc_evict_waiters); 8176 8177 /* 8178 * Free any buffers that were tagged for destruction. This needs 8179 * to occur before arc_state_fini() runs and destroys the aggsum 8180 * values which are updated when freeing scatter ABDs. 8181 */ 8182 l2arc_do_free_on_write(); 8183 8184 /* 8185 * buf_fini() must proceed arc_state_fini() because buf_fin() may 8186 * trigger the release of kmem magazines, which can callback to 8187 * arc_space_return() which accesses aggsums freed in act_state_fini(). 8188 */ 8189 buf_fini(); 8190 arc_state_fini(); 8191 8192 arc_unregister_hotplug(); 8193 8194 /* 8195 * We destroy the zthrs after all the ARC state has been 8196 * torn down to avoid the case of them receiving any 8197 * wakeup() signals after they are destroyed. 8198 */ 8199 zthr_destroy(arc_evict_zthr); 8200 zthr_destroy(arc_reap_zthr); 8201 8202 ASSERT0(arc_loaned_bytes); 8203 } 8204 8205 /* 8206 * Level 2 ARC 8207 * 8208 * The level 2 ARC (L2ARC) is a cache layer in-between main memory and disk. 8209 * It uses dedicated storage devices to hold cached data, which are populated 8210 * using large infrequent writes. The main role of this cache is to boost 8211 * the performance of random read workloads. The intended L2ARC devices 8212 * include short-stroked disks, solid state disks, and other media with 8213 * substantially faster read latency than disk. 8214 * 8215 * +-----------------------+ 8216 * | ARC | 8217 * +-----------------------+ 8218 * | ^ ^ 8219 * | | | 8220 * l2arc_feed_thread() arc_read() 8221 * | | | 8222 * | l2arc read | 8223 * V | | 8224 * +---------------+ | 8225 * | L2ARC | | 8226 * +---------------+ | 8227 * | ^ | 8228 * l2arc_write() | | 8229 * | | | 8230 * V | | 8231 * +-------+ +-------+ 8232 * | vdev | | vdev | 8233 * | cache | | cache | 8234 * +-------+ +-------+ 8235 * +=========+ .-----. 8236 * : L2ARC : |-_____-| 8237 * : devices : | Disks | 8238 * +=========+ `-_____-' 8239 * 8240 * Read requests are satisfied from the following sources, in order: 8241 * 8242 * 1) ARC 8243 * 2) vdev cache of L2ARC devices 8244 * 3) L2ARC devices 8245 * 4) vdev cache of disks 8246 * 5) disks 8247 * 8248 * Some L2ARC device types exhibit extremely slow write performance. 8249 * To accommodate for this there are some significant differences between 8250 * the L2ARC and traditional cache design: 8251 * 8252 * 1. There is no eviction path from the ARC to the L2ARC. Evictions from 8253 * the ARC behave as usual, freeing buffers and placing headers on ghost 8254 * lists. The ARC does not send buffers to the L2ARC during eviction as 8255 * this would add inflated write latencies for all ARC memory pressure. 8256 * 8257 * 2. The L2ARC attempts to cache data from the ARC before it is evicted. 8258 * It does this by periodically scanning buffers from the eviction-end of 8259 * the MFU and MRU ARC lists, copying them to the L2ARC devices if they are 8260 * not already there. It scans until a headroom of buffers is satisfied, 8261 * which itself is a buffer for ARC eviction. If a compressible buffer is 8262 * found during scanning and selected for writing to an L2ARC device, we 8263 * temporarily boost scanning headroom during the next scan cycle to make 8264 * sure we adapt to compression effects (which might significantly reduce 8265 * the data volume we write to L2ARC). The thread that does this is 8266 * l2arc_feed_thread(), illustrated below; example sizes are included to 8267 * provide a better sense of ratio than this diagram: 8268 * 8269 * head --> tail 8270 * +---------------------+----------+ 8271 * ARC_mfu |:::::#:::::::::::::::|o#o###o###|-->. # already on L2ARC 8272 * +---------------------+----------+ | o L2ARC eligible 8273 * ARC_mru |:#:::::::::::::::::::|#o#ooo####|-->| : ARC buffer 8274 * +---------------------+----------+ | 8275 * 15.9 Gbytes ^ 32 Mbytes | 8276 * headroom | 8277 * l2arc_feed_thread() 8278 * | 8279 * l2arc write hand <--[oooo]--' 8280 * | 8 Mbyte 8281 * | write max 8282 * V 8283 * +==============================+ 8284 * L2ARC dev |####|#|###|###| |####| ... | 8285 * +==============================+ 8286 * 32 Gbytes 8287 * 8288 * 3. If an ARC buffer is copied to the L2ARC but then hit instead of 8289 * evicted, then the L2ARC has cached a buffer much sooner than it probably 8290 * needed to, potentially wasting L2ARC device bandwidth and storage. It is 8291 * safe to say that this is an uncommon case, since buffers at the end of 8292 * the ARC lists have moved there due to inactivity. 8293 * 8294 * 4. If the ARC evicts faster than the L2ARC can maintain a headroom, 8295 * then the L2ARC simply misses copying some buffers. This serves as a 8296 * pressure valve to prevent heavy read workloads from both stalling the ARC 8297 * with waits and clogging the L2ARC with writes. This also helps prevent 8298 * the potential for the L2ARC to churn if it attempts to cache content too 8299 * quickly, such as during backups of the entire pool. 8300 * 8301 * 5. After system boot and before the ARC has filled main memory, there are 8302 * no evictions from the ARC and so the tails of the ARC_mfu and ARC_mru 8303 * lists can remain mostly static. Instead of searching from tail of these 8304 * lists as pictured, the l2arc_feed_thread() will search from the list heads 8305 * for eligible buffers, greatly increasing its chance of finding them. 8306 * 8307 * The L2ARC device write speed is also boosted during this time so that 8308 * the L2ARC warms up faster. Since there have been no ARC evictions yet, 8309 * there are no L2ARC reads, and no fear of degrading read performance 8310 * through increased writes. 8311 * 8312 * 6. Writes to the L2ARC devices are grouped and sent in-sequence, so that 8313 * the vdev queue can aggregate them into larger and fewer writes. Each 8314 * device is written to in a rotor fashion, sweeping writes through 8315 * available space then repeating. 8316 * 8317 * 7. The L2ARC does not store dirty content. It never needs to flush 8318 * write buffers back to disk based storage. 8319 * 8320 * 8. If an ARC buffer is written (and dirtied) which also exists in the 8321 * L2ARC, the now stale L2ARC buffer is immediately dropped. 8322 * 8323 * The performance of the L2ARC can be tweaked by a number of tunables, which 8324 * may be necessary for different workloads: 8325 * 8326 * l2arc_write_max max write bytes per interval 8327 * l2arc_write_boost extra write bytes during device warmup 8328 * l2arc_noprefetch skip caching prefetched buffers 8329 * l2arc_headroom number of max device writes to precache 8330 * l2arc_headroom_boost when we find compressed buffers during ARC 8331 * scanning, we multiply headroom by this 8332 * percentage factor for the next scan cycle, 8333 * since more compressed buffers are likely to 8334 * be present 8335 * l2arc_feed_secs seconds between L2ARC writing 8336 * 8337 * Tunables may be removed or added as future performance improvements are 8338 * integrated, and also may become zpool properties. 8339 * 8340 * There are three key functions that control how the L2ARC warms up: 8341 * 8342 * l2arc_write_eligible() check if a buffer is eligible to cache 8343 * l2arc_write_size() calculate how much to write 8344 * l2arc_write_interval() calculate sleep delay between writes 8345 * 8346 * These three functions determine what to write, how much, and how quickly 8347 * to send writes. 8348 * 8349 * L2ARC persistence: 8350 * 8351 * When writing buffers to L2ARC, we periodically add some metadata to 8352 * make sure we can pick them up after reboot, thus dramatically reducing 8353 * the impact that any downtime has on the performance of storage systems 8354 * with large caches. 8355 * 8356 * The implementation works fairly simply by integrating the following two 8357 * modifications: 8358 * 8359 * *) When writing to the L2ARC, we occasionally write a "l2arc log block", 8360 * which is an additional piece of metadata which describes what's been 8361 * written. This allows us to rebuild the arc_buf_hdr_t structures of the 8362 * main ARC buffers. There are 2 linked-lists of log blocks headed by 8363 * dh_start_lbps[2]. We alternate which chain we append to, so they are 8364 * time-wise and offset-wise interleaved, but that is an optimization rather 8365 * than for correctness. The log block also includes a pointer to the 8366 * previous block in its chain. 8367 * 8368 * *) We reserve SPA_MINBLOCKSIZE of space at the start of each L2ARC device 8369 * for our header bookkeeping purposes. This contains a device header, 8370 * which contains our top-level reference structures. We update it each 8371 * time we write a new log block, so that we're able to locate it in the 8372 * L2ARC device. If this write results in an inconsistent device header 8373 * (e.g. due to power failure), we detect this by verifying the header's 8374 * checksum and simply fail to reconstruct the L2ARC after reboot. 8375 * 8376 * Implementation diagram: 8377 * 8378 * +=== L2ARC device (not to scale) ======================================+ 8379 * | ___two newest log block pointers__.__________ | 8380 * | / \dh_start_lbps[1] | 8381 * | / \ \dh_start_lbps[0]| 8382 * |.___/__. V V | 8383 * ||L2 dev|....|lb |bufs |lb |bufs |lb |bufs |lb |bufs |lb |---(empty)---| 8384 * || hdr| ^ /^ /^ / / | 8385 * |+------+ ...--\-------/ \-----/--\------/ / | 8386 * | \--------------/ \--------------/ | 8387 * +======================================================================+ 8388 * 8389 * As can be seen on the diagram, rather than using a simple linked list, 8390 * we use a pair of linked lists with alternating elements. This is a 8391 * performance enhancement due to the fact that we only find out the 8392 * address of the next log block access once the current block has been 8393 * completely read in. Obviously, this hurts performance, because we'd be 8394 * keeping the device's I/O queue at only a 1 operation deep, thus 8395 * incurring a large amount of I/O round-trip latency. Having two lists 8396 * allows us to fetch two log blocks ahead of where we are currently 8397 * rebuilding L2ARC buffers. 8398 * 8399 * On-device data structures: 8400 * 8401 * L2ARC device header: l2arc_dev_hdr_phys_t 8402 * L2ARC log block: l2arc_log_blk_phys_t 8403 * 8404 * L2ARC reconstruction: 8405 * 8406 * When writing data, we simply write in the standard rotary fashion, 8407 * evicting buffers as we go and simply writing new data over them (writing 8408 * a new log block every now and then). This obviously means that once we 8409 * loop around the end of the device, we will start cutting into an already 8410 * committed log block (and its referenced data buffers), like so: 8411 * 8412 * current write head__ __old tail 8413 * \ / 8414 * V V 8415 * <--|bufs |lb |bufs |lb | |bufs |lb |bufs |lb |--> 8416 * ^ ^^^^^^^^^___________________________________ 8417 * | \ 8418 * <<nextwrite>> may overwrite this blk and/or its bufs --' 8419 * 8420 * When importing the pool, we detect this situation and use it to stop 8421 * our scanning process (see l2arc_rebuild). 8422 * 8423 * There is one significant caveat to consider when rebuilding ARC contents 8424 * from an L2ARC device: what about invalidated buffers? Given the above 8425 * construction, we cannot update blocks which we've already written to amend 8426 * them to remove buffers which were invalidated. Thus, during reconstruction, 8427 * we might be populating the cache with buffers for data that's not on the 8428 * main pool anymore, or may have been overwritten! 8429 * 8430 * As it turns out, this isn't a problem. Every arc_read request includes 8431 * both the DVA and, crucially, the birth TXG of the BP the caller is 8432 * looking for. So even if the cache were populated by completely rotten 8433 * blocks for data that had been long deleted and/or overwritten, we'll 8434 * never actually return bad data from the cache, since the DVA with the 8435 * birth TXG uniquely identify a block in space and time - once created, 8436 * a block is immutable on disk. The worst thing we have done is wasted 8437 * some time and memory at l2arc rebuild to reconstruct outdated ARC 8438 * entries that will get dropped from the l2arc as it is being updated 8439 * with new blocks. 8440 * 8441 * L2ARC buffers that have been evicted by l2arc_evict() ahead of the write 8442 * hand are not restored. This is done by saving the offset (in bytes) 8443 * l2arc_evict() has evicted to in the L2ARC device header and taking it 8444 * into account when restoring buffers. 8445 */ 8446 8447 static boolean_t 8448 l2arc_write_eligible(uint64_t spa_guid, arc_buf_hdr_t *hdr) 8449 { 8450 /* 8451 * A buffer is *not* eligible for the L2ARC if it: 8452 * 1. belongs to a different spa. 8453 * 2. is already cached on the L2ARC. 8454 * 3. has an I/O in progress (it may be an incomplete read). 8455 * 4. is flagged not eligible (zfs property). 8456 */ 8457 if (hdr->b_spa != spa_guid || HDR_HAS_L2HDR(hdr) || 8458 HDR_IO_IN_PROGRESS(hdr) || !HDR_L2CACHE(hdr)) 8459 return (B_FALSE); 8460 8461 return (B_TRUE); 8462 } 8463 8464 static uint64_t 8465 l2arc_write_size(l2arc_dev_t *dev) 8466 { 8467 uint64_t size, dev_size, tsize; 8468 8469 /* 8470 * Make sure our globals have meaningful values in case the user 8471 * altered them. 8472 */ 8473 size = l2arc_write_max; 8474 if (size == 0) { 8475 cmn_err(CE_NOTE, "Bad value for l2arc_write_max, value must " 8476 "be greater than zero, resetting it to the default (%d)", 8477 L2ARC_WRITE_SIZE); 8478 size = l2arc_write_max = L2ARC_WRITE_SIZE; 8479 } 8480 8481 if (arc_warm == B_FALSE) 8482 size += l2arc_write_boost; 8483 8484 /* 8485 * Make sure the write size does not exceed the size of the cache 8486 * device. This is important in l2arc_evict(), otherwise infinite 8487 * iteration can occur. 8488 */ 8489 dev_size = dev->l2ad_end - dev->l2ad_start; 8490 tsize = size + l2arc_log_blk_overhead(size, dev); 8491 if (dev->l2ad_vdev->vdev_has_trim && l2arc_trim_ahead > 0) 8492 tsize += MAX(64 * 1024 * 1024, 8493 (tsize * l2arc_trim_ahead) / 100); 8494 8495 if (tsize >= dev_size) { 8496 cmn_err(CE_NOTE, "l2arc_write_max or l2arc_write_boost " 8497 "plus the overhead of log blocks (persistent L2ARC, " 8498 "%llu bytes) exceeds the size of the cache device " 8499 "(guid %llu), resetting them to the default (%d)", 8500 (u_longlong_t)l2arc_log_blk_overhead(size, dev), 8501 (u_longlong_t)dev->l2ad_vdev->vdev_guid, L2ARC_WRITE_SIZE); 8502 size = l2arc_write_max = l2arc_write_boost = L2ARC_WRITE_SIZE; 8503 8504 if (arc_warm == B_FALSE) 8505 size += l2arc_write_boost; 8506 } 8507 8508 return (size); 8509 8510 } 8511 8512 static clock_t 8513 l2arc_write_interval(clock_t began, uint64_t wanted, uint64_t wrote) 8514 { 8515 clock_t interval, next, now; 8516 8517 /* 8518 * If the ARC lists are busy, increase our write rate; if the 8519 * lists are stale, idle back. This is achieved by checking 8520 * how much we previously wrote - if it was more than half of 8521 * what we wanted, schedule the next write much sooner. 8522 */ 8523 if (l2arc_feed_again && wrote > (wanted / 2)) 8524 interval = (hz * l2arc_feed_min_ms) / 1000; 8525 else 8526 interval = hz * l2arc_feed_secs; 8527 8528 now = ddi_get_lbolt(); 8529 next = MAX(now, MIN(now + interval, began + interval)); 8530 8531 return (next); 8532 } 8533 8534 /* 8535 * Cycle through L2ARC devices. This is how L2ARC load balances. 8536 * If a device is returned, this also returns holding the spa config lock. 8537 */ 8538 static l2arc_dev_t * 8539 l2arc_dev_get_next(void) 8540 { 8541 l2arc_dev_t *first, *next = NULL; 8542 8543 /* 8544 * Lock out the removal of spas (spa_namespace_lock), then removal 8545 * of cache devices (l2arc_dev_mtx). Once a device has been selected, 8546 * both locks will be dropped and a spa config lock held instead. 8547 */ 8548 mutex_enter(&spa_namespace_lock); 8549 mutex_enter(&l2arc_dev_mtx); 8550 8551 /* if there are no vdevs, there is nothing to do */ 8552 if (l2arc_ndev == 0) 8553 goto out; 8554 8555 first = NULL; 8556 next = l2arc_dev_last; 8557 do { 8558 /* loop around the list looking for a non-faulted vdev */ 8559 if (next == NULL) { 8560 next = list_head(l2arc_dev_list); 8561 } else { 8562 next = list_next(l2arc_dev_list, next); 8563 if (next == NULL) 8564 next = list_head(l2arc_dev_list); 8565 } 8566 8567 /* if we have come back to the start, bail out */ 8568 if (first == NULL) 8569 first = next; 8570 else if (next == first) 8571 break; 8572 8573 ASSERT3P(next, !=, NULL); 8574 } while (vdev_is_dead(next->l2ad_vdev) || next->l2ad_rebuild || 8575 next->l2ad_trim_all); 8576 8577 /* if we were unable to find any usable vdevs, return NULL */ 8578 if (vdev_is_dead(next->l2ad_vdev) || next->l2ad_rebuild || 8579 next->l2ad_trim_all) 8580 next = NULL; 8581 8582 l2arc_dev_last = next; 8583 8584 out: 8585 mutex_exit(&l2arc_dev_mtx); 8586 8587 /* 8588 * Grab the config lock to prevent the 'next' device from being 8589 * removed while we are writing to it. 8590 */ 8591 if (next != NULL) 8592 spa_config_enter(next->l2ad_spa, SCL_L2ARC, next, RW_READER); 8593 mutex_exit(&spa_namespace_lock); 8594 8595 return (next); 8596 } 8597 8598 /* 8599 * Free buffers that were tagged for destruction. 8600 */ 8601 static void 8602 l2arc_do_free_on_write(void) 8603 { 8604 list_t *buflist; 8605 l2arc_data_free_t *df, *df_prev; 8606 8607 mutex_enter(&l2arc_free_on_write_mtx); 8608 buflist = l2arc_free_on_write; 8609 8610 for (df = list_tail(buflist); df; df = df_prev) { 8611 df_prev = list_prev(buflist, df); 8612 ASSERT3P(df->l2df_abd, !=, NULL); 8613 abd_free(df->l2df_abd); 8614 list_remove(buflist, df); 8615 kmem_free(df, sizeof (l2arc_data_free_t)); 8616 } 8617 8618 mutex_exit(&l2arc_free_on_write_mtx); 8619 } 8620 8621 /* 8622 * A write to a cache device has completed. Update all headers to allow 8623 * reads from these buffers to begin. 8624 */ 8625 static void 8626 l2arc_write_done(zio_t *zio) 8627 { 8628 l2arc_write_callback_t *cb; 8629 l2arc_lb_abd_buf_t *abd_buf; 8630 l2arc_lb_ptr_buf_t *lb_ptr_buf; 8631 l2arc_dev_t *dev; 8632 l2arc_dev_hdr_phys_t *l2dhdr; 8633 list_t *buflist; 8634 arc_buf_hdr_t *head, *hdr, *hdr_prev; 8635 kmutex_t *hash_lock; 8636 int64_t bytes_dropped = 0; 8637 8638 cb = zio->io_private; 8639 ASSERT3P(cb, !=, NULL); 8640 dev = cb->l2wcb_dev; 8641 l2dhdr = dev->l2ad_dev_hdr; 8642 ASSERT3P(dev, !=, NULL); 8643 head = cb->l2wcb_head; 8644 ASSERT3P(head, !=, NULL); 8645 buflist = &dev->l2ad_buflist; 8646 ASSERT3P(buflist, !=, NULL); 8647 DTRACE_PROBE2(l2arc__iodone, zio_t *, zio, 8648 l2arc_write_callback_t *, cb); 8649 8650 /* 8651 * All writes completed, or an error was hit. 8652 */ 8653 top: 8654 mutex_enter(&dev->l2ad_mtx); 8655 for (hdr = list_prev(buflist, head); hdr; hdr = hdr_prev) { 8656 hdr_prev = list_prev(buflist, hdr); 8657 8658 hash_lock = HDR_LOCK(hdr); 8659 8660 /* 8661 * We cannot use mutex_enter or else we can deadlock 8662 * with l2arc_write_buffers (due to swapping the order 8663 * the hash lock and l2ad_mtx are taken). 8664 */ 8665 if (!mutex_tryenter(hash_lock)) { 8666 /* 8667 * Missed the hash lock. We must retry so we 8668 * don't leave the ARC_FLAG_L2_WRITING bit set. 8669 */ 8670 ARCSTAT_BUMP(arcstat_l2_writes_lock_retry); 8671 8672 /* 8673 * We don't want to rescan the headers we've 8674 * already marked as having been written out, so 8675 * we reinsert the head node so we can pick up 8676 * where we left off. 8677 */ 8678 list_remove(buflist, head); 8679 list_insert_after(buflist, hdr, head); 8680 8681 mutex_exit(&dev->l2ad_mtx); 8682 8683 /* 8684 * We wait for the hash lock to become available 8685 * to try and prevent busy waiting, and increase 8686 * the chance we'll be able to acquire the lock 8687 * the next time around. 8688 */ 8689 mutex_enter(hash_lock); 8690 mutex_exit(hash_lock); 8691 goto top; 8692 } 8693 8694 /* 8695 * We could not have been moved into the arc_l2c_only 8696 * state while in-flight due to our ARC_FLAG_L2_WRITING 8697 * bit being set. Let's just ensure that's being enforced. 8698 */ 8699 ASSERT(HDR_HAS_L1HDR(hdr)); 8700 8701 /* 8702 * Skipped - drop L2ARC entry and mark the header as no 8703 * longer L2 eligibile. 8704 */ 8705 if (zio->io_error != 0) { 8706 /* 8707 * Error - drop L2ARC entry. 8708 */ 8709 list_remove(buflist, hdr); 8710 arc_hdr_clear_flags(hdr, ARC_FLAG_HAS_L2HDR); 8711 8712 uint64_t psize = HDR_GET_PSIZE(hdr); 8713 l2arc_hdr_arcstats_decrement(hdr); 8714 8715 bytes_dropped += 8716 vdev_psize_to_asize(dev->l2ad_vdev, psize); 8717 (void) zfs_refcount_remove_many(&dev->l2ad_alloc, 8718 arc_hdr_size(hdr), hdr); 8719 } 8720 8721 /* 8722 * Allow ARC to begin reads and ghost list evictions to 8723 * this L2ARC entry. 8724 */ 8725 arc_hdr_clear_flags(hdr, ARC_FLAG_L2_WRITING); 8726 8727 mutex_exit(hash_lock); 8728 } 8729 8730 /* 8731 * Free the allocated abd buffers for writing the log blocks. 8732 * If the zio failed reclaim the allocated space and remove the 8733 * pointers to these log blocks from the log block pointer list 8734 * of the L2ARC device. 8735 */ 8736 while ((abd_buf = list_remove_tail(&cb->l2wcb_abd_list)) != NULL) { 8737 abd_free(abd_buf->abd); 8738 zio_buf_free(abd_buf, sizeof (*abd_buf)); 8739 if (zio->io_error != 0) { 8740 lb_ptr_buf = list_remove_head(&dev->l2ad_lbptr_list); 8741 /* 8742 * L2BLK_GET_PSIZE returns aligned size for log 8743 * blocks. 8744 */ 8745 uint64_t asize = 8746 L2BLK_GET_PSIZE((lb_ptr_buf->lb_ptr)->lbp_prop); 8747 bytes_dropped += asize; 8748 ARCSTAT_INCR(arcstat_l2_log_blk_asize, -asize); 8749 ARCSTAT_BUMPDOWN(arcstat_l2_log_blk_count); 8750 zfs_refcount_remove_many(&dev->l2ad_lb_asize, asize, 8751 lb_ptr_buf); 8752 zfs_refcount_remove(&dev->l2ad_lb_count, lb_ptr_buf); 8753 kmem_free(lb_ptr_buf->lb_ptr, 8754 sizeof (l2arc_log_blkptr_t)); 8755 kmem_free(lb_ptr_buf, sizeof (l2arc_lb_ptr_buf_t)); 8756 } 8757 } 8758 list_destroy(&cb->l2wcb_abd_list); 8759 8760 if (zio->io_error != 0) { 8761 ARCSTAT_BUMP(arcstat_l2_writes_error); 8762 8763 /* 8764 * Restore the lbps array in the header to its previous state. 8765 * If the list of log block pointers is empty, zero out the 8766 * log block pointers in the device header. 8767 */ 8768 lb_ptr_buf = list_head(&dev->l2ad_lbptr_list); 8769 for (int i = 0; i < 2; i++) { 8770 if (lb_ptr_buf == NULL) { 8771 /* 8772 * If the list is empty zero out the device 8773 * header. Otherwise zero out the second log 8774 * block pointer in the header. 8775 */ 8776 if (i == 0) { 8777 memset(l2dhdr, 0, 8778 dev->l2ad_dev_hdr_asize); 8779 } else { 8780 memset(&l2dhdr->dh_start_lbps[i], 0, 8781 sizeof (l2arc_log_blkptr_t)); 8782 } 8783 break; 8784 } 8785 memcpy(&l2dhdr->dh_start_lbps[i], lb_ptr_buf->lb_ptr, 8786 sizeof (l2arc_log_blkptr_t)); 8787 lb_ptr_buf = list_next(&dev->l2ad_lbptr_list, 8788 lb_ptr_buf); 8789 } 8790 } 8791 8792 ARCSTAT_BUMP(arcstat_l2_writes_done); 8793 list_remove(buflist, head); 8794 ASSERT(!HDR_HAS_L1HDR(head)); 8795 kmem_cache_free(hdr_l2only_cache, head); 8796 mutex_exit(&dev->l2ad_mtx); 8797 8798 ASSERT(dev->l2ad_vdev != NULL); 8799 vdev_space_update(dev->l2ad_vdev, -bytes_dropped, 0, 0); 8800 8801 l2arc_do_free_on_write(); 8802 8803 kmem_free(cb, sizeof (l2arc_write_callback_t)); 8804 } 8805 8806 static int 8807 l2arc_untransform(zio_t *zio, l2arc_read_callback_t *cb) 8808 { 8809 int ret; 8810 spa_t *spa = zio->io_spa; 8811 arc_buf_hdr_t *hdr = cb->l2rcb_hdr; 8812 blkptr_t *bp = zio->io_bp; 8813 uint8_t salt[ZIO_DATA_SALT_LEN]; 8814 uint8_t iv[ZIO_DATA_IV_LEN]; 8815 uint8_t mac[ZIO_DATA_MAC_LEN]; 8816 boolean_t no_crypt = B_FALSE; 8817 8818 /* 8819 * ZIL data is never be written to the L2ARC, so we don't need 8820 * special handling for its unique MAC storage. 8821 */ 8822 ASSERT3U(BP_GET_TYPE(bp), !=, DMU_OT_INTENT_LOG); 8823 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 8824 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 8825 8826 /* 8827 * If the data was encrypted, decrypt it now. Note that 8828 * we must check the bp here and not the hdr, since the 8829 * hdr does not have its encryption parameters updated 8830 * until arc_read_done(). 8831 */ 8832 if (BP_IS_ENCRYPTED(bp)) { 8833 abd_t *eabd = arc_get_data_abd(hdr, arc_hdr_size(hdr), hdr, 8834 ARC_HDR_DO_ADAPT | ARC_HDR_USE_RESERVE); 8835 8836 zio_crypt_decode_params_bp(bp, salt, iv); 8837 zio_crypt_decode_mac_bp(bp, mac); 8838 8839 ret = spa_do_crypt_abd(B_FALSE, spa, &cb->l2rcb_zb, 8840 BP_GET_TYPE(bp), BP_GET_DEDUP(bp), BP_SHOULD_BYTESWAP(bp), 8841 salt, iv, mac, HDR_GET_PSIZE(hdr), eabd, 8842 hdr->b_l1hdr.b_pabd, &no_crypt); 8843 if (ret != 0) { 8844 arc_free_data_abd(hdr, eabd, arc_hdr_size(hdr), hdr); 8845 goto error; 8846 } 8847 8848 /* 8849 * If we actually performed decryption, replace b_pabd 8850 * with the decrypted data. Otherwise we can just throw 8851 * our decryption buffer away. 8852 */ 8853 if (!no_crypt) { 8854 arc_free_data_abd(hdr, hdr->b_l1hdr.b_pabd, 8855 arc_hdr_size(hdr), hdr); 8856 hdr->b_l1hdr.b_pabd = eabd; 8857 zio->io_abd = eabd; 8858 } else { 8859 arc_free_data_abd(hdr, eabd, arc_hdr_size(hdr), hdr); 8860 } 8861 } 8862 8863 /* 8864 * If the L2ARC block was compressed, but ARC compression 8865 * is disabled we decompress the data into a new buffer and 8866 * replace the existing data. 8867 */ 8868 if (HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 8869 !HDR_COMPRESSION_ENABLED(hdr)) { 8870 abd_t *cabd = arc_get_data_abd(hdr, arc_hdr_size(hdr), hdr, 8871 ARC_HDR_DO_ADAPT | ARC_HDR_USE_RESERVE); 8872 void *tmp = abd_borrow_buf(cabd, arc_hdr_size(hdr)); 8873 8874 ret = zio_decompress_data(HDR_GET_COMPRESS(hdr), 8875 hdr->b_l1hdr.b_pabd, tmp, HDR_GET_PSIZE(hdr), 8876 HDR_GET_LSIZE(hdr), &hdr->b_complevel); 8877 if (ret != 0) { 8878 abd_return_buf_copy(cabd, tmp, arc_hdr_size(hdr)); 8879 arc_free_data_abd(hdr, cabd, arc_hdr_size(hdr), hdr); 8880 goto error; 8881 } 8882 8883 abd_return_buf_copy(cabd, tmp, arc_hdr_size(hdr)); 8884 arc_free_data_abd(hdr, hdr->b_l1hdr.b_pabd, 8885 arc_hdr_size(hdr), hdr); 8886 hdr->b_l1hdr.b_pabd = cabd; 8887 zio->io_abd = cabd; 8888 zio->io_size = HDR_GET_LSIZE(hdr); 8889 } 8890 8891 return (0); 8892 8893 error: 8894 return (ret); 8895 } 8896 8897 8898 /* 8899 * A read to a cache device completed. Validate buffer contents before 8900 * handing over to the regular ARC routines. 8901 */ 8902 static void 8903 l2arc_read_done(zio_t *zio) 8904 { 8905 int tfm_error = 0; 8906 l2arc_read_callback_t *cb = zio->io_private; 8907 arc_buf_hdr_t *hdr; 8908 kmutex_t *hash_lock; 8909 boolean_t valid_cksum; 8910 boolean_t using_rdata = (BP_IS_ENCRYPTED(&cb->l2rcb_bp) && 8911 (cb->l2rcb_flags & ZIO_FLAG_RAW_ENCRYPT)); 8912 8913 ASSERT3P(zio->io_vd, !=, NULL); 8914 ASSERT(zio->io_flags & ZIO_FLAG_DONT_PROPAGATE); 8915 8916 spa_config_exit(zio->io_spa, SCL_L2ARC, zio->io_vd); 8917 8918 ASSERT3P(cb, !=, NULL); 8919 hdr = cb->l2rcb_hdr; 8920 ASSERT3P(hdr, !=, NULL); 8921 8922 hash_lock = HDR_LOCK(hdr); 8923 mutex_enter(hash_lock); 8924 ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); 8925 8926 /* 8927 * If the data was read into a temporary buffer, 8928 * move it and free the buffer. 8929 */ 8930 if (cb->l2rcb_abd != NULL) { 8931 ASSERT3U(arc_hdr_size(hdr), <, zio->io_size); 8932 if (zio->io_error == 0) { 8933 if (using_rdata) { 8934 abd_copy(hdr->b_crypt_hdr.b_rabd, 8935 cb->l2rcb_abd, arc_hdr_size(hdr)); 8936 } else { 8937 abd_copy(hdr->b_l1hdr.b_pabd, 8938 cb->l2rcb_abd, arc_hdr_size(hdr)); 8939 } 8940 } 8941 8942 /* 8943 * The following must be done regardless of whether 8944 * there was an error: 8945 * - free the temporary buffer 8946 * - point zio to the real ARC buffer 8947 * - set zio size accordingly 8948 * These are required because zio is either re-used for 8949 * an I/O of the block in the case of the error 8950 * or the zio is passed to arc_read_done() and it 8951 * needs real data. 8952 */ 8953 abd_free(cb->l2rcb_abd); 8954 zio->io_size = zio->io_orig_size = arc_hdr_size(hdr); 8955 8956 if (using_rdata) { 8957 ASSERT(HDR_HAS_RABD(hdr)); 8958 zio->io_abd = zio->io_orig_abd = 8959 hdr->b_crypt_hdr.b_rabd; 8960 } else { 8961 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 8962 zio->io_abd = zio->io_orig_abd = hdr->b_l1hdr.b_pabd; 8963 } 8964 } 8965 8966 ASSERT3P(zio->io_abd, !=, NULL); 8967 8968 /* 8969 * Check this survived the L2ARC journey. 8970 */ 8971 ASSERT(zio->io_abd == hdr->b_l1hdr.b_pabd || 8972 (HDR_HAS_RABD(hdr) && zio->io_abd == hdr->b_crypt_hdr.b_rabd)); 8973 zio->io_bp_copy = cb->l2rcb_bp; /* XXX fix in L2ARC 2.0 */ 8974 zio->io_bp = &zio->io_bp_copy; /* XXX fix in L2ARC 2.0 */ 8975 zio->io_prop.zp_complevel = hdr->b_complevel; 8976 8977 valid_cksum = arc_cksum_is_equal(hdr, zio); 8978 8979 /* 8980 * b_rabd will always match the data as it exists on disk if it is 8981 * being used. Therefore if we are reading into b_rabd we do not 8982 * attempt to untransform the data. 8983 */ 8984 if (valid_cksum && !using_rdata) 8985 tfm_error = l2arc_untransform(zio, cb); 8986 8987 if (valid_cksum && tfm_error == 0 && zio->io_error == 0 && 8988 !HDR_L2_EVICTED(hdr)) { 8989 mutex_exit(hash_lock); 8990 zio->io_private = hdr; 8991 arc_read_done(zio); 8992 } else { 8993 /* 8994 * Buffer didn't survive caching. Increment stats and 8995 * reissue to the original storage device. 8996 */ 8997 if (zio->io_error != 0) { 8998 ARCSTAT_BUMP(arcstat_l2_io_error); 8999 } else { 9000 zio->io_error = SET_ERROR(EIO); 9001 } 9002 if (!valid_cksum || tfm_error != 0) 9003 ARCSTAT_BUMP(arcstat_l2_cksum_bad); 9004 9005 /* 9006 * If there's no waiter, issue an async i/o to the primary 9007 * storage now. If there *is* a waiter, the caller must 9008 * issue the i/o in a context where it's OK to block. 9009 */ 9010 if (zio->io_waiter == NULL) { 9011 zio_t *pio = zio_unique_parent(zio); 9012 void *abd = (using_rdata) ? 9013 hdr->b_crypt_hdr.b_rabd : hdr->b_l1hdr.b_pabd; 9014 9015 ASSERT(!pio || pio->io_child_type == ZIO_CHILD_LOGICAL); 9016 9017 zio = zio_read(pio, zio->io_spa, zio->io_bp, 9018 abd, zio->io_size, arc_read_done, 9019 hdr, zio->io_priority, cb->l2rcb_flags, 9020 &cb->l2rcb_zb); 9021 9022 /* 9023 * Original ZIO will be freed, so we need to update 9024 * ARC header with the new ZIO pointer to be used 9025 * by zio_change_priority() in arc_read(). 9026 */ 9027 for (struct arc_callback *acb = hdr->b_l1hdr.b_acb; 9028 acb != NULL; acb = acb->acb_next) 9029 acb->acb_zio_head = zio; 9030 9031 mutex_exit(hash_lock); 9032 zio_nowait(zio); 9033 } else { 9034 mutex_exit(hash_lock); 9035 } 9036 } 9037 9038 kmem_free(cb, sizeof (l2arc_read_callback_t)); 9039 } 9040 9041 /* 9042 * This is the list priority from which the L2ARC will search for pages to 9043 * cache. This is used within loops (0..3) to cycle through lists in the 9044 * desired order. This order can have a significant effect on cache 9045 * performance. 9046 * 9047 * Currently the metadata lists are hit first, MFU then MRU, followed by 9048 * the data lists. This function returns a locked list, and also returns 9049 * the lock pointer. 9050 */ 9051 static multilist_sublist_t * 9052 l2arc_sublist_lock(int list_num) 9053 { 9054 multilist_t *ml = NULL; 9055 unsigned int idx; 9056 9057 ASSERT(list_num >= 0 && list_num < L2ARC_FEED_TYPES); 9058 9059 switch (list_num) { 9060 case 0: 9061 ml = &arc_mfu->arcs_list[ARC_BUFC_METADATA]; 9062 break; 9063 case 1: 9064 ml = &arc_mru->arcs_list[ARC_BUFC_METADATA]; 9065 break; 9066 case 2: 9067 ml = &arc_mfu->arcs_list[ARC_BUFC_DATA]; 9068 break; 9069 case 3: 9070 ml = &arc_mru->arcs_list[ARC_BUFC_DATA]; 9071 break; 9072 default: 9073 return (NULL); 9074 } 9075 9076 /* 9077 * Return a randomly-selected sublist. This is acceptable 9078 * because the caller feeds only a little bit of data for each 9079 * call (8MB). Subsequent calls will result in different 9080 * sublists being selected. 9081 */ 9082 idx = multilist_get_random_index(ml); 9083 return (multilist_sublist_lock(ml, idx)); 9084 } 9085 9086 /* 9087 * Calculates the maximum overhead of L2ARC metadata log blocks for a given 9088 * L2ARC write size. l2arc_evict and l2arc_write_size need to include this 9089 * overhead in processing to make sure there is enough headroom available 9090 * when writing buffers. 9091 */ 9092 static inline uint64_t 9093 l2arc_log_blk_overhead(uint64_t write_sz, l2arc_dev_t *dev) 9094 { 9095 if (dev->l2ad_log_entries == 0) { 9096 return (0); 9097 } else { 9098 uint64_t log_entries = write_sz >> SPA_MINBLOCKSHIFT; 9099 9100 uint64_t log_blocks = (log_entries + 9101 dev->l2ad_log_entries - 1) / 9102 dev->l2ad_log_entries; 9103 9104 return (vdev_psize_to_asize(dev->l2ad_vdev, 9105 sizeof (l2arc_log_blk_phys_t)) * log_blocks); 9106 } 9107 } 9108 9109 /* 9110 * Evict buffers from the device write hand to the distance specified in 9111 * bytes. This distance may span populated buffers, it may span nothing. 9112 * This is clearing a region on the L2ARC device ready for writing. 9113 * If the 'all' boolean is set, every buffer is evicted. 9114 */ 9115 static void 9116 l2arc_evict(l2arc_dev_t *dev, uint64_t distance, boolean_t all) 9117 { 9118 list_t *buflist; 9119 arc_buf_hdr_t *hdr, *hdr_prev; 9120 kmutex_t *hash_lock; 9121 uint64_t taddr; 9122 l2arc_lb_ptr_buf_t *lb_ptr_buf, *lb_ptr_buf_prev; 9123 vdev_t *vd = dev->l2ad_vdev; 9124 boolean_t rerun; 9125 9126 buflist = &dev->l2ad_buflist; 9127 9128 /* 9129 * We need to add in the worst case scenario of log block overhead. 9130 */ 9131 distance += l2arc_log_blk_overhead(distance, dev); 9132 if (vd->vdev_has_trim && l2arc_trim_ahead > 0) { 9133 /* 9134 * Trim ahead of the write size 64MB or (l2arc_trim_ahead/100) 9135 * times the write size, whichever is greater. 9136 */ 9137 distance += MAX(64 * 1024 * 1024, 9138 (distance * l2arc_trim_ahead) / 100); 9139 } 9140 9141 top: 9142 rerun = B_FALSE; 9143 if (dev->l2ad_hand >= (dev->l2ad_end - distance)) { 9144 /* 9145 * When there is no space to accommodate upcoming writes, 9146 * evict to the end. Then bump the write and evict hands 9147 * to the start and iterate. This iteration does not 9148 * happen indefinitely as we make sure in 9149 * l2arc_write_size() that when the write hand is reset, 9150 * the write size does not exceed the end of the device. 9151 */ 9152 rerun = B_TRUE; 9153 taddr = dev->l2ad_end; 9154 } else { 9155 taddr = dev->l2ad_hand + distance; 9156 } 9157 DTRACE_PROBE4(l2arc__evict, l2arc_dev_t *, dev, list_t *, buflist, 9158 uint64_t, taddr, boolean_t, all); 9159 9160 if (!all) { 9161 /* 9162 * This check has to be placed after deciding whether to 9163 * iterate (rerun). 9164 */ 9165 if (dev->l2ad_first) { 9166 /* 9167 * This is the first sweep through the device. There is 9168 * nothing to evict. We have already trimmmed the 9169 * whole device. 9170 */ 9171 goto out; 9172 } else { 9173 /* 9174 * Trim the space to be evicted. 9175 */ 9176 if (vd->vdev_has_trim && dev->l2ad_evict < taddr && 9177 l2arc_trim_ahead > 0) { 9178 /* 9179 * We have to drop the spa_config lock because 9180 * vdev_trim_range() will acquire it. 9181 * l2ad_evict already accounts for the label 9182 * size. To prevent vdev_trim_ranges() from 9183 * adding it again, we subtract it from 9184 * l2ad_evict. 9185 */ 9186 spa_config_exit(dev->l2ad_spa, SCL_L2ARC, dev); 9187 vdev_trim_simple(vd, 9188 dev->l2ad_evict - VDEV_LABEL_START_SIZE, 9189 taddr - dev->l2ad_evict); 9190 spa_config_enter(dev->l2ad_spa, SCL_L2ARC, dev, 9191 RW_READER); 9192 } 9193 9194 /* 9195 * When rebuilding L2ARC we retrieve the evict hand 9196 * from the header of the device. Of note, l2arc_evict() 9197 * does not actually delete buffers from the cache 9198 * device, but trimming may do so depending on the 9199 * hardware implementation. Thus keeping track of the 9200 * evict hand is useful. 9201 */ 9202 dev->l2ad_evict = MAX(dev->l2ad_evict, taddr); 9203 } 9204 } 9205 9206 retry: 9207 mutex_enter(&dev->l2ad_mtx); 9208 /* 9209 * We have to account for evicted log blocks. Run vdev_space_update() 9210 * on log blocks whose offset (in bytes) is before the evicted offset 9211 * (in bytes) by searching in the list of pointers to log blocks 9212 * present in the L2ARC device. 9213 */ 9214 for (lb_ptr_buf = list_tail(&dev->l2ad_lbptr_list); lb_ptr_buf; 9215 lb_ptr_buf = lb_ptr_buf_prev) { 9216 9217 lb_ptr_buf_prev = list_prev(&dev->l2ad_lbptr_list, lb_ptr_buf); 9218 9219 /* L2BLK_GET_PSIZE returns aligned size for log blocks */ 9220 uint64_t asize = L2BLK_GET_PSIZE( 9221 (lb_ptr_buf->lb_ptr)->lbp_prop); 9222 9223 /* 9224 * We don't worry about log blocks left behind (ie 9225 * lbp_payload_start < l2ad_hand) because l2arc_write_buffers() 9226 * will never write more than l2arc_evict() evicts. 9227 */ 9228 if (!all && l2arc_log_blkptr_valid(dev, lb_ptr_buf->lb_ptr)) { 9229 break; 9230 } else { 9231 vdev_space_update(vd, -asize, 0, 0); 9232 ARCSTAT_INCR(arcstat_l2_log_blk_asize, -asize); 9233 ARCSTAT_BUMPDOWN(arcstat_l2_log_blk_count); 9234 zfs_refcount_remove_many(&dev->l2ad_lb_asize, asize, 9235 lb_ptr_buf); 9236 zfs_refcount_remove(&dev->l2ad_lb_count, lb_ptr_buf); 9237 list_remove(&dev->l2ad_lbptr_list, lb_ptr_buf); 9238 kmem_free(lb_ptr_buf->lb_ptr, 9239 sizeof (l2arc_log_blkptr_t)); 9240 kmem_free(lb_ptr_buf, sizeof (l2arc_lb_ptr_buf_t)); 9241 } 9242 } 9243 9244 for (hdr = list_tail(buflist); hdr; hdr = hdr_prev) { 9245 hdr_prev = list_prev(buflist, hdr); 9246 9247 ASSERT(!HDR_EMPTY(hdr)); 9248 hash_lock = HDR_LOCK(hdr); 9249 9250 /* 9251 * We cannot use mutex_enter or else we can deadlock 9252 * with l2arc_write_buffers (due to swapping the order 9253 * the hash lock and l2ad_mtx are taken). 9254 */ 9255 if (!mutex_tryenter(hash_lock)) { 9256 /* 9257 * Missed the hash lock. Retry. 9258 */ 9259 ARCSTAT_BUMP(arcstat_l2_evict_lock_retry); 9260 mutex_exit(&dev->l2ad_mtx); 9261 mutex_enter(hash_lock); 9262 mutex_exit(hash_lock); 9263 goto retry; 9264 } 9265 9266 /* 9267 * A header can't be on this list if it doesn't have L2 header. 9268 */ 9269 ASSERT(HDR_HAS_L2HDR(hdr)); 9270 9271 /* Ensure this header has finished being written. */ 9272 ASSERT(!HDR_L2_WRITING(hdr)); 9273 ASSERT(!HDR_L2_WRITE_HEAD(hdr)); 9274 9275 if (!all && (hdr->b_l2hdr.b_daddr >= dev->l2ad_evict || 9276 hdr->b_l2hdr.b_daddr < dev->l2ad_hand)) { 9277 /* 9278 * We've evicted to the target address, 9279 * or the end of the device. 9280 */ 9281 mutex_exit(hash_lock); 9282 break; 9283 } 9284 9285 if (!HDR_HAS_L1HDR(hdr)) { 9286 ASSERT(!HDR_L2_READING(hdr)); 9287 /* 9288 * This doesn't exist in the ARC. Destroy. 9289 * arc_hdr_destroy() will call list_remove() 9290 * and decrement arcstat_l2_lsize. 9291 */ 9292 arc_change_state(arc_anon, hdr); 9293 arc_hdr_destroy(hdr); 9294 } else { 9295 ASSERT(hdr->b_l1hdr.b_state != arc_l2c_only); 9296 ARCSTAT_BUMP(arcstat_l2_evict_l1cached); 9297 /* 9298 * Invalidate issued or about to be issued 9299 * reads, since we may be about to write 9300 * over this location. 9301 */ 9302 if (HDR_L2_READING(hdr)) { 9303 ARCSTAT_BUMP(arcstat_l2_evict_reading); 9304 arc_hdr_set_flags(hdr, ARC_FLAG_L2_EVICTED); 9305 } 9306 9307 arc_hdr_l2hdr_destroy(hdr); 9308 } 9309 mutex_exit(hash_lock); 9310 } 9311 mutex_exit(&dev->l2ad_mtx); 9312 9313 out: 9314 /* 9315 * We need to check if we evict all buffers, otherwise we may iterate 9316 * unnecessarily. 9317 */ 9318 if (!all && rerun) { 9319 /* 9320 * Bump device hand to the device start if it is approaching the 9321 * end. l2arc_evict() has already evicted ahead for this case. 9322 */ 9323 dev->l2ad_hand = dev->l2ad_start; 9324 dev->l2ad_evict = dev->l2ad_start; 9325 dev->l2ad_first = B_FALSE; 9326 goto top; 9327 } 9328 9329 if (!all) { 9330 /* 9331 * In case of cache device removal (all) the following 9332 * assertions may be violated without functional consequences 9333 * as the device is about to be removed. 9334 */ 9335 ASSERT3U(dev->l2ad_hand + distance, <, dev->l2ad_end); 9336 if (!dev->l2ad_first) 9337 ASSERT3U(dev->l2ad_hand, <, dev->l2ad_evict); 9338 } 9339 } 9340 9341 /* 9342 * Handle any abd transforms that might be required for writing to the L2ARC. 9343 * If successful, this function will always return an abd with the data 9344 * transformed as it is on disk in a new abd of asize bytes. 9345 */ 9346 static int 9347 l2arc_apply_transforms(spa_t *spa, arc_buf_hdr_t *hdr, uint64_t asize, 9348 abd_t **abd_out) 9349 { 9350 int ret; 9351 void *tmp = NULL; 9352 abd_t *cabd = NULL, *eabd = NULL, *to_write = hdr->b_l1hdr.b_pabd; 9353 enum zio_compress compress = HDR_GET_COMPRESS(hdr); 9354 uint64_t psize = HDR_GET_PSIZE(hdr); 9355 uint64_t size = arc_hdr_size(hdr); 9356 boolean_t ismd = HDR_ISTYPE_METADATA(hdr); 9357 boolean_t bswap = (hdr->b_l1hdr.b_byteswap != DMU_BSWAP_NUMFUNCS); 9358 dsl_crypto_key_t *dck = NULL; 9359 uint8_t mac[ZIO_DATA_MAC_LEN] = { 0 }; 9360 boolean_t no_crypt = B_FALSE; 9361 9362 ASSERT((HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 9363 !HDR_COMPRESSION_ENABLED(hdr)) || 9364 HDR_ENCRYPTED(hdr) || HDR_SHARED_DATA(hdr) || psize != asize); 9365 ASSERT3U(psize, <=, asize); 9366 9367 /* 9368 * If this data simply needs its own buffer, we simply allocate it 9369 * and copy the data. This may be done to eliminate a dependency on a 9370 * shared buffer or to reallocate the buffer to match asize. 9371 */ 9372 if (HDR_HAS_RABD(hdr) && asize != psize) { 9373 ASSERT3U(asize, >=, psize); 9374 to_write = abd_alloc_for_io(asize, ismd); 9375 abd_copy(to_write, hdr->b_crypt_hdr.b_rabd, psize); 9376 if (psize != asize) 9377 abd_zero_off(to_write, psize, asize - psize); 9378 goto out; 9379 } 9380 9381 if ((compress == ZIO_COMPRESS_OFF || HDR_COMPRESSION_ENABLED(hdr)) && 9382 !HDR_ENCRYPTED(hdr)) { 9383 ASSERT3U(size, ==, psize); 9384 to_write = abd_alloc_for_io(asize, ismd); 9385 abd_copy(to_write, hdr->b_l1hdr.b_pabd, size); 9386 if (size != asize) 9387 abd_zero_off(to_write, size, asize - size); 9388 goto out; 9389 } 9390 9391 if (compress != ZIO_COMPRESS_OFF && !HDR_COMPRESSION_ENABLED(hdr)) { 9392 /* 9393 * In some cases, we can wind up with size > asize, so 9394 * we need to opt for the larger allocation option here. 9395 * 9396 * (We also need abd_return_buf_copy in all cases because 9397 * it's an ASSERT() to modify the buffer before returning it 9398 * with arc_return_buf(), and all the compressors 9399 * write things before deciding to fail compression in nearly 9400 * every case.) 9401 */ 9402 cabd = abd_alloc_for_io(size, ismd); 9403 tmp = abd_borrow_buf(cabd, size); 9404 9405 psize = zio_compress_data(compress, to_write, tmp, size, 9406 hdr->b_complevel); 9407 9408 if (psize >= asize) { 9409 psize = HDR_GET_PSIZE(hdr); 9410 abd_return_buf_copy(cabd, tmp, size); 9411 HDR_SET_COMPRESS(hdr, ZIO_COMPRESS_OFF); 9412 to_write = cabd; 9413 abd_copy(to_write, hdr->b_l1hdr.b_pabd, psize); 9414 if (psize != asize) 9415 abd_zero_off(to_write, psize, asize - psize); 9416 goto encrypt; 9417 } 9418 ASSERT3U(psize, <=, HDR_GET_PSIZE(hdr)); 9419 if (psize < asize) 9420 memset((char *)tmp + psize, 0, asize - psize); 9421 psize = HDR_GET_PSIZE(hdr); 9422 abd_return_buf_copy(cabd, tmp, size); 9423 to_write = cabd; 9424 } 9425 9426 encrypt: 9427 if (HDR_ENCRYPTED(hdr)) { 9428 eabd = abd_alloc_for_io(asize, ismd); 9429 9430 /* 9431 * If the dataset was disowned before the buffer 9432 * made it to this point, the key to re-encrypt 9433 * it won't be available. In this case we simply 9434 * won't write the buffer to the L2ARC. 9435 */ 9436 ret = spa_keystore_lookup_key(spa, hdr->b_crypt_hdr.b_dsobj, 9437 FTAG, &dck); 9438 if (ret != 0) 9439 goto error; 9440 9441 ret = zio_do_crypt_abd(B_TRUE, &dck->dck_key, 9442 hdr->b_crypt_hdr.b_ot, bswap, hdr->b_crypt_hdr.b_salt, 9443 hdr->b_crypt_hdr.b_iv, mac, psize, to_write, eabd, 9444 &no_crypt); 9445 if (ret != 0) 9446 goto error; 9447 9448 if (no_crypt) 9449 abd_copy(eabd, to_write, psize); 9450 9451 if (psize != asize) 9452 abd_zero_off(eabd, psize, asize - psize); 9453 9454 /* assert that the MAC we got here matches the one we saved */ 9455 ASSERT0(memcmp(mac, hdr->b_crypt_hdr.b_mac, ZIO_DATA_MAC_LEN)); 9456 spa_keystore_dsl_key_rele(spa, dck, FTAG); 9457 9458 if (to_write == cabd) 9459 abd_free(cabd); 9460 9461 to_write = eabd; 9462 } 9463 9464 out: 9465 ASSERT3P(to_write, !=, hdr->b_l1hdr.b_pabd); 9466 *abd_out = to_write; 9467 return (0); 9468 9469 error: 9470 if (dck != NULL) 9471 spa_keystore_dsl_key_rele(spa, dck, FTAG); 9472 if (cabd != NULL) 9473 abd_free(cabd); 9474 if (eabd != NULL) 9475 abd_free(eabd); 9476 9477 *abd_out = NULL; 9478 return (ret); 9479 } 9480 9481 static void 9482 l2arc_blk_fetch_done(zio_t *zio) 9483 { 9484 l2arc_read_callback_t *cb; 9485 9486 cb = zio->io_private; 9487 if (cb->l2rcb_abd != NULL) 9488 abd_free(cb->l2rcb_abd); 9489 kmem_free(cb, sizeof (l2arc_read_callback_t)); 9490 } 9491 9492 /* 9493 * Find and write ARC buffers to the L2ARC device. 9494 * 9495 * An ARC_FLAG_L2_WRITING flag is set so that the L2ARC buffers are not valid 9496 * for reading until they have completed writing. 9497 * The headroom_boost is an in-out parameter used to maintain headroom boost 9498 * state between calls to this function. 9499 * 9500 * Returns the number of bytes actually written (which may be smaller than 9501 * the delta by which the device hand has changed due to alignment and the 9502 * writing of log blocks). 9503 */ 9504 static uint64_t 9505 l2arc_write_buffers(spa_t *spa, l2arc_dev_t *dev, uint64_t target_sz) 9506 { 9507 arc_buf_hdr_t *hdr, *hdr_prev, *head; 9508 uint64_t write_asize, write_psize, write_lsize, headroom; 9509 boolean_t full; 9510 l2arc_write_callback_t *cb = NULL; 9511 zio_t *pio, *wzio; 9512 uint64_t guid = spa_load_guid(spa); 9513 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 9514 9515 ASSERT3P(dev->l2ad_vdev, !=, NULL); 9516 9517 pio = NULL; 9518 write_lsize = write_asize = write_psize = 0; 9519 full = B_FALSE; 9520 head = kmem_cache_alloc(hdr_l2only_cache, KM_PUSHPAGE); 9521 arc_hdr_set_flags(head, ARC_FLAG_L2_WRITE_HEAD | ARC_FLAG_HAS_L2HDR); 9522 9523 /* 9524 * Copy buffers for L2ARC writing. 9525 */ 9526 for (int pass = 0; pass < L2ARC_FEED_TYPES; pass++) { 9527 /* 9528 * If pass == 1 or 3, we cache MRU metadata and data 9529 * respectively. 9530 */ 9531 if (l2arc_mfuonly) { 9532 if (pass == 1 || pass == 3) 9533 continue; 9534 } 9535 9536 multilist_sublist_t *mls = l2arc_sublist_lock(pass); 9537 uint64_t passed_sz = 0; 9538 9539 VERIFY3P(mls, !=, NULL); 9540 9541 /* 9542 * L2ARC fast warmup. 9543 * 9544 * Until the ARC is warm and starts to evict, read from the 9545 * head of the ARC lists rather than the tail. 9546 */ 9547 if (arc_warm == B_FALSE) 9548 hdr = multilist_sublist_head(mls); 9549 else 9550 hdr = multilist_sublist_tail(mls); 9551 9552 headroom = target_sz * l2arc_headroom; 9553 if (zfs_compressed_arc_enabled) 9554 headroom = (headroom * l2arc_headroom_boost) / 100; 9555 9556 for (; hdr; hdr = hdr_prev) { 9557 kmutex_t *hash_lock; 9558 abd_t *to_write = NULL; 9559 9560 if (arc_warm == B_FALSE) 9561 hdr_prev = multilist_sublist_next(mls, hdr); 9562 else 9563 hdr_prev = multilist_sublist_prev(mls, hdr); 9564 9565 hash_lock = HDR_LOCK(hdr); 9566 if (!mutex_tryenter(hash_lock)) { 9567 /* 9568 * Skip this buffer rather than waiting. 9569 */ 9570 continue; 9571 } 9572 9573 passed_sz += HDR_GET_LSIZE(hdr); 9574 if (l2arc_headroom != 0 && passed_sz > headroom) { 9575 /* 9576 * Searched too far. 9577 */ 9578 mutex_exit(hash_lock); 9579 break; 9580 } 9581 9582 if (!l2arc_write_eligible(guid, hdr)) { 9583 mutex_exit(hash_lock); 9584 continue; 9585 } 9586 9587 ASSERT(HDR_HAS_L1HDR(hdr)); 9588 9589 ASSERT3U(HDR_GET_PSIZE(hdr), >, 0); 9590 ASSERT3U(arc_hdr_size(hdr), >, 0); 9591 ASSERT(hdr->b_l1hdr.b_pabd != NULL || 9592 HDR_HAS_RABD(hdr)); 9593 uint64_t psize = HDR_GET_PSIZE(hdr); 9594 uint64_t asize = vdev_psize_to_asize(dev->l2ad_vdev, 9595 psize); 9596 9597 if ((write_asize + asize) > target_sz) { 9598 full = B_TRUE; 9599 mutex_exit(hash_lock); 9600 break; 9601 } 9602 9603 /* 9604 * We rely on the L1 portion of the header below, so 9605 * it's invalid for this header to have been evicted out 9606 * of the ghost cache, prior to being written out. The 9607 * ARC_FLAG_L2_WRITING bit ensures this won't happen. 9608 */ 9609 arc_hdr_set_flags(hdr, ARC_FLAG_L2_WRITING); 9610 9611 /* 9612 * If this header has b_rabd, we can use this since it 9613 * must always match the data exactly as it exists on 9614 * disk. Otherwise, the L2ARC can normally use the 9615 * hdr's data, but if we're sharing data between the 9616 * hdr and one of its bufs, L2ARC needs its own copy of 9617 * the data so that the ZIO below can't race with the 9618 * buf consumer. To ensure that this copy will be 9619 * available for the lifetime of the ZIO and be cleaned 9620 * up afterwards, we add it to the l2arc_free_on_write 9621 * queue. If we need to apply any transforms to the 9622 * data (compression, encryption) we will also need the 9623 * extra buffer. 9624 */ 9625 if (HDR_HAS_RABD(hdr) && psize == asize) { 9626 to_write = hdr->b_crypt_hdr.b_rabd; 9627 } else if ((HDR_COMPRESSION_ENABLED(hdr) || 9628 HDR_GET_COMPRESS(hdr) == ZIO_COMPRESS_OFF) && 9629 !HDR_ENCRYPTED(hdr) && !HDR_SHARED_DATA(hdr) && 9630 psize == asize) { 9631 to_write = hdr->b_l1hdr.b_pabd; 9632 } else { 9633 int ret; 9634 arc_buf_contents_t type = arc_buf_type(hdr); 9635 9636 ret = l2arc_apply_transforms(spa, hdr, asize, 9637 &to_write); 9638 if (ret != 0) { 9639 arc_hdr_clear_flags(hdr, 9640 ARC_FLAG_L2_WRITING); 9641 mutex_exit(hash_lock); 9642 continue; 9643 } 9644 9645 l2arc_free_abd_on_write(to_write, asize, type); 9646 } 9647 9648 if (pio == NULL) { 9649 /* 9650 * Insert a dummy header on the buflist so 9651 * l2arc_write_done() can find where the 9652 * write buffers begin without searching. 9653 */ 9654 mutex_enter(&dev->l2ad_mtx); 9655 list_insert_head(&dev->l2ad_buflist, head); 9656 mutex_exit(&dev->l2ad_mtx); 9657 9658 cb = kmem_alloc( 9659 sizeof (l2arc_write_callback_t), KM_SLEEP); 9660 cb->l2wcb_dev = dev; 9661 cb->l2wcb_head = head; 9662 /* 9663 * Create a list to save allocated abd buffers 9664 * for l2arc_log_blk_commit(). 9665 */ 9666 list_create(&cb->l2wcb_abd_list, 9667 sizeof (l2arc_lb_abd_buf_t), 9668 offsetof(l2arc_lb_abd_buf_t, node)); 9669 pio = zio_root(spa, l2arc_write_done, cb, 9670 ZIO_FLAG_CANFAIL); 9671 } 9672 9673 hdr->b_l2hdr.b_dev = dev; 9674 hdr->b_l2hdr.b_hits = 0; 9675 9676 hdr->b_l2hdr.b_daddr = dev->l2ad_hand; 9677 hdr->b_l2hdr.b_arcs_state = 9678 hdr->b_l1hdr.b_state->arcs_state; 9679 arc_hdr_set_flags(hdr, ARC_FLAG_HAS_L2HDR); 9680 9681 mutex_enter(&dev->l2ad_mtx); 9682 list_insert_head(&dev->l2ad_buflist, hdr); 9683 mutex_exit(&dev->l2ad_mtx); 9684 9685 (void) zfs_refcount_add_many(&dev->l2ad_alloc, 9686 arc_hdr_size(hdr), hdr); 9687 9688 wzio = zio_write_phys(pio, dev->l2ad_vdev, 9689 hdr->b_l2hdr.b_daddr, asize, to_write, 9690 ZIO_CHECKSUM_OFF, NULL, hdr, 9691 ZIO_PRIORITY_ASYNC_WRITE, 9692 ZIO_FLAG_CANFAIL, B_FALSE); 9693 9694 write_lsize += HDR_GET_LSIZE(hdr); 9695 DTRACE_PROBE2(l2arc__write, vdev_t *, dev->l2ad_vdev, 9696 zio_t *, wzio); 9697 9698 write_psize += psize; 9699 write_asize += asize; 9700 dev->l2ad_hand += asize; 9701 l2arc_hdr_arcstats_increment(hdr); 9702 vdev_space_update(dev->l2ad_vdev, asize, 0, 0); 9703 9704 mutex_exit(hash_lock); 9705 9706 /* 9707 * Append buf info to current log and commit if full. 9708 * arcstat_l2_{size,asize} kstats are updated 9709 * internally. 9710 */ 9711 if (l2arc_log_blk_insert(dev, hdr)) 9712 l2arc_log_blk_commit(dev, pio, cb); 9713 9714 zio_nowait(wzio); 9715 } 9716 9717 multilist_sublist_unlock(mls); 9718 9719 if (full == B_TRUE) 9720 break; 9721 } 9722 9723 /* No buffers selected for writing? */ 9724 if (pio == NULL) { 9725 ASSERT0(write_lsize); 9726 ASSERT(!HDR_HAS_L1HDR(head)); 9727 kmem_cache_free(hdr_l2only_cache, head); 9728 9729 /* 9730 * Although we did not write any buffers l2ad_evict may 9731 * have advanced. 9732 */ 9733 if (dev->l2ad_evict != l2dhdr->dh_evict) 9734 l2arc_dev_hdr_update(dev); 9735 9736 return (0); 9737 } 9738 9739 if (!dev->l2ad_first) 9740 ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); 9741 9742 ASSERT3U(write_asize, <=, target_sz); 9743 ARCSTAT_BUMP(arcstat_l2_writes_sent); 9744 ARCSTAT_INCR(arcstat_l2_write_bytes, write_psize); 9745 9746 dev->l2ad_writing = B_TRUE; 9747 (void) zio_wait(pio); 9748 dev->l2ad_writing = B_FALSE; 9749 9750 /* 9751 * Update the device header after the zio completes as 9752 * l2arc_write_done() may have updated the memory holding the log block 9753 * pointers in the device header. 9754 */ 9755 l2arc_dev_hdr_update(dev); 9756 9757 return (write_asize); 9758 } 9759 9760 static boolean_t 9761 l2arc_hdr_limit_reached(void) 9762 { 9763 int64_t s = aggsum_upper_bound(&arc_sums.arcstat_l2_hdr_size); 9764 9765 return (arc_reclaim_needed() || (s > arc_meta_limit * 3 / 4) || 9766 (s > (arc_warm ? arc_c : arc_c_max) * l2arc_meta_percent / 100)); 9767 } 9768 9769 /* 9770 * This thread feeds the L2ARC at regular intervals. This is the beating 9771 * heart of the L2ARC. 9772 */ 9773 static __attribute__((noreturn)) void 9774 l2arc_feed_thread(void *unused) 9775 { 9776 (void) unused; 9777 callb_cpr_t cpr; 9778 l2arc_dev_t *dev; 9779 spa_t *spa; 9780 uint64_t size, wrote; 9781 clock_t begin, next = ddi_get_lbolt(); 9782 fstrans_cookie_t cookie; 9783 9784 CALLB_CPR_INIT(&cpr, &l2arc_feed_thr_lock, callb_generic_cpr, FTAG); 9785 9786 mutex_enter(&l2arc_feed_thr_lock); 9787 9788 cookie = spl_fstrans_mark(); 9789 while (l2arc_thread_exit == 0) { 9790 CALLB_CPR_SAFE_BEGIN(&cpr); 9791 (void) cv_timedwait_idle(&l2arc_feed_thr_cv, 9792 &l2arc_feed_thr_lock, next); 9793 CALLB_CPR_SAFE_END(&cpr, &l2arc_feed_thr_lock); 9794 next = ddi_get_lbolt() + hz; 9795 9796 /* 9797 * Quick check for L2ARC devices. 9798 */ 9799 mutex_enter(&l2arc_dev_mtx); 9800 if (l2arc_ndev == 0) { 9801 mutex_exit(&l2arc_dev_mtx); 9802 continue; 9803 } 9804 mutex_exit(&l2arc_dev_mtx); 9805 begin = ddi_get_lbolt(); 9806 9807 /* 9808 * This selects the next l2arc device to write to, and in 9809 * doing so the next spa to feed from: dev->l2ad_spa. This 9810 * will return NULL if there are now no l2arc devices or if 9811 * they are all faulted. 9812 * 9813 * If a device is returned, its spa's config lock is also 9814 * held to prevent device removal. l2arc_dev_get_next() 9815 * will grab and release l2arc_dev_mtx. 9816 */ 9817 if ((dev = l2arc_dev_get_next()) == NULL) 9818 continue; 9819 9820 spa = dev->l2ad_spa; 9821 ASSERT3P(spa, !=, NULL); 9822 9823 /* 9824 * If the pool is read-only then force the feed thread to 9825 * sleep a little longer. 9826 */ 9827 if (!spa_writeable(spa)) { 9828 next = ddi_get_lbolt() + 5 * l2arc_feed_secs * hz; 9829 spa_config_exit(spa, SCL_L2ARC, dev); 9830 continue; 9831 } 9832 9833 /* 9834 * Avoid contributing to memory pressure. 9835 */ 9836 if (l2arc_hdr_limit_reached()) { 9837 ARCSTAT_BUMP(arcstat_l2_abort_lowmem); 9838 spa_config_exit(spa, SCL_L2ARC, dev); 9839 continue; 9840 } 9841 9842 ARCSTAT_BUMP(arcstat_l2_feeds); 9843 9844 size = l2arc_write_size(dev); 9845 9846 /* 9847 * Evict L2ARC buffers that will be overwritten. 9848 */ 9849 l2arc_evict(dev, size, B_FALSE); 9850 9851 /* 9852 * Write ARC buffers. 9853 */ 9854 wrote = l2arc_write_buffers(spa, dev, size); 9855 9856 /* 9857 * Calculate interval between writes. 9858 */ 9859 next = l2arc_write_interval(begin, size, wrote); 9860 spa_config_exit(spa, SCL_L2ARC, dev); 9861 } 9862 spl_fstrans_unmark(cookie); 9863 9864 l2arc_thread_exit = 0; 9865 cv_broadcast(&l2arc_feed_thr_cv); 9866 CALLB_CPR_EXIT(&cpr); /* drops l2arc_feed_thr_lock */ 9867 thread_exit(); 9868 } 9869 9870 boolean_t 9871 l2arc_vdev_present(vdev_t *vd) 9872 { 9873 return (l2arc_vdev_get(vd) != NULL); 9874 } 9875 9876 /* 9877 * Returns the l2arc_dev_t associated with a particular vdev_t or NULL if 9878 * the vdev_t isn't an L2ARC device. 9879 */ 9880 l2arc_dev_t * 9881 l2arc_vdev_get(vdev_t *vd) 9882 { 9883 l2arc_dev_t *dev; 9884 9885 mutex_enter(&l2arc_dev_mtx); 9886 for (dev = list_head(l2arc_dev_list); dev != NULL; 9887 dev = list_next(l2arc_dev_list, dev)) { 9888 if (dev->l2ad_vdev == vd) 9889 break; 9890 } 9891 mutex_exit(&l2arc_dev_mtx); 9892 9893 return (dev); 9894 } 9895 9896 static void 9897 l2arc_rebuild_dev(l2arc_dev_t *dev, boolean_t reopen) 9898 { 9899 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 9900 uint64_t l2dhdr_asize = dev->l2ad_dev_hdr_asize; 9901 spa_t *spa = dev->l2ad_spa; 9902 9903 /* 9904 * The L2ARC has to hold at least the payload of one log block for 9905 * them to be restored (persistent L2ARC). The payload of a log block 9906 * depends on the amount of its log entries. We always write log blocks 9907 * with 1022 entries. How many of them are committed or restored depends 9908 * on the size of the L2ARC device. Thus the maximum payload of 9909 * one log block is 1022 * SPA_MAXBLOCKSIZE = 16GB. If the L2ARC device 9910 * is less than that, we reduce the amount of committed and restored 9911 * log entries per block so as to enable persistence. 9912 */ 9913 if (dev->l2ad_end < l2arc_rebuild_blocks_min_l2size) { 9914 dev->l2ad_log_entries = 0; 9915 } else { 9916 dev->l2ad_log_entries = MIN((dev->l2ad_end - 9917 dev->l2ad_start) >> SPA_MAXBLOCKSHIFT, 9918 L2ARC_LOG_BLK_MAX_ENTRIES); 9919 } 9920 9921 /* 9922 * Read the device header, if an error is returned do not rebuild L2ARC. 9923 */ 9924 if (l2arc_dev_hdr_read(dev) == 0 && dev->l2ad_log_entries > 0) { 9925 /* 9926 * If we are onlining a cache device (vdev_reopen) that was 9927 * still present (l2arc_vdev_present()) and rebuild is enabled, 9928 * we should evict all ARC buffers and pointers to log blocks 9929 * and reclaim their space before restoring its contents to 9930 * L2ARC. 9931 */ 9932 if (reopen) { 9933 if (!l2arc_rebuild_enabled) { 9934 return; 9935 } else { 9936 l2arc_evict(dev, 0, B_TRUE); 9937 /* start a new log block */ 9938 dev->l2ad_log_ent_idx = 0; 9939 dev->l2ad_log_blk_payload_asize = 0; 9940 dev->l2ad_log_blk_payload_start = 0; 9941 } 9942 } 9943 /* 9944 * Just mark the device as pending for a rebuild. We won't 9945 * be starting a rebuild in line here as it would block pool 9946 * import. Instead spa_load_impl will hand that off to an 9947 * async task which will call l2arc_spa_rebuild_start. 9948 */ 9949 dev->l2ad_rebuild = B_TRUE; 9950 } else if (spa_writeable(spa)) { 9951 /* 9952 * In this case TRIM the whole device if l2arc_trim_ahead > 0, 9953 * otherwise create a new header. We zero out the memory holding 9954 * the header to reset dh_start_lbps. If we TRIM the whole 9955 * device the new header will be written by 9956 * vdev_trim_l2arc_thread() at the end of the TRIM to update the 9957 * trim_state in the header too. When reading the header, if 9958 * trim_state is not VDEV_TRIM_COMPLETE and l2arc_trim_ahead > 0 9959 * we opt to TRIM the whole device again. 9960 */ 9961 if (l2arc_trim_ahead > 0) { 9962 dev->l2ad_trim_all = B_TRUE; 9963 } else { 9964 memset(l2dhdr, 0, l2dhdr_asize); 9965 l2arc_dev_hdr_update(dev); 9966 } 9967 } 9968 } 9969 9970 /* 9971 * Add a vdev for use by the L2ARC. By this point the spa has already 9972 * validated the vdev and opened it. 9973 */ 9974 void 9975 l2arc_add_vdev(spa_t *spa, vdev_t *vd) 9976 { 9977 l2arc_dev_t *adddev; 9978 uint64_t l2dhdr_asize; 9979 9980 ASSERT(!l2arc_vdev_present(vd)); 9981 9982 /* 9983 * Create a new l2arc device entry. 9984 */ 9985 adddev = vmem_zalloc(sizeof (l2arc_dev_t), KM_SLEEP); 9986 adddev->l2ad_spa = spa; 9987 adddev->l2ad_vdev = vd; 9988 /* leave extra size for an l2arc device header */ 9989 l2dhdr_asize = adddev->l2ad_dev_hdr_asize = 9990 MAX(sizeof (*adddev->l2ad_dev_hdr), 1 << vd->vdev_ashift); 9991 adddev->l2ad_start = VDEV_LABEL_START_SIZE + l2dhdr_asize; 9992 adddev->l2ad_end = VDEV_LABEL_START_SIZE + vdev_get_min_asize(vd); 9993 ASSERT3U(adddev->l2ad_start, <, adddev->l2ad_end); 9994 adddev->l2ad_hand = adddev->l2ad_start; 9995 adddev->l2ad_evict = adddev->l2ad_start; 9996 adddev->l2ad_first = B_TRUE; 9997 adddev->l2ad_writing = B_FALSE; 9998 adddev->l2ad_trim_all = B_FALSE; 9999 list_link_init(&adddev->l2ad_node); 10000 adddev->l2ad_dev_hdr = kmem_zalloc(l2dhdr_asize, KM_SLEEP); 10001 10002 mutex_init(&adddev->l2ad_mtx, NULL, MUTEX_DEFAULT, NULL); 10003 /* 10004 * This is a list of all ARC buffers that are still valid on the 10005 * device. 10006 */ 10007 list_create(&adddev->l2ad_buflist, sizeof (arc_buf_hdr_t), 10008 offsetof(arc_buf_hdr_t, b_l2hdr.b_l2node)); 10009 10010 /* 10011 * This is a list of pointers to log blocks that are still present 10012 * on the device. 10013 */ 10014 list_create(&adddev->l2ad_lbptr_list, sizeof (l2arc_lb_ptr_buf_t), 10015 offsetof(l2arc_lb_ptr_buf_t, node)); 10016 10017 vdev_space_update(vd, 0, 0, adddev->l2ad_end - adddev->l2ad_hand); 10018 zfs_refcount_create(&adddev->l2ad_alloc); 10019 zfs_refcount_create(&adddev->l2ad_lb_asize); 10020 zfs_refcount_create(&adddev->l2ad_lb_count); 10021 10022 /* 10023 * Decide if dev is eligible for L2ARC rebuild or whole device 10024 * trimming. This has to happen before the device is added in the 10025 * cache device list and l2arc_dev_mtx is released. Otherwise 10026 * l2arc_feed_thread() might already start writing on the 10027 * device. 10028 */ 10029 l2arc_rebuild_dev(adddev, B_FALSE); 10030 10031 /* 10032 * Add device to global list 10033 */ 10034 mutex_enter(&l2arc_dev_mtx); 10035 list_insert_head(l2arc_dev_list, adddev); 10036 atomic_inc_64(&l2arc_ndev); 10037 mutex_exit(&l2arc_dev_mtx); 10038 } 10039 10040 /* 10041 * Decide if a vdev is eligible for L2ARC rebuild, called from vdev_reopen() 10042 * in case of onlining a cache device. 10043 */ 10044 void 10045 l2arc_rebuild_vdev(vdev_t *vd, boolean_t reopen) 10046 { 10047 l2arc_dev_t *dev = NULL; 10048 10049 dev = l2arc_vdev_get(vd); 10050 ASSERT3P(dev, !=, NULL); 10051 10052 /* 10053 * In contrast to l2arc_add_vdev() we do not have to worry about 10054 * l2arc_feed_thread() invalidating previous content when onlining a 10055 * cache device. The device parameters (l2ad*) are not cleared when 10056 * offlining the device and writing new buffers will not invalidate 10057 * all previous content. In worst case only buffers that have not had 10058 * their log block written to the device will be lost. 10059 * When onlining the cache device (ie offline->online without exporting 10060 * the pool in between) this happens: 10061 * vdev_reopen() -> vdev_open() -> l2arc_rebuild_vdev() 10062 * | | 10063 * vdev_is_dead() = B_FALSE l2ad_rebuild = B_TRUE 10064 * During the time where vdev_is_dead = B_FALSE and until l2ad_rebuild 10065 * is set to B_TRUE we might write additional buffers to the device. 10066 */ 10067 l2arc_rebuild_dev(dev, reopen); 10068 } 10069 10070 /* 10071 * Remove a vdev from the L2ARC. 10072 */ 10073 void 10074 l2arc_remove_vdev(vdev_t *vd) 10075 { 10076 l2arc_dev_t *remdev = NULL; 10077 10078 /* 10079 * Find the device by vdev 10080 */ 10081 remdev = l2arc_vdev_get(vd); 10082 ASSERT3P(remdev, !=, NULL); 10083 10084 /* 10085 * Cancel any ongoing or scheduled rebuild. 10086 */ 10087 mutex_enter(&l2arc_rebuild_thr_lock); 10088 if (remdev->l2ad_rebuild_began == B_TRUE) { 10089 remdev->l2ad_rebuild_cancel = B_TRUE; 10090 while (remdev->l2ad_rebuild == B_TRUE) 10091 cv_wait(&l2arc_rebuild_thr_cv, &l2arc_rebuild_thr_lock); 10092 } 10093 mutex_exit(&l2arc_rebuild_thr_lock); 10094 10095 /* 10096 * Remove device from global list 10097 */ 10098 mutex_enter(&l2arc_dev_mtx); 10099 list_remove(l2arc_dev_list, remdev); 10100 l2arc_dev_last = NULL; /* may have been invalidated */ 10101 atomic_dec_64(&l2arc_ndev); 10102 mutex_exit(&l2arc_dev_mtx); 10103 10104 /* 10105 * Clear all buflists and ARC references. L2ARC device flush. 10106 */ 10107 l2arc_evict(remdev, 0, B_TRUE); 10108 list_destroy(&remdev->l2ad_buflist); 10109 ASSERT(list_is_empty(&remdev->l2ad_lbptr_list)); 10110 list_destroy(&remdev->l2ad_lbptr_list); 10111 mutex_destroy(&remdev->l2ad_mtx); 10112 zfs_refcount_destroy(&remdev->l2ad_alloc); 10113 zfs_refcount_destroy(&remdev->l2ad_lb_asize); 10114 zfs_refcount_destroy(&remdev->l2ad_lb_count); 10115 kmem_free(remdev->l2ad_dev_hdr, remdev->l2ad_dev_hdr_asize); 10116 vmem_free(remdev, sizeof (l2arc_dev_t)); 10117 } 10118 10119 void 10120 l2arc_init(void) 10121 { 10122 l2arc_thread_exit = 0; 10123 l2arc_ndev = 0; 10124 10125 mutex_init(&l2arc_feed_thr_lock, NULL, MUTEX_DEFAULT, NULL); 10126 cv_init(&l2arc_feed_thr_cv, NULL, CV_DEFAULT, NULL); 10127 mutex_init(&l2arc_rebuild_thr_lock, NULL, MUTEX_DEFAULT, NULL); 10128 cv_init(&l2arc_rebuild_thr_cv, NULL, CV_DEFAULT, NULL); 10129 mutex_init(&l2arc_dev_mtx, NULL, MUTEX_DEFAULT, NULL); 10130 mutex_init(&l2arc_free_on_write_mtx, NULL, MUTEX_DEFAULT, NULL); 10131 10132 l2arc_dev_list = &L2ARC_dev_list; 10133 l2arc_free_on_write = &L2ARC_free_on_write; 10134 list_create(l2arc_dev_list, sizeof (l2arc_dev_t), 10135 offsetof(l2arc_dev_t, l2ad_node)); 10136 list_create(l2arc_free_on_write, sizeof (l2arc_data_free_t), 10137 offsetof(l2arc_data_free_t, l2df_list_node)); 10138 } 10139 10140 void 10141 l2arc_fini(void) 10142 { 10143 mutex_destroy(&l2arc_feed_thr_lock); 10144 cv_destroy(&l2arc_feed_thr_cv); 10145 mutex_destroy(&l2arc_rebuild_thr_lock); 10146 cv_destroy(&l2arc_rebuild_thr_cv); 10147 mutex_destroy(&l2arc_dev_mtx); 10148 mutex_destroy(&l2arc_free_on_write_mtx); 10149 10150 list_destroy(l2arc_dev_list); 10151 list_destroy(l2arc_free_on_write); 10152 } 10153 10154 void 10155 l2arc_start(void) 10156 { 10157 if (!(spa_mode_global & SPA_MODE_WRITE)) 10158 return; 10159 10160 (void) thread_create(NULL, 0, l2arc_feed_thread, NULL, 0, &p0, 10161 TS_RUN, defclsyspri); 10162 } 10163 10164 void 10165 l2arc_stop(void) 10166 { 10167 if (!(spa_mode_global & SPA_MODE_WRITE)) 10168 return; 10169 10170 mutex_enter(&l2arc_feed_thr_lock); 10171 cv_signal(&l2arc_feed_thr_cv); /* kick thread out of startup */ 10172 l2arc_thread_exit = 1; 10173 while (l2arc_thread_exit != 0) 10174 cv_wait(&l2arc_feed_thr_cv, &l2arc_feed_thr_lock); 10175 mutex_exit(&l2arc_feed_thr_lock); 10176 } 10177 10178 /* 10179 * Punches out rebuild threads for the L2ARC devices in a spa. This should 10180 * be called after pool import from the spa async thread, since starting 10181 * these threads directly from spa_import() will make them part of the 10182 * "zpool import" context and delay process exit (and thus pool import). 10183 */ 10184 void 10185 l2arc_spa_rebuild_start(spa_t *spa) 10186 { 10187 ASSERT(MUTEX_HELD(&spa_namespace_lock)); 10188 10189 /* 10190 * Locate the spa's l2arc devices and kick off rebuild threads. 10191 */ 10192 for (int i = 0; i < spa->spa_l2cache.sav_count; i++) { 10193 l2arc_dev_t *dev = 10194 l2arc_vdev_get(spa->spa_l2cache.sav_vdevs[i]); 10195 if (dev == NULL) { 10196 /* Don't attempt a rebuild if the vdev is UNAVAIL */ 10197 continue; 10198 } 10199 mutex_enter(&l2arc_rebuild_thr_lock); 10200 if (dev->l2ad_rebuild && !dev->l2ad_rebuild_cancel) { 10201 dev->l2ad_rebuild_began = B_TRUE; 10202 (void) thread_create(NULL, 0, l2arc_dev_rebuild_thread, 10203 dev, 0, &p0, TS_RUN, minclsyspri); 10204 } 10205 mutex_exit(&l2arc_rebuild_thr_lock); 10206 } 10207 } 10208 10209 /* 10210 * Main entry point for L2ARC rebuilding. 10211 */ 10212 static __attribute__((noreturn)) void 10213 l2arc_dev_rebuild_thread(void *arg) 10214 { 10215 l2arc_dev_t *dev = arg; 10216 10217 VERIFY(!dev->l2ad_rebuild_cancel); 10218 VERIFY(dev->l2ad_rebuild); 10219 (void) l2arc_rebuild(dev); 10220 mutex_enter(&l2arc_rebuild_thr_lock); 10221 dev->l2ad_rebuild_began = B_FALSE; 10222 dev->l2ad_rebuild = B_FALSE; 10223 mutex_exit(&l2arc_rebuild_thr_lock); 10224 10225 thread_exit(); 10226 } 10227 10228 /* 10229 * This function implements the actual L2ARC metadata rebuild. It: 10230 * starts reading the log block chain and restores each block's contents 10231 * to memory (reconstructing arc_buf_hdr_t's). 10232 * 10233 * Operation stops under any of the following conditions: 10234 * 10235 * 1) We reach the end of the log block chain. 10236 * 2) We encounter *any* error condition (cksum errors, io errors) 10237 */ 10238 static int 10239 l2arc_rebuild(l2arc_dev_t *dev) 10240 { 10241 vdev_t *vd = dev->l2ad_vdev; 10242 spa_t *spa = vd->vdev_spa; 10243 int err = 0; 10244 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 10245 l2arc_log_blk_phys_t *this_lb, *next_lb; 10246 zio_t *this_io = NULL, *next_io = NULL; 10247 l2arc_log_blkptr_t lbps[2]; 10248 l2arc_lb_ptr_buf_t *lb_ptr_buf; 10249 boolean_t lock_held; 10250 10251 this_lb = vmem_zalloc(sizeof (*this_lb), KM_SLEEP); 10252 next_lb = vmem_zalloc(sizeof (*next_lb), KM_SLEEP); 10253 10254 /* 10255 * We prevent device removal while issuing reads to the device, 10256 * then during the rebuilding phases we drop this lock again so 10257 * that a spa_unload or device remove can be initiated - this is 10258 * safe, because the spa will signal us to stop before removing 10259 * our device and wait for us to stop. 10260 */ 10261 spa_config_enter(spa, SCL_L2ARC, vd, RW_READER); 10262 lock_held = B_TRUE; 10263 10264 /* 10265 * Retrieve the persistent L2ARC device state. 10266 * L2BLK_GET_PSIZE returns aligned size for log blocks. 10267 */ 10268 dev->l2ad_evict = MAX(l2dhdr->dh_evict, dev->l2ad_start); 10269 dev->l2ad_hand = MAX(l2dhdr->dh_start_lbps[0].lbp_daddr + 10270 L2BLK_GET_PSIZE((&l2dhdr->dh_start_lbps[0])->lbp_prop), 10271 dev->l2ad_start); 10272 dev->l2ad_first = !!(l2dhdr->dh_flags & L2ARC_DEV_HDR_EVICT_FIRST); 10273 10274 vd->vdev_trim_action_time = l2dhdr->dh_trim_action_time; 10275 vd->vdev_trim_state = l2dhdr->dh_trim_state; 10276 10277 /* 10278 * In case the zfs module parameter l2arc_rebuild_enabled is false 10279 * we do not start the rebuild process. 10280 */ 10281 if (!l2arc_rebuild_enabled) 10282 goto out; 10283 10284 /* Prepare the rebuild process */ 10285 memcpy(lbps, l2dhdr->dh_start_lbps, sizeof (lbps)); 10286 10287 /* Start the rebuild process */ 10288 for (;;) { 10289 if (!l2arc_log_blkptr_valid(dev, &lbps[0])) 10290 break; 10291 10292 if ((err = l2arc_log_blk_read(dev, &lbps[0], &lbps[1], 10293 this_lb, next_lb, this_io, &next_io)) != 0) 10294 goto out; 10295 10296 /* 10297 * Our memory pressure valve. If the system is running low 10298 * on memory, rather than swamping memory with new ARC buf 10299 * hdrs, we opt not to rebuild the L2ARC. At this point, 10300 * however, we have already set up our L2ARC dev to chain in 10301 * new metadata log blocks, so the user may choose to offline/ 10302 * online the L2ARC dev at a later time (or re-import the pool) 10303 * to reconstruct it (when there's less memory pressure). 10304 */ 10305 if (l2arc_hdr_limit_reached()) { 10306 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_lowmem); 10307 cmn_err(CE_NOTE, "System running low on memory, " 10308 "aborting L2ARC rebuild."); 10309 err = SET_ERROR(ENOMEM); 10310 goto out; 10311 } 10312 10313 spa_config_exit(spa, SCL_L2ARC, vd); 10314 lock_held = B_FALSE; 10315 10316 /* 10317 * Now that we know that the next_lb checks out alright, we 10318 * can start reconstruction from this log block. 10319 * L2BLK_GET_PSIZE returns aligned size for log blocks. 10320 */ 10321 uint64_t asize = L2BLK_GET_PSIZE((&lbps[0])->lbp_prop); 10322 l2arc_log_blk_restore(dev, this_lb, asize); 10323 10324 /* 10325 * log block restored, include its pointer in the list of 10326 * pointers to log blocks present in the L2ARC device. 10327 */ 10328 lb_ptr_buf = kmem_zalloc(sizeof (l2arc_lb_ptr_buf_t), KM_SLEEP); 10329 lb_ptr_buf->lb_ptr = kmem_zalloc(sizeof (l2arc_log_blkptr_t), 10330 KM_SLEEP); 10331 memcpy(lb_ptr_buf->lb_ptr, &lbps[0], 10332 sizeof (l2arc_log_blkptr_t)); 10333 mutex_enter(&dev->l2ad_mtx); 10334 list_insert_tail(&dev->l2ad_lbptr_list, lb_ptr_buf); 10335 ARCSTAT_INCR(arcstat_l2_log_blk_asize, asize); 10336 ARCSTAT_BUMP(arcstat_l2_log_blk_count); 10337 zfs_refcount_add_many(&dev->l2ad_lb_asize, asize, lb_ptr_buf); 10338 zfs_refcount_add(&dev->l2ad_lb_count, lb_ptr_buf); 10339 mutex_exit(&dev->l2ad_mtx); 10340 vdev_space_update(vd, asize, 0, 0); 10341 10342 /* 10343 * Protection against loops of log blocks: 10344 * 10345 * l2ad_hand l2ad_evict 10346 * V V 10347 * l2ad_start |=======================================| l2ad_end 10348 * -----|||----|||---|||----||| 10349 * (3) (2) (1) (0) 10350 * ---|||---|||----|||---||| 10351 * (7) (6) (5) (4) 10352 * 10353 * In this situation the pointer of log block (4) passes 10354 * l2arc_log_blkptr_valid() but the log block should not be 10355 * restored as it is overwritten by the payload of log block 10356 * (0). Only log blocks (0)-(3) should be restored. We check 10357 * whether l2ad_evict lies in between the payload starting 10358 * offset of the next log block (lbps[1].lbp_payload_start) 10359 * and the payload starting offset of the present log block 10360 * (lbps[0].lbp_payload_start). If true and this isn't the 10361 * first pass, we are looping from the beginning and we should 10362 * stop. 10363 */ 10364 if (l2arc_range_check_overlap(lbps[1].lbp_payload_start, 10365 lbps[0].lbp_payload_start, dev->l2ad_evict) && 10366 !dev->l2ad_first) 10367 goto out; 10368 10369 kpreempt(KPREEMPT_SYNC); 10370 for (;;) { 10371 mutex_enter(&l2arc_rebuild_thr_lock); 10372 if (dev->l2ad_rebuild_cancel) { 10373 dev->l2ad_rebuild = B_FALSE; 10374 cv_signal(&l2arc_rebuild_thr_cv); 10375 mutex_exit(&l2arc_rebuild_thr_lock); 10376 err = SET_ERROR(ECANCELED); 10377 goto out; 10378 } 10379 mutex_exit(&l2arc_rebuild_thr_lock); 10380 if (spa_config_tryenter(spa, SCL_L2ARC, vd, 10381 RW_READER)) { 10382 lock_held = B_TRUE; 10383 break; 10384 } 10385 /* 10386 * L2ARC config lock held by somebody in writer, 10387 * possibly due to them trying to remove us. They'll 10388 * likely to want us to shut down, so after a little 10389 * delay, we check l2ad_rebuild_cancel and retry 10390 * the lock again. 10391 */ 10392 delay(1); 10393 } 10394 10395 /* 10396 * Continue with the next log block. 10397 */ 10398 lbps[0] = lbps[1]; 10399 lbps[1] = this_lb->lb_prev_lbp; 10400 PTR_SWAP(this_lb, next_lb); 10401 this_io = next_io; 10402 next_io = NULL; 10403 } 10404 10405 if (this_io != NULL) 10406 l2arc_log_blk_fetch_abort(this_io); 10407 out: 10408 if (next_io != NULL) 10409 l2arc_log_blk_fetch_abort(next_io); 10410 vmem_free(this_lb, sizeof (*this_lb)); 10411 vmem_free(next_lb, sizeof (*next_lb)); 10412 10413 if (!l2arc_rebuild_enabled) { 10414 spa_history_log_internal(spa, "L2ARC rebuild", NULL, 10415 "disabled"); 10416 } else if (err == 0 && zfs_refcount_count(&dev->l2ad_lb_count) > 0) { 10417 ARCSTAT_BUMP(arcstat_l2_rebuild_success); 10418 spa_history_log_internal(spa, "L2ARC rebuild", NULL, 10419 "successful, restored %llu blocks", 10420 (u_longlong_t)zfs_refcount_count(&dev->l2ad_lb_count)); 10421 } else if (err == 0 && zfs_refcount_count(&dev->l2ad_lb_count) == 0) { 10422 /* 10423 * No error but also nothing restored, meaning the lbps array 10424 * in the device header points to invalid/non-present log 10425 * blocks. Reset the header. 10426 */ 10427 spa_history_log_internal(spa, "L2ARC rebuild", NULL, 10428 "no valid log blocks"); 10429 memset(l2dhdr, 0, dev->l2ad_dev_hdr_asize); 10430 l2arc_dev_hdr_update(dev); 10431 } else if (err == ECANCELED) { 10432 /* 10433 * In case the rebuild was canceled do not log to spa history 10434 * log as the pool may be in the process of being removed. 10435 */ 10436 zfs_dbgmsg("L2ARC rebuild aborted, restored %llu blocks", 10437 (u_longlong_t)zfs_refcount_count(&dev->l2ad_lb_count)); 10438 } else if (err != 0) { 10439 spa_history_log_internal(spa, "L2ARC rebuild", NULL, 10440 "aborted, restored %llu blocks", 10441 (u_longlong_t)zfs_refcount_count(&dev->l2ad_lb_count)); 10442 } 10443 10444 if (lock_held) 10445 spa_config_exit(spa, SCL_L2ARC, vd); 10446 10447 return (err); 10448 } 10449 10450 /* 10451 * Attempts to read the device header on the provided L2ARC device and writes 10452 * it to `hdr'. On success, this function returns 0, otherwise the appropriate 10453 * error code is returned. 10454 */ 10455 static int 10456 l2arc_dev_hdr_read(l2arc_dev_t *dev) 10457 { 10458 int err; 10459 uint64_t guid; 10460 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 10461 const uint64_t l2dhdr_asize = dev->l2ad_dev_hdr_asize; 10462 abd_t *abd; 10463 10464 guid = spa_guid(dev->l2ad_vdev->vdev_spa); 10465 10466 abd = abd_get_from_buf(l2dhdr, l2dhdr_asize); 10467 10468 err = zio_wait(zio_read_phys(NULL, dev->l2ad_vdev, 10469 VDEV_LABEL_START_SIZE, l2dhdr_asize, abd, 10470 ZIO_CHECKSUM_LABEL, NULL, NULL, ZIO_PRIORITY_SYNC_READ, 10471 ZIO_FLAG_DONT_CACHE | ZIO_FLAG_CANFAIL | 10472 ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY | 10473 ZIO_FLAG_SPECULATIVE, B_FALSE)); 10474 10475 abd_free(abd); 10476 10477 if (err != 0) { 10478 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_dh_errors); 10479 zfs_dbgmsg("L2ARC IO error (%d) while reading device header, " 10480 "vdev guid: %llu", err, 10481 (u_longlong_t)dev->l2ad_vdev->vdev_guid); 10482 return (err); 10483 } 10484 10485 if (l2dhdr->dh_magic == BSWAP_64(L2ARC_DEV_HDR_MAGIC)) 10486 byteswap_uint64_array(l2dhdr, sizeof (*l2dhdr)); 10487 10488 if (l2dhdr->dh_magic != L2ARC_DEV_HDR_MAGIC || 10489 l2dhdr->dh_spa_guid != guid || 10490 l2dhdr->dh_vdev_guid != dev->l2ad_vdev->vdev_guid || 10491 l2dhdr->dh_version != L2ARC_PERSISTENT_VERSION || 10492 l2dhdr->dh_log_entries != dev->l2ad_log_entries || 10493 l2dhdr->dh_end != dev->l2ad_end || 10494 !l2arc_range_check_overlap(dev->l2ad_start, dev->l2ad_end, 10495 l2dhdr->dh_evict) || 10496 (l2dhdr->dh_trim_state != VDEV_TRIM_COMPLETE && 10497 l2arc_trim_ahead > 0)) { 10498 /* 10499 * Attempt to rebuild a device containing no actual dev hdr 10500 * or containing a header from some other pool or from another 10501 * version of persistent L2ARC. 10502 */ 10503 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_unsupported); 10504 return (SET_ERROR(ENOTSUP)); 10505 } 10506 10507 return (0); 10508 } 10509 10510 /* 10511 * Reads L2ARC log blocks from storage and validates their contents. 10512 * 10513 * This function implements a simple fetcher to make sure that while 10514 * we're processing one buffer the L2ARC is already fetching the next 10515 * one in the chain. 10516 * 10517 * The arguments this_lp and next_lp point to the current and next log block 10518 * address in the block chain. Similarly, this_lb and next_lb hold the 10519 * l2arc_log_blk_phys_t's of the current and next L2ARC blk. 10520 * 10521 * The `this_io' and `next_io' arguments are used for block fetching. 10522 * When issuing the first blk IO during rebuild, you should pass NULL for 10523 * `this_io'. This function will then issue a sync IO to read the block and 10524 * also issue an async IO to fetch the next block in the block chain. The 10525 * fetched IO is returned in `next_io'. On subsequent calls to this 10526 * function, pass the value returned in `next_io' from the previous call 10527 * as `this_io' and a fresh `next_io' pointer to hold the next fetch IO. 10528 * Prior to the call, you should initialize your `next_io' pointer to be 10529 * NULL. If no fetch IO was issued, the pointer is left set at NULL. 10530 * 10531 * On success, this function returns 0, otherwise it returns an appropriate 10532 * error code. On error the fetching IO is aborted and cleared before 10533 * returning from this function. Therefore, if we return `success', the 10534 * caller can assume that we have taken care of cleanup of fetch IOs. 10535 */ 10536 static int 10537 l2arc_log_blk_read(l2arc_dev_t *dev, 10538 const l2arc_log_blkptr_t *this_lbp, const l2arc_log_blkptr_t *next_lbp, 10539 l2arc_log_blk_phys_t *this_lb, l2arc_log_blk_phys_t *next_lb, 10540 zio_t *this_io, zio_t **next_io) 10541 { 10542 int err = 0; 10543 zio_cksum_t cksum; 10544 abd_t *abd = NULL; 10545 uint64_t asize; 10546 10547 ASSERT(this_lbp != NULL && next_lbp != NULL); 10548 ASSERT(this_lb != NULL && next_lb != NULL); 10549 ASSERT(next_io != NULL && *next_io == NULL); 10550 ASSERT(l2arc_log_blkptr_valid(dev, this_lbp)); 10551 10552 /* 10553 * Check to see if we have issued the IO for this log block in a 10554 * previous run. If not, this is the first call, so issue it now. 10555 */ 10556 if (this_io == NULL) { 10557 this_io = l2arc_log_blk_fetch(dev->l2ad_vdev, this_lbp, 10558 this_lb); 10559 } 10560 10561 /* 10562 * Peek to see if we can start issuing the next IO immediately. 10563 */ 10564 if (l2arc_log_blkptr_valid(dev, next_lbp)) { 10565 /* 10566 * Start issuing IO for the next log block early - this 10567 * should help keep the L2ARC device busy while we 10568 * decompress and restore this log block. 10569 */ 10570 *next_io = l2arc_log_blk_fetch(dev->l2ad_vdev, next_lbp, 10571 next_lb); 10572 } 10573 10574 /* Wait for the IO to read this log block to complete */ 10575 if ((err = zio_wait(this_io)) != 0) { 10576 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_io_errors); 10577 zfs_dbgmsg("L2ARC IO error (%d) while reading log block, " 10578 "offset: %llu, vdev guid: %llu", err, 10579 (u_longlong_t)this_lbp->lbp_daddr, 10580 (u_longlong_t)dev->l2ad_vdev->vdev_guid); 10581 goto cleanup; 10582 } 10583 10584 /* 10585 * Make sure the buffer checks out. 10586 * L2BLK_GET_PSIZE returns aligned size for log blocks. 10587 */ 10588 asize = L2BLK_GET_PSIZE((this_lbp)->lbp_prop); 10589 fletcher_4_native(this_lb, asize, NULL, &cksum); 10590 if (!ZIO_CHECKSUM_EQUAL(cksum, this_lbp->lbp_cksum)) { 10591 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_cksum_lb_errors); 10592 zfs_dbgmsg("L2ARC log block cksum failed, offset: %llu, " 10593 "vdev guid: %llu, l2ad_hand: %llu, l2ad_evict: %llu", 10594 (u_longlong_t)this_lbp->lbp_daddr, 10595 (u_longlong_t)dev->l2ad_vdev->vdev_guid, 10596 (u_longlong_t)dev->l2ad_hand, 10597 (u_longlong_t)dev->l2ad_evict); 10598 err = SET_ERROR(ECKSUM); 10599 goto cleanup; 10600 } 10601 10602 /* Now we can take our time decoding this buffer */ 10603 switch (L2BLK_GET_COMPRESS((this_lbp)->lbp_prop)) { 10604 case ZIO_COMPRESS_OFF: 10605 break; 10606 case ZIO_COMPRESS_LZ4: 10607 abd = abd_alloc_for_io(asize, B_TRUE); 10608 abd_copy_from_buf_off(abd, this_lb, 0, asize); 10609 if ((err = zio_decompress_data( 10610 L2BLK_GET_COMPRESS((this_lbp)->lbp_prop), 10611 abd, this_lb, asize, sizeof (*this_lb), NULL)) != 0) { 10612 err = SET_ERROR(EINVAL); 10613 goto cleanup; 10614 } 10615 break; 10616 default: 10617 err = SET_ERROR(EINVAL); 10618 goto cleanup; 10619 } 10620 if (this_lb->lb_magic == BSWAP_64(L2ARC_LOG_BLK_MAGIC)) 10621 byteswap_uint64_array(this_lb, sizeof (*this_lb)); 10622 if (this_lb->lb_magic != L2ARC_LOG_BLK_MAGIC) { 10623 err = SET_ERROR(EINVAL); 10624 goto cleanup; 10625 } 10626 cleanup: 10627 /* Abort an in-flight fetch I/O in case of error */ 10628 if (err != 0 && *next_io != NULL) { 10629 l2arc_log_blk_fetch_abort(*next_io); 10630 *next_io = NULL; 10631 } 10632 if (abd != NULL) 10633 abd_free(abd); 10634 return (err); 10635 } 10636 10637 /* 10638 * Restores the payload of a log block to ARC. This creates empty ARC hdr 10639 * entries which only contain an l2arc hdr, essentially restoring the 10640 * buffers to their L2ARC evicted state. This function also updates space 10641 * usage on the L2ARC vdev to make sure it tracks restored buffers. 10642 */ 10643 static void 10644 l2arc_log_blk_restore(l2arc_dev_t *dev, const l2arc_log_blk_phys_t *lb, 10645 uint64_t lb_asize) 10646 { 10647 uint64_t size = 0, asize = 0; 10648 uint64_t log_entries = dev->l2ad_log_entries; 10649 10650 /* 10651 * Usually arc_adapt() is called only for data, not headers, but 10652 * since we may allocate significant amount of memory here, let ARC 10653 * grow its arc_c. 10654 */ 10655 arc_adapt(log_entries * HDR_L2ONLY_SIZE, arc_l2c_only); 10656 10657 for (int i = log_entries - 1; i >= 0; i--) { 10658 /* 10659 * Restore goes in the reverse temporal direction to preserve 10660 * correct temporal ordering of buffers in the l2ad_buflist. 10661 * l2arc_hdr_restore also does a list_insert_tail instead of 10662 * list_insert_head on the l2ad_buflist: 10663 * 10664 * LIST l2ad_buflist LIST 10665 * HEAD <------ (time) ------ TAIL 10666 * direction +-----+-----+-----+-----+-----+ direction 10667 * of l2arc <== | buf | buf | buf | buf | buf | ===> of rebuild 10668 * fill +-----+-----+-----+-----+-----+ 10669 * ^ ^ 10670 * | | 10671 * | | 10672 * l2arc_feed_thread l2arc_rebuild 10673 * will place new bufs here restores bufs here 10674 * 10675 * During l2arc_rebuild() the device is not used by 10676 * l2arc_feed_thread() as dev->l2ad_rebuild is set to true. 10677 */ 10678 size += L2BLK_GET_LSIZE((&lb->lb_entries[i])->le_prop); 10679 asize += vdev_psize_to_asize(dev->l2ad_vdev, 10680 L2BLK_GET_PSIZE((&lb->lb_entries[i])->le_prop)); 10681 l2arc_hdr_restore(&lb->lb_entries[i], dev); 10682 } 10683 10684 /* 10685 * Record rebuild stats: 10686 * size Logical size of restored buffers in the L2ARC 10687 * asize Aligned size of restored buffers in the L2ARC 10688 */ 10689 ARCSTAT_INCR(arcstat_l2_rebuild_size, size); 10690 ARCSTAT_INCR(arcstat_l2_rebuild_asize, asize); 10691 ARCSTAT_INCR(arcstat_l2_rebuild_bufs, log_entries); 10692 ARCSTAT_F_AVG(arcstat_l2_log_blk_avg_asize, lb_asize); 10693 ARCSTAT_F_AVG(arcstat_l2_data_to_meta_ratio, asize / lb_asize); 10694 ARCSTAT_BUMP(arcstat_l2_rebuild_log_blks); 10695 } 10696 10697 /* 10698 * Restores a single ARC buf hdr from a log entry. The ARC buffer is put 10699 * into a state indicating that it has been evicted to L2ARC. 10700 */ 10701 static void 10702 l2arc_hdr_restore(const l2arc_log_ent_phys_t *le, l2arc_dev_t *dev) 10703 { 10704 arc_buf_hdr_t *hdr, *exists; 10705 kmutex_t *hash_lock; 10706 arc_buf_contents_t type = L2BLK_GET_TYPE((le)->le_prop); 10707 uint64_t asize; 10708 10709 /* 10710 * Do all the allocation before grabbing any locks, this lets us 10711 * sleep if memory is full and we don't have to deal with failed 10712 * allocations. 10713 */ 10714 hdr = arc_buf_alloc_l2only(L2BLK_GET_LSIZE((le)->le_prop), type, 10715 dev, le->le_dva, le->le_daddr, 10716 L2BLK_GET_PSIZE((le)->le_prop), le->le_birth, 10717 L2BLK_GET_COMPRESS((le)->le_prop), le->le_complevel, 10718 L2BLK_GET_PROTECTED((le)->le_prop), 10719 L2BLK_GET_PREFETCH((le)->le_prop), 10720 L2BLK_GET_STATE((le)->le_prop)); 10721 asize = vdev_psize_to_asize(dev->l2ad_vdev, 10722 L2BLK_GET_PSIZE((le)->le_prop)); 10723 10724 /* 10725 * vdev_space_update() has to be called before arc_hdr_destroy() to 10726 * avoid underflow since the latter also calls vdev_space_update(). 10727 */ 10728 l2arc_hdr_arcstats_increment(hdr); 10729 vdev_space_update(dev->l2ad_vdev, asize, 0, 0); 10730 10731 mutex_enter(&dev->l2ad_mtx); 10732 list_insert_tail(&dev->l2ad_buflist, hdr); 10733 (void) zfs_refcount_add_many(&dev->l2ad_alloc, arc_hdr_size(hdr), hdr); 10734 mutex_exit(&dev->l2ad_mtx); 10735 10736 exists = buf_hash_insert(hdr, &hash_lock); 10737 if (exists) { 10738 /* Buffer was already cached, no need to restore it. */ 10739 arc_hdr_destroy(hdr); 10740 /* 10741 * If the buffer is already cached, check whether it has 10742 * L2ARC metadata. If not, enter them and update the flag. 10743 * This is important is case of onlining a cache device, since 10744 * we previously evicted all L2ARC metadata from ARC. 10745 */ 10746 if (!HDR_HAS_L2HDR(exists)) { 10747 arc_hdr_set_flags(exists, ARC_FLAG_HAS_L2HDR); 10748 exists->b_l2hdr.b_dev = dev; 10749 exists->b_l2hdr.b_daddr = le->le_daddr; 10750 exists->b_l2hdr.b_arcs_state = 10751 L2BLK_GET_STATE((le)->le_prop); 10752 mutex_enter(&dev->l2ad_mtx); 10753 list_insert_tail(&dev->l2ad_buflist, exists); 10754 (void) zfs_refcount_add_many(&dev->l2ad_alloc, 10755 arc_hdr_size(exists), exists); 10756 mutex_exit(&dev->l2ad_mtx); 10757 l2arc_hdr_arcstats_increment(exists); 10758 vdev_space_update(dev->l2ad_vdev, asize, 0, 0); 10759 } 10760 ARCSTAT_BUMP(arcstat_l2_rebuild_bufs_precached); 10761 } 10762 10763 mutex_exit(hash_lock); 10764 } 10765 10766 /* 10767 * Starts an asynchronous read IO to read a log block. This is used in log 10768 * block reconstruction to start reading the next block before we are done 10769 * decoding and reconstructing the current block, to keep the l2arc device 10770 * nice and hot with read IO to process. 10771 * The returned zio will contain a newly allocated memory buffers for the IO 10772 * data which should then be freed by the caller once the zio is no longer 10773 * needed (i.e. due to it having completed). If you wish to abort this 10774 * zio, you should do so using l2arc_log_blk_fetch_abort, which takes 10775 * care of disposing of the allocated buffers correctly. 10776 */ 10777 static zio_t * 10778 l2arc_log_blk_fetch(vdev_t *vd, const l2arc_log_blkptr_t *lbp, 10779 l2arc_log_blk_phys_t *lb) 10780 { 10781 uint32_t asize; 10782 zio_t *pio; 10783 l2arc_read_callback_t *cb; 10784 10785 /* L2BLK_GET_PSIZE returns aligned size for log blocks */ 10786 asize = L2BLK_GET_PSIZE((lbp)->lbp_prop); 10787 ASSERT(asize <= sizeof (l2arc_log_blk_phys_t)); 10788 10789 cb = kmem_zalloc(sizeof (l2arc_read_callback_t), KM_SLEEP); 10790 cb->l2rcb_abd = abd_get_from_buf(lb, asize); 10791 pio = zio_root(vd->vdev_spa, l2arc_blk_fetch_done, cb, 10792 ZIO_FLAG_DONT_CACHE | ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE | 10793 ZIO_FLAG_DONT_RETRY); 10794 (void) zio_nowait(zio_read_phys(pio, vd, lbp->lbp_daddr, asize, 10795 cb->l2rcb_abd, ZIO_CHECKSUM_OFF, NULL, NULL, 10796 ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_DONT_CACHE | ZIO_FLAG_CANFAIL | 10797 ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY, B_FALSE)); 10798 10799 return (pio); 10800 } 10801 10802 /* 10803 * Aborts a zio returned from l2arc_log_blk_fetch and frees the data 10804 * buffers allocated for it. 10805 */ 10806 static void 10807 l2arc_log_blk_fetch_abort(zio_t *zio) 10808 { 10809 (void) zio_wait(zio); 10810 } 10811 10812 /* 10813 * Creates a zio to update the device header on an l2arc device. 10814 */ 10815 void 10816 l2arc_dev_hdr_update(l2arc_dev_t *dev) 10817 { 10818 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 10819 const uint64_t l2dhdr_asize = dev->l2ad_dev_hdr_asize; 10820 abd_t *abd; 10821 int err; 10822 10823 VERIFY(spa_config_held(dev->l2ad_spa, SCL_STATE_ALL, RW_READER)); 10824 10825 l2dhdr->dh_magic = L2ARC_DEV_HDR_MAGIC; 10826 l2dhdr->dh_version = L2ARC_PERSISTENT_VERSION; 10827 l2dhdr->dh_spa_guid = spa_guid(dev->l2ad_vdev->vdev_spa); 10828 l2dhdr->dh_vdev_guid = dev->l2ad_vdev->vdev_guid; 10829 l2dhdr->dh_log_entries = dev->l2ad_log_entries; 10830 l2dhdr->dh_evict = dev->l2ad_evict; 10831 l2dhdr->dh_start = dev->l2ad_start; 10832 l2dhdr->dh_end = dev->l2ad_end; 10833 l2dhdr->dh_lb_asize = zfs_refcount_count(&dev->l2ad_lb_asize); 10834 l2dhdr->dh_lb_count = zfs_refcount_count(&dev->l2ad_lb_count); 10835 l2dhdr->dh_flags = 0; 10836 l2dhdr->dh_trim_action_time = dev->l2ad_vdev->vdev_trim_action_time; 10837 l2dhdr->dh_trim_state = dev->l2ad_vdev->vdev_trim_state; 10838 if (dev->l2ad_first) 10839 l2dhdr->dh_flags |= L2ARC_DEV_HDR_EVICT_FIRST; 10840 10841 abd = abd_get_from_buf(l2dhdr, l2dhdr_asize); 10842 10843 err = zio_wait(zio_write_phys(NULL, dev->l2ad_vdev, 10844 VDEV_LABEL_START_SIZE, l2dhdr_asize, abd, ZIO_CHECKSUM_LABEL, NULL, 10845 NULL, ZIO_PRIORITY_ASYNC_WRITE, ZIO_FLAG_CANFAIL, B_FALSE)); 10846 10847 abd_free(abd); 10848 10849 if (err != 0) { 10850 zfs_dbgmsg("L2ARC IO error (%d) while writing device header, " 10851 "vdev guid: %llu", err, 10852 (u_longlong_t)dev->l2ad_vdev->vdev_guid); 10853 } 10854 } 10855 10856 /* 10857 * Commits a log block to the L2ARC device. This routine is invoked from 10858 * l2arc_write_buffers when the log block fills up. 10859 * This function allocates some memory to temporarily hold the serialized 10860 * buffer to be written. This is then released in l2arc_write_done. 10861 */ 10862 static void 10863 l2arc_log_blk_commit(l2arc_dev_t *dev, zio_t *pio, l2arc_write_callback_t *cb) 10864 { 10865 l2arc_log_blk_phys_t *lb = &dev->l2ad_log_blk; 10866 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 10867 uint64_t psize, asize; 10868 zio_t *wzio; 10869 l2arc_lb_abd_buf_t *abd_buf; 10870 uint8_t *tmpbuf; 10871 l2arc_lb_ptr_buf_t *lb_ptr_buf; 10872 10873 VERIFY3S(dev->l2ad_log_ent_idx, ==, dev->l2ad_log_entries); 10874 10875 tmpbuf = zio_buf_alloc(sizeof (*lb)); 10876 abd_buf = zio_buf_alloc(sizeof (*abd_buf)); 10877 abd_buf->abd = abd_get_from_buf(lb, sizeof (*lb)); 10878 lb_ptr_buf = kmem_zalloc(sizeof (l2arc_lb_ptr_buf_t), KM_SLEEP); 10879 lb_ptr_buf->lb_ptr = kmem_zalloc(sizeof (l2arc_log_blkptr_t), KM_SLEEP); 10880 10881 /* link the buffer into the block chain */ 10882 lb->lb_prev_lbp = l2dhdr->dh_start_lbps[1]; 10883 lb->lb_magic = L2ARC_LOG_BLK_MAGIC; 10884 10885 /* 10886 * l2arc_log_blk_commit() may be called multiple times during a single 10887 * l2arc_write_buffers() call. Save the allocated abd buffers in a list 10888 * so we can free them in l2arc_write_done() later on. 10889 */ 10890 list_insert_tail(&cb->l2wcb_abd_list, abd_buf); 10891 10892 /* try to compress the buffer */ 10893 psize = zio_compress_data(ZIO_COMPRESS_LZ4, 10894 abd_buf->abd, tmpbuf, sizeof (*lb), 0); 10895 10896 /* a log block is never entirely zero */ 10897 ASSERT(psize != 0); 10898 asize = vdev_psize_to_asize(dev->l2ad_vdev, psize); 10899 ASSERT(asize <= sizeof (*lb)); 10900 10901 /* 10902 * Update the start log block pointer in the device header to point 10903 * to the log block we're about to write. 10904 */ 10905 l2dhdr->dh_start_lbps[1] = l2dhdr->dh_start_lbps[0]; 10906 l2dhdr->dh_start_lbps[0].lbp_daddr = dev->l2ad_hand; 10907 l2dhdr->dh_start_lbps[0].lbp_payload_asize = 10908 dev->l2ad_log_blk_payload_asize; 10909 l2dhdr->dh_start_lbps[0].lbp_payload_start = 10910 dev->l2ad_log_blk_payload_start; 10911 L2BLK_SET_LSIZE( 10912 (&l2dhdr->dh_start_lbps[0])->lbp_prop, sizeof (*lb)); 10913 L2BLK_SET_PSIZE( 10914 (&l2dhdr->dh_start_lbps[0])->lbp_prop, asize); 10915 L2BLK_SET_CHECKSUM( 10916 (&l2dhdr->dh_start_lbps[0])->lbp_prop, 10917 ZIO_CHECKSUM_FLETCHER_4); 10918 if (asize < sizeof (*lb)) { 10919 /* compression succeeded */ 10920 memset(tmpbuf + psize, 0, asize - psize); 10921 L2BLK_SET_COMPRESS( 10922 (&l2dhdr->dh_start_lbps[0])->lbp_prop, 10923 ZIO_COMPRESS_LZ4); 10924 } else { 10925 /* compression failed */ 10926 memcpy(tmpbuf, lb, sizeof (*lb)); 10927 L2BLK_SET_COMPRESS( 10928 (&l2dhdr->dh_start_lbps[0])->lbp_prop, 10929 ZIO_COMPRESS_OFF); 10930 } 10931 10932 /* checksum what we're about to write */ 10933 fletcher_4_native(tmpbuf, asize, NULL, 10934 &l2dhdr->dh_start_lbps[0].lbp_cksum); 10935 10936 abd_free(abd_buf->abd); 10937 10938 /* perform the write itself */ 10939 abd_buf->abd = abd_get_from_buf(tmpbuf, sizeof (*lb)); 10940 abd_take_ownership_of_buf(abd_buf->abd, B_TRUE); 10941 wzio = zio_write_phys(pio, dev->l2ad_vdev, dev->l2ad_hand, 10942 asize, abd_buf->abd, ZIO_CHECKSUM_OFF, NULL, NULL, 10943 ZIO_PRIORITY_ASYNC_WRITE, ZIO_FLAG_CANFAIL, B_FALSE); 10944 DTRACE_PROBE2(l2arc__write, vdev_t *, dev->l2ad_vdev, zio_t *, wzio); 10945 (void) zio_nowait(wzio); 10946 10947 dev->l2ad_hand += asize; 10948 /* 10949 * Include the committed log block's pointer in the list of pointers 10950 * to log blocks present in the L2ARC device. 10951 */ 10952 memcpy(lb_ptr_buf->lb_ptr, &l2dhdr->dh_start_lbps[0], 10953 sizeof (l2arc_log_blkptr_t)); 10954 mutex_enter(&dev->l2ad_mtx); 10955 list_insert_head(&dev->l2ad_lbptr_list, lb_ptr_buf); 10956 ARCSTAT_INCR(arcstat_l2_log_blk_asize, asize); 10957 ARCSTAT_BUMP(arcstat_l2_log_blk_count); 10958 zfs_refcount_add_many(&dev->l2ad_lb_asize, asize, lb_ptr_buf); 10959 zfs_refcount_add(&dev->l2ad_lb_count, lb_ptr_buf); 10960 mutex_exit(&dev->l2ad_mtx); 10961 vdev_space_update(dev->l2ad_vdev, asize, 0, 0); 10962 10963 /* bump the kstats */ 10964 ARCSTAT_INCR(arcstat_l2_write_bytes, asize); 10965 ARCSTAT_BUMP(arcstat_l2_log_blk_writes); 10966 ARCSTAT_F_AVG(arcstat_l2_log_blk_avg_asize, asize); 10967 ARCSTAT_F_AVG(arcstat_l2_data_to_meta_ratio, 10968 dev->l2ad_log_blk_payload_asize / asize); 10969 10970 /* start a new log block */ 10971 dev->l2ad_log_ent_idx = 0; 10972 dev->l2ad_log_blk_payload_asize = 0; 10973 dev->l2ad_log_blk_payload_start = 0; 10974 } 10975 10976 /* 10977 * Validates an L2ARC log block address to make sure that it can be read 10978 * from the provided L2ARC device. 10979 */ 10980 boolean_t 10981 l2arc_log_blkptr_valid(l2arc_dev_t *dev, const l2arc_log_blkptr_t *lbp) 10982 { 10983 /* L2BLK_GET_PSIZE returns aligned size for log blocks */ 10984 uint64_t asize = L2BLK_GET_PSIZE((lbp)->lbp_prop); 10985 uint64_t end = lbp->lbp_daddr + asize - 1; 10986 uint64_t start = lbp->lbp_payload_start; 10987 boolean_t evicted = B_FALSE; 10988 10989 /* 10990 * A log block is valid if all of the following conditions are true: 10991 * - it fits entirely (including its payload) between l2ad_start and 10992 * l2ad_end 10993 * - it has a valid size 10994 * - neither the log block itself nor part of its payload was evicted 10995 * by l2arc_evict(): 10996 * 10997 * l2ad_hand l2ad_evict 10998 * | | lbp_daddr 10999 * | start | | end 11000 * | | | | | 11001 * V V V V V 11002 * l2ad_start ============================================ l2ad_end 11003 * --------------------------|||| 11004 * ^ ^ 11005 * | log block 11006 * payload 11007 */ 11008 11009 evicted = 11010 l2arc_range_check_overlap(start, end, dev->l2ad_hand) || 11011 l2arc_range_check_overlap(start, end, dev->l2ad_evict) || 11012 l2arc_range_check_overlap(dev->l2ad_hand, dev->l2ad_evict, start) || 11013 l2arc_range_check_overlap(dev->l2ad_hand, dev->l2ad_evict, end); 11014 11015 return (start >= dev->l2ad_start && end <= dev->l2ad_end && 11016 asize > 0 && asize <= sizeof (l2arc_log_blk_phys_t) && 11017 (!evicted || dev->l2ad_first)); 11018 } 11019 11020 /* 11021 * Inserts ARC buffer header `hdr' into the current L2ARC log block on 11022 * the device. The buffer being inserted must be present in L2ARC. 11023 * Returns B_TRUE if the L2ARC log block is full and needs to be committed 11024 * to L2ARC, or B_FALSE if it still has room for more ARC buffers. 11025 */ 11026 static boolean_t 11027 l2arc_log_blk_insert(l2arc_dev_t *dev, const arc_buf_hdr_t *hdr) 11028 { 11029 l2arc_log_blk_phys_t *lb = &dev->l2ad_log_blk; 11030 l2arc_log_ent_phys_t *le; 11031 11032 if (dev->l2ad_log_entries == 0) 11033 return (B_FALSE); 11034 11035 int index = dev->l2ad_log_ent_idx++; 11036 11037 ASSERT3S(index, <, dev->l2ad_log_entries); 11038 ASSERT(HDR_HAS_L2HDR(hdr)); 11039 11040 le = &lb->lb_entries[index]; 11041 memset(le, 0, sizeof (*le)); 11042 le->le_dva = hdr->b_dva; 11043 le->le_birth = hdr->b_birth; 11044 le->le_daddr = hdr->b_l2hdr.b_daddr; 11045 if (index == 0) 11046 dev->l2ad_log_blk_payload_start = le->le_daddr; 11047 L2BLK_SET_LSIZE((le)->le_prop, HDR_GET_LSIZE(hdr)); 11048 L2BLK_SET_PSIZE((le)->le_prop, HDR_GET_PSIZE(hdr)); 11049 L2BLK_SET_COMPRESS((le)->le_prop, HDR_GET_COMPRESS(hdr)); 11050 le->le_complevel = hdr->b_complevel; 11051 L2BLK_SET_TYPE((le)->le_prop, hdr->b_type); 11052 L2BLK_SET_PROTECTED((le)->le_prop, !!(HDR_PROTECTED(hdr))); 11053 L2BLK_SET_PREFETCH((le)->le_prop, !!(HDR_PREFETCH(hdr))); 11054 L2BLK_SET_STATE((le)->le_prop, hdr->b_l1hdr.b_state->arcs_state); 11055 11056 dev->l2ad_log_blk_payload_asize += vdev_psize_to_asize(dev->l2ad_vdev, 11057 HDR_GET_PSIZE(hdr)); 11058 11059 return (dev->l2ad_log_ent_idx == dev->l2ad_log_entries); 11060 } 11061 11062 /* 11063 * Checks whether a given L2ARC device address sits in a time-sequential 11064 * range. The trick here is that the L2ARC is a rotary buffer, so we can't 11065 * just do a range comparison, we need to handle the situation in which the 11066 * range wraps around the end of the L2ARC device. Arguments: 11067 * bottom -- Lower end of the range to check (written to earlier). 11068 * top -- Upper end of the range to check (written to later). 11069 * check -- The address for which we want to determine if it sits in 11070 * between the top and bottom. 11071 * 11072 * The 3-way conditional below represents the following cases: 11073 * 11074 * bottom < top : Sequentially ordered case: 11075 * <check>--------+-------------------+ 11076 * | (overlap here?) | 11077 * L2ARC dev V V 11078 * |---------------<bottom>============<top>--------------| 11079 * 11080 * bottom > top: Looped-around case: 11081 * <check>--------+------------------+ 11082 * | (overlap here?) | 11083 * L2ARC dev V V 11084 * |===============<top>---------------<bottom>===========| 11085 * ^ ^ 11086 * | (or here?) | 11087 * +---------------+---------<check> 11088 * 11089 * top == bottom : Just a single address comparison. 11090 */ 11091 boolean_t 11092 l2arc_range_check_overlap(uint64_t bottom, uint64_t top, uint64_t check) 11093 { 11094 if (bottom < top) 11095 return (bottom <= check && check <= top); 11096 else if (bottom > top) 11097 return (check <= top || bottom <= check); 11098 else 11099 return (check == top); 11100 } 11101 11102 EXPORT_SYMBOL(arc_buf_size); 11103 EXPORT_SYMBOL(arc_write); 11104 EXPORT_SYMBOL(arc_read); 11105 EXPORT_SYMBOL(arc_buf_info); 11106 EXPORT_SYMBOL(arc_getbuf_func); 11107 EXPORT_SYMBOL(arc_add_prune_callback); 11108 EXPORT_SYMBOL(arc_remove_prune_callback); 11109 11110 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, min, param_set_arc_min, 11111 spl_param_get_u64, ZMOD_RW, "Minimum ARC size in bytes"); 11112 11113 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, max, param_set_arc_max, 11114 spl_param_get_u64, ZMOD_RW, "Maximum ARC size in bytes"); 11115 11116 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, meta_limit, param_set_arc_u64, 11117 spl_param_get_u64, ZMOD_RW, "Metadata limit for ARC size in bytes"); 11118 11119 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, meta_limit_percent, 11120 param_set_arc_int, param_get_uint, ZMOD_RW, 11121 "Percent of ARC size for ARC meta limit"); 11122 11123 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, meta_min, param_set_arc_u64, 11124 spl_param_get_u64, ZMOD_RW, "Minimum ARC metadata size in bytes"); 11125 11126 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, meta_prune, INT, ZMOD_RW, 11127 "Meta objects to scan for prune"); 11128 11129 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, meta_adjust_restarts, UINT, ZMOD_RW, 11130 "Limit number of restarts in arc_evict_meta"); 11131 11132 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, meta_strategy, UINT, ZMOD_RW, 11133 "Meta reclaim strategy"); 11134 11135 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, grow_retry, param_set_arc_int, 11136 param_get_uint, ZMOD_RW, "Seconds before growing ARC size"); 11137 11138 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, p_dampener_disable, INT, ZMOD_RW, 11139 "Disable arc_p adapt dampener"); 11140 11141 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, shrink_shift, param_set_arc_int, 11142 param_get_uint, ZMOD_RW, "log2(fraction of ARC to reclaim)"); 11143 11144 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, pc_percent, UINT, ZMOD_RW, 11145 "Percent of pagecache to reclaim ARC to"); 11146 11147 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, p_min_shift, param_set_arc_int, 11148 param_get_uint, ZMOD_RW, "arc_c shift to calc min/max arc_p"); 11149 11150 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, average_blocksize, UINT, ZMOD_RD, 11151 "Target average block size"); 11152 11153 ZFS_MODULE_PARAM(zfs, zfs_, compressed_arc_enabled, INT, ZMOD_RW, 11154 "Disable compressed ARC buffers"); 11155 11156 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, min_prefetch_ms, param_set_arc_int, 11157 param_get_uint, ZMOD_RW, "Min life of prefetch block in ms"); 11158 11159 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, min_prescient_prefetch_ms, 11160 param_set_arc_int, param_get_uint, ZMOD_RW, 11161 "Min life of prescient prefetched block in ms"); 11162 11163 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, write_max, U64, ZMOD_RW, 11164 "Max write bytes per interval"); 11165 11166 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, write_boost, U64, ZMOD_RW, 11167 "Extra write bytes during device warmup"); 11168 11169 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, headroom, U64, ZMOD_RW, 11170 "Number of max device writes to precache"); 11171 11172 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, headroom_boost, U64, ZMOD_RW, 11173 "Compressed l2arc_headroom multiplier"); 11174 11175 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, trim_ahead, U64, ZMOD_RW, 11176 "TRIM ahead L2ARC write size multiplier"); 11177 11178 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, feed_secs, U64, ZMOD_RW, 11179 "Seconds between L2ARC writing"); 11180 11181 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, feed_min_ms, U64, ZMOD_RW, 11182 "Min feed interval in milliseconds"); 11183 11184 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, noprefetch, INT, ZMOD_RW, 11185 "Skip caching prefetched buffers"); 11186 11187 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, feed_again, INT, ZMOD_RW, 11188 "Turbo L2ARC warmup"); 11189 11190 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, norw, INT, ZMOD_RW, 11191 "No reads during writes"); 11192 11193 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, meta_percent, UINT, ZMOD_RW, 11194 "Percent of ARC size allowed for L2ARC-only headers"); 11195 11196 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, rebuild_enabled, INT, ZMOD_RW, 11197 "Rebuild the L2ARC when importing a pool"); 11198 11199 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, rebuild_blocks_min_l2size, U64, ZMOD_RW, 11200 "Min size in bytes to write rebuild log blocks in L2ARC"); 11201 11202 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, mfuonly, INT, ZMOD_RW, 11203 "Cache only MFU data from ARC into L2ARC"); 11204 11205 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, exclude_special, INT, ZMOD_RW, 11206 "Exclude dbufs on special vdevs from being cached to L2ARC if set."); 11207 11208 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, lotsfree_percent, param_set_arc_int, 11209 param_get_uint, ZMOD_RW, "System free memory I/O throttle in bytes"); 11210 11211 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, sys_free, param_set_arc_u64, 11212 spl_param_get_u64, ZMOD_RW, "System free memory target size in bytes"); 11213 11214 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, dnode_limit, param_set_arc_u64, 11215 spl_param_get_u64, ZMOD_RW, "Minimum bytes of dnodes in ARC"); 11216 11217 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, dnode_limit_percent, 11218 param_set_arc_int, param_get_uint, ZMOD_RW, 11219 "Percent of ARC meta buffers for dnodes"); 11220 11221 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, dnode_reduce_percent, UINT, ZMOD_RW, 11222 "Percentage of excess dnodes to try to unpin"); 11223 11224 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, eviction_pct, UINT, ZMOD_RW, 11225 "When full, ARC allocation waits for eviction of this % of alloc size"); 11226 11227 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, evict_batch_limit, UINT, ZMOD_RW, 11228 "The number of headers to evict per sublist before moving to the next"); 11229 11230 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, prune_task_threads, INT, ZMOD_RW, 11231 "Number of arc_prune threads"); 11232