1 /* 2 * CDDL HEADER START 3 * 4 * The contents of this file are subject to the terms of the 5 * Common Development and Distribution License (the "License"). 6 * You may not use this file except in compliance with the License. 7 * 8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9 * or https://opensource.org/licenses/CDDL-1.0. 10 * See the License for the specific language governing permissions 11 * and limitations under the License. 12 * 13 * When distributing Covered Code, include this CDDL HEADER in each 14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15 * If applicable, add the following below this CDDL HEADER, with the 16 * fields enclosed by brackets "[]" replaced with your own identifying 17 * information: Portions Copyright [yyyy] [name of copyright owner] 18 * 19 * CDDL HEADER END 20 */ 21 /* 22 * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. 23 * Copyright (c) 2018, Joyent, Inc. 24 * Copyright (c) 2011, 2020, Delphix. All rights reserved. 25 * Copyright (c) 2014, Saso Kiselkov. All rights reserved. 26 * Copyright (c) 2017, Nexenta Systems, Inc. All rights reserved. 27 * Copyright (c) 2019, loli10K <ezomori.nozomu@gmail.com>. All rights reserved. 28 * Copyright (c) 2020, George Amanakis. All rights reserved. 29 * Copyright (c) 2019, Klara Inc. 30 * Copyright (c) 2019, Allan Jude 31 * Copyright (c) 2020, The FreeBSD Foundation [1] 32 * 33 * [1] Portions of this software were developed by Allan Jude 34 * under sponsorship from the FreeBSD Foundation. 35 */ 36 37 /* 38 * DVA-based Adjustable Replacement Cache 39 * 40 * While much of the theory of operation used here is 41 * based on the self-tuning, low overhead replacement cache 42 * presented by Megiddo and Modha at FAST 2003, there are some 43 * significant differences: 44 * 45 * 1. The Megiddo and Modha model assumes any page is evictable. 46 * Pages in its cache cannot be "locked" into memory. This makes 47 * the eviction algorithm simple: evict the last page in the list. 48 * This also make the performance characteristics easy to reason 49 * about. Our cache is not so simple. At any given moment, some 50 * subset of the blocks in the cache are un-evictable because we 51 * have handed out a reference to them. Blocks are only evictable 52 * when there are no external references active. This makes 53 * eviction far more problematic: we choose to evict the evictable 54 * blocks that are the "lowest" in the list. 55 * 56 * There are times when it is not possible to evict the requested 57 * space. In these circumstances we are unable to adjust the cache 58 * size. To prevent the cache growing unbounded at these times we 59 * implement a "cache throttle" that slows the flow of new data 60 * into the cache until we can make space available. 61 * 62 * 2. The Megiddo and Modha model assumes a fixed cache size. 63 * Pages are evicted when the cache is full and there is a cache 64 * miss. Our model has a variable sized cache. It grows with 65 * high use, but also tries to react to memory pressure from the 66 * operating system: decreasing its size when system memory is 67 * tight. 68 * 69 * 3. The Megiddo and Modha model assumes a fixed page size. All 70 * elements of the cache are therefore exactly the same size. So 71 * when adjusting the cache size following a cache miss, its simply 72 * a matter of choosing a single page to evict. In our model, we 73 * have variable sized cache blocks (ranging from 512 bytes to 74 * 128K bytes). We therefore choose a set of blocks to evict to make 75 * space for a cache miss that approximates as closely as possible 76 * the space used by the new block. 77 * 78 * See also: "ARC: A Self-Tuning, Low Overhead Replacement Cache" 79 * by N. Megiddo & D. Modha, FAST 2003 80 */ 81 82 /* 83 * The locking model: 84 * 85 * A new reference to a cache buffer can be obtained in two 86 * ways: 1) via a hash table lookup using the DVA as a key, 87 * or 2) via one of the ARC lists. The arc_read() interface 88 * uses method 1, while the internal ARC algorithms for 89 * adjusting the cache use method 2. We therefore provide two 90 * types of locks: 1) the hash table lock array, and 2) the 91 * ARC list locks. 92 * 93 * Buffers do not have their own mutexes, rather they rely on the 94 * hash table mutexes for the bulk of their protection (i.e. most 95 * fields in the arc_buf_hdr_t are protected by these mutexes). 96 * 97 * buf_hash_find() returns the appropriate mutex (held) when it 98 * locates the requested buffer in the hash table. It returns 99 * NULL for the mutex if the buffer was not in the table. 100 * 101 * buf_hash_remove() expects the appropriate hash mutex to be 102 * already held before it is invoked. 103 * 104 * Each ARC state also has a mutex which is used to protect the 105 * buffer list associated with the state. When attempting to 106 * obtain a hash table lock while holding an ARC list lock you 107 * must use: mutex_tryenter() to avoid deadlock. Also note that 108 * the active state mutex must be held before the ghost state mutex. 109 * 110 * It as also possible to register a callback which is run when the 111 * metadata limit is reached and no buffers can be safely evicted. In 112 * this case the arc user should drop a reference on some arc buffers so 113 * they can be reclaimed. For example, when using the ZPL each dentry 114 * holds a references on a znode. These dentries must be pruned before 115 * the arc buffer holding the znode can be safely evicted. 116 * 117 * Note that the majority of the performance stats are manipulated 118 * with atomic operations. 119 * 120 * The L2ARC uses the l2ad_mtx on each vdev for the following: 121 * 122 * - L2ARC buflist creation 123 * - L2ARC buflist eviction 124 * - L2ARC write completion, which walks L2ARC buflists 125 * - ARC header destruction, as it removes from L2ARC buflists 126 * - ARC header release, as it removes from L2ARC buflists 127 */ 128 129 /* 130 * ARC operation: 131 * 132 * Every block that is in the ARC is tracked by an arc_buf_hdr_t structure. 133 * This structure can point either to a block that is still in the cache or to 134 * one that is only accessible in an L2 ARC device, or it can provide 135 * information about a block that was recently evicted. If a block is 136 * only accessible in the L2ARC, then the arc_buf_hdr_t only has enough 137 * information to retrieve it from the L2ARC device. This information is 138 * stored in the l2arc_buf_hdr_t sub-structure of the arc_buf_hdr_t. A block 139 * that is in this state cannot access the data directly. 140 * 141 * Blocks that are actively being referenced or have not been evicted 142 * are cached in the L1ARC. The L1ARC (l1arc_buf_hdr_t) is a structure within 143 * the arc_buf_hdr_t that will point to the data block in memory. A block can 144 * only be read by a consumer if it has an l1arc_buf_hdr_t. The L1ARC 145 * caches data in two ways -- in a list of ARC buffers (arc_buf_t) and 146 * also in the arc_buf_hdr_t's private physical data block pointer (b_pabd). 147 * 148 * The L1ARC's data pointer may or may not be uncompressed. The ARC has the 149 * ability to store the physical data (b_pabd) associated with the DVA of the 150 * arc_buf_hdr_t. Since the b_pabd is a copy of the on-disk physical block, 151 * it will match its on-disk compression characteristics. This behavior can be 152 * disabled by setting 'zfs_compressed_arc_enabled' to B_FALSE. When the 153 * compressed ARC functionality is disabled, the b_pabd will point to an 154 * uncompressed version of the on-disk data. 155 * 156 * Data in the L1ARC is not accessed by consumers of the ARC directly. Each 157 * arc_buf_hdr_t can have multiple ARC buffers (arc_buf_t) which reference it. 158 * Each ARC buffer (arc_buf_t) is being actively accessed by a specific ARC 159 * consumer. The ARC will provide references to this data and will keep it 160 * cached until it is no longer in use. The ARC caches only the L1ARC's physical 161 * data block and will evict any arc_buf_t that is no longer referenced. The 162 * amount of memory consumed by the arc_buf_ts' data buffers can be seen via the 163 * "overhead_size" kstat. 164 * 165 * Depending on the consumer, an arc_buf_t can be requested in uncompressed or 166 * compressed form. The typical case is that consumers will want uncompressed 167 * data, and when that happens a new data buffer is allocated where the data is 168 * decompressed for them to use. Currently the only consumer who wants 169 * compressed arc_buf_t's is "zfs send", when it streams data exactly as it 170 * exists on disk. When this happens, the arc_buf_t's data buffer is shared 171 * with the arc_buf_hdr_t. 172 * 173 * Here is a diagram showing an arc_buf_hdr_t referenced by two arc_buf_t's. The 174 * first one is owned by a compressed send consumer (and therefore references 175 * the same compressed data buffer as the arc_buf_hdr_t) and the second could be 176 * used by any other consumer (and has its own uncompressed copy of the data 177 * buffer). 178 * 179 * arc_buf_hdr_t 180 * +-----------+ 181 * | fields | 182 * | common to | 183 * | L1- and | 184 * | L2ARC | 185 * +-----------+ 186 * | l2arc_buf_hdr_t 187 * | | 188 * +-----------+ 189 * | l1arc_buf_hdr_t 190 * | | arc_buf_t 191 * | b_buf +------------>+-----------+ arc_buf_t 192 * | b_pabd +-+ |b_next +---->+-----------+ 193 * +-----------+ | |-----------| |b_next +-->NULL 194 * | |b_comp = T | +-----------+ 195 * | |b_data +-+ |b_comp = F | 196 * | +-----------+ | |b_data +-+ 197 * +->+------+ | +-----------+ | 198 * compressed | | | | 199 * data | |<--------------+ | uncompressed 200 * +------+ compressed, | data 201 * shared +-->+------+ 202 * data | | 203 * | | 204 * +------+ 205 * 206 * When a consumer reads a block, the ARC must first look to see if the 207 * arc_buf_hdr_t is cached. If the hdr is cached then the ARC allocates a new 208 * arc_buf_t and either copies uncompressed data into a new data buffer from an 209 * existing uncompressed arc_buf_t, decompresses the hdr's b_pabd buffer into a 210 * new data buffer, or shares the hdr's b_pabd buffer, depending on whether the 211 * hdr is compressed and the desired compression characteristics of the 212 * arc_buf_t consumer. If the arc_buf_t ends up sharing data with the 213 * arc_buf_hdr_t and both of them are uncompressed then the arc_buf_t must be 214 * the last buffer in the hdr's b_buf list, however a shared compressed buf can 215 * be anywhere in the hdr's list. 216 * 217 * The diagram below shows an example of an uncompressed ARC hdr that is 218 * sharing its data with an arc_buf_t (note that the shared uncompressed buf is 219 * the last element in the buf list): 220 * 221 * arc_buf_hdr_t 222 * +-----------+ 223 * | | 224 * | | 225 * | | 226 * +-----------+ 227 * l2arc_buf_hdr_t| | 228 * | | 229 * +-----------+ 230 * l1arc_buf_hdr_t| | 231 * | | arc_buf_t (shared) 232 * | b_buf +------------>+---------+ arc_buf_t 233 * | | |b_next +---->+---------+ 234 * | b_pabd +-+ |---------| |b_next +-->NULL 235 * +-----------+ | | | +---------+ 236 * | |b_data +-+ | | 237 * | +---------+ | |b_data +-+ 238 * +->+------+ | +---------+ | 239 * | | | | 240 * uncompressed | | | | 241 * data +------+ | | 242 * ^ +->+------+ | 243 * | uncompressed | | | 244 * | data | | | 245 * | +------+ | 246 * +---------------------------------+ 247 * 248 * Writing to the ARC requires that the ARC first discard the hdr's b_pabd 249 * since the physical block is about to be rewritten. The new data contents 250 * will be contained in the arc_buf_t. As the I/O pipeline performs the write, 251 * it may compress the data before writing it to disk. The ARC will be called 252 * with the transformed data and will memcpy the transformed on-disk block into 253 * a newly allocated b_pabd. Writes are always done into buffers which have 254 * either been loaned (and hence are new and don't have other readers) or 255 * buffers which have been released (and hence have their own hdr, if there 256 * were originally other readers of the buf's original hdr). This ensures that 257 * the ARC only needs to update a single buf and its hdr after a write occurs. 258 * 259 * When the L2ARC is in use, it will also take advantage of the b_pabd. The 260 * L2ARC will always write the contents of b_pabd to the L2ARC. This means 261 * that when compressed ARC is enabled that the L2ARC blocks are identical 262 * to the on-disk block in the main data pool. This provides a significant 263 * advantage since the ARC can leverage the bp's checksum when reading from the 264 * L2ARC to determine if the contents are valid. However, if the compressed 265 * ARC is disabled, then the L2ARC's block must be transformed to look 266 * like the physical block in the main data pool before comparing the 267 * checksum and determining its validity. 268 * 269 * The L1ARC has a slightly different system for storing encrypted data. 270 * Raw (encrypted + possibly compressed) data has a few subtle differences from 271 * data that is just compressed. The biggest difference is that it is not 272 * possible to decrypt encrypted data (or vice-versa) if the keys aren't loaded. 273 * The other difference is that encryption cannot be treated as a suggestion. 274 * If a caller would prefer compressed data, but they actually wind up with 275 * uncompressed data the worst thing that could happen is there might be a 276 * performance hit. If the caller requests encrypted data, however, we must be 277 * sure they actually get it or else secret information could be leaked. Raw 278 * data is stored in hdr->b_crypt_hdr.b_rabd. An encrypted header, therefore, 279 * may have both an encrypted version and a decrypted version of its data at 280 * once. When a caller needs a raw arc_buf_t, it is allocated and the data is 281 * copied out of this header. To avoid complications with b_pabd, raw buffers 282 * cannot be shared. 283 */ 284 285 #include <sys/spa.h> 286 #include <sys/zio.h> 287 #include <sys/spa_impl.h> 288 #include <sys/zio_compress.h> 289 #include <sys/zio_checksum.h> 290 #include <sys/zfs_context.h> 291 #include <sys/arc.h> 292 #include <sys/zfs_refcount.h> 293 #include <sys/vdev.h> 294 #include <sys/vdev_impl.h> 295 #include <sys/dsl_pool.h> 296 #include <sys/multilist.h> 297 #include <sys/abd.h> 298 #include <sys/zil.h> 299 #include <sys/fm/fs/zfs.h> 300 #include <sys/callb.h> 301 #include <sys/kstat.h> 302 #include <sys/zthr.h> 303 #include <zfs_fletcher.h> 304 #include <sys/arc_impl.h> 305 #include <sys/trace_zfs.h> 306 #include <sys/aggsum.h> 307 #include <sys/wmsum.h> 308 #include <cityhash.h> 309 #include <sys/vdev_trim.h> 310 #include <sys/zfs_racct.h> 311 #include <sys/zstd/zstd.h> 312 313 #ifndef _KERNEL 314 /* set with ZFS_DEBUG=watch, to enable watchpoints on frozen buffers */ 315 boolean_t arc_watch = B_FALSE; 316 #endif 317 318 /* 319 * This thread's job is to keep enough free memory in the system, by 320 * calling arc_kmem_reap_soon() plus arc_reduce_target_size(), which improves 321 * arc_available_memory(). 322 */ 323 static zthr_t *arc_reap_zthr; 324 325 /* 326 * This thread's job is to keep arc_size under arc_c, by calling 327 * arc_evict(), which improves arc_is_overflowing(). 328 */ 329 static zthr_t *arc_evict_zthr; 330 static arc_buf_hdr_t **arc_state_evict_markers; 331 static int arc_state_evict_marker_count; 332 333 static kmutex_t arc_evict_lock; 334 static boolean_t arc_evict_needed = B_FALSE; 335 static clock_t arc_last_uncached_flush; 336 337 /* 338 * Count of bytes evicted since boot. 339 */ 340 static uint64_t arc_evict_count; 341 342 /* 343 * List of arc_evict_waiter_t's, representing threads waiting for the 344 * arc_evict_count to reach specific values. 345 */ 346 static list_t arc_evict_waiters; 347 348 /* 349 * When arc_is_overflowing(), arc_get_data_impl() waits for this percent of 350 * the requested amount of data to be evicted. For example, by default for 351 * every 2KB that's evicted, 1KB of it may be "reused" by a new allocation. 352 * Since this is above 100%, it ensures that progress is made towards getting 353 * arc_size under arc_c. Since this is finite, it ensures that allocations 354 * can still happen, even during the potentially long time that arc_size is 355 * more than arc_c. 356 */ 357 static uint_t zfs_arc_eviction_pct = 200; 358 359 /* 360 * The number of headers to evict in arc_evict_state_impl() before 361 * dropping the sublist lock and evicting from another sublist. A lower 362 * value means we're more likely to evict the "correct" header (i.e. the 363 * oldest header in the arc state), but comes with higher overhead 364 * (i.e. more invocations of arc_evict_state_impl()). 365 */ 366 static uint_t zfs_arc_evict_batch_limit = 10; 367 368 /* number of seconds before growing cache again */ 369 uint_t arc_grow_retry = 5; 370 371 /* 372 * Minimum time between calls to arc_kmem_reap_soon(). 373 */ 374 static const int arc_kmem_cache_reap_retry_ms = 1000; 375 376 /* shift of arc_c for calculating overflow limit in arc_get_data_impl */ 377 static int zfs_arc_overflow_shift = 8; 378 379 /* log2(fraction of arc to reclaim) */ 380 uint_t arc_shrink_shift = 7; 381 382 /* percent of pagecache to reclaim arc to */ 383 #ifdef _KERNEL 384 uint_t zfs_arc_pc_percent = 0; 385 #endif 386 387 /* 388 * log2(fraction of ARC which must be free to allow growing). 389 * I.e. If there is less than arc_c >> arc_no_grow_shift free memory, 390 * when reading a new block into the ARC, we will evict an equal-sized block 391 * from the ARC. 392 * 393 * This must be less than arc_shrink_shift, so that when we shrink the ARC, 394 * we will still not allow it to grow. 395 */ 396 uint_t arc_no_grow_shift = 5; 397 398 399 /* 400 * minimum lifespan of a prefetch block in clock ticks 401 * (initialized in arc_init()) 402 */ 403 static uint_t arc_min_prefetch_ms; 404 static uint_t arc_min_prescient_prefetch_ms; 405 406 /* 407 * If this percent of memory is free, don't throttle. 408 */ 409 uint_t arc_lotsfree_percent = 10; 410 411 /* 412 * The arc has filled available memory and has now warmed up. 413 */ 414 boolean_t arc_warm; 415 416 /* 417 * These tunables are for performance analysis. 418 */ 419 uint64_t zfs_arc_max = 0; 420 uint64_t zfs_arc_min = 0; 421 static uint64_t zfs_arc_dnode_limit = 0; 422 static uint_t zfs_arc_dnode_reduce_percent = 10; 423 static uint_t zfs_arc_grow_retry = 0; 424 static uint_t zfs_arc_shrink_shift = 0; 425 uint_t zfs_arc_average_blocksize = 8 * 1024; /* 8KB */ 426 427 /* 428 * ARC dirty data constraints for arc_tempreserve_space() throttle: 429 * * total dirty data limit 430 * * anon block dirty limit 431 * * each pool's anon allowance 432 */ 433 static const unsigned long zfs_arc_dirty_limit_percent = 50; 434 static const unsigned long zfs_arc_anon_limit_percent = 25; 435 static const unsigned long zfs_arc_pool_dirty_percent = 20; 436 437 /* 438 * Enable or disable compressed arc buffers. 439 */ 440 int zfs_compressed_arc_enabled = B_TRUE; 441 442 /* 443 * Balance between metadata and data on ghost hits. Values above 100 444 * increase metadata caching by proportionally reducing effect of ghost 445 * data hits on target data/metadata rate. 446 */ 447 static uint_t zfs_arc_meta_balance = 500; 448 449 /* 450 * Percentage that can be consumed by dnodes of ARC meta buffers. 451 */ 452 static uint_t zfs_arc_dnode_limit_percent = 10; 453 454 /* 455 * These tunables are Linux-specific 456 */ 457 static uint64_t zfs_arc_sys_free = 0; 458 static uint_t zfs_arc_min_prefetch_ms = 0; 459 static uint_t zfs_arc_min_prescient_prefetch_ms = 0; 460 static uint_t zfs_arc_lotsfree_percent = 10; 461 462 /* 463 * Number of arc_prune threads 464 */ 465 static int zfs_arc_prune_task_threads = 1; 466 467 /* The 7 states: */ 468 arc_state_t ARC_anon; 469 arc_state_t ARC_mru; 470 arc_state_t ARC_mru_ghost; 471 arc_state_t ARC_mfu; 472 arc_state_t ARC_mfu_ghost; 473 arc_state_t ARC_l2c_only; 474 arc_state_t ARC_uncached; 475 476 arc_stats_t arc_stats = { 477 { "hits", KSTAT_DATA_UINT64 }, 478 { "iohits", KSTAT_DATA_UINT64 }, 479 { "misses", KSTAT_DATA_UINT64 }, 480 { "demand_data_hits", KSTAT_DATA_UINT64 }, 481 { "demand_data_iohits", KSTAT_DATA_UINT64 }, 482 { "demand_data_misses", KSTAT_DATA_UINT64 }, 483 { "demand_metadata_hits", KSTAT_DATA_UINT64 }, 484 { "demand_metadata_iohits", KSTAT_DATA_UINT64 }, 485 { "demand_metadata_misses", KSTAT_DATA_UINT64 }, 486 { "prefetch_data_hits", KSTAT_DATA_UINT64 }, 487 { "prefetch_data_iohits", KSTAT_DATA_UINT64 }, 488 { "prefetch_data_misses", KSTAT_DATA_UINT64 }, 489 { "prefetch_metadata_hits", KSTAT_DATA_UINT64 }, 490 { "prefetch_metadata_iohits", KSTAT_DATA_UINT64 }, 491 { "prefetch_metadata_misses", KSTAT_DATA_UINT64 }, 492 { "mru_hits", KSTAT_DATA_UINT64 }, 493 { "mru_ghost_hits", KSTAT_DATA_UINT64 }, 494 { "mfu_hits", KSTAT_DATA_UINT64 }, 495 { "mfu_ghost_hits", KSTAT_DATA_UINT64 }, 496 { "uncached_hits", KSTAT_DATA_UINT64 }, 497 { "deleted", KSTAT_DATA_UINT64 }, 498 { "mutex_miss", KSTAT_DATA_UINT64 }, 499 { "access_skip", KSTAT_DATA_UINT64 }, 500 { "evict_skip", KSTAT_DATA_UINT64 }, 501 { "evict_not_enough", KSTAT_DATA_UINT64 }, 502 { "evict_l2_cached", KSTAT_DATA_UINT64 }, 503 { "evict_l2_eligible", KSTAT_DATA_UINT64 }, 504 { "evict_l2_eligible_mfu", KSTAT_DATA_UINT64 }, 505 { "evict_l2_eligible_mru", KSTAT_DATA_UINT64 }, 506 { "evict_l2_ineligible", KSTAT_DATA_UINT64 }, 507 { "evict_l2_skip", KSTAT_DATA_UINT64 }, 508 { "hash_elements", KSTAT_DATA_UINT64 }, 509 { "hash_elements_max", KSTAT_DATA_UINT64 }, 510 { "hash_collisions", KSTAT_DATA_UINT64 }, 511 { "hash_chains", KSTAT_DATA_UINT64 }, 512 { "hash_chain_max", KSTAT_DATA_UINT64 }, 513 { "meta", KSTAT_DATA_UINT64 }, 514 { "pd", KSTAT_DATA_UINT64 }, 515 { "pm", KSTAT_DATA_UINT64 }, 516 { "c", KSTAT_DATA_UINT64 }, 517 { "c_min", KSTAT_DATA_UINT64 }, 518 { "c_max", KSTAT_DATA_UINT64 }, 519 { "size", KSTAT_DATA_UINT64 }, 520 { "compressed_size", KSTAT_DATA_UINT64 }, 521 { "uncompressed_size", KSTAT_DATA_UINT64 }, 522 { "overhead_size", KSTAT_DATA_UINT64 }, 523 { "hdr_size", KSTAT_DATA_UINT64 }, 524 { "data_size", KSTAT_DATA_UINT64 }, 525 { "metadata_size", KSTAT_DATA_UINT64 }, 526 { "dbuf_size", KSTAT_DATA_UINT64 }, 527 { "dnode_size", KSTAT_DATA_UINT64 }, 528 { "bonus_size", KSTAT_DATA_UINT64 }, 529 #if defined(COMPAT_FREEBSD11) 530 { "other_size", KSTAT_DATA_UINT64 }, 531 #endif 532 { "anon_size", KSTAT_DATA_UINT64 }, 533 { "anon_data", KSTAT_DATA_UINT64 }, 534 { "anon_metadata", KSTAT_DATA_UINT64 }, 535 { "anon_evictable_data", KSTAT_DATA_UINT64 }, 536 { "anon_evictable_metadata", KSTAT_DATA_UINT64 }, 537 { "mru_size", KSTAT_DATA_UINT64 }, 538 { "mru_data", KSTAT_DATA_UINT64 }, 539 { "mru_metadata", KSTAT_DATA_UINT64 }, 540 { "mru_evictable_data", KSTAT_DATA_UINT64 }, 541 { "mru_evictable_metadata", KSTAT_DATA_UINT64 }, 542 { "mru_ghost_size", KSTAT_DATA_UINT64 }, 543 { "mru_ghost_data", KSTAT_DATA_UINT64 }, 544 { "mru_ghost_metadata", KSTAT_DATA_UINT64 }, 545 { "mru_ghost_evictable_data", KSTAT_DATA_UINT64 }, 546 { "mru_ghost_evictable_metadata", KSTAT_DATA_UINT64 }, 547 { "mfu_size", KSTAT_DATA_UINT64 }, 548 { "mfu_data", KSTAT_DATA_UINT64 }, 549 { "mfu_metadata", KSTAT_DATA_UINT64 }, 550 { "mfu_evictable_data", KSTAT_DATA_UINT64 }, 551 { "mfu_evictable_metadata", KSTAT_DATA_UINT64 }, 552 { "mfu_ghost_size", KSTAT_DATA_UINT64 }, 553 { "mfu_ghost_data", KSTAT_DATA_UINT64 }, 554 { "mfu_ghost_metadata", KSTAT_DATA_UINT64 }, 555 { "mfu_ghost_evictable_data", KSTAT_DATA_UINT64 }, 556 { "mfu_ghost_evictable_metadata", KSTAT_DATA_UINT64 }, 557 { "uncached_size", KSTAT_DATA_UINT64 }, 558 { "uncached_data", KSTAT_DATA_UINT64 }, 559 { "uncached_metadata", KSTAT_DATA_UINT64 }, 560 { "uncached_evictable_data", KSTAT_DATA_UINT64 }, 561 { "uncached_evictable_metadata", KSTAT_DATA_UINT64 }, 562 { "l2_hits", KSTAT_DATA_UINT64 }, 563 { "l2_misses", KSTAT_DATA_UINT64 }, 564 { "l2_prefetch_asize", KSTAT_DATA_UINT64 }, 565 { "l2_mru_asize", KSTAT_DATA_UINT64 }, 566 { "l2_mfu_asize", KSTAT_DATA_UINT64 }, 567 { "l2_bufc_data_asize", KSTAT_DATA_UINT64 }, 568 { "l2_bufc_metadata_asize", KSTAT_DATA_UINT64 }, 569 { "l2_feeds", KSTAT_DATA_UINT64 }, 570 { "l2_rw_clash", KSTAT_DATA_UINT64 }, 571 { "l2_read_bytes", KSTAT_DATA_UINT64 }, 572 { "l2_write_bytes", KSTAT_DATA_UINT64 }, 573 { "l2_writes_sent", KSTAT_DATA_UINT64 }, 574 { "l2_writes_done", KSTAT_DATA_UINT64 }, 575 { "l2_writes_error", KSTAT_DATA_UINT64 }, 576 { "l2_writes_lock_retry", KSTAT_DATA_UINT64 }, 577 { "l2_evict_lock_retry", KSTAT_DATA_UINT64 }, 578 { "l2_evict_reading", KSTAT_DATA_UINT64 }, 579 { "l2_evict_l1cached", KSTAT_DATA_UINT64 }, 580 { "l2_free_on_write", KSTAT_DATA_UINT64 }, 581 { "l2_abort_lowmem", KSTAT_DATA_UINT64 }, 582 { "l2_cksum_bad", KSTAT_DATA_UINT64 }, 583 { "l2_io_error", KSTAT_DATA_UINT64 }, 584 { "l2_size", KSTAT_DATA_UINT64 }, 585 { "l2_asize", KSTAT_DATA_UINT64 }, 586 { "l2_hdr_size", KSTAT_DATA_UINT64 }, 587 { "l2_log_blk_writes", KSTAT_DATA_UINT64 }, 588 { "l2_log_blk_avg_asize", KSTAT_DATA_UINT64 }, 589 { "l2_log_blk_asize", KSTAT_DATA_UINT64 }, 590 { "l2_log_blk_count", KSTAT_DATA_UINT64 }, 591 { "l2_data_to_meta_ratio", KSTAT_DATA_UINT64 }, 592 { "l2_rebuild_success", KSTAT_DATA_UINT64 }, 593 { "l2_rebuild_unsupported", KSTAT_DATA_UINT64 }, 594 { "l2_rebuild_io_errors", KSTAT_DATA_UINT64 }, 595 { "l2_rebuild_dh_errors", KSTAT_DATA_UINT64 }, 596 { "l2_rebuild_cksum_lb_errors", KSTAT_DATA_UINT64 }, 597 { "l2_rebuild_lowmem", KSTAT_DATA_UINT64 }, 598 { "l2_rebuild_size", KSTAT_DATA_UINT64 }, 599 { "l2_rebuild_asize", KSTAT_DATA_UINT64 }, 600 { "l2_rebuild_bufs", KSTAT_DATA_UINT64 }, 601 { "l2_rebuild_bufs_precached", KSTAT_DATA_UINT64 }, 602 { "l2_rebuild_log_blks", KSTAT_DATA_UINT64 }, 603 { "memory_throttle_count", KSTAT_DATA_UINT64 }, 604 { "memory_direct_count", KSTAT_DATA_UINT64 }, 605 { "memory_indirect_count", KSTAT_DATA_UINT64 }, 606 { "memory_all_bytes", KSTAT_DATA_UINT64 }, 607 { "memory_free_bytes", KSTAT_DATA_UINT64 }, 608 { "memory_available_bytes", KSTAT_DATA_INT64 }, 609 { "arc_no_grow", KSTAT_DATA_UINT64 }, 610 { "arc_tempreserve", KSTAT_DATA_UINT64 }, 611 { "arc_loaned_bytes", KSTAT_DATA_UINT64 }, 612 { "arc_prune", KSTAT_DATA_UINT64 }, 613 { "arc_meta_used", KSTAT_DATA_UINT64 }, 614 { "arc_dnode_limit", KSTAT_DATA_UINT64 }, 615 { "async_upgrade_sync", KSTAT_DATA_UINT64 }, 616 { "predictive_prefetch", KSTAT_DATA_UINT64 }, 617 { "demand_hit_predictive_prefetch", KSTAT_DATA_UINT64 }, 618 { "demand_iohit_predictive_prefetch", KSTAT_DATA_UINT64 }, 619 { "prescient_prefetch", KSTAT_DATA_UINT64 }, 620 { "demand_hit_prescient_prefetch", KSTAT_DATA_UINT64 }, 621 { "demand_iohit_prescient_prefetch", KSTAT_DATA_UINT64 }, 622 { "arc_need_free", KSTAT_DATA_UINT64 }, 623 { "arc_sys_free", KSTAT_DATA_UINT64 }, 624 { "arc_raw_size", KSTAT_DATA_UINT64 }, 625 { "cached_only_in_progress", KSTAT_DATA_UINT64 }, 626 { "abd_chunk_waste_size", KSTAT_DATA_UINT64 }, 627 }; 628 629 arc_sums_t arc_sums; 630 631 #define ARCSTAT_MAX(stat, val) { \ 632 uint64_t m; \ 633 while ((val) > (m = arc_stats.stat.value.ui64) && \ 634 (m != atomic_cas_64(&arc_stats.stat.value.ui64, m, (val)))) \ 635 continue; \ 636 } 637 638 /* 639 * We define a macro to allow ARC hits/misses to be easily broken down by 640 * two separate conditions, giving a total of four different subtypes for 641 * each of hits and misses (so eight statistics total). 642 */ 643 #define ARCSTAT_CONDSTAT(cond1, stat1, notstat1, cond2, stat2, notstat2, stat) \ 644 if (cond1) { \ 645 if (cond2) { \ 646 ARCSTAT_BUMP(arcstat_##stat1##_##stat2##_##stat); \ 647 } else { \ 648 ARCSTAT_BUMP(arcstat_##stat1##_##notstat2##_##stat); \ 649 } \ 650 } else { \ 651 if (cond2) { \ 652 ARCSTAT_BUMP(arcstat_##notstat1##_##stat2##_##stat); \ 653 } else { \ 654 ARCSTAT_BUMP(arcstat_##notstat1##_##notstat2##_##stat);\ 655 } \ 656 } 657 658 /* 659 * This macro allows us to use kstats as floating averages. Each time we 660 * update this kstat, we first factor it and the update value by 661 * ARCSTAT_AVG_FACTOR to shrink the new value's contribution to the overall 662 * average. This macro assumes that integer loads and stores are atomic, but 663 * is not safe for multiple writers updating the kstat in parallel (only the 664 * last writer's update will remain). 665 */ 666 #define ARCSTAT_F_AVG_FACTOR 3 667 #define ARCSTAT_F_AVG(stat, value) \ 668 do { \ 669 uint64_t x = ARCSTAT(stat); \ 670 x = x - x / ARCSTAT_F_AVG_FACTOR + \ 671 (value) / ARCSTAT_F_AVG_FACTOR; \ 672 ARCSTAT(stat) = x; \ 673 } while (0) 674 675 static kstat_t *arc_ksp; 676 677 /* 678 * There are several ARC variables that are critical to export as kstats -- 679 * but we don't want to have to grovel around in the kstat whenever we wish to 680 * manipulate them. For these variables, we therefore define them to be in 681 * terms of the statistic variable. This assures that we are not introducing 682 * the possibility of inconsistency by having shadow copies of the variables, 683 * while still allowing the code to be readable. 684 */ 685 #define arc_tempreserve ARCSTAT(arcstat_tempreserve) 686 #define arc_loaned_bytes ARCSTAT(arcstat_loaned_bytes) 687 #define arc_dnode_limit ARCSTAT(arcstat_dnode_limit) /* max size for dnodes */ 688 #define arc_need_free ARCSTAT(arcstat_need_free) /* waiting to be evicted */ 689 690 hrtime_t arc_growtime; 691 list_t arc_prune_list; 692 kmutex_t arc_prune_mtx; 693 taskq_t *arc_prune_taskq; 694 695 #define GHOST_STATE(state) \ 696 ((state) == arc_mru_ghost || (state) == arc_mfu_ghost || \ 697 (state) == arc_l2c_only) 698 699 #define HDR_IN_HASH_TABLE(hdr) ((hdr)->b_flags & ARC_FLAG_IN_HASH_TABLE) 700 #define HDR_IO_IN_PROGRESS(hdr) ((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS) 701 #define HDR_IO_ERROR(hdr) ((hdr)->b_flags & ARC_FLAG_IO_ERROR) 702 #define HDR_PREFETCH(hdr) ((hdr)->b_flags & ARC_FLAG_PREFETCH) 703 #define HDR_PRESCIENT_PREFETCH(hdr) \ 704 ((hdr)->b_flags & ARC_FLAG_PRESCIENT_PREFETCH) 705 #define HDR_COMPRESSION_ENABLED(hdr) \ 706 ((hdr)->b_flags & ARC_FLAG_COMPRESSED_ARC) 707 708 #define HDR_L2CACHE(hdr) ((hdr)->b_flags & ARC_FLAG_L2CACHE) 709 #define HDR_UNCACHED(hdr) ((hdr)->b_flags & ARC_FLAG_UNCACHED) 710 #define HDR_L2_READING(hdr) \ 711 (((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS) && \ 712 ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR)) 713 #define HDR_L2_WRITING(hdr) ((hdr)->b_flags & ARC_FLAG_L2_WRITING) 714 #define HDR_L2_EVICTED(hdr) ((hdr)->b_flags & ARC_FLAG_L2_EVICTED) 715 #define HDR_L2_WRITE_HEAD(hdr) ((hdr)->b_flags & ARC_FLAG_L2_WRITE_HEAD) 716 #define HDR_PROTECTED(hdr) ((hdr)->b_flags & ARC_FLAG_PROTECTED) 717 #define HDR_NOAUTH(hdr) ((hdr)->b_flags & ARC_FLAG_NOAUTH) 718 #define HDR_SHARED_DATA(hdr) ((hdr)->b_flags & ARC_FLAG_SHARED_DATA) 719 720 #define HDR_ISTYPE_METADATA(hdr) \ 721 ((hdr)->b_flags & ARC_FLAG_BUFC_METADATA) 722 #define HDR_ISTYPE_DATA(hdr) (!HDR_ISTYPE_METADATA(hdr)) 723 724 #define HDR_HAS_L1HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L1HDR) 725 #define HDR_HAS_L2HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR) 726 #define HDR_HAS_RABD(hdr) \ 727 (HDR_HAS_L1HDR(hdr) && HDR_PROTECTED(hdr) && \ 728 (hdr)->b_crypt_hdr.b_rabd != NULL) 729 #define HDR_ENCRYPTED(hdr) \ 730 (HDR_PROTECTED(hdr) && DMU_OT_IS_ENCRYPTED((hdr)->b_crypt_hdr.b_ot)) 731 #define HDR_AUTHENTICATED(hdr) \ 732 (HDR_PROTECTED(hdr) && !DMU_OT_IS_ENCRYPTED((hdr)->b_crypt_hdr.b_ot)) 733 734 /* For storing compression mode in b_flags */ 735 #define HDR_COMPRESS_OFFSET (highbit64(ARC_FLAG_COMPRESS_0) - 1) 736 737 #define HDR_GET_COMPRESS(hdr) ((enum zio_compress)BF32_GET((hdr)->b_flags, \ 738 HDR_COMPRESS_OFFSET, SPA_COMPRESSBITS)) 739 #define HDR_SET_COMPRESS(hdr, cmp) BF32_SET((hdr)->b_flags, \ 740 HDR_COMPRESS_OFFSET, SPA_COMPRESSBITS, (cmp)); 741 742 #define ARC_BUF_LAST(buf) ((buf)->b_next == NULL) 743 #define ARC_BUF_SHARED(buf) ((buf)->b_flags & ARC_BUF_FLAG_SHARED) 744 #define ARC_BUF_COMPRESSED(buf) ((buf)->b_flags & ARC_BUF_FLAG_COMPRESSED) 745 #define ARC_BUF_ENCRYPTED(buf) ((buf)->b_flags & ARC_BUF_FLAG_ENCRYPTED) 746 747 /* 748 * Other sizes 749 */ 750 751 #define HDR_FULL_SIZE ((int64_t)sizeof (arc_buf_hdr_t)) 752 #define HDR_L2ONLY_SIZE ((int64_t)offsetof(arc_buf_hdr_t, b_l1hdr)) 753 754 /* 755 * Hash table routines 756 */ 757 758 #define BUF_LOCKS 2048 759 typedef struct buf_hash_table { 760 uint64_t ht_mask; 761 arc_buf_hdr_t **ht_table; 762 kmutex_t ht_locks[BUF_LOCKS] ____cacheline_aligned; 763 } buf_hash_table_t; 764 765 static buf_hash_table_t buf_hash_table; 766 767 #define BUF_HASH_INDEX(spa, dva, birth) \ 768 (buf_hash(spa, dva, birth) & buf_hash_table.ht_mask) 769 #define BUF_HASH_LOCK(idx) (&buf_hash_table.ht_locks[idx & (BUF_LOCKS-1)]) 770 #define HDR_LOCK(hdr) \ 771 (BUF_HASH_LOCK(BUF_HASH_INDEX(hdr->b_spa, &hdr->b_dva, hdr->b_birth))) 772 773 uint64_t zfs_crc64_table[256]; 774 775 /* 776 * Level 2 ARC 777 */ 778 779 #define L2ARC_WRITE_SIZE (32 * 1024 * 1024) /* initial write max */ 780 #define L2ARC_HEADROOM 8 /* num of writes */ 781 782 /* 783 * If we discover during ARC scan any buffers to be compressed, we boost 784 * our headroom for the next scanning cycle by this percentage multiple. 785 */ 786 #define L2ARC_HEADROOM_BOOST 200 787 #define L2ARC_FEED_SECS 1 /* caching interval secs */ 788 #define L2ARC_FEED_MIN_MS 200 /* min caching interval ms */ 789 790 /* 791 * We can feed L2ARC from two states of ARC buffers, mru and mfu, 792 * and each of the state has two types: data and metadata. 793 */ 794 #define L2ARC_FEED_TYPES 4 795 796 /* L2ARC Performance Tunables */ 797 uint64_t l2arc_write_max = L2ARC_WRITE_SIZE; /* def max write size */ 798 uint64_t l2arc_write_boost = L2ARC_WRITE_SIZE; /* extra warmup write */ 799 uint64_t l2arc_headroom = L2ARC_HEADROOM; /* # of dev writes */ 800 uint64_t l2arc_headroom_boost = L2ARC_HEADROOM_BOOST; 801 uint64_t l2arc_feed_secs = L2ARC_FEED_SECS; /* interval seconds */ 802 uint64_t l2arc_feed_min_ms = L2ARC_FEED_MIN_MS; /* min interval msecs */ 803 int l2arc_noprefetch = B_TRUE; /* don't cache prefetch bufs */ 804 int l2arc_feed_again = B_TRUE; /* turbo warmup */ 805 int l2arc_norw = B_FALSE; /* no reads during writes */ 806 static uint_t l2arc_meta_percent = 33; /* limit on headers size */ 807 808 /* 809 * L2ARC Internals 810 */ 811 static list_t L2ARC_dev_list; /* device list */ 812 static list_t *l2arc_dev_list; /* device list pointer */ 813 static kmutex_t l2arc_dev_mtx; /* device list mutex */ 814 static l2arc_dev_t *l2arc_dev_last; /* last device used */ 815 static list_t L2ARC_free_on_write; /* free after write buf list */ 816 static list_t *l2arc_free_on_write; /* free after write list ptr */ 817 static kmutex_t l2arc_free_on_write_mtx; /* mutex for list */ 818 static uint64_t l2arc_ndev; /* number of devices */ 819 820 typedef struct l2arc_read_callback { 821 arc_buf_hdr_t *l2rcb_hdr; /* read header */ 822 blkptr_t l2rcb_bp; /* original blkptr */ 823 zbookmark_phys_t l2rcb_zb; /* original bookmark */ 824 int l2rcb_flags; /* original flags */ 825 abd_t *l2rcb_abd; /* temporary buffer */ 826 } l2arc_read_callback_t; 827 828 typedef struct l2arc_data_free { 829 /* protected by l2arc_free_on_write_mtx */ 830 abd_t *l2df_abd; 831 size_t l2df_size; 832 arc_buf_contents_t l2df_type; 833 list_node_t l2df_list_node; 834 } l2arc_data_free_t; 835 836 typedef enum arc_fill_flags { 837 ARC_FILL_LOCKED = 1 << 0, /* hdr lock is held */ 838 ARC_FILL_COMPRESSED = 1 << 1, /* fill with compressed data */ 839 ARC_FILL_ENCRYPTED = 1 << 2, /* fill with encrypted data */ 840 ARC_FILL_NOAUTH = 1 << 3, /* don't attempt to authenticate */ 841 ARC_FILL_IN_PLACE = 1 << 4 /* fill in place (special case) */ 842 } arc_fill_flags_t; 843 844 typedef enum arc_ovf_level { 845 ARC_OVF_NONE, /* ARC within target size. */ 846 ARC_OVF_SOME, /* ARC is slightly overflowed. */ 847 ARC_OVF_SEVERE /* ARC is severely overflowed. */ 848 } arc_ovf_level_t; 849 850 static kmutex_t l2arc_feed_thr_lock; 851 static kcondvar_t l2arc_feed_thr_cv; 852 static uint8_t l2arc_thread_exit; 853 854 static kmutex_t l2arc_rebuild_thr_lock; 855 static kcondvar_t l2arc_rebuild_thr_cv; 856 857 enum arc_hdr_alloc_flags { 858 ARC_HDR_ALLOC_RDATA = 0x1, 859 ARC_HDR_USE_RESERVE = 0x4, 860 ARC_HDR_ALLOC_LINEAR = 0x8, 861 }; 862 863 864 static abd_t *arc_get_data_abd(arc_buf_hdr_t *, uint64_t, const void *, int); 865 static void *arc_get_data_buf(arc_buf_hdr_t *, uint64_t, const void *); 866 static void arc_get_data_impl(arc_buf_hdr_t *, uint64_t, const void *, int); 867 static void arc_free_data_abd(arc_buf_hdr_t *, abd_t *, uint64_t, const void *); 868 static void arc_free_data_buf(arc_buf_hdr_t *, void *, uint64_t, const void *); 869 static void arc_free_data_impl(arc_buf_hdr_t *hdr, uint64_t size, 870 const void *tag); 871 static void arc_hdr_free_abd(arc_buf_hdr_t *, boolean_t); 872 static void arc_hdr_alloc_abd(arc_buf_hdr_t *, int); 873 static void arc_hdr_destroy(arc_buf_hdr_t *); 874 static void arc_access(arc_buf_hdr_t *, arc_flags_t, boolean_t); 875 static void arc_buf_watch(arc_buf_t *); 876 static void arc_change_state(arc_state_t *, arc_buf_hdr_t *); 877 878 static arc_buf_contents_t arc_buf_type(arc_buf_hdr_t *); 879 static uint32_t arc_bufc_to_flags(arc_buf_contents_t); 880 static inline void arc_hdr_set_flags(arc_buf_hdr_t *hdr, arc_flags_t flags); 881 static inline void arc_hdr_clear_flags(arc_buf_hdr_t *hdr, arc_flags_t flags); 882 883 static boolean_t l2arc_write_eligible(uint64_t, arc_buf_hdr_t *); 884 static void l2arc_read_done(zio_t *); 885 static void l2arc_do_free_on_write(void); 886 static void l2arc_hdr_arcstats_update(arc_buf_hdr_t *hdr, boolean_t incr, 887 boolean_t state_only); 888 889 static void arc_prune_async(uint64_t adjust); 890 891 #define l2arc_hdr_arcstats_increment(hdr) \ 892 l2arc_hdr_arcstats_update((hdr), B_TRUE, B_FALSE) 893 #define l2arc_hdr_arcstats_decrement(hdr) \ 894 l2arc_hdr_arcstats_update((hdr), B_FALSE, B_FALSE) 895 #define l2arc_hdr_arcstats_increment_state(hdr) \ 896 l2arc_hdr_arcstats_update((hdr), B_TRUE, B_TRUE) 897 #define l2arc_hdr_arcstats_decrement_state(hdr) \ 898 l2arc_hdr_arcstats_update((hdr), B_FALSE, B_TRUE) 899 900 /* 901 * l2arc_exclude_special : A zfs module parameter that controls whether buffers 902 * present on special vdevs are eligibile for caching in L2ARC. If 903 * set to 1, exclude dbufs on special vdevs from being cached to 904 * L2ARC. 905 */ 906 int l2arc_exclude_special = 0; 907 908 /* 909 * l2arc_mfuonly : A ZFS module parameter that controls whether only MFU 910 * metadata and data are cached from ARC into L2ARC. 911 */ 912 static int l2arc_mfuonly = 0; 913 914 /* 915 * L2ARC TRIM 916 * l2arc_trim_ahead : A ZFS module parameter that controls how much ahead of 917 * the current write size (l2arc_write_max) we should TRIM if we 918 * have filled the device. It is defined as a percentage of the 919 * write size. If set to 100 we trim twice the space required to 920 * accommodate upcoming writes. A minimum of 64MB will be trimmed. 921 * It also enables TRIM of the whole L2ARC device upon creation or 922 * addition to an existing pool or if the header of the device is 923 * invalid upon importing a pool or onlining a cache device. The 924 * default is 0, which disables TRIM on L2ARC altogether as it can 925 * put significant stress on the underlying storage devices. This 926 * will vary depending of how well the specific device handles 927 * these commands. 928 */ 929 static uint64_t l2arc_trim_ahead = 0; 930 931 /* 932 * Performance tuning of L2ARC persistence: 933 * 934 * l2arc_rebuild_enabled : A ZFS module parameter that controls whether adding 935 * an L2ARC device (either at pool import or later) will attempt 936 * to rebuild L2ARC buffer contents. 937 * l2arc_rebuild_blocks_min_l2size : A ZFS module parameter that controls 938 * whether log blocks are written to the L2ARC device. If the L2ARC 939 * device is less than 1GB, the amount of data l2arc_evict() 940 * evicts is significant compared to the amount of restored L2ARC 941 * data. In this case do not write log blocks in L2ARC in order 942 * not to waste space. 943 */ 944 static int l2arc_rebuild_enabled = B_TRUE; 945 static uint64_t l2arc_rebuild_blocks_min_l2size = 1024 * 1024 * 1024; 946 947 /* L2ARC persistence rebuild control routines. */ 948 void l2arc_rebuild_vdev(vdev_t *vd, boolean_t reopen); 949 static __attribute__((noreturn)) void l2arc_dev_rebuild_thread(void *arg); 950 static int l2arc_rebuild(l2arc_dev_t *dev); 951 952 /* L2ARC persistence read I/O routines. */ 953 static int l2arc_dev_hdr_read(l2arc_dev_t *dev); 954 static int l2arc_log_blk_read(l2arc_dev_t *dev, 955 const l2arc_log_blkptr_t *this_lp, const l2arc_log_blkptr_t *next_lp, 956 l2arc_log_blk_phys_t *this_lb, l2arc_log_blk_phys_t *next_lb, 957 zio_t *this_io, zio_t **next_io); 958 static zio_t *l2arc_log_blk_fetch(vdev_t *vd, 959 const l2arc_log_blkptr_t *lp, l2arc_log_blk_phys_t *lb); 960 static void l2arc_log_blk_fetch_abort(zio_t *zio); 961 962 /* L2ARC persistence block restoration routines. */ 963 static void l2arc_log_blk_restore(l2arc_dev_t *dev, 964 const l2arc_log_blk_phys_t *lb, uint64_t lb_asize); 965 static void l2arc_hdr_restore(const l2arc_log_ent_phys_t *le, 966 l2arc_dev_t *dev); 967 968 /* L2ARC persistence write I/O routines. */ 969 static uint64_t l2arc_log_blk_commit(l2arc_dev_t *dev, zio_t *pio, 970 l2arc_write_callback_t *cb); 971 972 /* L2ARC persistence auxiliary routines. */ 973 boolean_t l2arc_log_blkptr_valid(l2arc_dev_t *dev, 974 const l2arc_log_blkptr_t *lbp); 975 static boolean_t l2arc_log_blk_insert(l2arc_dev_t *dev, 976 const arc_buf_hdr_t *ab); 977 boolean_t l2arc_range_check_overlap(uint64_t bottom, 978 uint64_t top, uint64_t check); 979 static void l2arc_blk_fetch_done(zio_t *zio); 980 static inline uint64_t 981 l2arc_log_blk_overhead(uint64_t write_sz, l2arc_dev_t *dev); 982 983 /* 984 * We use Cityhash for this. It's fast, and has good hash properties without 985 * requiring any large static buffers. 986 */ 987 static uint64_t 988 buf_hash(uint64_t spa, const dva_t *dva, uint64_t birth) 989 { 990 return (cityhash4(spa, dva->dva_word[0], dva->dva_word[1], birth)); 991 } 992 993 #define HDR_EMPTY(hdr) \ 994 ((hdr)->b_dva.dva_word[0] == 0 && \ 995 (hdr)->b_dva.dva_word[1] == 0) 996 997 #define HDR_EMPTY_OR_LOCKED(hdr) \ 998 (HDR_EMPTY(hdr) || MUTEX_HELD(HDR_LOCK(hdr))) 999 1000 #define HDR_EQUAL(spa, dva, birth, hdr) \ 1001 ((hdr)->b_dva.dva_word[0] == (dva)->dva_word[0]) && \ 1002 ((hdr)->b_dva.dva_word[1] == (dva)->dva_word[1]) && \ 1003 ((hdr)->b_birth == birth) && ((hdr)->b_spa == spa) 1004 1005 static void 1006 buf_discard_identity(arc_buf_hdr_t *hdr) 1007 { 1008 hdr->b_dva.dva_word[0] = 0; 1009 hdr->b_dva.dva_word[1] = 0; 1010 hdr->b_birth = 0; 1011 } 1012 1013 static arc_buf_hdr_t * 1014 buf_hash_find(uint64_t spa, const blkptr_t *bp, kmutex_t **lockp) 1015 { 1016 const dva_t *dva = BP_IDENTITY(bp); 1017 uint64_t birth = BP_GET_BIRTH(bp); 1018 uint64_t idx = BUF_HASH_INDEX(spa, dva, birth); 1019 kmutex_t *hash_lock = BUF_HASH_LOCK(idx); 1020 arc_buf_hdr_t *hdr; 1021 1022 mutex_enter(hash_lock); 1023 for (hdr = buf_hash_table.ht_table[idx]; hdr != NULL; 1024 hdr = hdr->b_hash_next) { 1025 if (HDR_EQUAL(spa, dva, birth, hdr)) { 1026 *lockp = hash_lock; 1027 return (hdr); 1028 } 1029 } 1030 mutex_exit(hash_lock); 1031 *lockp = NULL; 1032 return (NULL); 1033 } 1034 1035 /* 1036 * Insert an entry into the hash table. If there is already an element 1037 * equal to elem in the hash table, then the already existing element 1038 * will be returned and the new element will not be inserted. 1039 * Otherwise returns NULL. 1040 * If lockp == NULL, the caller is assumed to already hold the hash lock. 1041 */ 1042 static arc_buf_hdr_t * 1043 buf_hash_insert(arc_buf_hdr_t *hdr, kmutex_t **lockp) 1044 { 1045 uint64_t idx = BUF_HASH_INDEX(hdr->b_spa, &hdr->b_dva, hdr->b_birth); 1046 kmutex_t *hash_lock = BUF_HASH_LOCK(idx); 1047 arc_buf_hdr_t *fhdr; 1048 uint32_t i; 1049 1050 ASSERT(!DVA_IS_EMPTY(&hdr->b_dva)); 1051 ASSERT(hdr->b_birth != 0); 1052 ASSERT(!HDR_IN_HASH_TABLE(hdr)); 1053 1054 if (lockp != NULL) { 1055 *lockp = hash_lock; 1056 mutex_enter(hash_lock); 1057 } else { 1058 ASSERT(MUTEX_HELD(hash_lock)); 1059 } 1060 1061 for (fhdr = buf_hash_table.ht_table[idx], i = 0; fhdr != NULL; 1062 fhdr = fhdr->b_hash_next, i++) { 1063 if (HDR_EQUAL(hdr->b_spa, &hdr->b_dva, hdr->b_birth, fhdr)) 1064 return (fhdr); 1065 } 1066 1067 hdr->b_hash_next = buf_hash_table.ht_table[idx]; 1068 buf_hash_table.ht_table[idx] = hdr; 1069 arc_hdr_set_flags(hdr, ARC_FLAG_IN_HASH_TABLE); 1070 1071 /* collect some hash table performance data */ 1072 if (i > 0) { 1073 ARCSTAT_BUMP(arcstat_hash_collisions); 1074 if (i == 1) 1075 ARCSTAT_BUMP(arcstat_hash_chains); 1076 1077 ARCSTAT_MAX(arcstat_hash_chain_max, i); 1078 } 1079 uint64_t he = atomic_inc_64_nv( 1080 &arc_stats.arcstat_hash_elements.value.ui64); 1081 ARCSTAT_MAX(arcstat_hash_elements_max, he); 1082 1083 return (NULL); 1084 } 1085 1086 static void 1087 buf_hash_remove(arc_buf_hdr_t *hdr) 1088 { 1089 arc_buf_hdr_t *fhdr, **hdrp; 1090 uint64_t idx = BUF_HASH_INDEX(hdr->b_spa, &hdr->b_dva, hdr->b_birth); 1091 1092 ASSERT(MUTEX_HELD(BUF_HASH_LOCK(idx))); 1093 ASSERT(HDR_IN_HASH_TABLE(hdr)); 1094 1095 hdrp = &buf_hash_table.ht_table[idx]; 1096 while ((fhdr = *hdrp) != hdr) { 1097 ASSERT3P(fhdr, !=, NULL); 1098 hdrp = &fhdr->b_hash_next; 1099 } 1100 *hdrp = hdr->b_hash_next; 1101 hdr->b_hash_next = NULL; 1102 arc_hdr_clear_flags(hdr, ARC_FLAG_IN_HASH_TABLE); 1103 1104 /* collect some hash table performance data */ 1105 atomic_dec_64(&arc_stats.arcstat_hash_elements.value.ui64); 1106 1107 if (buf_hash_table.ht_table[idx] && 1108 buf_hash_table.ht_table[idx]->b_hash_next == NULL) 1109 ARCSTAT_BUMPDOWN(arcstat_hash_chains); 1110 } 1111 1112 /* 1113 * Global data structures and functions for the buf kmem cache. 1114 */ 1115 1116 static kmem_cache_t *hdr_full_cache; 1117 static kmem_cache_t *hdr_l2only_cache; 1118 static kmem_cache_t *buf_cache; 1119 1120 static void 1121 buf_fini(void) 1122 { 1123 #if defined(_KERNEL) 1124 /* 1125 * Large allocations which do not require contiguous pages 1126 * should be using vmem_free() in the linux kernel\ 1127 */ 1128 vmem_free(buf_hash_table.ht_table, 1129 (buf_hash_table.ht_mask + 1) * sizeof (void *)); 1130 #else 1131 kmem_free(buf_hash_table.ht_table, 1132 (buf_hash_table.ht_mask + 1) * sizeof (void *)); 1133 #endif 1134 for (int i = 0; i < BUF_LOCKS; i++) 1135 mutex_destroy(BUF_HASH_LOCK(i)); 1136 kmem_cache_destroy(hdr_full_cache); 1137 kmem_cache_destroy(hdr_l2only_cache); 1138 kmem_cache_destroy(buf_cache); 1139 } 1140 1141 /* 1142 * Constructor callback - called when the cache is empty 1143 * and a new buf is requested. 1144 */ 1145 static int 1146 hdr_full_cons(void *vbuf, void *unused, int kmflag) 1147 { 1148 (void) unused, (void) kmflag; 1149 arc_buf_hdr_t *hdr = vbuf; 1150 1151 memset(hdr, 0, HDR_FULL_SIZE); 1152 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_NUMFUNCS; 1153 zfs_refcount_create(&hdr->b_l1hdr.b_refcnt); 1154 #ifdef ZFS_DEBUG 1155 mutex_init(&hdr->b_l1hdr.b_freeze_lock, NULL, MUTEX_DEFAULT, NULL); 1156 #endif 1157 multilist_link_init(&hdr->b_l1hdr.b_arc_node); 1158 list_link_init(&hdr->b_l2hdr.b_l2node); 1159 arc_space_consume(HDR_FULL_SIZE, ARC_SPACE_HDRS); 1160 1161 return (0); 1162 } 1163 1164 static int 1165 hdr_l2only_cons(void *vbuf, void *unused, int kmflag) 1166 { 1167 (void) unused, (void) kmflag; 1168 arc_buf_hdr_t *hdr = vbuf; 1169 1170 memset(hdr, 0, HDR_L2ONLY_SIZE); 1171 arc_space_consume(HDR_L2ONLY_SIZE, ARC_SPACE_L2HDRS); 1172 1173 return (0); 1174 } 1175 1176 static int 1177 buf_cons(void *vbuf, void *unused, int kmflag) 1178 { 1179 (void) unused, (void) kmflag; 1180 arc_buf_t *buf = vbuf; 1181 1182 memset(buf, 0, sizeof (arc_buf_t)); 1183 arc_space_consume(sizeof (arc_buf_t), ARC_SPACE_HDRS); 1184 1185 return (0); 1186 } 1187 1188 /* 1189 * Destructor callback - called when a cached buf is 1190 * no longer required. 1191 */ 1192 static void 1193 hdr_full_dest(void *vbuf, void *unused) 1194 { 1195 (void) unused; 1196 arc_buf_hdr_t *hdr = vbuf; 1197 1198 ASSERT(HDR_EMPTY(hdr)); 1199 zfs_refcount_destroy(&hdr->b_l1hdr.b_refcnt); 1200 #ifdef ZFS_DEBUG 1201 mutex_destroy(&hdr->b_l1hdr.b_freeze_lock); 1202 #endif 1203 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 1204 arc_space_return(HDR_FULL_SIZE, ARC_SPACE_HDRS); 1205 } 1206 1207 static void 1208 hdr_l2only_dest(void *vbuf, void *unused) 1209 { 1210 (void) unused; 1211 arc_buf_hdr_t *hdr = vbuf; 1212 1213 ASSERT(HDR_EMPTY(hdr)); 1214 arc_space_return(HDR_L2ONLY_SIZE, ARC_SPACE_L2HDRS); 1215 } 1216 1217 static void 1218 buf_dest(void *vbuf, void *unused) 1219 { 1220 (void) unused; 1221 (void) vbuf; 1222 1223 arc_space_return(sizeof (arc_buf_t), ARC_SPACE_HDRS); 1224 } 1225 1226 static void 1227 buf_init(void) 1228 { 1229 uint64_t *ct = NULL; 1230 uint64_t hsize = 1ULL << 12; 1231 int i, j; 1232 1233 /* 1234 * The hash table is big enough to fill all of physical memory 1235 * with an average block size of zfs_arc_average_blocksize (default 8K). 1236 * By default, the table will take up 1237 * totalmem * sizeof(void*) / 8K (1MB per GB with 8-byte pointers). 1238 */ 1239 while (hsize * zfs_arc_average_blocksize < arc_all_memory()) 1240 hsize <<= 1; 1241 retry: 1242 buf_hash_table.ht_mask = hsize - 1; 1243 #if defined(_KERNEL) 1244 /* 1245 * Large allocations which do not require contiguous pages 1246 * should be using vmem_alloc() in the linux kernel 1247 */ 1248 buf_hash_table.ht_table = 1249 vmem_zalloc(hsize * sizeof (void*), KM_SLEEP); 1250 #else 1251 buf_hash_table.ht_table = 1252 kmem_zalloc(hsize * sizeof (void*), KM_NOSLEEP); 1253 #endif 1254 if (buf_hash_table.ht_table == NULL) { 1255 ASSERT(hsize > (1ULL << 8)); 1256 hsize >>= 1; 1257 goto retry; 1258 } 1259 1260 hdr_full_cache = kmem_cache_create("arc_buf_hdr_t_full", HDR_FULL_SIZE, 1261 0, hdr_full_cons, hdr_full_dest, NULL, NULL, NULL, 0); 1262 hdr_l2only_cache = kmem_cache_create("arc_buf_hdr_t_l2only", 1263 HDR_L2ONLY_SIZE, 0, hdr_l2only_cons, hdr_l2only_dest, NULL, 1264 NULL, NULL, 0); 1265 buf_cache = kmem_cache_create("arc_buf_t", sizeof (arc_buf_t), 1266 0, buf_cons, buf_dest, NULL, NULL, NULL, 0); 1267 1268 for (i = 0; i < 256; i++) 1269 for (ct = zfs_crc64_table + i, *ct = i, j = 8; j > 0; j--) 1270 *ct = (*ct >> 1) ^ (-(*ct & 1) & ZFS_CRC64_POLY); 1271 1272 for (i = 0; i < BUF_LOCKS; i++) 1273 mutex_init(BUF_HASH_LOCK(i), NULL, MUTEX_DEFAULT, NULL); 1274 } 1275 1276 #define ARC_MINTIME (hz>>4) /* 62 ms */ 1277 1278 /* 1279 * This is the size that the buf occupies in memory. If the buf is compressed, 1280 * it will correspond to the compressed size. You should use this method of 1281 * getting the buf size unless you explicitly need the logical size. 1282 */ 1283 uint64_t 1284 arc_buf_size(arc_buf_t *buf) 1285 { 1286 return (ARC_BUF_COMPRESSED(buf) ? 1287 HDR_GET_PSIZE(buf->b_hdr) : HDR_GET_LSIZE(buf->b_hdr)); 1288 } 1289 1290 uint64_t 1291 arc_buf_lsize(arc_buf_t *buf) 1292 { 1293 return (HDR_GET_LSIZE(buf->b_hdr)); 1294 } 1295 1296 /* 1297 * This function will return B_TRUE if the buffer is encrypted in memory. 1298 * This buffer can be decrypted by calling arc_untransform(). 1299 */ 1300 boolean_t 1301 arc_is_encrypted(arc_buf_t *buf) 1302 { 1303 return (ARC_BUF_ENCRYPTED(buf) != 0); 1304 } 1305 1306 /* 1307 * Returns B_TRUE if the buffer represents data that has not had its MAC 1308 * verified yet. 1309 */ 1310 boolean_t 1311 arc_is_unauthenticated(arc_buf_t *buf) 1312 { 1313 return (HDR_NOAUTH(buf->b_hdr) != 0); 1314 } 1315 1316 void 1317 arc_get_raw_params(arc_buf_t *buf, boolean_t *byteorder, uint8_t *salt, 1318 uint8_t *iv, uint8_t *mac) 1319 { 1320 arc_buf_hdr_t *hdr = buf->b_hdr; 1321 1322 ASSERT(HDR_PROTECTED(hdr)); 1323 1324 memcpy(salt, hdr->b_crypt_hdr.b_salt, ZIO_DATA_SALT_LEN); 1325 memcpy(iv, hdr->b_crypt_hdr.b_iv, ZIO_DATA_IV_LEN); 1326 memcpy(mac, hdr->b_crypt_hdr.b_mac, ZIO_DATA_MAC_LEN); 1327 *byteorder = (hdr->b_l1hdr.b_byteswap == DMU_BSWAP_NUMFUNCS) ? 1328 ZFS_HOST_BYTEORDER : !ZFS_HOST_BYTEORDER; 1329 } 1330 1331 /* 1332 * Indicates how this buffer is compressed in memory. If it is not compressed 1333 * the value will be ZIO_COMPRESS_OFF. It can be made normally readable with 1334 * arc_untransform() as long as it is also unencrypted. 1335 */ 1336 enum zio_compress 1337 arc_get_compression(arc_buf_t *buf) 1338 { 1339 return (ARC_BUF_COMPRESSED(buf) ? 1340 HDR_GET_COMPRESS(buf->b_hdr) : ZIO_COMPRESS_OFF); 1341 } 1342 1343 /* 1344 * Return the compression algorithm used to store this data in the ARC. If ARC 1345 * compression is enabled or this is an encrypted block, this will be the same 1346 * as what's used to store it on-disk. Otherwise, this will be ZIO_COMPRESS_OFF. 1347 */ 1348 static inline enum zio_compress 1349 arc_hdr_get_compress(arc_buf_hdr_t *hdr) 1350 { 1351 return (HDR_COMPRESSION_ENABLED(hdr) ? 1352 HDR_GET_COMPRESS(hdr) : ZIO_COMPRESS_OFF); 1353 } 1354 1355 uint8_t 1356 arc_get_complevel(arc_buf_t *buf) 1357 { 1358 return (buf->b_hdr->b_complevel); 1359 } 1360 1361 static inline boolean_t 1362 arc_buf_is_shared(arc_buf_t *buf) 1363 { 1364 boolean_t shared = (buf->b_data != NULL && 1365 buf->b_hdr->b_l1hdr.b_pabd != NULL && 1366 abd_is_linear(buf->b_hdr->b_l1hdr.b_pabd) && 1367 buf->b_data == abd_to_buf(buf->b_hdr->b_l1hdr.b_pabd)); 1368 IMPLY(shared, HDR_SHARED_DATA(buf->b_hdr)); 1369 EQUIV(shared, ARC_BUF_SHARED(buf)); 1370 IMPLY(shared, ARC_BUF_COMPRESSED(buf) || ARC_BUF_LAST(buf)); 1371 1372 /* 1373 * It would be nice to assert arc_can_share() too, but the "hdr isn't 1374 * already being shared" requirement prevents us from doing that. 1375 */ 1376 1377 return (shared); 1378 } 1379 1380 /* 1381 * Free the checksum associated with this header. If there is no checksum, this 1382 * is a no-op. 1383 */ 1384 static inline void 1385 arc_cksum_free(arc_buf_hdr_t *hdr) 1386 { 1387 #ifdef ZFS_DEBUG 1388 ASSERT(HDR_HAS_L1HDR(hdr)); 1389 1390 mutex_enter(&hdr->b_l1hdr.b_freeze_lock); 1391 if (hdr->b_l1hdr.b_freeze_cksum != NULL) { 1392 kmem_free(hdr->b_l1hdr.b_freeze_cksum, sizeof (zio_cksum_t)); 1393 hdr->b_l1hdr.b_freeze_cksum = NULL; 1394 } 1395 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1396 #endif 1397 } 1398 1399 /* 1400 * Return true iff at least one of the bufs on hdr is not compressed. 1401 * Encrypted buffers count as compressed. 1402 */ 1403 static boolean_t 1404 arc_hdr_has_uncompressed_buf(arc_buf_hdr_t *hdr) 1405 { 1406 ASSERT(hdr->b_l1hdr.b_state == arc_anon || HDR_EMPTY_OR_LOCKED(hdr)); 1407 1408 for (arc_buf_t *b = hdr->b_l1hdr.b_buf; b != NULL; b = b->b_next) { 1409 if (!ARC_BUF_COMPRESSED(b)) { 1410 return (B_TRUE); 1411 } 1412 } 1413 return (B_FALSE); 1414 } 1415 1416 1417 /* 1418 * If we've turned on the ZFS_DEBUG_MODIFY flag, verify that the buf's data 1419 * matches the checksum that is stored in the hdr. If there is no checksum, 1420 * or if the buf is compressed, this is a no-op. 1421 */ 1422 static void 1423 arc_cksum_verify(arc_buf_t *buf) 1424 { 1425 #ifdef ZFS_DEBUG 1426 arc_buf_hdr_t *hdr = buf->b_hdr; 1427 zio_cksum_t zc; 1428 1429 if (!(zfs_flags & ZFS_DEBUG_MODIFY)) 1430 return; 1431 1432 if (ARC_BUF_COMPRESSED(buf)) 1433 return; 1434 1435 ASSERT(HDR_HAS_L1HDR(hdr)); 1436 1437 mutex_enter(&hdr->b_l1hdr.b_freeze_lock); 1438 1439 if (hdr->b_l1hdr.b_freeze_cksum == NULL || HDR_IO_ERROR(hdr)) { 1440 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1441 return; 1442 } 1443 1444 fletcher_2_native(buf->b_data, arc_buf_size(buf), NULL, &zc); 1445 if (!ZIO_CHECKSUM_EQUAL(*hdr->b_l1hdr.b_freeze_cksum, zc)) 1446 panic("buffer modified while frozen!"); 1447 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1448 #endif 1449 } 1450 1451 /* 1452 * This function makes the assumption that data stored in the L2ARC 1453 * will be transformed exactly as it is in the main pool. Because of 1454 * this we can verify the checksum against the reading process's bp. 1455 */ 1456 static boolean_t 1457 arc_cksum_is_equal(arc_buf_hdr_t *hdr, zio_t *zio) 1458 { 1459 ASSERT(!BP_IS_EMBEDDED(zio->io_bp)); 1460 VERIFY3U(BP_GET_PSIZE(zio->io_bp), ==, HDR_GET_PSIZE(hdr)); 1461 1462 /* 1463 * Block pointers always store the checksum for the logical data. 1464 * If the block pointer has the gang bit set, then the checksum 1465 * it represents is for the reconstituted data and not for an 1466 * individual gang member. The zio pipeline, however, must be able to 1467 * determine the checksum of each of the gang constituents so it 1468 * treats the checksum comparison differently than what we need 1469 * for l2arc blocks. This prevents us from using the 1470 * zio_checksum_error() interface directly. Instead we must call the 1471 * zio_checksum_error_impl() so that we can ensure the checksum is 1472 * generated using the correct checksum algorithm and accounts for the 1473 * logical I/O size and not just a gang fragment. 1474 */ 1475 return (zio_checksum_error_impl(zio->io_spa, zio->io_bp, 1476 BP_GET_CHECKSUM(zio->io_bp), zio->io_abd, zio->io_size, 1477 zio->io_offset, NULL) == 0); 1478 } 1479 1480 /* 1481 * Given a buf full of data, if ZFS_DEBUG_MODIFY is enabled this computes a 1482 * checksum and attaches it to the buf's hdr so that we can ensure that the buf 1483 * isn't modified later on. If buf is compressed or there is already a checksum 1484 * on the hdr, this is a no-op (we only checksum uncompressed bufs). 1485 */ 1486 static void 1487 arc_cksum_compute(arc_buf_t *buf) 1488 { 1489 if (!(zfs_flags & ZFS_DEBUG_MODIFY)) 1490 return; 1491 1492 #ifdef ZFS_DEBUG 1493 arc_buf_hdr_t *hdr = buf->b_hdr; 1494 ASSERT(HDR_HAS_L1HDR(hdr)); 1495 mutex_enter(&hdr->b_l1hdr.b_freeze_lock); 1496 if (hdr->b_l1hdr.b_freeze_cksum != NULL || ARC_BUF_COMPRESSED(buf)) { 1497 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1498 return; 1499 } 1500 1501 ASSERT(!ARC_BUF_ENCRYPTED(buf)); 1502 ASSERT(!ARC_BUF_COMPRESSED(buf)); 1503 hdr->b_l1hdr.b_freeze_cksum = kmem_alloc(sizeof (zio_cksum_t), 1504 KM_SLEEP); 1505 fletcher_2_native(buf->b_data, arc_buf_size(buf), NULL, 1506 hdr->b_l1hdr.b_freeze_cksum); 1507 mutex_exit(&hdr->b_l1hdr.b_freeze_lock); 1508 #endif 1509 arc_buf_watch(buf); 1510 } 1511 1512 #ifndef _KERNEL 1513 void 1514 arc_buf_sigsegv(int sig, siginfo_t *si, void *unused) 1515 { 1516 (void) sig, (void) unused; 1517 panic("Got SIGSEGV at address: 0x%lx\n", (long)si->si_addr); 1518 } 1519 #endif 1520 1521 static void 1522 arc_buf_unwatch(arc_buf_t *buf) 1523 { 1524 #ifndef _KERNEL 1525 if (arc_watch) { 1526 ASSERT0(mprotect(buf->b_data, arc_buf_size(buf), 1527 PROT_READ | PROT_WRITE)); 1528 } 1529 #else 1530 (void) buf; 1531 #endif 1532 } 1533 1534 static void 1535 arc_buf_watch(arc_buf_t *buf) 1536 { 1537 #ifndef _KERNEL 1538 if (arc_watch) 1539 ASSERT0(mprotect(buf->b_data, arc_buf_size(buf), 1540 PROT_READ)); 1541 #else 1542 (void) buf; 1543 #endif 1544 } 1545 1546 static arc_buf_contents_t 1547 arc_buf_type(arc_buf_hdr_t *hdr) 1548 { 1549 arc_buf_contents_t type; 1550 if (HDR_ISTYPE_METADATA(hdr)) { 1551 type = ARC_BUFC_METADATA; 1552 } else { 1553 type = ARC_BUFC_DATA; 1554 } 1555 VERIFY3U(hdr->b_type, ==, type); 1556 return (type); 1557 } 1558 1559 boolean_t 1560 arc_is_metadata(arc_buf_t *buf) 1561 { 1562 return (HDR_ISTYPE_METADATA(buf->b_hdr) != 0); 1563 } 1564 1565 static uint32_t 1566 arc_bufc_to_flags(arc_buf_contents_t type) 1567 { 1568 switch (type) { 1569 case ARC_BUFC_DATA: 1570 /* metadata field is 0 if buffer contains normal data */ 1571 return (0); 1572 case ARC_BUFC_METADATA: 1573 return (ARC_FLAG_BUFC_METADATA); 1574 default: 1575 break; 1576 } 1577 panic("undefined ARC buffer type!"); 1578 return ((uint32_t)-1); 1579 } 1580 1581 void 1582 arc_buf_thaw(arc_buf_t *buf) 1583 { 1584 arc_buf_hdr_t *hdr = buf->b_hdr; 1585 1586 ASSERT3P(hdr->b_l1hdr.b_state, ==, arc_anon); 1587 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 1588 1589 arc_cksum_verify(buf); 1590 1591 /* 1592 * Compressed buffers do not manipulate the b_freeze_cksum. 1593 */ 1594 if (ARC_BUF_COMPRESSED(buf)) 1595 return; 1596 1597 ASSERT(HDR_HAS_L1HDR(hdr)); 1598 arc_cksum_free(hdr); 1599 arc_buf_unwatch(buf); 1600 } 1601 1602 void 1603 arc_buf_freeze(arc_buf_t *buf) 1604 { 1605 if (!(zfs_flags & ZFS_DEBUG_MODIFY)) 1606 return; 1607 1608 if (ARC_BUF_COMPRESSED(buf)) 1609 return; 1610 1611 ASSERT(HDR_HAS_L1HDR(buf->b_hdr)); 1612 arc_cksum_compute(buf); 1613 } 1614 1615 /* 1616 * The arc_buf_hdr_t's b_flags should never be modified directly. Instead, 1617 * the following functions should be used to ensure that the flags are 1618 * updated in a thread-safe way. When manipulating the flags either 1619 * the hash_lock must be held or the hdr must be undiscoverable. This 1620 * ensures that we're not racing with any other threads when updating 1621 * the flags. 1622 */ 1623 static inline void 1624 arc_hdr_set_flags(arc_buf_hdr_t *hdr, arc_flags_t flags) 1625 { 1626 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1627 hdr->b_flags |= flags; 1628 } 1629 1630 static inline void 1631 arc_hdr_clear_flags(arc_buf_hdr_t *hdr, arc_flags_t flags) 1632 { 1633 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1634 hdr->b_flags &= ~flags; 1635 } 1636 1637 /* 1638 * Setting the compression bits in the arc_buf_hdr_t's b_flags is 1639 * done in a special way since we have to clear and set bits 1640 * at the same time. Consumers that wish to set the compression bits 1641 * must use this function to ensure that the flags are updated in 1642 * thread-safe manner. 1643 */ 1644 static void 1645 arc_hdr_set_compress(arc_buf_hdr_t *hdr, enum zio_compress cmp) 1646 { 1647 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1648 1649 /* 1650 * Holes and embedded blocks will always have a psize = 0 so 1651 * we ignore the compression of the blkptr and set the 1652 * want to uncompress them. Mark them as uncompressed. 1653 */ 1654 if (!zfs_compressed_arc_enabled || HDR_GET_PSIZE(hdr) == 0) { 1655 arc_hdr_clear_flags(hdr, ARC_FLAG_COMPRESSED_ARC); 1656 ASSERT(!HDR_COMPRESSION_ENABLED(hdr)); 1657 } else { 1658 arc_hdr_set_flags(hdr, ARC_FLAG_COMPRESSED_ARC); 1659 ASSERT(HDR_COMPRESSION_ENABLED(hdr)); 1660 } 1661 1662 HDR_SET_COMPRESS(hdr, cmp); 1663 ASSERT3U(HDR_GET_COMPRESS(hdr), ==, cmp); 1664 } 1665 1666 /* 1667 * Looks for another buf on the same hdr which has the data decompressed, copies 1668 * from it, and returns true. If no such buf exists, returns false. 1669 */ 1670 static boolean_t 1671 arc_buf_try_copy_decompressed_data(arc_buf_t *buf) 1672 { 1673 arc_buf_hdr_t *hdr = buf->b_hdr; 1674 boolean_t copied = B_FALSE; 1675 1676 ASSERT(HDR_HAS_L1HDR(hdr)); 1677 ASSERT3P(buf->b_data, !=, NULL); 1678 ASSERT(!ARC_BUF_COMPRESSED(buf)); 1679 1680 for (arc_buf_t *from = hdr->b_l1hdr.b_buf; from != NULL; 1681 from = from->b_next) { 1682 /* can't use our own data buffer */ 1683 if (from == buf) { 1684 continue; 1685 } 1686 1687 if (!ARC_BUF_COMPRESSED(from)) { 1688 memcpy(buf->b_data, from->b_data, arc_buf_size(buf)); 1689 copied = B_TRUE; 1690 break; 1691 } 1692 } 1693 1694 #ifdef ZFS_DEBUG 1695 /* 1696 * There were no decompressed bufs, so there should not be a 1697 * checksum on the hdr either. 1698 */ 1699 if (zfs_flags & ZFS_DEBUG_MODIFY) 1700 EQUIV(!copied, hdr->b_l1hdr.b_freeze_cksum == NULL); 1701 #endif 1702 1703 return (copied); 1704 } 1705 1706 /* 1707 * Allocates an ARC buf header that's in an evicted & L2-cached state. 1708 * This is used during l2arc reconstruction to make empty ARC buffers 1709 * which circumvent the regular disk->arc->l2arc path and instead come 1710 * into being in the reverse order, i.e. l2arc->arc. 1711 */ 1712 static arc_buf_hdr_t * 1713 arc_buf_alloc_l2only(size_t size, arc_buf_contents_t type, l2arc_dev_t *dev, 1714 dva_t dva, uint64_t daddr, int32_t psize, uint64_t birth, 1715 enum zio_compress compress, uint8_t complevel, boolean_t protected, 1716 boolean_t prefetch, arc_state_type_t arcs_state) 1717 { 1718 arc_buf_hdr_t *hdr; 1719 1720 ASSERT(size != 0); 1721 hdr = kmem_cache_alloc(hdr_l2only_cache, KM_SLEEP); 1722 hdr->b_birth = birth; 1723 hdr->b_type = type; 1724 hdr->b_flags = 0; 1725 arc_hdr_set_flags(hdr, arc_bufc_to_flags(type) | ARC_FLAG_HAS_L2HDR); 1726 HDR_SET_LSIZE(hdr, size); 1727 HDR_SET_PSIZE(hdr, psize); 1728 arc_hdr_set_compress(hdr, compress); 1729 hdr->b_complevel = complevel; 1730 if (protected) 1731 arc_hdr_set_flags(hdr, ARC_FLAG_PROTECTED); 1732 if (prefetch) 1733 arc_hdr_set_flags(hdr, ARC_FLAG_PREFETCH); 1734 hdr->b_spa = spa_load_guid(dev->l2ad_vdev->vdev_spa); 1735 1736 hdr->b_dva = dva; 1737 1738 hdr->b_l2hdr.b_dev = dev; 1739 hdr->b_l2hdr.b_daddr = daddr; 1740 hdr->b_l2hdr.b_arcs_state = arcs_state; 1741 1742 return (hdr); 1743 } 1744 1745 /* 1746 * Return the size of the block, b_pabd, that is stored in the arc_buf_hdr_t. 1747 */ 1748 static uint64_t 1749 arc_hdr_size(arc_buf_hdr_t *hdr) 1750 { 1751 uint64_t size; 1752 1753 if (arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF && 1754 HDR_GET_PSIZE(hdr) > 0) { 1755 size = HDR_GET_PSIZE(hdr); 1756 } else { 1757 ASSERT3U(HDR_GET_LSIZE(hdr), !=, 0); 1758 size = HDR_GET_LSIZE(hdr); 1759 } 1760 return (size); 1761 } 1762 1763 static int 1764 arc_hdr_authenticate(arc_buf_hdr_t *hdr, spa_t *spa, uint64_t dsobj) 1765 { 1766 int ret; 1767 uint64_t csize; 1768 uint64_t lsize = HDR_GET_LSIZE(hdr); 1769 uint64_t psize = HDR_GET_PSIZE(hdr); 1770 void *tmpbuf = NULL; 1771 abd_t *abd = hdr->b_l1hdr.b_pabd; 1772 1773 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1774 ASSERT(HDR_AUTHENTICATED(hdr)); 1775 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 1776 1777 /* 1778 * The MAC is calculated on the compressed data that is stored on disk. 1779 * However, if compressed arc is disabled we will only have the 1780 * decompressed data available to us now. Compress it into a temporary 1781 * abd so we can verify the MAC. The performance overhead of this will 1782 * be relatively low, since most objects in an encrypted objset will 1783 * be encrypted (instead of authenticated) anyway. 1784 */ 1785 if (HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 1786 !HDR_COMPRESSION_ENABLED(hdr)) { 1787 1788 csize = zio_compress_data(HDR_GET_COMPRESS(hdr), 1789 hdr->b_l1hdr.b_pabd, &tmpbuf, lsize, hdr->b_complevel); 1790 ASSERT3P(tmpbuf, !=, NULL); 1791 ASSERT3U(csize, <=, psize); 1792 abd = abd_get_from_buf(tmpbuf, lsize); 1793 abd_take_ownership_of_buf(abd, B_TRUE); 1794 abd_zero_off(abd, csize, psize - csize); 1795 } 1796 1797 /* 1798 * Authentication is best effort. We authenticate whenever the key is 1799 * available. If we succeed we clear ARC_FLAG_NOAUTH. 1800 */ 1801 if (hdr->b_crypt_hdr.b_ot == DMU_OT_OBJSET) { 1802 ASSERT3U(HDR_GET_COMPRESS(hdr), ==, ZIO_COMPRESS_OFF); 1803 ASSERT3U(lsize, ==, psize); 1804 ret = spa_do_crypt_objset_mac_abd(B_FALSE, spa, dsobj, abd, 1805 psize, hdr->b_l1hdr.b_byteswap != DMU_BSWAP_NUMFUNCS); 1806 } else { 1807 ret = spa_do_crypt_mac_abd(B_FALSE, spa, dsobj, abd, psize, 1808 hdr->b_crypt_hdr.b_mac); 1809 } 1810 1811 if (ret == 0) 1812 arc_hdr_clear_flags(hdr, ARC_FLAG_NOAUTH); 1813 else if (ret != ENOENT) 1814 goto error; 1815 1816 if (tmpbuf != NULL) 1817 abd_free(abd); 1818 1819 return (0); 1820 1821 error: 1822 if (tmpbuf != NULL) 1823 abd_free(abd); 1824 1825 return (ret); 1826 } 1827 1828 /* 1829 * This function will take a header that only has raw encrypted data in 1830 * b_crypt_hdr.b_rabd and decrypt it into a new buffer which is stored in 1831 * b_l1hdr.b_pabd. If designated in the header flags, this function will 1832 * also decompress the data. 1833 */ 1834 static int 1835 arc_hdr_decrypt(arc_buf_hdr_t *hdr, spa_t *spa, const zbookmark_phys_t *zb) 1836 { 1837 int ret; 1838 abd_t *cabd = NULL; 1839 void *tmp = NULL; 1840 boolean_t no_crypt = B_FALSE; 1841 boolean_t bswap = (hdr->b_l1hdr.b_byteswap != DMU_BSWAP_NUMFUNCS); 1842 1843 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1844 ASSERT(HDR_ENCRYPTED(hdr)); 1845 1846 arc_hdr_alloc_abd(hdr, 0); 1847 1848 ret = spa_do_crypt_abd(B_FALSE, spa, zb, hdr->b_crypt_hdr.b_ot, 1849 B_FALSE, bswap, hdr->b_crypt_hdr.b_salt, hdr->b_crypt_hdr.b_iv, 1850 hdr->b_crypt_hdr.b_mac, HDR_GET_PSIZE(hdr), hdr->b_l1hdr.b_pabd, 1851 hdr->b_crypt_hdr.b_rabd, &no_crypt); 1852 if (ret != 0) 1853 goto error; 1854 1855 if (no_crypt) { 1856 abd_copy(hdr->b_l1hdr.b_pabd, hdr->b_crypt_hdr.b_rabd, 1857 HDR_GET_PSIZE(hdr)); 1858 } 1859 1860 /* 1861 * If this header has disabled arc compression but the b_pabd is 1862 * compressed after decrypting it, we need to decompress the newly 1863 * decrypted data. 1864 */ 1865 if (HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 1866 !HDR_COMPRESSION_ENABLED(hdr)) { 1867 /* 1868 * We want to make sure that we are correctly honoring the 1869 * zfs_abd_scatter_enabled setting, so we allocate an abd here 1870 * and then loan a buffer from it, rather than allocating a 1871 * linear buffer and wrapping it in an abd later. 1872 */ 1873 cabd = arc_get_data_abd(hdr, arc_hdr_size(hdr), hdr, 0); 1874 tmp = abd_borrow_buf(cabd, arc_hdr_size(hdr)); 1875 1876 ret = zio_decompress_data(HDR_GET_COMPRESS(hdr), 1877 hdr->b_l1hdr.b_pabd, tmp, HDR_GET_PSIZE(hdr), 1878 HDR_GET_LSIZE(hdr), &hdr->b_complevel); 1879 if (ret != 0) { 1880 abd_return_buf(cabd, tmp, arc_hdr_size(hdr)); 1881 goto error; 1882 } 1883 1884 abd_return_buf_copy(cabd, tmp, arc_hdr_size(hdr)); 1885 arc_free_data_abd(hdr, hdr->b_l1hdr.b_pabd, 1886 arc_hdr_size(hdr), hdr); 1887 hdr->b_l1hdr.b_pabd = cabd; 1888 } 1889 1890 return (0); 1891 1892 error: 1893 arc_hdr_free_abd(hdr, B_FALSE); 1894 if (cabd != NULL) 1895 arc_free_data_buf(hdr, cabd, arc_hdr_size(hdr), hdr); 1896 1897 return (ret); 1898 } 1899 1900 /* 1901 * This function is called during arc_buf_fill() to prepare the header's 1902 * abd plaintext pointer for use. This involves authenticated protected 1903 * data and decrypting encrypted data into the plaintext abd. 1904 */ 1905 static int 1906 arc_fill_hdr_crypt(arc_buf_hdr_t *hdr, kmutex_t *hash_lock, spa_t *spa, 1907 const zbookmark_phys_t *zb, boolean_t noauth) 1908 { 1909 int ret; 1910 1911 ASSERT(HDR_PROTECTED(hdr)); 1912 1913 if (hash_lock != NULL) 1914 mutex_enter(hash_lock); 1915 1916 if (HDR_NOAUTH(hdr) && !noauth) { 1917 /* 1918 * The caller requested authenticated data but our data has 1919 * not been authenticated yet. Verify the MAC now if we can. 1920 */ 1921 ret = arc_hdr_authenticate(hdr, spa, zb->zb_objset); 1922 if (ret != 0) 1923 goto error; 1924 } else if (HDR_HAS_RABD(hdr) && hdr->b_l1hdr.b_pabd == NULL) { 1925 /* 1926 * If we only have the encrypted version of the data, but the 1927 * unencrypted version was requested we take this opportunity 1928 * to store the decrypted version in the header for future use. 1929 */ 1930 ret = arc_hdr_decrypt(hdr, spa, zb); 1931 if (ret != 0) 1932 goto error; 1933 } 1934 1935 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 1936 1937 if (hash_lock != NULL) 1938 mutex_exit(hash_lock); 1939 1940 return (0); 1941 1942 error: 1943 if (hash_lock != NULL) 1944 mutex_exit(hash_lock); 1945 1946 return (ret); 1947 } 1948 1949 /* 1950 * This function is used by the dbuf code to decrypt bonus buffers in place. 1951 * The dbuf code itself doesn't have any locking for decrypting a shared dnode 1952 * block, so we use the hash lock here to protect against concurrent calls to 1953 * arc_buf_fill(). 1954 */ 1955 static void 1956 arc_buf_untransform_in_place(arc_buf_t *buf) 1957 { 1958 arc_buf_hdr_t *hdr = buf->b_hdr; 1959 1960 ASSERT(HDR_ENCRYPTED(hdr)); 1961 ASSERT3U(hdr->b_crypt_hdr.b_ot, ==, DMU_OT_DNODE); 1962 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 1963 ASSERT3PF(hdr->b_l1hdr.b_pabd, !=, NULL, "hdr %px buf %px", hdr, buf); 1964 1965 zio_crypt_copy_dnode_bonus(hdr->b_l1hdr.b_pabd, buf->b_data, 1966 arc_buf_size(buf)); 1967 buf->b_flags &= ~ARC_BUF_FLAG_ENCRYPTED; 1968 buf->b_flags &= ~ARC_BUF_FLAG_COMPRESSED; 1969 } 1970 1971 /* 1972 * Given a buf that has a data buffer attached to it, this function will 1973 * efficiently fill the buf with data of the specified compression setting from 1974 * the hdr and update the hdr's b_freeze_cksum if necessary. If the buf and hdr 1975 * are already sharing a data buf, no copy is performed. 1976 * 1977 * If the buf is marked as compressed but uncompressed data was requested, this 1978 * will allocate a new data buffer for the buf, remove that flag, and fill the 1979 * buf with uncompressed data. You can't request a compressed buf on a hdr with 1980 * uncompressed data, and (since we haven't added support for it yet) if you 1981 * want compressed data your buf must already be marked as compressed and have 1982 * the correct-sized data buffer. 1983 */ 1984 static int 1985 arc_buf_fill(arc_buf_t *buf, spa_t *spa, const zbookmark_phys_t *zb, 1986 arc_fill_flags_t flags) 1987 { 1988 int error = 0; 1989 arc_buf_hdr_t *hdr = buf->b_hdr; 1990 boolean_t hdr_compressed = 1991 (arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF); 1992 boolean_t compressed = (flags & ARC_FILL_COMPRESSED) != 0; 1993 boolean_t encrypted = (flags & ARC_FILL_ENCRYPTED) != 0; 1994 dmu_object_byteswap_t bswap = hdr->b_l1hdr.b_byteswap; 1995 kmutex_t *hash_lock = (flags & ARC_FILL_LOCKED) ? NULL : HDR_LOCK(hdr); 1996 1997 ASSERT3P(buf->b_data, !=, NULL); 1998 IMPLY(compressed, hdr_compressed || ARC_BUF_ENCRYPTED(buf)); 1999 IMPLY(compressed, ARC_BUF_COMPRESSED(buf)); 2000 IMPLY(encrypted, HDR_ENCRYPTED(hdr)); 2001 IMPLY(encrypted, ARC_BUF_ENCRYPTED(buf)); 2002 IMPLY(encrypted, ARC_BUF_COMPRESSED(buf)); 2003 IMPLY(encrypted, !arc_buf_is_shared(buf)); 2004 2005 /* 2006 * If the caller wanted encrypted data we just need to copy it from 2007 * b_rabd and potentially byteswap it. We won't be able to do any 2008 * further transforms on it. 2009 */ 2010 if (encrypted) { 2011 ASSERT(HDR_HAS_RABD(hdr)); 2012 abd_copy_to_buf(buf->b_data, hdr->b_crypt_hdr.b_rabd, 2013 HDR_GET_PSIZE(hdr)); 2014 goto byteswap; 2015 } 2016 2017 /* 2018 * Adjust encrypted and authenticated headers to accommodate 2019 * the request if needed. Dnode blocks (ARC_FILL_IN_PLACE) are 2020 * allowed to fail decryption due to keys not being loaded 2021 * without being marked as an IO error. 2022 */ 2023 if (HDR_PROTECTED(hdr)) { 2024 error = arc_fill_hdr_crypt(hdr, hash_lock, spa, 2025 zb, !!(flags & ARC_FILL_NOAUTH)); 2026 if (error == EACCES && (flags & ARC_FILL_IN_PLACE) != 0) { 2027 return (error); 2028 } else if (error != 0) { 2029 if (hash_lock != NULL) 2030 mutex_enter(hash_lock); 2031 arc_hdr_set_flags(hdr, ARC_FLAG_IO_ERROR); 2032 if (hash_lock != NULL) 2033 mutex_exit(hash_lock); 2034 return (error); 2035 } 2036 } 2037 2038 /* 2039 * There is a special case here for dnode blocks which are 2040 * decrypting their bonus buffers. These blocks may request to 2041 * be decrypted in-place. This is necessary because there may 2042 * be many dnodes pointing into this buffer and there is 2043 * currently no method to synchronize replacing the backing 2044 * b_data buffer and updating all of the pointers. Here we use 2045 * the hash lock to ensure there are no races. If the need 2046 * arises for other types to be decrypted in-place, they must 2047 * add handling here as well. 2048 */ 2049 if ((flags & ARC_FILL_IN_PLACE) != 0) { 2050 ASSERT(!hdr_compressed); 2051 ASSERT(!compressed); 2052 ASSERT(!encrypted); 2053 2054 if (HDR_ENCRYPTED(hdr) && ARC_BUF_ENCRYPTED(buf)) { 2055 ASSERT3U(hdr->b_crypt_hdr.b_ot, ==, DMU_OT_DNODE); 2056 2057 if (hash_lock != NULL) 2058 mutex_enter(hash_lock); 2059 arc_buf_untransform_in_place(buf); 2060 if (hash_lock != NULL) 2061 mutex_exit(hash_lock); 2062 2063 /* Compute the hdr's checksum if necessary */ 2064 arc_cksum_compute(buf); 2065 } 2066 2067 return (0); 2068 } 2069 2070 if (hdr_compressed == compressed) { 2071 if (ARC_BUF_SHARED(buf)) { 2072 ASSERT(arc_buf_is_shared(buf)); 2073 } else { 2074 abd_copy_to_buf(buf->b_data, hdr->b_l1hdr.b_pabd, 2075 arc_buf_size(buf)); 2076 } 2077 } else { 2078 ASSERT(hdr_compressed); 2079 ASSERT(!compressed); 2080 2081 /* 2082 * If the buf is sharing its data with the hdr, unlink it and 2083 * allocate a new data buffer for the buf. 2084 */ 2085 if (ARC_BUF_SHARED(buf)) { 2086 ASSERTF(ARC_BUF_COMPRESSED(buf), 2087 "buf %p was uncompressed", buf); 2088 2089 /* We need to give the buf its own b_data */ 2090 buf->b_flags &= ~ARC_BUF_FLAG_SHARED; 2091 buf->b_data = 2092 arc_get_data_buf(hdr, HDR_GET_LSIZE(hdr), buf); 2093 arc_hdr_clear_flags(hdr, ARC_FLAG_SHARED_DATA); 2094 2095 /* Previously overhead was 0; just add new overhead */ 2096 ARCSTAT_INCR(arcstat_overhead_size, HDR_GET_LSIZE(hdr)); 2097 } else if (ARC_BUF_COMPRESSED(buf)) { 2098 ASSERT(!arc_buf_is_shared(buf)); 2099 2100 /* We need to reallocate the buf's b_data */ 2101 arc_free_data_buf(hdr, buf->b_data, HDR_GET_PSIZE(hdr), 2102 buf); 2103 buf->b_data = 2104 arc_get_data_buf(hdr, HDR_GET_LSIZE(hdr), buf); 2105 2106 /* We increased the size of b_data; update overhead */ 2107 ARCSTAT_INCR(arcstat_overhead_size, 2108 HDR_GET_LSIZE(hdr) - HDR_GET_PSIZE(hdr)); 2109 } 2110 2111 /* 2112 * Regardless of the buf's previous compression settings, it 2113 * should not be compressed at the end of this function. 2114 */ 2115 buf->b_flags &= ~ARC_BUF_FLAG_COMPRESSED; 2116 2117 /* 2118 * Try copying the data from another buf which already has a 2119 * decompressed version. If that's not possible, it's time to 2120 * bite the bullet and decompress the data from the hdr. 2121 */ 2122 if (arc_buf_try_copy_decompressed_data(buf)) { 2123 /* Skip byteswapping and checksumming (already done) */ 2124 return (0); 2125 } else { 2126 error = zio_decompress_data(HDR_GET_COMPRESS(hdr), 2127 hdr->b_l1hdr.b_pabd, buf->b_data, 2128 HDR_GET_PSIZE(hdr), HDR_GET_LSIZE(hdr), 2129 &hdr->b_complevel); 2130 2131 /* 2132 * Absent hardware errors or software bugs, this should 2133 * be impossible, but log it anyway so we can debug it. 2134 */ 2135 if (error != 0) { 2136 zfs_dbgmsg( 2137 "hdr %px, compress %d, psize %d, lsize %d", 2138 hdr, arc_hdr_get_compress(hdr), 2139 HDR_GET_PSIZE(hdr), HDR_GET_LSIZE(hdr)); 2140 if (hash_lock != NULL) 2141 mutex_enter(hash_lock); 2142 arc_hdr_set_flags(hdr, ARC_FLAG_IO_ERROR); 2143 if (hash_lock != NULL) 2144 mutex_exit(hash_lock); 2145 return (SET_ERROR(EIO)); 2146 } 2147 } 2148 } 2149 2150 byteswap: 2151 /* Byteswap the buf's data if necessary */ 2152 if (bswap != DMU_BSWAP_NUMFUNCS) { 2153 ASSERT(!HDR_SHARED_DATA(hdr)); 2154 ASSERT3U(bswap, <, DMU_BSWAP_NUMFUNCS); 2155 dmu_ot_byteswap[bswap].ob_func(buf->b_data, HDR_GET_LSIZE(hdr)); 2156 } 2157 2158 /* Compute the hdr's checksum if necessary */ 2159 arc_cksum_compute(buf); 2160 2161 return (0); 2162 } 2163 2164 /* 2165 * If this function is being called to decrypt an encrypted buffer or verify an 2166 * authenticated one, the key must be loaded and a mapping must be made 2167 * available in the keystore via spa_keystore_create_mapping() or one of its 2168 * callers. 2169 */ 2170 int 2171 arc_untransform(arc_buf_t *buf, spa_t *spa, const zbookmark_phys_t *zb, 2172 boolean_t in_place) 2173 { 2174 int ret; 2175 arc_fill_flags_t flags = 0; 2176 2177 if (in_place) 2178 flags |= ARC_FILL_IN_PLACE; 2179 2180 ret = arc_buf_fill(buf, spa, zb, flags); 2181 if (ret == ECKSUM) { 2182 /* 2183 * Convert authentication and decryption errors to EIO 2184 * (and generate an ereport) before leaving the ARC. 2185 */ 2186 ret = SET_ERROR(EIO); 2187 spa_log_error(spa, zb, buf->b_hdr->b_birth); 2188 (void) zfs_ereport_post(FM_EREPORT_ZFS_AUTHENTICATION, 2189 spa, NULL, zb, NULL, 0); 2190 } 2191 2192 return (ret); 2193 } 2194 2195 /* 2196 * Increment the amount of evictable space in the arc_state_t's refcount. 2197 * We account for the space used by the hdr and the arc buf individually 2198 * so that we can add and remove them from the refcount individually. 2199 */ 2200 static void 2201 arc_evictable_space_increment(arc_buf_hdr_t *hdr, arc_state_t *state) 2202 { 2203 arc_buf_contents_t type = arc_buf_type(hdr); 2204 2205 ASSERT(HDR_HAS_L1HDR(hdr)); 2206 2207 if (GHOST_STATE(state)) { 2208 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 2209 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2210 ASSERT(!HDR_HAS_RABD(hdr)); 2211 (void) zfs_refcount_add_many(&state->arcs_esize[type], 2212 HDR_GET_LSIZE(hdr), hdr); 2213 return; 2214 } 2215 2216 if (hdr->b_l1hdr.b_pabd != NULL) { 2217 (void) zfs_refcount_add_many(&state->arcs_esize[type], 2218 arc_hdr_size(hdr), hdr); 2219 } 2220 if (HDR_HAS_RABD(hdr)) { 2221 (void) zfs_refcount_add_many(&state->arcs_esize[type], 2222 HDR_GET_PSIZE(hdr), hdr); 2223 } 2224 2225 for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; 2226 buf = buf->b_next) { 2227 if (ARC_BUF_SHARED(buf)) 2228 continue; 2229 (void) zfs_refcount_add_many(&state->arcs_esize[type], 2230 arc_buf_size(buf), buf); 2231 } 2232 } 2233 2234 /* 2235 * Decrement the amount of evictable space in the arc_state_t's refcount. 2236 * We account for the space used by the hdr and the arc buf individually 2237 * so that we can add and remove them from the refcount individually. 2238 */ 2239 static void 2240 arc_evictable_space_decrement(arc_buf_hdr_t *hdr, arc_state_t *state) 2241 { 2242 arc_buf_contents_t type = arc_buf_type(hdr); 2243 2244 ASSERT(HDR_HAS_L1HDR(hdr)); 2245 2246 if (GHOST_STATE(state)) { 2247 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 2248 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2249 ASSERT(!HDR_HAS_RABD(hdr)); 2250 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2251 HDR_GET_LSIZE(hdr), hdr); 2252 return; 2253 } 2254 2255 if (hdr->b_l1hdr.b_pabd != NULL) { 2256 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2257 arc_hdr_size(hdr), hdr); 2258 } 2259 if (HDR_HAS_RABD(hdr)) { 2260 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2261 HDR_GET_PSIZE(hdr), hdr); 2262 } 2263 2264 for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; 2265 buf = buf->b_next) { 2266 if (ARC_BUF_SHARED(buf)) 2267 continue; 2268 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2269 arc_buf_size(buf), buf); 2270 } 2271 } 2272 2273 /* 2274 * Add a reference to this hdr indicating that someone is actively 2275 * referencing that memory. When the refcount transitions from 0 to 1, 2276 * we remove it from the respective arc_state_t list to indicate that 2277 * it is not evictable. 2278 */ 2279 static void 2280 add_reference(arc_buf_hdr_t *hdr, const void *tag) 2281 { 2282 arc_state_t *state = hdr->b_l1hdr.b_state; 2283 2284 ASSERT(HDR_HAS_L1HDR(hdr)); 2285 if (!HDR_EMPTY(hdr) && !MUTEX_HELD(HDR_LOCK(hdr))) { 2286 ASSERT(state == arc_anon); 2287 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 2288 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 2289 } 2290 2291 if ((zfs_refcount_add(&hdr->b_l1hdr.b_refcnt, tag) == 1) && 2292 state != arc_anon && state != arc_l2c_only) { 2293 /* We don't use the L2-only state list. */ 2294 multilist_remove(&state->arcs_list[arc_buf_type(hdr)], hdr); 2295 arc_evictable_space_decrement(hdr, state); 2296 } 2297 } 2298 2299 /* 2300 * Remove a reference from this hdr. When the reference transitions from 2301 * 1 to 0 and we're not anonymous, then we add this hdr to the arc_state_t's 2302 * list making it eligible for eviction. 2303 */ 2304 static int 2305 remove_reference(arc_buf_hdr_t *hdr, const void *tag) 2306 { 2307 int cnt; 2308 arc_state_t *state = hdr->b_l1hdr.b_state; 2309 2310 ASSERT(HDR_HAS_L1HDR(hdr)); 2311 ASSERT(state == arc_anon || MUTEX_HELD(HDR_LOCK(hdr))); 2312 ASSERT(!GHOST_STATE(state)); /* arc_l2c_only counts as a ghost. */ 2313 2314 if ((cnt = zfs_refcount_remove(&hdr->b_l1hdr.b_refcnt, tag)) != 0) 2315 return (cnt); 2316 2317 if (state == arc_anon) { 2318 arc_hdr_destroy(hdr); 2319 return (0); 2320 } 2321 if (state == arc_uncached && !HDR_PREFETCH(hdr)) { 2322 arc_change_state(arc_anon, hdr); 2323 arc_hdr_destroy(hdr); 2324 return (0); 2325 } 2326 multilist_insert(&state->arcs_list[arc_buf_type(hdr)], hdr); 2327 arc_evictable_space_increment(hdr, state); 2328 return (0); 2329 } 2330 2331 /* 2332 * Returns detailed information about a specific arc buffer. When the 2333 * state_index argument is set the function will calculate the arc header 2334 * list position for its arc state. Since this requires a linear traversal 2335 * callers are strongly encourage not to do this. However, it can be helpful 2336 * for targeted analysis so the functionality is provided. 2337 */ 2338 void 2339 arc_buf_info(arc_buf_t *ab, arc_buf_info_t *abi, int state_index) 2340 { 2341 (void) state_index; 2342 arc_buf_hdr_t *hdr = ab->b_hdr; 2343 l1arc_buf_hdr_t *l1hdr = NULL; 2344 l2arc_buf_hdr_t *l2hdr = NULL; 2345 arc_state_t *state = NULL; 2346 2347 memset(abi, 0, sizeof (arc_buf_info_t)); 2348 2349 if (hdr == NULL) 2350 return; 2351 2352 abi->abi_flags = hdr->b_flags; 2353 2354 if (HDR_HAS_L1HDR(hdr)) { 2355 l1hdr = &hdr->b_l1hdr; 2356 state = l1hdr->b_state; 2357 } 2358 if (HDR_HAS_L2HDR(hdr)) 2359 l2hdr = &hdr->b_l2hdr; 2360 2361 if (l1hdr) { 2362 abi->abi_bufcnt = 0; 2363 for (arc_buf_t *buf = l1hdr->b_buf; buf; buf = buf->b_next) 2364 abi->abi_bufcnt++; 2365 abi->abi_access = l1hdr->b_arc_access; 2366 abi->abi_mru_hits = l1hdr->b_mru_hits; 2367 abi->abi_mru_ghost_hits = l1hdr->b_mru_ghost_hits; 2368 abi->abi_mfu_hits = l1hdr->b_mfu_hits; 2369 abi->abi_mfu_ghost_hits = l1hdr->b_mfu_ghost_hits; 2370 abi->abi_holds = zfs_refcount_count(&l1hdr->b_refcnt); 2371 } 2372 2373 if (l2hdr) { 2374 abi->abi_l2arc_dattr = l2hdr->b_daddr; 2375 abi->abi_l2arc_hits = l2hdr->b_hits; 2376 } 2377 2378 abi->abi_state_type = state ? state->arcs_state : ARC_STATE_ANON; 2379 abi->abi_state_contents = arc_buf_type(hdr); 2380 abi->abi_size = arc_hdr_size(hdr); 2381 } 2382 2383 /* 2384 * Move the supplied buffer to the indicated state. The hash lock 2385 * for the buffer must be held by the caller. 2386 */ 2387 static void 2388 arc_change_state(arc_state_t *new_state, arc_buf_hdr_t *hdr) 2389 { 2390 arc_state_t *old_state; 2391 int64_t refcnt; 2392 boolean_t update_old, update_new; 2393 arc_buf_contents_t type = arc_buf_type(hdr); 2394 2395 /* 2396 * We almost always have an L1 hdr here, since we call arc_hdr_realloc() 2397 * in arc_read() when bringing a buffer out of the L2ARC. However, the 2398 * L1 hdr doesn't always exist when we change state to arc_anon before 2399 * destroying a header, in which case reallocating to add the L1 hdr is 2400 * pointless. 2401 */ 2402 if (HDR_HAS_L1HDR(hdr)) { 2403 old_state = hdr->b_l1hdr.b_state; 2404 refcnt = zfs_refcount_count(&hdr->b_l1hdr.b_refcnt); 2405 update_old = (hdr->b_l1hdr.b_buf != NULL || 2406 hdr->b_l1hdr.b_pabd != NULL || HDR_HAS_RABD(hdr)); 2407 2408 IMPLY(GHOST_STATE(old_state), hdr->b_l1hdr.b_buf == NULL); 2409 IMPLY(GHOST_STATE(new_state), hdr->b_l1hdr.b_buf == NULL); 2410 IMPLY(old_state == arc_anon, hdr->b_l1hdr.b_buf == NULL || 2411 ARC_BUF_LAST(hdr->b_l1hdr.b_buf)); 2412 } else { 2413 old_state = arc_l2c_only; 2414 refcnt = 0; 2415 update_old = B_FALSE; 2416 } 2417 update_new = update_old; 2418 if (GHOST_STATE(old_state)) 2419 update_old = B_TRUE; 2420 if (GHOST_STATE(new_state)) 2421 update_new = B_TRUE; 2422 2423 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 2424 ASSERT3P(new_state, !=, old_state); 2425 2426 /* 2427 * If this buffer is evictable, transfer it from the 2428 * old state list to the new state list. 2429 */ 2430 if (refcnt == 0) { 2431 if (old_state != arc_anon && old_state != arc_l2c_only) { 2432 ASSERT(HDR_HAS_L1HDR(hdr)); 2433 /* remove_reference() saves on insert. */ 2434 if (multilist_link_active(&hdr->b_l1hdr.b_arc_node)) { 2435 multilist_remove(&old_state->arcs_list[type], 2436 hdr); 2437 arc_evictable_space_decrement(hdr, old_state); 2438 } 2439 } 2440 if (new_state != arc_anon && new_state != arc_l2c_only) { 2441 /* 2442 * An L1 header always exists here, since if we're 2443 * moving to some L1-cached state (i.e. not l2c_only or 2444 * anonymous), we realloc the header to add an L1hdr 2445 * beforehand. 2446 */ 2447 ASSERT(HDR_HAS_L1HDR(hdr)); 2448 multilist_insert(&new_state->arcs_list[type], hdr); 2449 arc_evictable_space_increment(hdr, new_state); 2450 } 2451 } 2452 2453 ASSERT(!HDR_EMPTY(hdr)); 2454 if (new_state == arc_anon && HDR_IN_HASH_TABLE(hdr)) 2455 buf_hash_remove(hdr); 2456 2457 /* adjust state sizes (ignore arc_l2c_only) */ 2458 2459 if (update_new && new_state != arc_l2c_only) { 2460 ASSERT(HDR_HAS_L1HDR(hdr)); 2461 if (GHOST_STATE(new_state)) { 2462 2463 /* 2464 * When moving a header to a ghost state, we first 2465 * remove all arc buffers. Thus, we'll have no arc 2466 * buffer to use for the reference. As a result, we 2467 * use the arc header pointer for the reference. 2468 */ 2469 (void) zfs_refcount_add_many( 2470 &new_state->arcs_size[type], 2471 HDR_GET_LSIZE(hdr), hdr); 2472 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2473 ASSERT(!HDR_HAS_RABD(hdr)); 2474 } else { 2475 2476 /* 2477 * Each individual buffer holds a unique reference, 2478 * thus we must remove each of these references one 2479 * at a time. 2480 */ 2481 for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; 2482 buf = buf->b_next) { 2483 2484 /* 2485 * When the arc_buf_t is sharing the data 2486 * block with the hdr, the owner of the 2487 * reference belongs to the hdr. Only 2488 * add to the refcount if the arc_buf_t is 2489 * not shared. 2490 */ 2491 if (ARC_BUF_SHARED(buf)) 2492 continue; 2493 2494 (void) zfs_refcount_add_many( 2495 &new_state->arcs_size[type], 2496 arc_buf_size(buf), buf); 2497 } 2498 2499 if (hdr->b_l1hdr.b_pabd != NULL) { 2500 (void) zfs_refcount_add_many( 2501 &new_state->arcs_size[type], 2502 arc_hdr_size(hdr), hdr); 2503 } 2504 2505 if (HDR_HAS_RABD(hdr)) { 2506 (void) zfs_refcount_add_many( 2507 &new_state->arcs_size[type], 2508 HDR_GET_PSIZE(hdr), hdr); 2509 } 2510 } 2511 } 2512 2513 if (update_old && old_state != arc_l2c_only) { 2514 ASSERT(HDR_HAS_L1HDR(hdr)); 2515 if (GHOST_STATE(old_state)) { 2516 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2517 ASSERT(!HDR_HAS_RABD(hdr)); 2518 2519 /* 2520 * When moving a header off of a ghost state, 2521 * the header will not contain any arc buffers. 2522 * We use the arc header pointer for the reference 2523 * which is exactly what we did when we put the 2524 * header on the ghost state. 2525 */ 2526 2527 (void) zfs_refcount_remove_many( 2528 &old_state->arcs_size[type], 2529 HDR_GET_LSIZE(hdr), hdr); 2530 } else { 2531 2532 /* 2533 * Each individual buffer holds a unique reference, 2534 * thus we must remove each of these references one 2535 * at a time. 2536 */ 2537 for (arc_buf_t *buf = hdr->b_l1hdr.b_buf; buf != NULL; 2538 buf = buf->b_next) { 2539 2540 /* 2541 * When the arc_buf_t is sharing the data 2542 * block with the hdr, the owner of the 2543 * reference belongs to the hdr. Only 2544 * add to the refcount if the arc_buf_t is 2545 * not shared. 2546 */ 2547 if (ARC_BUF_SHARED(buf)) 2548 continue; 2549 2550 (void) zfs_refcount_remove_many( 2551 &old_state->arcs_size[type], 2552 arc_buf_size(buf), buf); 2553 } 2554 ASSERT(hdr->b_l1hdr.b_pabd != NULL || 2555 HDR_HAS_RABD(hdr)); 2556 2557 if (hdr->b_l1hdr.b_pabd != NULL) { 2558 (void) zfs_refcount_remove_many( 2559 &old_state->arcs_size[type], 2560 arc_hdr_size(hdr), hdr); 2561 } 2562 2563 if (HDR_HAS_RABD(hdr)) { 2564 (void) zfs_refcount_remove_many( 2565 &old_state->arcs_size[type], 2566 HDR_GET_PSIZE(hdr), hdr); 2567 } 2568 } 2569 } 2570 2571 if (HDR_HAS_L1HDR(hdr)) { 2572 hdr->b_l1hdr.b_state = new_state; 2573 2574 if (HDR_HAS_L2HDR(hdr) && new_state != arc_l2c_only) { 2575 l2arc_hdr_arcstats_decrement_state(hdr); 2576 hdr->b_l2hdr.b_arcs_state = new_state->arcs_state; 2577 l2arc_hdr_arcstats_increment_state(hdr); 2578 } 2579 } 2580 } 2581 2582 void 2583 arc_space_consume(uint64_t space, arc_space_type_t type) 2584 { 2585 ASSERT(type >= 0 && type < ARC_SPACE_NUMTYPES); 2586 2587 switch (type) { 2588 default: 2589 break; 2590 case ARC_SPACE_DATA: 2591 ARCSTAT_INCR(arcstat_data_size, space); 2592 break; 2593 case ARC_SPACE_META: 2594 ARCSTAT_INCR(arcstat_metadata_size, space); 2595 break; 2596 case ARC_SPACE_BONUS: 2597 ARCSTAT_INCR(arcstat_bonus_size, space); 2598 break; 2599 case ARC_SPACE_DNODE: 2600 ARCSTAT_INCR(arcstat_dnode_size, space); 2601 break; 2602 case ARC_SPACE_DBUF: 2603 ARCSTAT_INCR(arcstat_dbuf_size, space); 2604 break; 2605 case ARC_SPACE_HDRS: 2606 ARCSTAT_INCR(arcstat_hdr_size, space); 2607 break; 2608 case ARC_SPACE_L2HDRS: 2609 aggsum_add(&arc_sums.arcstat_l2_hdr_size, space); 2610 break; 2611 case ARC_SPACE_ABD_CHUNK_WASTE: 2612 /* 2613 * Note: this includes space wasted by all scatter ABD's, not 2614 * just those allocated by the ARC. But the vast majority of 2615 * scatter ABD's come from the ARC, because other users are 2616 * very short-lived. 2617 */ 2618 ARCSTAT_INCR(arcstat_abd_chunk_waste_size, space); 2619 break; 2620 } 2621 2622 if (type != ARC_SPACE_DATA && type != ARC_SPACE_ABD_CHUNK_WASTE) 2623 ARCSTAT_INCR(arcstat_meta_used, space); 2624 2625 aggsum_add(&arc_sums.arcstat_size, space); 2626 } 2627 2628 void 2629 arc_space_return(uint64_t space, arc_space_type_t type) 2630 { 2631 ASSERT(type >= 0 && type < ARC_SPACE_NUMTYPES); 2632 2633 switch (type) { 2634 default: 2635 break; 2636 case ARC_SPACE_DATA: 2637 ARCSTAT_INCR(arcstat_data_size, -space); 2638 break; 2639 case ARC_SPACE_META: 2640 ARCSTAT_INCR(arcstat_metadata_size, -space); 2641 break; 2642 case ARC_SPACE_BONUS: 2643 ARCSTAT_INCR(arcstat_bonus_size, -space); 2644 break; 2645 case ARC_SPACE_DNODE: 2646 ARCSTAT_INCR(arcstat_dnode_size, -space); 2647 break; 2648 case ARC_SPACE_DBUF: 2649 ARCSTAT_INCR(arcstat_dbuf_size, -space); 2650 break; 2651 case ARC_SPACE_HDRS: 2652 ARCSTAT_INCR(arcstat_hdr_size, -space); 2653 break; 2654 case ARC_SPACE_L2HDRS: 2655 aggsum_add(&arc_sums.arcstat_l2_hdr_size, -space); 2656 break; 2657 case ARC_SPACE_ABD_CHUNK_WASTE: 2658 ARCSTAT_INCR(arcstat_abd_chunk_waste_size, -space); 2659 break; 2660 } 2661 2662 if (type != ARC_SPACE_DATA && type != ARC_SPACE_ABD_CHUNK_WASTE) 2663 ARCSTAT_INCR(arcstat_meta_used, -space); 2664 2665 ASSERT(aggsum_compare(&arc_sums.arcstat_size, space) >= 0); 2666 aggsum_add(&arc_sums.arcstat_size, -space); 2667 } 2668 2669 /* 2670 * Given a hdr and a buf, returns whether that buf can share its b_data buffer 2671 * with the hdr's b_pabd. 2672 */ 2673 static boolean_t 2674 arc_can_share(arc_buf_hdr_t *hdr, arc_buf_t *buf) 2675 { 2676 /* 2677 * The criteria for sharing a hdr's data are: 2678 * 1. the buffer is not encrypted 2679 * 2. the hdr's compression matches the buf's compression 2680 * 3. the hdr doesn't need to be byteswapped 2681 * 4. the hdr isn't already being shared 2682 * 5. the buf is either compressed or it is the last buf in the hdr list 2683 * 2684 * Criterion #5 maintains the invariant that shared uncompressed 2685 * bufs must be the final buf in the hdr's b_buf list. Reading this, you 2686 * might ask, "if a compressed buf is allocated first, won't that be the 2687 * last thing in the list?", but in that case it's impossible to create 2688 * a shared uncompressed buf anyway (because the hdr must be compressed 2689 * to have the compressed buf). You might also think that #3 is 2690 * sufficient to make this guarantee, however it's possible 2691 * (specifically in the rare L2ARC write race mentioned in 2692 * arc_buf_alloc_impl()) there will be an existing uncompressed buf that 2693 * is shareable, but wasn't at the time of its allocation. Rather than 2694 * allow a new shared uncompressed buf to be created and then shuffle 2695 * the list around to make it the last element, this simply disallows 2696 * sharing if the new buf isn't the first to be added. 2697 */ 2698 ASSERT3P(buf->b_hdr, ==, hdr); 2699 boolean_t hdr_compressed = 2700 arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF; 2701 boolean_t buf_compressed = ARC_BUF_COMPRESSED(buf) != 0; 2702 return (!ARC_BUF_ENCRYPTED(buf) && 2703 buf_compressed == hdr_compressed && 2704 hdr->b_l1hdr.b_byteswap == DMU_BSWAP_NUMFUNCS && 2705 !HDR_SHARED_DATA(hdr) && 2706 (ARC_BUF_LAST(buf) || ARC_BUF_COMPRESSED(buf))); 2707 } 2708 2709 /* 2710 * Allocate a buf for this hdr. If you care about the data that's in the hdr, 2711 * or if you want a compressed buffer, pass those flags in. Returns 0 if the 2712 * copy was made successfully, or an error code otherwise. 2713 */ 2714 static int 2715 arc_buf_alloc_impl(arc_buf_hdr_t *hdr, spa_t *spa, const zbookmark_phys_t *zb, 2716 const void *tag, boolean_t encrypted, boolean_t compressed, 2717 boolean_t noauth, boolean_t fill, arc_buf_t **ret) 2718 { 2719 arc_buf_t *buf; 2720 arc_fill_flags_t flags = ARC_FILL_LOCKED; 2721 2722 ASSERT(HDR_HAS_L1HDR(hdr)); 2723 ASSERT3U(HDR_GET_LSIZE(hdr), >, 0); 2724 VERIFY(hdr->b_type == ARC_BUFC_DATA || 2725 hdr->b_type == ARC_BUFC_METADATA); 2726 ASSERT3P(ret, !=, NULL); 2727 ASSERT3P(*ret, ==, NULL); 2728 IMPLY(encrypted, compressed); 2729 2730 buf = *ret = kmem_cache_alloc(buf_cache, KM_PUSHPAGE); 2731 buf->b_hdr = hdr; 2732 buf->b_data = NULL; 2733 buf->b_next = hdr->b_l1hdr.b_buf; 2734 buf->b_flags = 0; 2735 2736 add_reference(hdr, tag); 2737 2738 /* 2739 * We're about to change the hdr's b_flags. We must either 2740 * hold the hash_lock or be undiscoverable. 2741 */ 2742 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 2743 2744 /* 2745 * Only honor requests for compressed bufs if the hdr is actually 2746 * compressed. This must be overridden if the buffer is encrypted since 2747 * encrypted buffers cannot be decompressed. 2748 */ 2749 if (encrypted) { 2750 buf->b_flags |= ARC_BUF_FLAG_COMPRESSED; 2751 buf->b_flags |= ARC_BUF_FLAG_ENCRYPTED; 2752 flags |= ARC_FILL_COMPRESSED | ARC_FILL_ENCRYPTED; 2753 } else if (compressed && 2754 arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF) { 2755 buf->b_flags |= ARC_BUF_FLAG_COMPRESSED; 2756 flags |= ARC_FILL_COMPRESSED; 2757 } 2758 2759 if (noauth) { 2760 ASSERT0(encrypted); 2761 flags |= ARC_FILL_NOAUTH; 2762 } 2763 2764 /* 2765 * If the hdr's data can be shared then we share the data buffer and 2766 * set the appropriate bit in the hdr's b_flags to indicate the hdr is 2767 * sharing it's b_pabd with the arc_buf_t. Otherwise, we allocate a new 2768 * buffer to store the buf's data. 2769 * 2770 * There are two additional restrictions here because we're sharing 2771 * hdr -> buf instead of the usual buf -> hdr. First, the hdr can't be 2772 * actively involved in an L2ARC write, because if this buf is used by 2773 * an arc_write() then the hdr's data buffer will be released when the 2774 * write completes, even though the L2ARC write might still be using it. 2775 * Second, the hdr's ABD must be linear so that the buf's user doesn't 2776 * need to be ABD-aware. It must be allocated via 2777 * zio_[data_]buf_alloc(), not as a page, because we need to be able 2778 * to abd_release_ownership_of_buf(), which isn't allowed on "linear 2779 * page" buffers because the ABD code needs to handle freeing them 2780 * specially. 2781 */ 2782 boolean_t can_share = arc_can_share(hdr, buf) && 2783 !HDR_L2_WRITING(hdr) && 2784 hdr->b_l1hdr.b_pabd != NULL && 2785 abd_is_linear(hdr->b_l1hdr.b_pabd) && 2786 !abd_is_linear_page(hdr->b_l1hdr.b_pabd); 2787 2788 /* Set up b_data and sharing */ 2789 if (can_share) { 2790 buf->b_data = abd_to_buf(hdr->b_l1hdr.b_pabd); 2791 buf->b_flags |= ARC_BUF_FLAG_SHARED; 2792 arc_hdr_set_flags(hdr, ARC_FLAG_SHARED_DATA); 2793 } else { 2794 buf->b_data = 2795 arc_get_data_buf(hdr, arc_buf_size(buf), buf); 2796 ARCSTAT_INCR(arcstat_overhead_size, arc_buf_size(buf)); 2797 } 2798 VERIFY3P(buf->b_data, !=, NULL); 2799 2800 hdr->b_l1hdr.b_buf = buf; 2801 2802 /* 2803 * If the user wants the data from the hdr, we need to either copy or 2804 * decompress the data. 2805 */ 2806 if (fill) { 2807 ASSERT3P(zb, !=, NULL); 2808 return (arc_buf_fill(buf, spa, zb, flags)); 2809 } 2810 2811 return (0); 2812 } 2813 2814 static const char *arc_onloan_tag = "onloan"; 2815 2816 static inline void 2817 arc_loaned_bytes_update(int64_t delta) 2818 { 2819 atomic_add_64(&arc_loaned_bytes, delta); 2820 2821 /* assert that it did not wrap around */ 2822 ASSERT3S(atomic_add_64_nv(&arc_loaned_bytes, 0), >=, 0); 2823 } 2824 2825 /* 2826 * Loan out an anonymous arc buffer. Loaned buffers are not counted as in 2827 * flight data by arc_tempreserve_space() until they are "returned". Loaned 2828 * buffers must be returned to the arc before they can be used by the DMU or 2829 * freed. 2830 */ 2831 arc_buf_t * 2832 arc_loan_buf(spa_t *spa, boolean_t is_metadata, int size) 2833 { 2834 arc_buf_t *buf = arc_alloc_buf(spa, arc_onloan_tag, 2835 is_metadata ? ARC_BUFC_METADATA : ARC_BUFC_DATA, size); 2836 2837 arc_loaned_bytes_update(arc_buf_size(buf)); 2838 2839 return (buf); 2840 } 2841 2842 arc_buf_t * 2843 arc_loan_compressed_buf(spa_t *spa, uint64_t psize, uint64_t lsize, 2844 enum zio_compress compression_type, uint8_t complevel) 2845 { 2846 arc_buf_t *buf = arc_alloc_compressed_buf(spa, arc_onloan_tag, 2847 psize, lsize, compression_type, complevel); 2848 2849 arc_loaned_bytes_update(arc_buf_size(buf)); 2850 2851 return (buf); 2852 } 2853 2854 arc_buf_t * 2855 arc_loan_raw_buf(spa_t *spa, uint64_t dsobj, boolean_t byteorder, 2856 const uint8_t *salt, const uint8_t *iv, const uint8_t *mac, 2857 dmu_object_type_t ot, uint64_t psize, uint64_t lsize, 2858 enum zio_compress compression_type, uint8_t complevel) 2859 { 2860 arc_buf_t *buf = arc_alloc_raw_buf(spa, arc_onloan_tag, dsobj, 2861 byteorder, salt, iv, mac, ot, psize, lsize, compression_type, 2862 complevel); 2863 2864 atomic_add_64(&arc_loaned_bytes, psize); 2865 return (buf); 2866 } 2867 2868 2869 /* 2870 * Return a loaned arc buffer to the arc. 2871 */ 2872 void 2873 arc_return_buf(arc_buf_t *buf, const void *tag) 2874 { 2875 arc_buf_hdr_t *hdr = buf->b_hdr; 2876 2877 ASSERT3P(buf->b_data, !=, NULL); 2878 ASSERT(HDR_HAS_L1HDR(hdr)); 2879 (void) zfs_refcount_add(&hdr->b_l1hdr.b_refcnt, tag); 2880 (void) zfs_refcount_remove(&hdr->b_l1hdr.b_refcnt, arc_onloan_tag); 2881 2882 arc_loaned_bytes_update(-arc_buf_size(buf)); 2883 } 2884 2885 /* Detach an arc_buf from a dbuf (tag) */ 2886 void 2887 arc_loan_inuse_buf(arc_buf_t *buf, const void *tag) 2888 { 2889 arc_buf_hdr_t *hdr = buf->b_hdr; 2890 2891 ASSERT3P(buf->b_data, !=, NULL); 2892 ASSERT(HDR_HAS_L1HDR(hdr)); 2893 (void) zfs_refcount_add(&hdr->b_l1hdr.b_refcnt, arc_onloan_tag); 2894 (void) zfs_refcount_remove(&hdr->b_l1hdr.b_refcnt, tag); 2895 2896 arc_loaned_bytes_update(arc_buf_size(buf)); 2897 } 2898 2899 static void 2900 l2arc_free_abd_on_write(abd_t *abd, size_t size, arc_buf_contents_t type) 2901 { 2902 l2arc_data_free_t *df = kmem_alloc(sizeof (*df), KM_SLEEP); 2903 2904 df->l2df_abd = abd; 2905 df->l2df_size = size; 2906 df->l2df_type = type; 2907 mutex_enter(&l2arc_free_on_write_mtx); 2908 list_insert_head(l2arc_free_on_write, df); 2909 mutex_exit(&l2arc_free_on_write_mtx); 2910 } 2911 2912 static void 2913 arc_hdr_free_on_write(arc_buf_hdr_t *hdr, boolean_t free_rdata) 2914 { 2915 arc_state_t *state = hdr->b_l1hdr.b_state; 2916 arc_buf_contents_t type = arc_buf_type(hdr); 2917 uint64_t size = (free_rdata) ? HDR_GET_PSIZE(hdr) : arc_hdr_size(hdr); 2918 2919 /* protected by hash lock, if in the hash table */ 2920 if (multilist_link_active(&hdr->b_l1hdr.b_arc_node)) { 2921 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 2922 ASSERT(state != arc_anon && state != arc_l2c_only); 2923 2924 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 2925 size, hdr); 2926 } 2927 (void) zfs_refcount_remove_many(&state->arcs_size[type], size, hdr); 2928 if (type == ARC_BUFC_METADATA) { 2929 arc_space_return(size, ARC_SPACE_META); 2930 } else { 2931 ASSERT(type == ARC_BUFC_DATA); 2932 arc_space_return(size, ARC_SPACE_DATA); 2933 } 2934 2935 if (free_rdata) { 2936 l2arc_free_abd_on_write(hdr->b_crypt_hdr.b_rabd, size, type); 2937 } else { 2938 l2arc_free_abd_on_write(hdr->b_l1hdr.b_pabd, size, type); 2939 } 2940 } 2941 2942 /* 2943 * Share the arc_buf_t's data with the hdr. Whenever we are sharing the 2944 * data buffer, we transfer the refcount ownership to the hdr and update 2945 * the appropriate kstats. 2946 */ 2947 static void 2948 arc_share_buf(arc_buf_hdr_t *hdr, arc_buf_t *buf) 2949 { 2950 ASSERT(arc_can_share(hdr, buf)); 2951 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 2952 ASSERT(!ARC_BUF_ENCRYPTED(buf)); 2953 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 2954 2955 /* 2956 * Start sharing the data buffer. We transfer the 2957 * refcount ownership to the hdr since it always owns 2958 * the refcount whenever an arc_buf_t is shared. 2959 */ 2960 zfs_refcount_transfer_ownership_many( 2961 &hdr->b_l1hdr.b_state->arcs_size[arc_buf_type(hdr)], 2962 arc_hdr_size(hdr), buf, hdr); 2963 hdr->b_l1hdr.b_pabd = abd_get_from_buf(buf->b_data, arc_buf_size(buf)); 2964 abd_take_ownership_of_buf(hdr->b_l1hdr.b_pabd, 2965 HDR_ISTYPE_METADATA(hdr)); 2966 arc_hdr_set_flags(hdr, ARC_FLAG_SHARED_DATA); 2967 buf->b_flags |= ARC_BUF_FLAG_SHARED; 2968 2969 /* 2970 * Since we've transferred ownership to the hdr we need 2971 * to increment its compressed and uncompressed kstats and 2972 * decrement the overhead size. 2973 */ 2974 ARCSTAT_INCR(arcstat_compressed_size, arc_hdr_size(hdr)); 2975 ARCSTAT_INCR(arcstat_uncompressed_size, HDR_GET_LSIZE(hdr)); 2976 ARCSTAT_INCR(arcstat_overhead_size, -arc_buf_size(buf)); 2977 } 2978 2979 static void 2980 arc_unshare_buf(arc_buf_hdr_t *hdr, arc_buf_t *buf) 2981 { 2982 ASSERT(arc_buf_is_shared(buf)); 2983 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 2984 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 2985 2986 /* 2987 * We are no longer sharing this buffer so we need 2988 * to transfer its ownership to the rightful owner. 2989 */ 2990 zfs_refcount_transfer_ownership_many( 2991 &hdr->b_l1hdr.b_state->arcs_size[arc_buf_type(hdr)], 2992 arc_hdr_size(hdr), hdr, buf); 2993 arc_hdr_clear_flags(hdr, ARC_FLAG_SHARED_DATA); 2994 abd_release_ownership_of_buf(hdr->b_l1hdr.b_pabd); 2995 abd_free(hdr->b_l1hdr.b_pabd); 2996 hdr->b_l1hdr.b_pabd = NULL; 2997 buf->b_flags &= ~ARC_BUF_FLAG_SHARED; 2998 2999 /* 3000 * Since the buffer is no longer shared between 3001 * the arc buf and the hdr, count it as overhead. 3002 */ 3003 ARCSTAT_INCR(arcstat_compressed_size, -arc_hdr_size(hdr)); 3004 ARCSTAT_INCR(arcstat_uncompressed_size, -HDR_GET_LSIZE(hdr)); 3005 ARCSTAT_INCR(arcstat_overhead_size, arc_buf_size(buf)); 3006 } 3007 3008 /* 3009 * Remove an arc_buf_t from the hdr's buf list and return the last 3010 * arc_buf_t on the list. If no buffers remain on the list then return 3011 * NULL. 3012 */ 3013 static arc_buf_t * 3014 arc_buf_remove(arc_buf_hdr_t *hdr, arc_buf_t *buf) 3015 { 3016 ASSERT(HDR_HAS_L1HDR(hdr)); 3017 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 3018 3019 arc_buf_t **bufp = &hdr->b_l1hdr.b_buf; 3020 arc_buf_t *lastbuf = NULL; 3021 3022 /* 3023 * Remove the buf from the hdr list and locate the last 3024 * remaining buffer on the list. 3025 */ 3026 while (*bufp != NULL) { 3027 if (*bufp == buf) 3028 *bufp = buf->b_next; 3029 3030 /* 3031 * If we've removed a buffer in the middle of 3032 * the list then update the lastbuf and update 3033 * bufp. 3034 */ 3035 if (*bufp != NULL) { 3036 lastbuf = *bufp; 3037 bufp = &(*bufp)->b_next; 3038 } 3039 } 3040 buf->b_next = NULL; 3041 ASSERT3P(lastbuf, !=, buf); 3042 IMPLY(lastbuf != NULL, ARC_BUF_LAST(lastbuf)); 3043 3044 return (lastbuf); 3045 } 3046 3047 /* 3048 * Free up buf->b_data and pull the arc_buf_t off of the arc_buf_hdr_t's 3049 * list and free it. 3050 */ 3051 static void 3052 arc_buf_destroy_impl(arc_buf_t *buf) 3053 { 3054 arc_buf_hdr_t *hdr = buf->b_hdr; 3055 3056 /* 3057 * Free up the data associated with the buf but only if we're not 3058 * sharing this with the hdr. If we are sharing it with the hdr, the 3059 * hdr is responsible for doing the free. 3060 */ 3061 if (buf->b_data != NULL) { 3062 /* 3063 * We're about to change the hdr's b_flags. We must either 3064 * hold the hash_lock or be undiscoverable. 3065 */ 3066 ASSERT(HDR_EMPTY_OR_LOCKED(hdr)); 3067 3068 arc_cksum_verify(buf); 3069 arc_buf_unwatch(buf); 3070 3071 if (ARC_BUF_SHARED(buf)) { 3072 arc_hdr_clear_flags(hdr, ARC_FLAG_SHARED_DATA); 3073 } else { 3074 ASSERT(!arc_buf_is_shared(buf)); 3075 uint64_t size = arc_buf_size(buf); 3076 arc_free_data_buf(hdr, buf->b_data, size, buf); 3077 ARCSTAT_INCR(arcstat_overhead_size, -size); 3078 } 3079 buf->b_data = NULL; 3080 3081 /* 3082 * If we have no more encrypted buffers and we've already 3083 * gotten a copy of the decrypted data we can free b_rabd 3084 * to save some space. 3085 */ 3086 if (ARC_BUF_ENCRYPTED(buf) && HDR_HAS_RABD(hdr) && 3087 hdr->b_l1hdr.b_pabd != NULL && !HDR_IO_IN_PROGRESS(hdr)) { 3088 arc_buf_t *b; 3089 for (b = hdr->b_l1hdr.b_buf; b; b = b->b_next) { 3090 if (b != buf && ARC_BUF_ENCRYPTED(b)) 3091 break; 3092 } 3093 if (b == NULL) 3094 arc_hdr_free_abd(hdr, B_TRUE); 3095 } 3096 } 3097 3098 arc_buf_t *lastbuf = arc_buf_remove(hdr, buf); 3099 3100 if (ARC_BUF_SHARED(buf) && !ARC_BUF_COMPRESSED(buf)) { 3101 /* 3102 * If the current arc_buf_t is sharing its data buffer with the 3103 * hdr, then reassign the hdr's b_pabd to share it with the new 3104 * buffer at the end of the list. The shared buffer is always 3105 * the last one on the hdr's buffer list. 3106 * 3107 * There is an equivalent case for compressed bufs, but since 3108 * they aren't guaranteed to be the last buf in the list and 3109 * that is an exceedingly rare case, we just allow that space be 3110 * wasted temporarily. We must also be careful not to share 3111 * encrypted buffers, since they cannot be shared. 3112 */ 3113 if (lastbuf != NULL && !ARC_BUF_ENCRYPTED(lastbuf)) { 3114 /* Only one buf can be shared at once */ 3115 ASSERT(!arc_buf_is_shared(lastbuf)); 3116 /* hdr is uncompressed so can't have compressed buf */ 3117 ASSERT(!ARC_BUF_COMPRESSED(lastbuf)); 3118 3119 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 3120 arc_hdr_free_abd(hdr, B_FALSE); 3121 3122 /* 3123 * We must setup a new shared block between the 3124 * last buffer and the hdr. The data would have 3125 * been allocated by the arc buf so we need to transfer 3126 * ownership to the hdr since it's now being shared. 3127 */ 3128 arc_share_buf(hdr, lastbuf); 3129 } 3130 } else if (HDR_SHARED_DATA(hdr)) { 3131 /* 3132 * Uncompressed shared buffers are always at the end 3133 * of the list. Compressed buffers don't have the 3134 * same requirements. This makes it hard to 3135 * simply assert that the lastbuf is shared so 3136 * we rely on the hdr's compression flags to determine 3137 * if we have a compressed, shared buffer. 3138 */ 3139 ASSERT3P(lastbuf, !=, NULL); 3140 ASSERT(arc_buf_is_shared(lastbuf) || 3141 arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF); 3142 } 3143 3144 /* 3145 * Free the checksum if we're removing the last uncompressed buf from 3146 * this hdr. 3147 */ 3148 if (!arc_hdr_has_uncompressed_buf(hdr)) { 3149 arc_cksum_free(hdr); 3150 } 3151 3152 /* clean up the buf */ 3153 buf->b_hdr = NULL; 3154 kmem_cache_free(buf_cache, buf); 3155 } 3156 3157 static void 3158 arc_hdr_alloc_abd(arc_buf_hdr_t *hdr, int alloc_flags) 3159 { 3160 uint64_t size; 3161 boolean_t alloc_rdata = ((alloc_flags & ARC_HDR_ALLOC_RDATA) != 0); 3162 3163 ASSERT3U(HDR_GET_LSIZE(hdr), >, 0); 3164 ASSERT(HDR_HAS_L1HDR(hdr)); 3165 ASSERT(!HDR_SHARED_DATA(hdr) || alloc_rdata); 3166 IMPLY(alloc_rdata, HDR_PROTECTED(hdr)); 3167 3168 if (alloc_rdata) { 3169 size = HDR_GET_PSIZE(hdr); 3170 ASSERT3P(hdr->b_crypt_hdr.b_rabd, ==, NULL); 3171 hdr->b_crypt_hdr.b_rabd = arc_get_data_abd(hdr, size, hdr, 3172 alloc_flags); 3173 ASSERT3P(hdr->b_crypt_hdr.b_rabd, !=, NULL); 3174 ARCSTAT_INCR(arcstat_raw_size, size); 3175 } else { 3176 size = arc_hdr_size(hdr); 3177 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 3178 hdr->b_l1hdr.b_pabd = arc_get_data_abd(hdr, size, hdr, 3179 alloc_flags); 3180 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 3181 } 3182 3183 ARCSTAT_INCR(arcstat_compressed_size, size); 3184 ARCSTAT_INCR(arcstat_uncompressed_size, HDR_GET_LSIZE(hdr)); 3185 } 3186 3187 static void 3188 arc_hdr_free_abd(arc_buf_hdr_t *hdr, boolean_t free_rdata) 3189 { 3190 uint64_t size = (free_rdata) ? HDR_GET_PSIZE(hdr) : arc_hdr_size(hdr); 3191 3192 ASSERT(HDR_HAS_L1HDR(hdr)); 3193 ASSERT(hdr->b_l1hdr.b_pabd != NULL || HDR_HAS_RABD(hdr)); 3194 IMPLY(free_rdata, HDR_HAS_RABD(hdr)); 3195 3196 /* 3197 * If the hdr is currently being written to the l2arc then 3198 * we defer freeing the data by adding it to the l2arc_free_on_write 3199 * list. The l2arc will free the data once it's finished 3200 * writing it to the l2arc device. 3201 */ 3202 if (HDR_L2_WRITING(hdr)) { 3203 arc_hdr_free_on_write(hdr, free_rdata); 3204 ARCSTAT_BUMP(arcstat_l2_free_on_write); 3205 } else if (free_rdata) { 3206 arc_free_data_abd(hdr, hdr->b_crypt_hdr.b_rabd, size, hdr); 3207 } else { 3208 arc_free_data_abd(hdr, hdr->b_l1hdr.b_pabd, size, hdr); 3209 } 3210 3211 if (free_rdata) { 3212 hdr->b_crypt_hdr.b_rabd = NULL; 3213 ARCSTAT_INCR(arcstat_raw_size, -size); 3214 } else { 3215 hdr->b_l1hdr.b_pabd = NULL; 3216 } 3217 3218 if (hdr->b_l1hdr.b_pabd == NULL && !HDR_HAS_RABD(hdr)) 3219 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_NUMFUNCS; 3220 3221 ARCSTAT_INCR(arcstat_compressed_size, -size); 3222 ARCSTAT_INCR(arcstat_uncompressed_size, -HDR_GET_LSIZE(hdr)); 3223 } 3224 3225 /* 3226 * Allocate empty anonymous ARC header. The header will get its identity 3227 * assigned and buffers attached later as part of read or write operations. 3228 * 3229 * In case of read arc_read() assigns header its identify (b_dva + b_birth), 3230 * inserts it into ARC hash to become globally visible and allocates physical 3231 * (b_pabd) or raw (b_rabd) ABD buffer to read into from disk. On disk read 3232 * completion arc_read_done() allocates ARC buffer(s) as needed, potentially 3233 * sharing one of them with the physical ABD buffer. 3234 * 3235 * In case of write arc_alloc_buf() allocates ARC buffer to be filled with 3236 * data. Then after compression and/or encryption arc_write_ready() allocates 3237 * and fills (or potentially shares) physical (b_pabd) or raw (b_rabd) ABD 3238 * buffer. On disk write completion arc_write_done() assigns the header its 3239 * new identity (b_dva + b_birth) and inserts into ARC hash. 3240 * 3241 * In case of partial overwrite the old data is read first as described. Then 3242 * arc_release() either allocates new anonymous ARC header and moves the ARC 3243 * buffer to it, or reuses the old ARC header by discarding its identity and 3244 * removing it from ARC hash. After buffer modification normal write process 3245 * follows as described. 3246 */ 3247 static arc_buf_hdr_t * 3248 arc_hdr_alloc(uint64_t spa, int32_t psize, int32_t lsize, 3249 boolean_t protected, enum zio_compress compression_type, uint8_t complevel, 3250 arc_buf_contents_t type) 3251 { 3252 arc_buf_hdr_t *hdr; 3253 3254 VERIFY(type == ARC_BUFC_DATA || type == ARC_BUFC_METADATA); 3255 hdr = kmem_cache_alloc(hdr_full_cache, KM_PUSHPAGE); 3256 3257 ASSERT(HDR_EMPTY(hdr)); 3258 #ifdef ZFS_DEBUG 3259 ASSERT3P(hdr->b_l1hdr.b_freeze_cksum, ==, NULL); 3260 #endif 3261 HDR_SET_PSIZE(hdr, psize); 3262 HDR_SET_LSIZE(hdr, lsize); 3263 hdr->b_spa = spa; 3264 hdr->b_type = type; 3265 hdr->b_flags = 0; 3266 arc_hdr_set_flags(hdr, arc_bufc_to_flags(type) | ARC_FLAG_HAS_L1HDR); 3267 arc_hdr_set_compress(hdr, compression_type); 3268 hdr->b_complevel = complevel; 3269 if (protected) 3270 arc_hdr_set_flags(hdr, ARC_FLAG_PROTECTED); 3271 3272 hdr->b_l1hdr.b_state = arc_anon; 3273 hdr->b_l1hdr.b_arc_access = 0; 3274 hdr->b_l1hdr.b_mru_hits = 0; 3275 hdr->b_l1hdr.b_mru_ghost_hits = 0; 3276 hdr->b_l1hdr.b_mfu_hits = 0; 3277 hdr->b_l1hdr.b_mfu_ghost_hits = 0; 3278 hdr->b_l1hdr.b_buf = NULL; 3279 3280 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 3281 3282 return (hdr); 3283 } 3284 3285 /* 3286 * Transition between the two allocation states for the arc_buf_hdr struct. 3287 * The arc_buf_hdr struct can be allocated with (hdr_full_cache) or without 3288 * (hdr_l2only_cache) the fields necessary for the L1 cache - the smaller 3289 * version is used when a cache buffer is only in the L2ARC in order to reduce 3290 * memory usage. 3291 */ 3292 static arc_buf_hdr_t * 3293 arc_hdr_realloc(arc_buf_hdr_t *hdr, kmem_cache_t *old, kmem_cache_t *new) 3294 { 3295 ASSERT(HDR_HAS_L2HDR(hdr)); 3296 3297 arc_buf_hdr_t *nhdr; 3298 l2arc_dev_t *dev = hdr->b_l2hdr.b_dev; 3299 3300 ASSERT((old == hdr_full_cache && new == hdr_l2only_cache) || 3301 (old == hdr_l2only_cache && new == hdr_full_cache)); 3302 3303 nhdr = kmem_cache_alloc(new, KM_PUSHPAGE); 3304 3305 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 3306 buf_hash_remove(hdr); 3307 3308 memcpy(nhdr, hdr, HDR_L2ONLY_SIZE); 3309 3310 if (new == hdr_full_cache) { 3311 arc_hdr_set_flags(nhdr, ARC_FLAG_HAS_L1HDR); 3312 /* 3313 * arc_access and arc_change_state need to be aware that a 3314 * header has just come out of L2ARC, so we set its state to 3315 * l2c_only even though it's about to change. 3316 */ 3317 nhdr->b_l1hdr.b_state = arc_l2c_only; 3318 3319 /* Verify previous threads set to NULL before freeing */ 3320 ASSERT3P(nhdr->b_l1hdr.b_pabd, ==, NULL); 3321 ASSERT(!HDR_HAS_RABD(hdr)); 3322 } else { 3323 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 3324 #ifdef ZFS_DEBUG 3325 ASSERT3P(hdr->b_l1hdr.b_freeze_cksum, ==, NULL); 3326 #endif 3327 3328 /* 3329 * If we've reached here, We must have been called from 3330 * arc_evict_hdr(), as such we should have already been 3331 * removed from any ghost list we were previously on 3332 * (which protects us from racing with arc_evict_state), 3333 * thus no locking is needed during this check. 3334 */ 3335 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 3336 3337 /* 3338 * A buffer must not be moved into the arc_l2c_only 3339 * state if it's not finished being written out to the 3340 * l2arc device. Otherwise, the b_l1hdr.b_pabd field 3341 * might try to be accessed, even though it was removed. 3342 */ 3343 VERIFY(!HDR_L2_WRITING(hdr)); 3344 VERIFY3P(hdr->b_l1hdr.b_pabd, ==, NULL); 3345 ASSERT(!HDR_HAS_RABD(hdr)); 3346 3347 arc_hdr_clear_flags(nhdr, ARC_FLAG_HAS_L1HDR); 3348 } 3349 /* 3350 * The header has been reallocated so we need to re-insert it into any 3351 * lists it was on. 3352 */ 3353 (void) buf_hash_insert(nhdr, NULL); 3354 3355 ASSERT(list_link_active(&hdr->b_l2hdr.b_l2node)); 3356 3357 mutex_enter(&dev->l2ad_mtx); 3358 3359 /* 3360 * We must place the realloc'ed header back into the list at 3361 * the same spot. Otherwise, if it's placed earlier in the list, 3362 * l2arc_write_buffers() could find it during the function's 3363 * write phase, and try to write it out to the l2arc. 3364 */ 3365 list_insert_after(&dev->l2ad_buflist, hdr, nhdr); 3366 list_remove(&dev->l2ad_buflist, hdr); 3367 3368 mutex_exit(&dev->l2ad_mtx); 3369 3370 /* 3371 * Since we're using the pointer address as the tag when 3372 * incrementing and decrementing the l2ad_alloc refcount, we 3373 * must remove the old pointer (that we're about to destroy) and 3374 * add the new pointer to the refcount. Otherwise we'd remove 3375 * the wrong pointer address when calling arc_hdr_destroy() later. 3376 */ 3377 3378 (void) zfs_refcount_remove_many(&dev->l2ad_alloc, 3379 arc_hdr_size(hdr), hdr); 3380 (void) zfs_refcount_add_many(&dev->l2ad_alloc, 3381 arc_hdr_size(nhdr), nhdr); 3382 3383 buf_discard_identity(hdr); 3384 kmem_cache_free(old, hdr); 3385 3386 return (nhdr); 3387 } 3388 3389 /* 3390 * This function is used by the send / receive code to convert a newly 3391 * allocated arc_buf_t to one that is suitable for a raw encrypted write. It 3392 * is also used to allow the root objset block to be updated without altering 3393 * its embedded MACs. Both block types will always be uncompressed so we do not 3394 * have to worry about compression type or psize. 3395 */ 3396 void 3397 arc_convert_to_raw(arc_buf_t *buf, uint64_t dsobj, boolean_t byteorder, 3398 dmu_object_type_t ot, const uint8_t *salt, const uint8_t *iv, 3399 const uint8_t *mac) 3400 { 3401 arc_buf_hdr_t *hdr = buf->b_hdr; 3402 3403 ASSERT(ot == DMU_OT_DNODE || ot == DMU_OT_OBJSET); 3404 ASSERT(HDR_HAS_L1HDR(hdr)); 3405 ASSERT3P(hdr->b_l1hdr.b_state, ==, arc_anon); 3406 3407 buf->b_flags |= (ARC_BUF_FLAG_COMPRESSED | ARC_BUF_FLAG_ENCRYPTED); 3408 arc_hdr_set_flags(hdr, ARC_FLAG_PROTECTED); 3409 hdr->b_crypt_hdr.b_dsobj = dsobj; 3410 hdr->b_crypt_hdr.b_ot = ot; 3411 hdr->b_l1hdr.b_byteswap = (byteorder == ZFS_HOST_BYTEORDER) ? 3412 DMU_BSWAP_NUMFUNCS : DMU_OT_BYTESWAP(ot); 3413 if (!arc_hdr_has_uncompressed_buf(hdr)) 3414 arc_cksum_free(hdr); 3415 3416 if (salt != NULL) 3417 memcpy(hdr->b_crypt_hdr.b_salt, salt, ZIO_DATA_SALT_LEN); 3418 if (iv != NULL) 3419 memcpy(hdr->b_crypt_hdr.b_iv, iv, ZIO_DATA_IV_LEN); 3420 if (mac != NULL) 3421 memcpy(hdr->b_crypt_hdr.b_mac, mac, ZIO_DATA_MAC_LEN); 3422 } 3423 3424 /* 3425 * Allocate a new arc_buf_hdr_t and arc_buf_t and return the buf to the caller. 3426 * The buf is returned thawed since we expect the consumer to modify it. 3427 */ 3428 arc_buf_t * 3429 arc_alloc_buf(spa_t *spa, const void *tag, arc_buf_contents_t type, 3430 int32_t size) 3431 { 3432 arc_buf_hdr_t *hdr = arc_hdr_alloc(spa_load_guid(spa), size, size, 3433 B_FALSE, ZIO_COMPRESS_OFF, 0, type); 3434 3435 arc_buf_t *buf = NULL; 3436 VERIFY0(arc_buf_alloc_impl(hdr, spa, NULL, tag, B_FALSE, B_FALSE, 3437 B_FALSE, B_FALSE, &buf)); 3438 arc_buf_thaw(buf); 3439 3440 return (buf); 3441 } 3442 3443 /* 3444 * Allocate a compressed buf in the same manner as arc_alloc_buf. Don't use this 3445 * for bufs containing metadata. 3446 */ 3447 arc_buf_t * 3448 arc_alloc_compressed_buf(spa_t *spa, const void *tag, uint64_t psize, 3449 uint64_t lsize, enum zio_compress compression_type, uint8_t complevel) 3450 { 3451 ASSERT3U(lsize, >, 0); 3452 ASSERT3U(lsize, >=, psize); 3453 ASSERT3U(compression_type, >, ZIO_COMPRESS_OFF); 3454 ASSERT3U(compression_type, <, ZIO_COMPRESS_FUNCTIONS); 3455 3456 arc_buf_hdr_t *hdr = arc_hdr_alloc(spa_load_guid(spa), psize, lsize, 3457 B_FALSE, compression_type, complevel, ARC_BUFC_DATA); 3458 3459 arc_buf_t *buf = NULL; 3460 VERIFY0(arc_buf_alloc_impl(hdr, spa, NULL, tag, B_FALSE, 3461 B_TRUE, B_FALSE, B_FALSE, &buf)); 3462 arc_buf_thaw(buf); 3463 3464 /* 3465 * To ensure that the hdr has the correct data in it if we call 3466 * arc_untransform() on this buf before it's been written to disk, 3467 * it's easiest if we just set up sharing between the buf and the hdr. 3468 */ 3469 arc_share_buf(hdr, buf); 3470 3471 return (buf); 3472 } 3473 3474 arc_buf_t * 3475 arc_alloc_raw_buf(spa_t *spa, const void *tag, uint64_t dsobj, 3476 boolean_t byteorder, const uint8_t *salt, const uint8_t *iv, 3477 const uint8_t *mac, dmu_object_type_t ot, uint64_t psize, uint64_t lsize, 3478 enum zio_compress compression_type, uint8_t complevel) 3479 { 3480 arc_buf_hdr_t *hdr; 3481 arc_buf_t *buf; 3482 arc_buf_contents_t type = DMU_OT_IS_METADATA(ot) ? 3483 ARC_BUFC_METADATA : ARC_BUFC_DATA; 3484 3485 ASSERT3U(lsize, >, 0); 3486 ASSERT3U(lsize, >=, psize); 3487 ASSERT3U(compression_type, >=, ZIO_COMPRESS_OFF); 3488 ASSERT3U(compression_type, <, ZIO_COMPRESS_FUNCTIONS); 3489 3490 hdr = arc_hdr_alloc(spa_load_guid(spa), psize, lsize, B_TRUE, 3491 compression_type, complevel, type); 3492 3493 hdr->b_crypt_hdr.b_dsobj = dsobj; 3494 hdr->b_crypt_hdr.b_ot = ot; 3495 hdr->b_l1hdr.b_byteswap = (byteorder == ZFS_HOST_BYTEORDER) ? 3496 DMU_BSWAP_NUMFUNCS : DMU_OT_BYTESWAP(ot); 3497 memcpy(hdr->b_crypt_hdr.b_salt, salt, ZIO_DATA_SALT_LEN); 3498 memcpy(hdr->b_crypt_hdr.b_iv, iv, ZIO_DATA_IV_LEN); 3499 memcpy(hdr->b_crypt_hdr.b_mac, mac, ZIO_DATA_MAC_LEN); 3500 3501 /* 3502 * This buffer will be considered encrypted even if the ot is not an 3503 * encrypted type. It will become authenticated instead in 3504 * arc_write_ready(). 3505 */ 3506 buf = NULL; 3507 VERIFY0(arc_buf_alloc_impl(hdr, spa, NULL, tag, B_TRUE, B_TRUE, 3508 B_FALSE, B_FALSE, &buf)); 3509 arc_buf_thaw(buf); 3510 3511 return (buf); 3512 } 3513 3514 static void 3515 l2arc_hdr_arcstats_update(arc_buf_hdr_t *hdr, boolean_t incr, 3516 boolean_t state_only) 3517 { 3518 l2arc_buf_hdr_t *l2hdr = &hdr->b_l2hdr; 3519 l2arc_dev_t *dev = l2hdr->b_dev; 3520 uint64_t lsize = HDR_GET_LSIZE(hdr); 3521 uint64_t psize = HDR_GET_PSIZE(hdr); 3522 uint64_t asize = vdev_psize_to_asize(dev->l2ad_vdev, psize); 3523 arc_buf_contents_t type = hdr->b_type; 3524 int64_t lsize_s; 3525 int64_t psize_s; 3526 int64_t asize_s; 3527 3528 if (incr) { 3529 lsize_s = lsize; 3530 psize_s = psize; 3531 asize_s = asize; 3532 } else { 3533 lsize_s = -lsize; 3534 psize_s = -psize; 3535 asize_s = -asize; 3536 } 3537 3538 /* If the buffer is a prefetch, count it as such. */ 3539 if (HDR_PREFETCH(hdr)) { 3540 ARCSTAT_INCR(arcstat_l2_prefetch_asize, asize_s); 3541 } else { 3542 /* 3543 * We use the value stored in the L2 header upon initial 3544 * caching in L2ARC. This value will be updated in case 3545 * an MRU/MRU_ghost buffer transitions to MFU but the L2ARC 3546 * metadata (log entry) cannot currently be updated. Having 3547 * the ARC state in the L2 header solves the problem of a 3548 * possibly absent L1 header (apparent in buffers restored 3549 * from persistent L2ARC). 3550 */ 3551 switch (hdr->b_l2hdr.b_arcs_state) { 3552 case ARC_STATE_MRU_GHOST: 3553 case ARC_STATE_MRU: 3554 ARCSTAT_INCR(arcstat_l2_mru_asize, asize_s); 3555 break; 3556 case ARC_STATE_MFU_GHOST: 3557 case ARC_STATE_MFU: 3558 ARCSTAT_INCR(arcstat_l2_mfu_asize, asize_s); 3559 break; 3560 default: 3561 break; 3562 } 3563 } 3564 3565 if (state_only) 3566 return; 3567 3568 ARCSTAT_INCR(arcstat_l2_psize, psize_s); 3569 ARCSTAT_INCR(arcstat_l2_lsize, lsize_s); 3570 3571 switch (type) { 3572 case ARC_BUFC_DATA: 3573 ARCSTAT_INCR(arcstat_l2_bufc_data_asize, asize_s); 3574 break; 3575 case ARC_BUFC_METADATA: 3576 ARCSTAT_INCR(arcstat_l2_bufc_metadata_asize, asize_s); 3577 break; 3578 default: 3579 break; 3580 } 3581 } 3582 3583 3584 static void 3585 arc_hdr_l2hdr_destroy(arc_buf_hdr_t *hdr) 3586 { 3587 l2arc_buf_hdr_t *l2hdr = &hdr->b_l2hdr; 3588 l2arc_dev_t *dev = l2hdr->b_dev; 3589 uint64_t psize = HDR_GET_PSIZE(hdr); 3590 uint64_t asize = vdev_psize_to_asize(dev->l2ad_vdev, psize); 3591 3592 ASSERT(MUTEX_HELD(&dev->l2ad_mtx)); 3593 ASSERT(HDR_HAS_L2HDR(hdr)); 3594 3595 list_remove(&dev->l2ad_buflist, hdr); 3596 3597 l2arc_hdr_arcstats_decrement(hdr); 3598 vdev_space_update(dev->l2ad_vdev, -asize, 0, 0); 3599 3600 (void) zfs_refcount_remove_many(&dev->l2ad_alloc, arc_hdr_size(hdr), 3601 hdr); 3602 arc_hdr_clear_flags(hdr, ARC_FLAG_HAS_L2HDR); 3603 } 3604 3605 static void 3606 arc_hdr_destroy(arc_buf_hdr_t *hdr) 3607 { 3608 if (HDR_HAS_L1HDR(hdr)) { 3609 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 3610 ASSERT3P(hdr->b_l1hdr.b_state, ==, arc_anon); 3611 } 3612 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 3613 ASSERT(!HDR_IN_HASH_TABLE(hdr)); 3614 3615 if (HDR_HAS_L2HDR(hdr)) { 3616 l2arc_dev_t *dev = hdr->b_l2hdr.b_dev; 3617 boolean_t buflist_held = MUTEX_HELD(&dev->l2ad_mtx); 3618 3619 if (!buflist_held) 3620 mutex_enter(&dev->l2ad_mtx); 3621 3622 /* 3623 * Even though we checked this conditional above, we 3624 * need to check this again now that we have the 3625 * l2ad_mtx. This is because we could be racing with 3626 * another thread calling l2arc_evict() which might have 3627 * destroyed this header's L2 portion as we were waiting 3628 * to acquire the l2ad_mtx. If that happens, we don't 3629 * want to re-destroy the header's L2 portion. 3630 */ 3631 if (HDR_HAS_L2HDR(hdr)) { 3632 3633 if (!HDR_EMPTY(hdr)) 3634 buf_discard_identity(hdr); 3635 3636 arc_hdr_l2hdr_destroy(hdr); 3637 } 3638 3639 if (!buflist_held) 3640 mutex_exit(&dev->l2ad_mtx); 3641 } 3642 3643 /* 3644 * The header's identify can only be safely discarded once it is no 3645 * longer discoverable. This requires removing it from the hash table 3646 * and the l2arc header list. After this point the hash lock can not 3647 * be used to protect the header. 3648 */ 3649 if (!HDR_EMPTY(hdr)) 3650 buf_discard_identity(hdr); 3651 3652 if (HDR_HAS_L1HDR(hdr)) { 3653 arc_cksum_free(hdr); 3654 3655 while (hdr->b_l1hdr.b_buf != NULL) 3656 arc_buf_destroy_impl(hdr->b_l1hdr.b_buf); 3657 3658 if (hdr->b_l1hdr.b_pabd != NULL) 3659 arc_hdr_free_abd(hdr, B_FALSE); 3660 3661 if (HDR_HAS_RABD(hdr)) 3662 arc_hdr_free_abd(hdr, B_TRUE); 3663 } 3664 3665 ASSERT3P(hdr->b_hash_next, ==, NULL); 3666 if (HDR_HAS_L1HDR(hdr)) { 3667 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 3668 ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); 3669 #ifdef ZFS_DEBUG 3670 ASSERT3P(hdr->b_l1hdr.b_freeze_cksum, ==, NULL); 3671 #endif 3672 kmem_cache_free(hdr_full_cache, hdr); 3673 } else { 3674 kmem_cache_free(hdr_l2only_cache, hdr); 3675 } 3676 } 3677 3678 void 3679 arc_buf_destroy(arc_buf_t *buf, const void *tag) 3680 { 3681 arc_buf_hdr_t *hdr = buf->b_hdr; 3682 3683 if (hdr->b_l1hdr.b_state == arc_anon) { 3684 ASSERT3P(hdr->b_l1hdr.b_buf, ==, buf); 3685 ASSERT(ARC_BUF_LAST(buf)); 3686 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 3687 VERIFY0(remove_reference(hdr, tag)); 3688 return; 3689 } 3690 3691 kmutex_t *hash_lock = HDR_LOCK(hdr); 3692 mutex_enter(hash_lock); 3693 3694 ASSERT3P(hdr, ==, buf->b_hdr); 3695 ASSERT3P(hdr->b_l1hdr.b_buf, !=, NULL); 3696 ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); 3697 ASSERT3P(hdr->b_l1hdr.b_state, !=, arc_anon); 3698 ASSERT3P(buf->b_data, !=, NULL); 3699 3700 arc_buf_destroy_impl(buf); 3701 (void) remove_reference(hdr, tag); 3702 mutex_exit(hash_lock); 3703 } 3704 3705 /* 3706 * Evict the arc_buf_hdr that is provided as a parameter. The resultant 3707 * state of the header is dependent on its state prior to entering this 3708 * function. The following transitions are possible: 3709 * 3710 * - arc_mru -> arc_mru_ghost 3711 * - arc_mfu -> arc_mfu_ghost 3712 * - arc_mru_ghost -> arc_l2c_only 3713 * - arc_mru_ghost -> deleted 3714 * - arc_mfu_ghost -> arc_l2c_only 3715 * - arc_mfu_ghost -> deleted 3716 * - arc_uncached -> deleted 3717 * 3718 * Return total size of evicted data buffers for eviction progress tracking. 3719 * When evicting from ghost states return logical buffer size to make eviction 3720 * progress at the same (or at least comparable) rate as from non-ghost states. 3721 * 3722 * Return *real_evicted for actual ARC size reduction to wake up threads 3723 * waiting for it. For non-ghost states it includes size of evicted data 3724 * buffers (the headers are not freed there). For ghost states it includes 3725 * only the evicted headers size. 3726 */ 3727 static int64_t 3728 arc_evict_hdr(arc_buf_hdr_t *hdr, uint64_t *real_evicted) 3729 { 3730 arc_state_t *evicted_state, *state; 3731 int64_t bytes_evicted = 0; 3732 uint_t min_lifetime = HDR_PRESCIENT_PREFETCH(hdr) ? 3733 arc_min_prescient_prefetch_ms : arc_min_prefetch_ms; 3734 3735 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 3736 ASSERT(HDR_HAS_L1HDR(hdr)); 3737 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 3738 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 3739 ASSERT0(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt)); 3740 3741 *real_evicted = 0; 3742 state = hdr->b_l1hdr.b_state; 3743 if (GHOST_STATE(state)) { 3744 3745 /* 3746 * l2arc_write_buffers() relies on a header's L1 portion 3747 * (i.e. its b_pabd field) during it's write phase. 3748 * Thus, we cannot push a header onto the arc_l2c_only 3749 * state (removing its L1 piece) until the header is 3750 * done being written to the l2arc. 3751 */ 3752 if (HDR_HAS_L2HDR(hdr) && HDR_L2_WRITING(hdr)) { 3753 ARCSTAT_BUMP(arcstat_evict_l2_skip); 3754 return (bytes_evicted); 3755 } 3756 3757 ARCSTAT_BUMP(arcstat_deleted); 3758 bytes_evicted += HDR_GET_LSIZE(hdr); 3759 3760 DTRACE_PROBE1(arc__delete, arc_buf_hdr_t *, hdr); 3761 3762 if (HDR_HAS_L2HDR(hdr)) { 3763 ASSERT(hdr->b_l1hdr.b_pabd == NULL); 3764 ASSERT(!HDR_HAS_RABD(hdr)); 3765 /* 3766 * This buffer is cached on the 2nd Level ARC; 3767 * don't destroy the header. 3768 */ 3769 arc_change_state(arc_l2c_only, hdr); 3770 /* 3771 * dropping from L1+L2 cached to L2-only, 3772 * realloc to remove the L1 header. 3773 */ 3774 (void) arc_hdr_realloc(hdr, hdr_full_cache, 3775 hdr_l2only_cache); 3776 *real_evicted += HDR_FULL_SIZE - HDR_L2ONLY_SIZE; 3777 } else { 3778 arc_change_state(arc_anon, hdr); 3779 arc_hdr_destroy(hdr); 3780 *real_evicted += HDR_FULL_SIZE; 3781 } 3782 return (bytes_evicted); 3783 } 3784 3785 ASSERT(state == arc_mru || state == arc_mfu || state == arc_uncached); 3786 evicted_state = (state == arc_uncached) ? arc_anon : 3787 ((state == arc_mru) ? arc_mru_ghost : arc_mfu_ghost); 3788 3789 /* prefetch buffers have a minimum lifespan */ 3790 if ((hdr->b_flags & (ARC_FLAG_PREFETCH | ARC_FLAG_INDIRECT)) && 3791 ddi_get_lbolt() - hdr->b_l1hdr.b_arc_access < 3792 MSEC_TO_TICK(min_lifetime)) { 3793 ARCSTAT_BUMP(arcstat_evict_skip); 3794 return (bytes_evicted); 3795 } 3796 3797 if (HDR_HAS_L2HDR(hdr)) { 3798 ARCSTAT_INCR(arcstat_evict_l2_cached, HDR_GET_LSIZE(hdr)); 3799 } else { 3800 if (l2arc_write_eligible(hdr->b_spa, hdr)) { 3801 ARCSTAT_INCR(arcstat_evict_l2_eligible, 3802 HDR_GET_LSIZE(hdr)); 3803 3804 switch (state->arcs_state) { 3805 case ARC_STATE_MRU: 3806 ARCSTAT_INCR( 3807 arcstat_evict_l2_eligible_mru, 3808 HDR_GET_LSIZE(hdr)); 3809 break; 3810 case ARC_STATE_MFU: 3811 ARCSTAT_INCR( 3812 arcstat_evict_l2_eligible_mfu, 3813 HDR_GET_LSIZE(hdr)); 3814 break; 3815 default: 3816 break; 3817 } 3818 } else { 3819 ARCSTAT_INCR(arcstat_evict_l2_ineligible, 3820 HDR_GET_LSIZE(hdr)); 3821 } 3822 } 3823 3824 bytes_evicted += arc_hdr_size(hdr); 3825 *real_evicted += arc_hdr_size(hdr); 3826 3827 /* 3828 * If this hdr is being evicted and has a compressed buffer then we 3829 * discard it here before we change states. This ensures that the 3830 * accounting is updated correctly in arc_free_data_impl(). 3831 */ 3832 if (hdr->b_l1hdr.b_pabd != NULL) 3833 arc_hdr_free_abd(hdr, B_FALSE); 3834 3835 if (HDR_HAS_RABD(hdr)) 3836 arc_hdr_free_abd(hdr, B_TRUE); 3837 3838 arc_change_state(evicted_state, hdr); 3839 DTRACE_PROBE1(arc__evict, arc_buf_hdr_t *, hdr); 3840 if (evicted_state == arc_anon) { 3841 arc_hdr_destroy(hdr); 3842 *real_evicted += HDR_FULL_SIZE; 3843 } else { 3844 ASSERT(HDR_IN_HASH_TABLE(hdr)); 3845 } 3846 3847 return (bytes_evicted); 3848 } 3849 3850 static void 3851 arc_set_need_free(void) 3852 { 3853 ASSERT(MUTEX_HELD(&arc_evict_lock)); 3854 int64_t remaining = arc_free_memory() - arc_sys_free / 2; 3855 arc_evict_waiter_t *aw = list_tail(&arc_evict_waiters); 3856 if (aw == NULL) { 3857 arc_need_free = MAX(-remaining, 0); 3858 } else { 3859 arc_need_free = 3860 MAX(-remaining, (int64_t)(aw->aew_count - arc_evict_count)); 3861 } 3862 } 3863 3864 static uint64_t 3865 arc_evict_state_impl(multilist_t *ml, int idx, arc_buf_hdr_t *marker, 3866 uint64_t spa, uint64_t bytes) 3867 { 3868 multilist_sublist_t *mls; 3869 uint64_t bytes_evicted = 0, real_evicted = 0; 3870 arc_buf_hdr_t *hdr; 3871 kmutex_t *hash_lock; 3872 uint_t evict_count = zfs_arc_evict_batch_limit; 3873 3874 ASSERT3P(marker, !=, NULL); 3875 3876 mls = multilist_sublist_lock_idx(ml, idx); 3877 3878 for (hdr = multilist_sublist_prev(mls, marker); likely(hdr != NULL); 3879 hdr = multilist_sublist_prev(mls, marker)) { 3880 if ((evict_count == 0) || (bytes_evicted >= bytes)) 3881 break; 3882 3883 /* 3884 * To keep our iteration location, move the marker 3885 * forward. Since we're not holding hdr's hash lock, we 3886 * must be very careful and not remove 'hdr' from the 3887 * sublist. Otherwise, other consumers might mistake the 3888 * 'hdr' as not being on a sublist when they call the 3889 * multilist_link_active() function (they all rely on 3890 * the hash lock protecting concurrent insertions and 3891 * removals). multilist_sublist_move_forward() was 3892 * specifically implemented to ensure this is the case 3893 * (only 'marker' will be removed and re-inserted). 3894 */ 3895 multilist_sublist_move_forward(mls, marker); 3896 3897 /* 3898 * The only case where the b_spa field should ever be 3899 * zero, is the marker headers inserted by 3900 * arc_evict_state(). It's possible for multiple threads 3901 * to be calling arc_evict_state() concurrently (e.g. 3902 * dsl_pool_close() and zio_inject_fault()), so we must 3903 * skip any markers we see from these other threads. 3904 */ 3905 if (hdr->b_spa == 0) 3906 continue; 3907 3908 /* we're only interested in evicting buffers of a certain spa */ 3909 if (spa != 0 && hdr->b_spa != spa) { 3910 ARCSTAT_BUMP(arcstat_evict_skip); 3911 continue; 3912 } 3913 3914 hash_lock = HDR_LOCK(hdr); 3915 3916 /* 3917 * We aren't calling this function from any code path 3918 * that would already be holding a hash lock, so we're 3919 * asserting on this assumption to be defensive in case 3920 * this ever changes. Without this check, it would be 3921 * possible to incorrectly increment arcstat_mutex_miss 3922 * below (e.g. if the code changed such that we called 3923 * this function with a hash lock held). 3924 */ 3925 ASSERT(!MUTEX_HELD(hash_lock)); 3926 3927 if (mutex_tryenter(hash_lock)) { 3928 uint64_t revicted; 3929 uint64_t evicted = arc_evict_hdr(hdr, &revicted); 3930 mutex_exit(hash_lock); 3931 3932 bytes_evicted += evicted; 3933 real_evicted += revicted; 3934 3935 /* 3936 * If evicted is zero, arc_evict_hdr() must have 3937 * decided to skip this header, don't increment 3938 * evict_count in this case. 3939 */ 3940 if (evicted != 0) 3941 evict_count--; 3942 3943 } else { 3944 ARCSTAT_BUMP(arcstat_mutex_miss); 3945 } 3946 } 3947 3948 multilist_sublist_unlock(mls); 3949 3950 /* 3951 * Increment the count of evicted bytes, and wake up any threads that 3952 * are waiting for the count to reach this value. Since the list is 3953 * ordered by ascending aew_count, we pop off the beginning of the 3954 * list until we reach the end, or a waiter that's past the current 3955 * "count". Doing this outside the loop reduces the number of times 3956 * we need to acquire the global arc_evict_lock. 3957 * 3958 * Only wake when there's sufficient free memory in the system 3959 * (specifically, arc_sys_free/2, which by default is a bit more than 3960 * 1/64th of RAM). See the comments in arc_wait_for_eviction(). 3961 */ 3962 mutex_enter(&arc_evict_lock); 3963 arc_evict_count += real_evicted; 3964 3965 if (arc_free_memory() > arc_sys_free / 2) { 3966 arc_evict_waiter_t *aw; 3967 while ((aw = list_head(&arc_evict_waiters)) != NULL && 3968 aw->aew_count <= arc_evict_count) { 3969 list_remove(&arc_evict_waiters, aw); 3970 cv_broadcast(&aw->aew_cv); 3971 } 3972 } 3973 arc_set_need_free(); 3974 mutex_exit(&arc_evict_lock); 3975 3976 /* 3977 * If the ARC size is reduced from arc_c_max to arc_c_min (especially 3978 * if the average cached block is small), eviction can be on-CPU for 3979 * many seconds. To ensure that other threads that may be bound to 3980 * this CPU are able to make progress, make a voluntary preemption 3981 * call here. 3982 */ 3983 kpreempt(KPREEMPT_SYNC); 3984 3985 return (bytes_evicted); 3986 } 3987 3988 static arc_buf_hdr_t * 3989 arc_state_alloc_marker(void) 3990 { 3991 arc_buf_hdr_t *marker = kmem_cache_alloc(hdr_full_cache, KM_SLEEP); 3992 3993 /* 3994 * A b_spa of 0 is used to indicate that this header is 3995 * a marker. This fact is used in arc_evict_state_impl(). 3996 */ 3997 marker->b_spa = 0; 3998 3999 return (marker); 4000 } 4001 4002 static void 4003 arc_state_free_marker(arc_buf_hdr_t *marker) 4004 { 4005 kmem_cache_free(hdr_full_cache, marker); 4006 } 4007 4008 /* 4009 * Allocate an array of buffer headers used as placeholders during arc state 4010 * eviction. 4011 */ 4012 static arc_buf_hdr_t ** 4013 arc_state_alloc_markers(int count) 4014 { 4015 arc_buf_hdr_t **markers; 4016 4017 markers = kmem_zalloc(sizeof (*markers) * count, KM_SLEEP); 4018 for (int i = 0; i < count; i++) 4019 markers[i] = arc_state_alloc_marker(); 4020 return (markers); 4021 } 4022 4023 static void 4024 arc_state_free_markers(arc_buf_hdr_t **markers, int count) 4025 { 4026 for (int i = 0; i < count; i++) 4027 arc_state_free_marker(markers[i]); 4028 kmem_free(markers, sizeof (*markers) * count); 4029 } 4030 4031 /* 4032 * Evict buffers from the given arc state, until we've removed the 4033 * specified number of bytes. Move the removed buffers to the 4034 * appropriate evict state. 4035 * 4036 * This function makes a "best effort". It skips over any buffers 4037 * it can't get a hash_lock on, and so, may not catch all candidates. 4038 * It may also return without evicting as much space as requested. 4039 * 4040 * If bytes is specified using the special value ARC_EVICT_ALL, this 4041 * will evict all available (i.e. unlocked and evictable) buffers from 4042 * the given arc state; which is used by arc_flush(). 4043 */ 4044 static uint64_t 4045 arc_evict_state(arc_state_t *state, arc_buf_contents_t type, uint64_t spa, 4046 uint64_t bytes) 4047 { 4048 uint64_t total_evicted = 0; 4049 multilist_t *ml = &state->arcs_list[type]; 4050 int num_sublists; 4051 arc_buf_hdr_t **markers; 4052 4053 num_sublists = multilist_get_num_sublists(ml); 4054 4055 /* 4056 * If we've tried to evict from each sublist, made some 4057 * progress, but still have not hit the target number of bytes 4058 * to evict, we want to keep trying. The markers allow us to 4059 * pick up where we left off for each individual sublist, rather 4060 * than starting from the tail each time. 4061 */ 4062 if (zthr_iscurthread(arc_evict_zthr)) { 4063 markers = arc_state_evict_markers; 4064 ASSERT3S(num_sublists, <=, arc_state_evict_marker_count); 4065 } else { 4066 markers = arc_state_alloc_markers(num_sublists); 4067 } 4068 for (int i = 0; i < num_sublists; i++) { 4069 multilist_sublist_t *mls; 4070 4071 mls = multilist_sublist_lock_idx(ml, i); 4072 multilist_sublist_insert_tail(mls, markers[i]); 4073 multilist_sublist_unlock(mls); 4074 } 4075 4076 /* 4077 * While we haven't hit our target number of bytes to evict, or 4078 * we're evicting all available buffers. 4079 */ 4080 while (total_evicted < bytes) { 4081 int sublist_idx = multilist_get_random_index(ml); 4082 uint64_t scan_evicted = 0; 4083 4084 /* 4085 * Start eviction using a randomly selected sublist, 4086 * this is to try and evenly balance eviction across all 4087 * sublists. Always starting at the same sublist 4088 * (e.g. index 0) would cause evictions to favor certain 4089 * sublists over others. 4090 */ 4091 for (int i = 0; i < num_sublists; i++) { 4092 uint64_t bytes_remaining; 4093 uint64_t bytes_evicted; 4094 4095 if (total_evicted < bytes) 4096 bytes_remaining = bytes - total_evicted; 4097 else 4098 break; 4099 4100 bytes_evicted = arc_evict_state_impl(ml, sublist_idx, 4101 markers[sublist_idx], spa, bytes_remaining); 4102 4103 scan_evicted += bytes_evicted; 4104 total_evicted += bytes_evicted; 4105 4106 /* we've reached the end, wrap to the beginning */ 4107 if (++sublist_idx >= num_sublists) 4108 sublist_idx = 0; 4109 } 4110 4111 /* 4112 * If we didn't evict anything during this scan, we have 4113 * no reason to believe we'll evict more during another 4114 * scan, so break the loop. 4115 */ 4116 if (scan_evicted == 0) { 4117 /* This isn't possible, let's make that obvious */ 4118 ASSERT3S(bytes, !=, 0); 4119 4120 /* 4121 * When bytes is ARC_EVICT_ALL, the only way to 4122 * break the loop is when scan_evicted is zero. 4123 * In that case, we actually have evicted enough, 4124 * so we don't want to increment the kstat. 4125 */ 4126 if (bytes != ARC_EVICT_ALL) { 4127 ASSERT3S(total_evicted, <, bytes); 4128 ARCSTAT_BUMP(arcstat_evict_not_enough); 4129 } 4130 4131 break; 4132 } 4133 } 4134 4135 for (int i = 0; i < num_sublists; i++) { 4136 multilist_sublist_t *mls = multilist_sublist_lock_idx(ml, i); 4137 multilist_sublist_remove(mls, markers[i]); 4138 multilist_sublist_unlock(mls); 4139 } 4140 if (markers != arc_state_evict_markers) 4141 arc_state_free_markers(markers, num_sublists); 4142 4143 return (total_evicted); 4144 } 4145 4146 /* 4147 * Flush all "evictable" data of the given type from the arc state 4148 * specified. This will not evict any "active" buffers (i.e. referenced). 4149 * 4150 * When 'retry' is set to B_FALSE, the function will make a single pass 4151 * over the state and evict any buffers that it can. Since it doesn't 4152 * continually retry the eviction, it might end up leaving some buffers 4153 * in the ARC due to lock misses. 4154 * 4155 * When 'retry' is set to B_TRUE, the function will continually retry the 4156 * eviction until *all* evictable buffers have been removed from the 4157 * state. As a result, if concurrent insertions into the state are 4158 * allowed (e.g. if the ARC isn't shutting down), this function might 4159 * wind up in an infinite loop, continually trying to evict buffers. 4160 */ 4161 static uint64_t 4162 arc_flush_state(arc_state_t *state, uint64_t spa, arc_buf_contents_t type, 4163 boolean_t retry) 4164 { 4165 uint64_t evicted = 0; 4166 4167 while (zfs_refcount_count(&state->arcs_esize[type]) != 0) { 4168 evicted += arc_evict_state(state, type, spa, ARC_EVICT_ALL); 4169 4170 if (!retry) 4171 break; 4172 } 4173 4174 return (evicted); 4175 } 4176 4177 /* 4178 * Evict the specified number of bytes from the state specified. This 4179 * function prevents us from trying to evict more from a state's list 4180 * than is "evictable", and to skip evicting altogether when passed a 4181 * negative value for "bytes". In contrast, arc_evict_state() will 4182 * evict everything it can, when passed a negative value for "bytes". 4183 */ 4184 static uint64_t 4185 arc_evict_impl(arc_state_t *state, arc_buf_contents_t type, int64_t bytes) 4186 { 4187 uint64_t delta; 4188 4189 if (bytes > 0 && zfs_refcount_count(&state->arcs_esize[type]) > 0) { 4190 delta = MIN(zfs_refcount_count(&state->arcs_esize[type]), 4191 bytes); 4192 return (arc_evict_state(state, type, 0, delta)); 4193 } 4194 4195 return (0); 4196 } 4197 4198 /* 4199 * Adjust specified fraction, taking into account initial ghost state(s) size, 4200 * ghost hit bytes towards increasing the fraction, ghost hit bytes towards 4201 * decreasing it, plus a balance factor, controlling the decrease rate, used 4202 * to balance metadata vs data. 4203 */ 4204 static uint64_t 4205 arc_evict_adj(uint64_t frac, uint64_t total, uint64_t up, uint64_t down, 4206 uint_t balance) 4207 { 4208 if (total < 8 || up + down == 0) 4209 return (frac); 4210 4211 /* 4212 * We should not have more ghost hits than ghost size, but they 4213 * may get close. Restrict maximum adjustment in that case. 4214 */ 4215 if (up + down >= total / 4) { 4216 uint64_t scale = (up + down) / (total / 8); 4217 up /= scale; 4218 down /= scale; 4219 } 4220 4221 /* Get maximal dynamic range by choosing optimal shifts. */ 4222 int s = highbit64(total); 4223 s = MIN(64 - s, 32); 4224 4225 uint64_t ofrac = (1ULL << 32) - frac; 4226 4227 if (frac >= 4 * ofrac) 4228 up /= frac / (2 * ofrac + 1); 4229 up = (up << s) / (total >> (32 - s)); 4230 if (ofrac >= 4 * frac) 4231 down /= ofrac / (2 * frac + 1); 4232 down = (down << s) / (total >> (32 - s)); 4233 down = down * 100 / balance; 4234 4235 return (frac + up - down); 4236 } 4237 4238 /* 4239 * Evict buffers from the cache, such that arcstat_size is capped by arc_c. 4240 */ 4241 static uint64_t 4242 arc_evict(void) 4243 { 4244 uint64_t asize, bytes, total_evicted = 0; 4245 int64_t e, mrud, mrum, mfud, mfum, w; 4246 static uint64_t ogrd, ogrm, ogfd, ogfm; 4247 static uint64_t gsrd, gsrm, gsfd, gsfm; 4248 uint64_t ngrd, ngrm, ngfd, ngfm; 4249 4250 /* Get current size of ARC states we can evict from. */ 4251 mrud = zfs_refcount_count(&arc_mru->arcs_size[ARC_BUFC_DATA]) + 4252 zfs_refcount_count(&arc_anon->arcs_size[ARC_BUFC_DATA]); 4253 mrum = zfs_refcount_count(&arc_mru->arcs_size[ARC_BUFC_METADATA]) + 4254 zfs_refcount_count(&arc_anon->arcs_size[ARC_BUFC_METADATA]); 4255 mfud = zfs_refcount_count(&arc_mfu->arcs_size[ARC_BUFC_DATA]); 4256 mfum = zfs_refcount_count(&arc_mfu->arcs_size[ARC_BUFC_METADATA]); 4257 uint64_t d = mrud + mfud; 4258 uint64_t m = mrum + mfum; 4259 uint64_t t = d + m; 4260 4261 /* Get ARC ghost hits since last eviction. */ 4262 ngrd = wmsum_value(&arc_mru_ghost->arcs_hits[ARC_BUFC_DATA]); 4263 uint64_t grd = ngrd - ogrd; 4264 ogrd = ngrd; 4265 ngrm = wmsum_value(&arc_mru_ghost->arcs_hits[ARC_BUFC_METADATA]); 4266 uint64_t grm = ngrm - ogrm; 4267 ogrm = ngrm; 4268 ngfd = wmsum_value(&arc_mfu_ghost->arcs_hits[ARC_BUFC_DATA]); 4269 uint64_t gfd = ngfd - ogfd; 4270 ogfd = ngfd; 4271 ngfm = wmsum_value(&arc_mfu_ghost->arcs_hits[ARC_BUFC_METADATA]); 4272 uint64_t gfm = ngfm - ogfm; 4273 ogfm = ngfm; 4274 4275 /* Adjust ARC states balance based on ghost hits. */ 4276 arc_meta = arc_evict_adj(arc_meta, gsrd + gsrm + gsfd + gsfm, 4277 grm + gfm, grd + gfd, zfs_arc_meta_balance); 4278 arc_pd = arc_evict_adj(arc_pd, gsrd + gsfd, grd, gfd, 100); 4279 arc_pm = arc_evict_adj(arc_pm, gsrm + gsfm, grm, gfm, 100); 4280 4281 asize = aggsum_value(&arc_sums.arcstat_size); 4282 int64_t wt = t - (asize - arc_c); 4283 4284 /* 4285 * Try to reduce pinned dnodes if more than 3/4 of wanted metadata 4286 * target is not evictable or if they go over arc_dnode_limit. 4287 */ 4288 int64_t prune = 0; 4289 int64_t dn = wmsum_value(&arc_sums.arcstat_dnode_size); 4290 w = wt * (int64_t)(arc_meta >> 16) >> 16; 4291 if (zfs_refcount_count(&arc_mru->arcs_size[ARC_BUFC_METADATA]) + 4292 zfs_refcount_count(&arc_mfu->arcs_size[ARC_BUFC_METADATA]) - 4293 zfs_refcount_count(&arc_mru->arcs_esize[ARC_BUFC_METADATA]) - 4294 zfs_refcount_count(&arc_mfu->arcs_esize[ARC_BUFC_METADATA]) > 4295 w * 3 / 4) { 4296 prune = dn / sizeof (dnode_t) * 4297 zfs_arc_dnode_reduce_percent / 100; 4298 } else if (dn > arc_dnode_limit) { 4299 prune = (dn - arc_dnode_limit) / sizeof (dnode_t) * 4300 zfs_arc_dnode_reduce_percent / 100; 4301 } 4302 if (prune > 0) 4303 arc_prune_async(prune); 4304 4305 /* Evict MRU metadata. */ 4306 w = wt * (int64_t)(arc_meta * arc_pm >> 48) >> 16; 4307 e = MIN((int64_t)(asize - arc_c), (int64_t)(mrum - w)); 4308 bytes = arc_evict_impl(arc_mru, ARC_BUFC_METADATA, e); 4309 total_evicted += bytes; 4310 mrum -= bytes; 4311 asize -= bytes; 4312 4313 /* Evict MFU metadata. */ 4314 w = wt * (int64_t)(arc_meta >> 16) >> 16; 4315 e = MIN((int64_t)(asize - arc_c), (int64_t)(m - w)); 4316 bytes = arc_evict_impl(arc_mfu, ARC_BUFC_METADATA, e); 4317 total_evicted += bytes; 4318 mfum -= bytes; 4319 asize -= bytes; 4320 4321 /* Evict MRU data. */ 4322 wt -= m - total_evicted; 4323 w = wt * (int64_t)(arc_pd >> 16) >> 16; 4324 e = MIN((int64_t)(asize - arc_c), (int64_t)(mrud - w)); 4325 bytes = arc_evict_impl(arc_mru, ARC_BUFC_DATA, e); 4326 total_evicted += bytes; 4327 mrud -= bytes; 4328 asize -= bytes; 4329 4330 /* Evict MFU data. */ 4331 e = asize - arc_c; 4332 bytes = arc_evict_impl(arc_mfu, ARC_BUFC_DATA, e); 4333 mfud -= bytes; 4334 total_evicted += bytes; 4335 4336 /* 4337 * Evict ghost lists 4338 * 4339 * Size of each state's ghost list represents how much that state 4340 * may grow by shrinking the other states. Would it need to shrink 4341 * other states to zero (that is unlikely), its ghost size would be 4342 * equal to sum of other three state sizes. But excessive ghost 4343 * size may result in false ghost hits (too far back), that may 4344 * never result in real cache hits if several states are competing. 4345 * So choose some arbitraty point of 1/2 of other state sizes. 4346 */ 4347 gsrd = (mrum + mfud + mfum) / 2; 4348 e = zfs_refcount_count(&arc_mru_ghost->arcs_size[ARC_BUFC_DATA]) - 4349 gsrd; 4350 (void) arc_evict_impl(arc_mru_ghost, ARC_BUFC_DATA, e); 4351 4352 gsrm = (mrud + mfud + mfum) / 2; 4353 e = zfs_refcount_count(&arc_mru_ghost->arcs_size[ARC_BUFC_METADATA]) - 4354 gsrm; 4355 (void) arc_evict_impl(arc_mru_ghost, ARC_BUFC_METADATA, e); 4356 4357 gsfd = (mrud + mrum + mfum) / 2; 4358 e = zfs_refcount_count(&arc_mfu_ghost->arcs_size[ARC_BUFC_DATA]) - 4359 gsfd; 4360 (void) arc_evict_impl(arc_mfu_ghost, ARC_BUFC_DATA, e); 4361 4362 gsfm = (mrud + mrum + mfud) / 2; 4363 e = zfs_refcount_count(&arc_mfu_ghost->arcs_size[ARC_BUFC_METADATA]) - 4364 gsfm; 4365 (void) arc_evict_impl(arc_mfu_ghost, ARC_BUFC_METADATA, e); 4366 4367 return (total_evicted); 4368 } 4369 4370 void 4371 arc_flush(spa_t *spa, boolean_t retry) 4372 { 4373 uint64_t guid = 0; 4374 4375 /* 4376 * If retry is B_TRUE, a spa must not be specified since we have 4377 * no good way to determine if all of a spa's buffers have been 4378 * evicted from an arc state. 4379 */ 4380 ASSERT(!retry || spa == NULL); 4381 4382 if (spa != NULL) 4383 guid = spa_load_guid(spa); 4384 4385 (void) arc_flush_state(arc_mru, guid, ARC_BUFC_DATA, retry); 4386 (void) arc_flush_state(arc_mru, guid, ARC_BUFC_METADATA, retry); 4387 4388 (void) arc_flush_state(arc_mfu, guid, ARC_BUFC_DATA, retry); 4389 (void) arc_flush_state(arc_mfu, guid, ARC_BUFC_METADATA, retry); 4390 4391 (void) arc_flush_state(arc_mru_ghost, guid, ARC_BUFC_DATA, retry); 4392 (void) arc_flush_state(arc_mru_ghost, guid, ARC_BUFC_METADATA, retry); 4393 4394 (void) arc_flush_state(arc_mfu_ghost, guid, ARC_BUFC_DATA, retry); 4395 (void) arc_flush_state(arc_mfu_ghost, guid, ARC_BUFC_METADATA, retry); 4396 4397 (void) arc_flush_state(arc_uncached, guid, ARC_BUFC_DATA, retry); 4398 (void) arc_flush_state(arc_uncached, guid, ARC_BUFC_METADATA, retry); 4399 } 4400 4401 void 4402 arc_reduce_target_size(int64_t to_free) 4403 { 4404 uint64_t c = arc_c; 4405 4406 if (c <= arc_c_min) 4407 return; 4408 4409 /* 4410 * All callers want the ARC to actually evict (at least) this much 4411 * memory. Therefore we reduce from the lower of the current size and 4412 * the target size. This way, even if arc_c is much higher than 4413 * arc_size (as can be the case after many calls to arc_freed(), we will 4414 * immediately have arc_c < arc_size and therefore the arc_evict_zthr 4415 * will evict. 4416 */ 4417 uint64_t asize = aggsum_value(&arc_sums.arcstat_size); 4418 if (asize < c) 4419 to_free += c - asize; 4420 arc_c = MAX((int64_t)c - to_free, (int64_t)arc_c_min); 4421 4422 /* See comment in arc_evict_cb_check() on why lock+flag */ 4423 mutex_enter(&arc_evict_lock); 4424 arc_evict_needed = B_TRUE; 4425 mutex_exit(&arc_evict_lock); 4426 zthr_wakeup(arc_evict_zthr); 4427 } 4428 4429 /* 4430 * Determine if the system is under memory pressure and is asking 4431 * to reclaim memory. A return value of B_TRUE indicates that the system 4432 * is under memory pressure and that the arc should adjust accordingly. 4433 */ 4434 boolean_t 4435 arc_reclaim_needed(void) 4436 { 4437 return (arc_available_memory() < 0); 4438 } 4439 4440 void 4441 arc_kmem_reap_soon(void) 4442 { 4443 size_t i; 4444 kmem_cache_t *prev_cache = NULL; 4445 kmem_cache_t *prev_data_cache = NULL; 4446 4447 #ifdef _KERNEL 4448 #if defined(_ILP32) 4449 /* 4450 * Reclaim unused memory from all kmem caches. 4451 */ 4452 kmem_reap(); 4453 #endif 4454 #endif 4455 4456 for (i = 0; i < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT; i++) { 4457 #if defined(_ILP32) 4458 /* reach upper limit of cache size on 32-bit */ 4459 if (zio_buf_cache[i] == NULL) 4460 break; 4461 #endif 4462 if (zio_buf_cache[i] != prev_cache) { 4463 prev_cache = zio_buf_cache[i]; 4464 kmem_cache_reap_now(zio_buf_cache[i]); 4465 } 4466 if (zio_data_buf_cache[i] != prev_data_cache) { 4467 prev_data_cache = zio_data_buf_cache[i]; 4468 kmem_cache_reap_now(zio_data_buf_cache[i]); 4469 } 4470 } 4471 kmem_cache_reap_now(buf_cache); 4472 kmem_cache_reap_now(hdr_full_cache); 4473 kmem_cache_reap_now(hdr_l2only_cache); 4474 kmem_cache_reap_now(zfs_btree_leaf_cache); 4475 abd_cache_reap_now(); 4476 } 4477 4478 static boolean_t 4479 arc_evict_cb_check(void *arg, zthr_t *zthr) 4480 { 4481 (void) arg, (void) zthr; 4482 4483 #ifdef ZFS_DEBUG 4484 /* 4485 * This is necessary in order to keep the kstat information 4486 * up to date for tools that display kstat data such as the 4487 * mdb ::arc dcmd and the Linux crash utility. These tools 4488 * typically do not call kstat's update function, but simply 4489 * dump out stats from the most recent update. Without 4490 * this call, these commands may show stale stats for the 4491 * anon, mru, mru_ghost, mfu, and mfu_ghost lists. Even 4492 * with this call, the data might be out of date if the 4493 * evict thread hasn't been woken recently; but that should 4494 * suffice. The arc_state_t structures can be queried 4495 * directly if more accurate information is needed. 4496 */ 4497 if (arc_ksp != NULL) 4498 arc_ksp->ks_update(arc_ksp, KSTAT_READ); 4499 #endif 4500 4501 /* 4502 * We have to rely on arc_wait_for_eviction() to tell us when to 4503 * evict, rather than checking if we are overflowing here, so that we 4504 * are sure to not leave arc_wait_for_eviction() waiting on aew_cv. 4505 * If we have become "not overflowing" since arc_wait_for_eviction() 4506 * checked, we need to wake it up. We could broadcast the CV here, 4507 * but arc_wait_for_eviction() may have not yet gone to sleep. We 4508 * would need to use a mutex to ensure that this function doesn't 4509 * broadcast until arc_wait_for_eviction() has gone to sleep (e.g. 4510 * the arc_evict_lock). However, the lock ordering of such a lock 4511 * would necessarily be incorrect with respect to the zthr_lock, 4512 * which is held before this function is called, and is held by 4513 * arc_wait_for_eviction() when it calls zthr_wakeup(). 4514 */ 4515 if (arc_evict_needed) 4516 return (B_TRUE); 4517 4518 /* 4519 * If we have buffers in uncached state, evict them periodically. 4520 */ 4521 return ((zfs_refcount_count(&arc_uncached->arcs_esize[ARC_BUFC_DATA]) + 4522 zfs_refcount_count(&arc_uncached->arcs_esize[ARC_BUFC_METADATA]) && 4523 ddi_get_lbolt() - arc_last_uncached_flush > 4524 MSEC_TO_TICK(arc_min_prefetch_ms / 2))); 4525 } 4526 4527 /* 4528 * Keep arc_size under arc_c by running arc_evict which evicts data 4529 * from the ARC. 4530 */ 4531 static void 4532 arc_evict_cb(void *arg, zthr_t *zthr) 4533 { 4534 (void) arg; 4535 4536 uint64_t evicted = 0; 4537 fstrans_cookie_t cookie = spl_fstrans_mark(); 4538 4539 /* Always try to evict from uncached state. */ 4540 arc_last_uncached_flush = ddi_get_lbolt(); 4541 evicted += arc_flush_state(arc_uncached, 0, ARC_BUFC_DATA, B_FALSE); 4542 evicted += arc_flush_state(arc_uncached, 0, ARC_BUFC_METADATA, B_FALSE); 4543 4544 /* Evict from other states only if told to. */ 4545 if (arc_evict_needed) 4546 evicted += arc_evict(); 4547 4548 /* 4549 * If evicted is zero, we couldn't evict anything 4550 * via arc_evict(). This could be due to hash lock 4551 * collisions, but more likely due to the majority of 4552 * arc buffers being unevictable. Therefore, even if 4553 * arc_size is above arc_c, another pass is unlikely to 4554 * be helpful and could potentially cause us to enter an 4555 * infinite loop. Additionally, zthr_iscancelled() is 4556 * checked here so that if the arc is shutting down, the 4557 * broadcast will wake any remaining arc evict waiters. 4558 * 4559 * Note we cancel using zthr instead of arc_evict_zthr 4560 * because the latter may not yet be initializd when the 4561 * callback is first invoked. 4562 */ 4563 mutex_enter(&arc_evict_lock); 4564 arc_evict_needed = !zthr_iscancelled(zthr) && 4565 evicted > 0 && aggsum_compare(&arc_sums.arcstat_size, arc_c) > 0; 4566 if (!arc_evict_needed) { 4567 /* 4568 * We're either no longer overflowing, or we 4569 * can't evict anything more, so we should wake 4570 * arc_get_data_impl() sooner. 4571 */ 4572 arc_evict_waiter_t *aw; 4573 while ((aw = list_remove_head(&arc_evict_waiters)) != NULL) { 4574 cv_broadcast(&aw->aew_cv); 4575 } 4576 arc_set_need_free(); 4577 } 4578 mutex_exit(&arc_evict_lock); 4579 spl_fstrans_unmark(cookie); 4580 } 4581 4582 static boolean_t 4583 arc_reap_cb_check(void *arg, zthr_t *zthr) 4584 { 4585 (void) arg, (void) zthr; 4586 4587 int64_t free_memory = arc_available_memory(); 4588 static int reap_cb_check_counter = 0; 4589 4590 /* 4591 * If a kmem reap is already active, don't schedule more. We must 4592 * check for this because kmem_cache_reap_soon() won't actually 4593 * block on the cache being reaped (this is to prevent callers from 4594 * becoming implicitly blocked by a system-wide kmem reap -- which, 4595 * on a system with many, many full magazines, can take minutes). 4596 */ 4597 if (!kmem_cache_reap_active() && free_memory < 0) { 4598 4599 arc_no_grow = B_TRUE; 4600 arc_warm = B_TRUE; 4601 /* 4602 * Wait at least zfs_grow_retry (default 5) seconds 4603 * before considering growing. 4604 */ 4605 arc_growtime = gethrtime() + SEC2NSEC(arc_grow_retry); 4606 return (B_TRUE); 4607 } else if (free_memory < arc_c >> arc_no_grow_shift) { 4608 arc_no_grow = B_TRUE; 4609 } else if (gethrtime() >= arc_growtime) { 4610 arc_no_grow = B_FALSE; 4611 } 4612 4613 /* 4614 * Called unconditionally every 60 seconds to reclaim unused 4615 * zstd compression and decompression context. This is done 4616 * here to avoid the need for an independent thread. 4617 */ 4618 if (!((reap_cb_check_counter++) % 60)) 4619 zfs_zstd_cache_reap_now(); 4620 4621 return (B_FALSE); 4622 } 4623 4624 /* 4625 * Keep enough free memory in the system by reaping the ARC's kmem 4626 * caches. To cause more slabs to be reapable, we may reduce the 4627 * target size of the cache (arc_c), causing the arc_evict_cb() 4628 * to free more buffers. 4629 */ 4630 static void 4631 arc_reap_cb(void *arg, zthr_t *zthr) 4632 { 4633 (void) arg, (void) zthr; 4634 4635 int64_t free_memory; 4636 fstrans_cookie_t cookie = spl_fstrans_mark(); 4637 4638 /* 4639 * Kick off asynchronous kmem_reap()'s of all our caches. 4640 */ 4641 arc_kmem_reap_soon(); 4642 4643 /* 4644 * Wait at least arc_kmem_cache_reap_retry_ms between 4645 * arc_kmem_reap_soon() calls. Without this check it is possible to 4646 * end up in a situation where we spend lots of time reaping 4647 * caches, while we're near arc_c_min. Waiting here also gives the 4648 * subsequent free memory check a chance of finding that the 4649 * asynchronous reap has already freed enough memory, and we don't 4650 * need to call arc_reduce_target_size(). 4651 */ 4652 delay((hz * arc_kmem_cache_reap_retry_ms + 999) / 1000); 4653 4654 /* 4655 * Reduce the target size as needed to maintain the amount of free 4656 * memory in the system at a fraction of the arc_size (1/128th by 4657 * default). If oversubscribed (free_memory < 0) then reduce the 4658 * target arc_size by the deficit amount plus the fractional 4659 * amount. If free memory is positive but less than the fractional 4660 * amount, reduce by what is needed to hit the fractional amount. 4661 */ 4662 free_memory = arc_available_memory(); 4663 4664 int64_t can_free = arc_c - arc_c_min; 4665 if (can_free > 0) { 4666 int64_t to_free = (can_free >> arc_shrink_shift) - free_memory; 4667 if (to_free > 0) 4668 arc_reduce_target_size(to_free); 4669 } 4670 spl_fstrans_unmark(cookie); 4671 } 4672 4673 #ifdef _KERNEL 4674 /* 4675 * Determine the amount of memory eligible for eviction contained in the 4676 * ARC. All clean data reported by the ghost lists can always be safely 4677 * evicted. Due to arc_c_min, the same does not hold for all clean data 4678 * contained by the regular mru and mfu lists. 4679 * 4680 * In the case of the regular mru and mfu lists, we need to report as 4681 * much clean data as possible, such that evicting that same reported 4682 * data will not bring arc_size below arc_c_min. Thus, in certain 4683 * circumstances, the total amount of clean data in the mru and mfu 4684 * lists might not actually be evictable. 4685 * 4686 * The following two distinct cases are accounted for: 4687 * 4688 * 1. The sum of the amount of dirty data contained by both the mru and 4689 * mfu lists, plus the ARC's other accounting (e.g. the anon list), 4690 * is greater than or equal to arc_c_min. 4691 * (i.e. amount of dirty data >= arc_c_min) 4692 * 4693 * This is the easy case; all clean data contained by the mru and mfu 4694 * lists is evictable. Evicting all clean data can only drop arc_size 4695 * to the amount of dirty data, which is greater than arc_c_min. 4696 * 4697 * 2. The sum of the amount of dirty data contained by both the mru and 4698 * mfu lists, plus the ARC's other accounting (e.g. the anon list), 4699 * is less than arc_c_min. 4700 * (i.e. arc_c_min > amount of dirty data) 4701 * 4702 * 2.1. arc_size is greater than or equal arc_c_min. 4703 * (i.e. arc_size >= arc_c_min > amount of dirty data) 4704 * 4705 * In this case, not all clean data from the regular mru and mfu 4706 * lists is actually evictable; we must leave enough clean data 4707 * to keep arc_size above arc_c_min. Thus, the maximum amount of 4708 * evictable data from the two lists combined, is exactly the 4709 * difference between arc_size and arc_c_min. 4710 * 4711 * 2.2. arc_size is less than arc_c_min 4712 * (i.e. arc_c_min > arc_size > amount of dirty data) 4713 * 4714 * In this case, none of the data contained in the mru and mfu 4715 * lists is evictable, even if it's clean. Since arc_size is 4716 * already below arc_c_min, evicting any more would only 4717 * increase this negative difference. 4718 */ 4719 4720 #endif /* _KERNEL */ 4721 4722 /* 4723 * Adapt arc info given the number of bytes we are trying to add and 4724 * the state that we are coming from. This function is only called 4725 * when we are adding new content to the cache. 4726 */ 4727 static void 4728 arc_adapt(uint64_t bytes) 4729 { 4730 /* 4731 * Wake reap thread if we do not have any available memory 4732 */ 4733 if (arc_reclaim_needed()) { 4734 zthr_wakeup(arc_reap_zthr); 4735 return; 4736 } 4737 4738 if (arc_no_grow) 4739 return; 4740 4741 if (arc_c >= arc_c_max) 4742 return; 4743 4744 /* 4745 * If we're within (2 * maxblocksize) bytes of the target 4746 * cache size, increment the target cache size 4747 */ 4748 if (aggsum_upper_bound(&arc_sums.arcstat_size) + 4749 2 * SPA_MAXBLOCKSIZE >= arc_c) { 4750 uint64_t dc = MAX(bytes, SPA_OLD_MAXBLOCKSIZE); 4751 if (atomic_add_64_nv(&arc_c, dc) > arc_c_max) 4752 arc_c = arc_c_max; 4753 } 4754 } 4755 4756 /* 4757 * Check if arc_size has grown past our upper threshold, determined by 4758 * zfs_arc_overflow_shift. 4759 */ 4760 static arc_ovf_level_t 4761 arc_is_overflowing(boolean_t use_reserve) 4762 { 4763 /* Always allow at least one block of overflow */ 4764 int64_t overflow = MAX(SPA_MAXBLOCKSIZE, 4765 arc_c >> zfs_arc_overflow_shift); 4766 4767 /* 4768 * We just compare the lower bound here for performance reasons. Our 4769 * primary goals are to make sure that the arc never grows without 4770 * bound, and that it can reach its maximum size. This check 4771 * accomplishes both goals. The maximum amount we could run over by is 4772 * 2 * aggsum_borrow_multiplier * NUM_CPUS * the average size of a block 4773 * in the ARC. In practice, that's in the tens of MB, which is low 4774 * enough to be safe. 4775 */ 4776 int64_t over = aggsum_lower_bound(&arc_sums.arcstat_size) - 4777 arc_c - overflow / 2; 4778 if (!use_reserve) 4779 overflow /= 2; 4780 return (over < 0 ? ARC_OVF_NONE : 4781 over < overflow ? ARC_OVF_SOME : ARC_OVF_SEVERE); 4782 } 4783 4784 static abd_t * 4785 arc_get_data_abd(arc_buf_hdr_t *hdr, uint64_t size, const void *tag, 4786 int alloc_flags) 4787 { 4788 arc_buf_contents_t type = arc_buf_type(hdr); 4789 4790 arc_get_data_impl(hdr, size, tag, alloc_flags); 4791 if (alloc_flags & ARC_HDR_ALLOC_LINEAR) 4792 return (abd_alloc_linear(size, type == ARC_BUFC_METADATA)); 4793 else 4794 return (abd_alloc(size, type == ARC_BUFC_METADATA)); 4795 } 4796 4797 static void * 4798 arc_get_data_buf(arc_buf_hdr_t *hdr, uint64_t size, const void *tag) 4799 { 4800 arc_buf_contents_t type = arc_buf_type(hdr); 4801 4802 arc_get_data_impl(hdr, size, tag, 0); 4803 if (type == ARC_BUFC_METADATA) { 4804 return (zio_buf_alloc(size)); 4805 } else { 4806 ASSERT(type == ARC_BUFC_DATA); 4807 return (zio_data_buf_alloc(size)); 4808 } 4809 } 4810 4811 /* 4812 * Wait for the specified amount of data (in bytes) to be evicted from the 4813 * ARC, and for there to be sufficient free memory in the system. Waiting for 4814 * eviction ensures that the memory used by the ARC decreases. Waiting for 4815 * free memory ensures that the system won't run out of free pages, regardless 4816 * of ARC behavior and settings. See arc_lowmem_init(). 4817 */ 4818 void 4819 arc_wait_for_eviction(uint64_t amount, boolean_t use_reserve) 4820 { 4821 switch (arc_is_overflowing(use_reserve)) { 4822 case ARC_OVF_NONE: 4823 return; 4824 case ARC_OVF_SOME: 4825 /* 4826 * This is a bit racy without taking arc_evict_lock, but the 4827 * worst that can happen is we either call zthr_wakeup() extra 4828 * time due to race with other thread here, or the set flag 4829 * get cleared by arc_evict_cb(), which is unlikely due to 4830 * big hysteresis, but also not important since at this level 4831 * of overflow the eviction is purely advisory. Same time 4832 * taking the global lock here every time without waiting for 4833 * the actual eviction creates a significant lock contention. 4834 */ 4835 if (!arc_evict_needed) { 4836 arc_evict_needed = B_TRUE; 4837 zthr_wakeup(arc_evict_zthr); 4838 } 4839 return; 4840 case ARC_OVF_SEVERE: 4841 default: 4842 { 4843 arc_evict_waiter_t aw; 4844 list_link_init(&aw.aew_node); 4845 cv_init(&aw.aew_cv, NULL, CV_DEFAULT, NULL); 4846 4847 uint64_t last_count = 0; 4848 mutex_enter(&arc_evict_lock); 4849 if (!list_is_empty(&arc_evict_waiters)) { 4850 arc_evict_waiter_t *last = 4851 list_tail(&arc_evict_waiters); 4852 last_count = last->aew_count; 4853 } else if (!arc_evict_needed) { 4854 arc_evict_needed = B_TRUE; 4855 zthr_wakeup(arc_evict_zthr); 4856 } 4857 /* 4858 * Note, the last waiter's count may be less than 4859 * arc_evict_count if we are low on memory in which 4860 * case arc_evict_state_impl() may have deferred 4861 * wakeups (but still incremented arc_evict_count). 4862 */ 4863 aw.aew_count = MAX(last_count, arc_evict_count) + amount; 4864 4865 list_insert_tail(&arc_evict_waiters, &aw); 4866 4867 arc_set_need_free(); 4868 4869 DTRACE_PROBE3(arc__wait__for__eviction, 4870 uint64_t, amount, 4871 uint64_t, arc_evict_count, 4872 uint64_t, aw.aew_count); 4873 4874 /* 4875 * We will be woken up either when arc_evict_count reaches 4876 * aew_count, or when the ARC is no longer overflowing and 4877 * eviction completes. 4878 * In case of "false" wakeup, we will still be on the list. 4879 */ 4880 do { 4881 cv_wait(&aw.aew_cv, &arc_evict_lock); 4882 } while (list_link_active(&aw.aew_node)); 4883 mutex_exit(&arc_evict_lock); 4884 4885 cv_destroy(&aw.aew_cv); 4886 } 4887 } 4888 } 4889 4890 /* 4891 * Allocate a block and return it to the caller. If we are hitting the 4892 * hard limit for the cache size, we must sleep, waiting for the eviction 4893 * thread to catch up. If we're past the target size but below the hard 4894 * limit, we'll only signal the reclaim thread and continue on. 4895 */ 4896 static void 4897 arc_get_data_impl(arc_buf_hdr_t *hdr, uint64_t size, const void *tag, 4898 int alloc_flags) 4899 { 4900 arc_adapt(size); 4901 4902 /* 4903 * If arc_size is currently overflowing, we must be adding data 4904 * faster than we are evicting. To ensure we don't compound the 4905 * problem by adding more data and forcing arc_size to grow even 4906 * further past it's target size, we wait for the eviction thread to 4907 * make some progress. We also wait for there to be sufficient free 4908 * memory in the system, as measured by arc_free_memory(). 4909 * 4910 * Specifically, we wait for zfs_arc_eviction_pct percent of the 4911 * requested size to be evicted. This should be more than 100%, to 4912 * ensure that that progress is also made towards getting arc_size 4913 * under arc_c. See the comment above zfs_arc_eviction_pct. 4914 */ 4915 arc_wait_for_eviction(size * zfs_arc_eviction_pct / 100, 4916 alloc_flags & ARC_HDR_USE_RESERVE); 4917 4918 arc_buf_contents_t type = arc_buf_type(hdr); 4919 if (type == ARC_BUFC_METADATA) { 4920 arc_space_consume(size, ARC_SPACE_META); 4921 } else { 4922 arc_space_consume(size, ARC_SPACE_DATA); 4923 } 4924 4925 /* 4926 * Update the state size. Note that ghost states have a 4927 * "ghost size" and so don't need to be updated. 4928 */ 4929 arc_state_t *state = hdr->b_l1hdr.b_state; 4930 if (!GHOST_STATE(state)) { 4931 4932 (void) zfs_refcount_add_many(&state->arcs_size[type], size, 4933 tag); 4934 4935 /* 4936 * If this is reached via arc_read, the link is 4937 * protected by the hash lock. If reached via 4938 * arc_buf_alloc, the header should not be accessed by 4939 * any other thread. And, if reached via arc_read_done, 4940 * the hash lock will protect it if it's found in the 4941 * hash table; otherwise no other thread should be 4942 * trying to [add|remove]_reference it. 4943 */ 4944 if (multilist_link_active(&hdr->b_l1hdr.b_arc_node)) { 4945 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 4946 (void) zfs_refcount_add_many(&state->arcs_esize[type], 4947 size, tag); 4948 } 4949 } 4950 } 4951 4952 static void 4953 arc_free_data_abd(arc_buf_hdr_t *hdr, abd_t *abd, uint64_t size, 4954 const void *tag) 4955 { 4956 arc_free_data_impl(hdr, size, tag); 4957 abd_free(abd); 4958 } 4959 4960 static void 4961 arc_free_data_buf(arc_buf_hdr_t *hdr, void *buf, uint64_t size, const void *tag) 4962 { 4963 arc_buf_contents_t type = arc_buf_type(hdr); 4964 4965 arc_free_data_impl(hdr, size, tag); 4966 if (type == ARC_BUFC_METADATA) { 4967 zio_buf_free(buf, size); 4968 } else { 4969 ASSERT(type == ARC_BUFC_DATA); 4970 zio_data_buf_free(buf, size); 4971 } 4972 } 4973 4974 /* 4975 * Free the arc data buffer. 4976 */ 4977 static void 4978 arc_free_data_impl(arc_buf_hdr_t *hdr, uint64_t size, const void *tag) 4979 { 4980 arc_state_t *state = hdr->b_l1hdr.b_state; 4981 arc_buf_contents_t type = arc_buf_type(hdr); 4982 4983 /* protected by hash lock, if in the hash table */ 4984 if (multilist_link_active(&hdr->b_l1hdr.b_arc_node)) { 4985 ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); 4986 ASSERT(state != arc_anon && state != arc_l2c_only); 4987 4988 (void) zfs_refcount_remove_many(&state->arcs_esize[type], 4989 size, tag); 4990 } 4991 (void) zfs_refcount_remove_many(&state->arcs_size[type], size, tag); 4992 4993 VERIFY3U(hdr->b_type, ==, type); 4994 if (type == ARC_BUFC_METADATA) { 4995 arc_space_return(size, ARC_SPACE_META); 4996 } else { 4997 ASSERT(type == ARC_BUFC_DATA); 4998 arc_space_return(size, ARC_SPACE_DATA); 4999 } 5000 } 5001 5002 /* 5003 * This routine is called whenever a buffer is accessed. 5004 */ 5005 static void 5006 arc_access(arc_buf_hdr_t *hdr, arc_flags_t arc_flags, boolean_t hit) 5007 { 5008 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 5009 ASSERT(HDR_HAS_L1HDR(hdr)); 5010 5011 /* 5012 * Update buffer prefetch status. 5013 */ 5014 boolean_t was_prefetch = HDR_PREFETCH(hdr); 5015 boolean_t now_prefetch = arc_flags & ARC_FLAG_PREFETCH; 5016 if (was_prefetch != now_prefetch) { 5017 if (was_prefetch) { 5018 ARCSTAT_CONDSTAT(hit, demand_hit, demand_iohit, 5019 HDR_PRESCIENT_PREFETCH(hdr), prescient, predictive, 5020 prefetch); 5021 } 5022 if (HDR_HAS_L2HDR(hdr)) 5023 l2arc_hdr_arcstats_decrement_state(hdr); 5024 if (was_prefetch) { 5025 arc_hdr_clear_flags(hdr, 5026 ARC_FLAG_PREFETCH | ARC_FLAG_PRESCIENT_PREFETCH); 5027 } else { 5028 arc_hdr_set_flags(hdr, ARC_FLAG_PREFETCH); 5029 } 5030 if (HDR_HAS_L2HDR(hdr)) 5031 l2arc_hdr_arcstats_increment_state(hdr); 5032 } 5033 if (now_prefetch) { 5034 if (arc_flags & ARC_FLAG_PRESCIENT_PREFETCH) { 5035 arc_hdr_set_flags(hdr, ARC_FLAG_PRESCIENT_PREFETCH); 5036 ARCSTAT_BUMP(arcstat_prescient_prefetch); 5037 } else { 5038 ARCSTAT_BUMP(arcstat_predictive_prefetch); 5039 } 5040 } 5041 if (arc_flags & ARC_FLAG_L2CACHE) 5042 arc_hdr_set_flags(hdr, ARC_FLAG_L2CACHE); 5043 5044 clock_t now = ddi_get_lbolt(); 5045 if (hdr->b_l1hdr.b_state == arc_anon) { 5046 arc_state_t *new_state; 5047 /* 5048 * This buffer is not in the cache, and does not appear in 5049 * our "ghost" lists. Add it to the MRU or uncached state. 5050 */ 5051 ASSERT0(hdr->b_l1hdr.b_arc_access); 5052 hdr->b_l1hdr.b_arc_access = now; 5053 if (HDR_UNCACHED(hdr)) { 5054 new_state = arc_uncached; 5055 DTRACE_PROBE1(new_state__uncached, arc_buf_hdr_t *, 5056 hdr); 5057 } else { 5058 new_state = arc_mru; 5059 DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, hdr); 5060 } 5061 arc_change_state(new_state, hdr); 5062 } else if (hdr->b_l1hdr.b_state == arc_mru) { 5063 /* 5064 * This buffer has been accessed once recently and either 5065 * its read is still in progress or it is in the cache. 5066 */ 5067 if (HDR_IO_IN_PROGRESS(hdr)) { 5068 hdr->b_l1hdr.b_arc_access = now; 5069 return; 5070 } 5071 hdr->b_l1hdr.b_mru_hits++; 5072 ARCSTAT_BUMP(arcstat_mru_hits); 5073 5074 /* 5075 * If the previous access was a prefetch, then it already 5076 * handled possible promotion, so nothing more to do for now. 5077 */ 5078 if (was_prefetch) { 5079 hdr->b_l1hdr.b_arc_access = now; 5080 return; 5081 } 5082 5083 /* 5084 * If more than ARC_MINTIME have passed from the previous 5085 * hit, promote the buffer to the MFU state. 5086 */ 5087 if (ddi_time_after(now, hdr->b_l1hdr.b_arc_access + 5088 ARC_MINTIME)) { 5089 hdr->b_l1hdr.b_arc_access = now; 5090 DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, hdr); 5091 arc_change_state(arc_mfu, hdr); 5092 } 5093 } else if (hdr->b_l1hdr.b_state == arc_mru_ghost) { 5094 arc_state_t *new_state; 5095 /* 5096 * This buffer has been accessed once recently, but was 5097 * evicted from the cache. Would we have bigger MRU, it 5098 * would be an MRU hit, so handle it the same way, except 5099 * we don't need to check the previous access time. 5100 */ 5101 hdr->b_l1hdr.b_mru_ghost_hits++; 5102 ARCSTAT_BUMP(arcstat_mru_ghost_hits); 5103 hdr->b_l1hdr.b_arc_access = now; 5104 wmsum_add(&arc_mru_ghost->arcs_hits[arc_buf_type(hdr)], 5105 arc_hdr_size(hdr)); 5106 if (was_prefetch) { 5107 new_state = arc_mru; 5108 DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, hdr); 5109 } else { 5110 new_state = arc_mfu; 5111 DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, hdr); 5112 } 5113 arc_change_state(new_state, hdr); 5114 } else if (hdr->b_l1hdr.b_state == arc_mfu) { 5115 /* 5116 * This buffer has been accessed more than once and either 5117 * still in the cache or being restored from one of ghosts. 5118 */ 5119 if (!HDR_IO_IN_PROGRESS(hdr)) { 5120 hdr->b_l1hdr.b_mfu_hits++; 5121 ARCSTAT_BUMP(arcstat_mfu_hits); 5122 } 5123 hdr->b_l1hdr.b_arc_access = now; 5124 } else if (hdr->b_l1hdr.b_state == arc_mfu_ghost) { 5125 /* 5126 * This buffer has been accessed more than once recently, but 5127 * has been evicted from the cache. Would we have bigger MFU 5128 * it would stay in cache, so move it back to MFU state. 5129 */ 5130 hdr->b_l1hdr.b_mfu_ghost_hits++; 5131 ARCSTAT_BUMP(arcstat_mfu_ghost_hits); 5132 hdr->b_l1hdr.b_arc_access = now; 5133 wmsum_add(&arc_mfu_ghost->arcs_hits[arc_buf_type(hdr)], 5134 arc_hdr_size(hdr)); 5135 DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, hdr); 5136 arc_change_state(arc_mfu, hdr); 5137 } else if (hdr->b_l1hdr.b_state == arc_uncached) { 5138 /* 5139 * This buffer is uncacheable, but we got a hit. Probably 5140 * a demand read after prefetch. Nothing more to do here. 5141 */ 5142 if (!HDR_IO_IN_PROGRESS(hdr)) 5143 ARCSTAT_BUMP(arcstat_uncached_hits); 5144 hdr->b_l1hdr.b_arc_access = now; 5145 } else if (hdr->b_l1hdr.b_state == arc_l2c_only) { 5146 /* 5147 * This buffer is on the 2nd Level ARC and was not accessed 5148 * for a long time, so treat it as new and put into MRU. 5149 */ 5150 hdr->b_l1hdr.b_arc_access = now; 5151 DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, hdr); 5152 arc_change_state(arc_mru, hdr); 5153 } else { 5154 cmn_err(CE_PANIC, "invalid arc state 0x%p", 5155 hdr->b_l1hdr.b_state); 5156 } 5157 } 5158 5159 /* 5160 * This routine is called by dbuf_hold() to update the arc_access() state 5161 * which otherwise would be skipped for entries in the dbuf cache. 5162 */ 5163 void 5164 arc_buf_access(arc_buf_t *buf) 5165 { 5166 arc_buf_hdr_t *hdr = buf->b_hdr; 5167 5168 /* 5169 * Avoid taking the hash_lock when possible as an optimization. 5170 * The header must be checked again under the hash_lock in order 5171 * to handle the case where it is concurrently being released. 5172 */ 5173 if (hdr->b_l1hdr.b_state == arc_anon || HDR_EMPTY(hdr)) 5174 return; 5175 5176 kmutex_t *hash_lock = HDR_LOCK(hdr); 5177 mutex_enter(hash_lock); 5178 5179 if (hdr->b_l1hdr.b_state == arc_anon || HDR_EMPTY(hdr)) { 5180 mutex_exit(hash_lock); 5181 ARCSTAT_BUMP(arcstat_access_skip); 5182 return; 5183 } 5184 5185 ASSERT(hdr->b_l1hdr.b_state == arc_mru || 5186 hdr->b_l1hdr.b_state == arc_mfu || 5187 hdr->b_l1hdr.b_state == arc_uncached); 5188 5189 DTRACE_PROBE1(arc__hit, arc_buf_hdr_t *, hdr); 5190 arc_access(hdr, 0, B_TRUE); 5191 mutex_exit(hash_lock); 5192 5193 ARCSTAT_BUMP(arcstat_hits); 5194 ARCSTAT_CONDSTAT(B_TRUE /* demand */, demand, prefetch, 5195 !HDR_ISTYPE_METADATA(hdr), data, metadata, hits); 5196 } 5197 5198 /* a generic arc_read_done_func_t which you can use */ 5199 void 5200 arc_bcopy_func(zio_t *zio, const zbookmark_phys_t *zb, const blkptr_t *bp, 5201 arc_buf_t *buf, void *arg) 5202 { 5203 (void) zio, (void) zb, (void) bp; 5204 5205 if (buf == NULL) 5206 return; 5207 5208 memcpy(arg, buf->b_data, arc_buf_size(buf)); 5209 arc_buf_destroy(buf, arg); 5210 } 5211 5212 /* a generic arc_read_done_func_t */ 5213 void 5214 arc_getbuf_func(zio_t *zio, const zbookmark_phys_t *zb, const blkptr_t *bp, 5215 arc_buf_t *buf, void *arg) 5216 { 5217 (void) zb, (void) bp; 5218 arc_buf_t **bufp = arg; 5219 5220 if (buf == NULL) { 5221 ASSERT(zio == NULL || zio->io_error != 0); 5222 *bufp = NULL; 5223 } else { 5224 ASSERT(zio == NULL || zio->io_error == 0); 5225 *bufp = buf; 5226 ASSERT(buf->b_data != NULL); 5227 } 5228 } 5229 5230 static void 5231 arc_hdr_verify(arc_buf_hdr_t *hdr, blkptr_t *bp) 5232 { 5233 if (BP_IS_HOLE(bp) || BP_IS_EMBEDDED(bp)) { 5234 ASSERT3U(HDR_GET_PSIZE(hdr), ==, 0); 5235 ASSERT3U(arc_hdr_get_compress(hdr), ==, ZIO_COMPRESS_OFF); 5236 } else { 5237 if (HDR_COMPRESSION_ENABLED(hdr)) { 5238 ASSERT3U(arc_hdr_get_compress(hdr), ==, 5239 BP_GET_COMPRESS(bp)); 5240 } 5241 ASSERT3U(HDR_GET_LSIZE(hdr), ==, BP_GET_LSIZE(bp)); 5242 ASSERT3U(HDR_GET_PSIZE(hdr), ==, BP_GET_PSIZE(bp)); 5243 ASSERT3U(!!HDR_PROTECTED(hdr), ==, BP_IS_PROTECTED(bp)); 5244 } 5245 } 5246 5247 static void 5248 arc_read_done(zio_t *zio) 5249 { 5250 blkptr_t *bp = zio->io_bp; 5251 arc_buf_hdr_t *hdr = zio->io_private; 5252 kmutex_t *hash_lock = NULL; 5253 arc_callback_t *callback_list; 5254 arc_callback_t *acb; 5255 5256 /* 5257 * The hdr was inserted into hash-table and removed from lists 5258 * prior to starting I/O. We should find this header, since 5259 * it's in the hash table, and it should be legit since it's 5260 * not possible to evict it during the I/O. The only possible 5261 * reason for it not to be found is if we were freed during the 5262 * read. 5263 */ 5264 if (HDR_IN_HASH_TABLE(hdr)) { 5265 arc_buf_hdr_t *found; 5266 5267 ASSERT3U(hdr->b_birth, ==, BP_GET_BIRTH(zio->io_bp)); 5268 ASSERT3U(hdr->b_dva.dva_word[0], ==, 5269 BP_IDENTITY(zio->io_bp)->dva_word[0]); 5270 ASSERT3U(hdr->b_dva.dva_word[1], ==, 5271 BP_IDENTITY(zio->io_bp)->dva_word[1]); 5272 5273 found = buf_hash_find(hdr->b_spa, zio->io_bp, &hash_lock); 5274 5275 ASSERT((found == hdr && 5276 DVA_EQUAL(&hdr->b_dva, BP_IDENTITY(zio->io_bp))) || 5277 (found == hdr && HDR_L2_READING(hdr))); 5278 ASSERT3P(hash_lock, !=, NULL); 5279 } 5280 5281 if (BP_IS_PROTECTED(bp)) { 5282 hdr->b_crypt_hdr.b_ot = BP_GET_TYPE(bp); 5283 hdr->b_crypt_hdr.b_dsobj = zio->io_bookmark.zb_objset; 5284 zio_crypt_decode_params_bp(bp, hdr->b_crypt_hdr.b_salt, 5285 hdr->b_crypt_hdr.b_iv); 5286 5287 if (zio->io_error == 0) { 5288 if (BP_GET_TYPE(bp) == DMU_OT_INTENT_LOG) { 5289 void *tmpbuf; 5290 5291 tmpbuf = abd_borrow_buf_copy(zio->io_abd, 5292 sizeof (zil_chain_t)); 5293 zio_crypt_decode_mac_zil(tmpbuf, 5294 hdr->b_crypt_hdr.b_mac); 5295 abd_return_buf(zio->io_abd, tmpbuf, 5296 sizeof (zil_chain_t)); 5297 } else { 5298 zio_crypt_decode_mac_bp(bp, 5299 hdr->b_crypt_hdr.b_mac); 5300 } 5301 } 5302 } 5303 5304 if (zio->io_error == 0) { 5305 /* byteswap if necessary */ 5306 if (BP_SHOULD_BYTESWAP(zio->io_bp)) { 5307 if (BP_GET_LEVEL(zio->io_bp) > 0) { 5308 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_UINT64; 5309 } else { 5310 hdr->b_l1hdr.b_byteswap = 5311 DMU_OT_BYTESWAP(BP_GET_TYPE(zio->io_bp)); 5312 } 5313 } else { 5314 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_NUMFUNCS; 5315 } 5316 if (!HDR_L2_READING(hdr)) { 5317 hdr->b_complevel = zio->io_prop.zp_complevel; 5318 } 5319 } 5320 5321 arc_hdr_clear_flags(hdr, ARC_FLAG_L2_EVICTED); 5322 if (l2arc_noprefetch && HDR_PREFETCH(hdr)) 5323 arc_hdr_clear_flags(hdr, ARC_FLAG_L2CACHE); 5324 5325 callback_list = hdr->b_l1hdr.b_acb; 5326 ASSERT3P(callback_list, !=, NULL); 5327 hdr->b_l1hdr.b_acb = NULL; 5328 5329 /* 5330 * If a read request has a callback (i.e. acb_done is not NULL), then we 5331 * make a buf containing the data according to the parameters which were 5332 * passed in. The implementation of arc_buf_alloc_impl() ensures that we 5333 * aren't needlessly decompressing the data multiple times. 5334 */ 5335 int callback_cnt = 0; 5336 for (acb = callback_list; acb != NULL; acb = acb->acb_next) { 5337 5338 /* We need the last one to call below in original order. */ 5339 callback_list = acb; 5340 5341 if (!acb->acb_done || acb->acb_nobuf) 5342 continue; 5343 5344 callback_cnt++; 5345 5346 if (zio->io_error != 0) 5347 continue; 5348 5349 int error = arc_buf_alloc_impl(hdr, zio->io_spa, 5350 &acb->acb_zb, acb->acb_private, acb->acb_encrypted, 5351 acb->acb_compressed, acb->acb_noauth, B_TRUE, 5352 &acb->acb_buf); 5353 5354 /* 5355 * Assert non-speculative zios didn't fail because an 5356 * encryption key wasn't loaded 5357 */ 5358 ASSERT((zio->io_flags & ZIO_FLAG_SPECULATIVE) || 5359 error != EACCES); 5360 5361 /* 5362 * If we failed to decrypt, report an error now (as the zio 5363 * layer would have done if it had done the transforms). 5364 */ 5365 if (error == ECKSUM) { 5366 ASSERT(BP_IS_PROTECTED(bp)); 5367 error = SET_ERROR(EIO); 5368 if ((zio->io_flags & ZIO_FLAG_SPECULATIVE) == 0) { 5369 spa_log_error(zio->io_spa, &acb->acb_zb, 5370 BP_GET_LOGICAL_BIRTH(zio->io_bp)); 5371 (void) zfs_ereport_post( 5372 FM_EREPORT_ZFS_AUTHENTICATION, 5373 zio->io_spa, NULL, &acb->acb_zb, zio, 0); 5374 } 5375 } 5376 5377 if (error != 0) { 5378 /* 5379 * Decompression or decryption failed. Set 5380 * io_error so that when we call acb_done 5381 * (below), we will indicate that the read 5382 * failed. Note that in the unusual case 5383 * where one callback is compressed and another 5384 * uncompressed, we will mark all of them 5385 * as failed, even though the uncompressed 5386 * one can't actually fail. In this case, 5387 * the hdr will not be anonymous, because 5388 * if there are multiple callbacks, it's 5389 * because multiple threads found the same 5390 * arc buf in the hash table. 5391 */ 5392 zio->io_error = error; 5393 } 5394 } 5395 5396 /* 5397 * If there are multiple callbacks, we must have the hash lock, 5398 * because the only way for multiple threads to find this hdr is 5399 * in the hash table. This ensures that if there are multiple 5400 * callbacks, the hdr is not anonymous. If it were anonymous, 5401 * we couldn't use arc_buf_destroy() in the error case below. 5402 */ 5403 ASSERT(callback_cnt < 2 || hash_lock != NULL); 5404 5405 if (zio->io_error == 0) { 5406 arc_hdr_verify(hdr, zio->io_bp); 5407 } else { 5408 arc_hdr_set_flags(hdr, ARC_FLAG_IO_ERROR); 5409 if (hdr->b_l1hdr.b_state != arc_anon) 5410 arc_change_state(arc_anon, hdr); 5411 if (HDR_IN_HASH_TABLE(hdr)) 5412 buf_hash_remove(hdr); 5413 } 5414 5415 arc_hdr_clear_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 5416 (void) remove_reference(hdr, hdr); 5417 5418 if (hash_lock != NULL) 5419 mutex_exit(hash_lock); 5420 5421 /* execute each callback and free its structure */ 5422 while ((acb = callback_list) != NULL) { 5423 if (acb->acb_done != NULL) { 5424 if (zio->io_error != 0 && acb->acb_buf != NULL) { 5425 /* 5426 * If arc_buf_alloc_impl() fails during 5427 * decompression, the buf will still be 5428 * allocated, and needs to be freed here. 5429 */ 5430 arc_buf_destroy(acb->acb_buf, 5431 acb->acb_private); 5432 acb->acb_buf = NULL; 5433 } 5434 acb->acb_done(zio, &zio->io_bookmark, zio->io_bp, 5435 acb->acb_buf, acb->acb_private); 5436 } 5437 5438 if (acb->acb_zio_dummy != NULL) { 5439 acb->acb_zio_dummy->io_error = zio->io_error; 5440 zio_nowait(acb->acb_zio_dummy); 5441 } 5442 5443 callback_list = acb->acb_prev; 5444 if (acb->acb_wait) { 5445 mutex_enter(&acb->acb_wait_lock); 5446 acb->acb_wait_error = zio->io_error; 5447 acb->acb_wait = B_FALSE; 5448 cv_signal(&acb->acb_wait_cv); 5449 mutex_exit(&acb->acb_wait_lock); 5450 /* acb will be freed by the waiting thread. */ 5451 } else { 5452 kmem_free(acb, sizeof (arc_callback_t)); 5453 } 5454 } 5455 } 5456 5457 /* 5458 * "Read" the block at the specified DVA (in bp) via the 5459 * cache. If the block is found in the cache, invoke the provided 5460 * callback immediately and return. Note that the `zio' parameter 5461 * in the callback will be NULL in this case, since no IO was 5462 * required. If the block is not in the cache pass the read request 5463 * on to the spa with a substitute callback function, so that the 5464 * requested block will be added to the cache. 5465 * 5466 * If a read request arrives for a block that has a read in-progress, 5467 * either wait for the in-progress read to complete (and return the 5468 * results); or, if this is a read with a "done" func, add a record 5469 * to the read to invoke the "done" func when the read completes, 5470 * and return; or just return. 5471 * 5472 * arc_read_done() will invoke all the requested "done" functions 5473 * for readers of this block. 5474 */ 5475 int 5476 arc_read(zio_t *pio, spa_t *spa, const blkptr_t *bp, 5477 arc_read_done_func_t *done, void *private, zio_priority_t priority, 5478 int zio_flags, arc_flags_t *arc_flags, const zbookmark_phys_t *zb) 5479 { 5480 arc_buf_hdr_t *hdr = NULL; 5481 kmutex_t *hash_lock = NULL; 5482 zio_t *rzio; 5483 uint64_t guid = spa_load_guid(spa); 5484 boolean_t compressed_read = (zio_flags & ZIO_FLAG_RAW_COMPRESS) != 0; 5485 boolean_t encrypted_read = BP_IS_ENCRYPTED(bp) && 5486 (zio_flags & ZIO_FLAG_RAW_ENCRYPT) != 0; 5487 boolean_t noauth_read = BP_IS_AUTHENTICATED(bp) && 5488 (zio_flags & ZIO_FLAG_RAW_ENCRYPT) != 0; 5489 boolean_t embedded_bp = !!BP_IS_EMBEDDED(bp); 5490 boolean_t no_buf = *arc_flags & ARC_FLAG_NO_BUF; 5491 arc_buf_t *buf = NULL; 5492 int rc = 0; 5493 5494 ASSERT(!embedded_bp || 5495 BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA); 5496 ASSERT(!BP_IS_HOLE(bp)); 5497 ASSERT(!BP_IS_REDACTED(bp)); 5498 5499 /* 5500 * Normally SPL_FSTRANS will already be set since kernel threads which 5501 * expect to call the DMU interfaces will set it when created. System 5502 * calls are similarly handled by setting/cleaning the bit in the 5503 * registered callback (module/os/.../zfs/zpl_*). 5504 * 5505 * External consumers such as Lustre which call the exported DMU 5506 * interfaces may not have set SPL_FSTRANS. To avoid a deadlock 5507 * on the hash_lock always set and clear the bit. 5508 */ 5509 fstrans_cookie_t cookie = spl_fstrans_mark(); 5510 top: 5511 /* 5512 * Verify the block pointer contents are reasonable. This should 5513 * always be the case since the blkptr is protected by a checksum. 5514 * However, if there is damage it's desirable to detect this early 5515 * and treat it as a checksum error. This allows an alternate blkptr 5516 * to be tried when one is available (e.g. ditto blocks). 5517 */ 5518 if (!zfs_blkptr_verify(spa, bp, (zio_flags & ZIO_FLAG_CONFIG_WRITER) ? 5519 BLK_CONFIG_HELD : BLK_CONFIG_NEEDED, BLK_VERIFY_LOG)) { 5520 rc = SET_ERROR(ECKSUM); 5521 goto done; 5522 } 5523 5524 if (!embedded_bp) { 5525 /* 5526 * Embedded BP's have no DVA and require no I/O to "read". 5527 * Create an anonymous arc buf to back it. 5528 */ 5529 hdr = buf_hash_find(guid, bp, &hash_lock); 5530 } 5531 5532 /* 5533 * Determine if we have an L1 cache hit or a cache miss. For simplicity 5534 * we maintain encrypted data separately from compressed / uncompressed 5535 * data. If the user is requesting raw encrypted data and we don't have 5536 * that in the header we will read from disk to guarantee that we can 5537 * get it even if the encryption keys aren't loaded. 5538 */ 5539 if (hdr != NULL && HDR_HAS_L1HDR(hdr) && (HDR_HAS_RABD(hdr) || 5540 (hdr->b_l1hdr.b_pabd != NULL && !encrypted_read))) { 5541 boolean_t is_data = !HDR_ISTYPE_METADATA(hdr); 5542 5543 if (HDR_IO_IN_PROGRESS(hdr)) { 5544 if (*arc_flags & ARC_FLAG_CACHED_ONLY) { 5545 mutex_exit(hash_lock); 5546 ARCSTAT_BUMP(arcstat_cached_only_in_progress); 5547 rc = SET_ERROR(ENOENT); 5548 goto done; 5549 } 5550 5551 zio_t *head_zio = hdr->b_l1hdr.b_acb->acb_zio_head; 5552 ASSERT3P(head_zio, !=, NULL); 5553 if ((hdr->b_flags & ARC_FLAG_PRIO_ASYNC_READ) && 5554 priority == ZIO_PRIORITY_SYNC_READ) { 5555 /* 5556 * This is a sync read that needs to wait for 5557 * an in-flight async read. Request that the 5558 * zio have its priority upgraded. 5559 */ 5560 zio_change_priority(head_zio, priority); 5561 DTRACE_PROBE1(arc__async__upgrade__sync, 5562 arc_buf_hdr_t *, hdr); 5563 ARCSTAT_BUMP(arcstat_async_upgrade_sync); 5564 } 5565 5566 DTRACE_PROBE1(arc__iohit, arc_buf_hdr_t *, hdr); 5567 arc_access(hdr, *arc_flags, B_FALSE); 5568 5569 /* 5570 * If there are multiple threads reading the same block 5571 * and that block is not yet in the ARC, then only one 5572 * thread will do the physical I/O and all other 5573 * threads will wait until that I/O completes. 5574 * Synchronous reads use the acb_wait_cv whereas nowait 5575 * reads register a callback. Both are signalled/called 5576 * in arc_read_done. 5577 * 5578 * Errors of the physical I/O may need to be propagated. 5579 * Synchronous read errors are returned here from 5580 * arc_read_done via acb_wait_error. Nowait reads 5581 * attach the acb_zio_dummy zio to pio and 5582 * arc_read_done propagates the physical I/O's io_error 5583 * to acb_zio_dummy, and thereby to pio. 5584 */ 5585 arc_callback_t *acb = NULL; 5586 if (done || pio || *arc_flags & ARC_FLAG_WAIT) { 5587 acb = kmem_zalloc(sizeof (arc_callback_t), 5588 KM_SLEEP); 5589 acb->acb_done = done; 5590 acb->acb_private = private; 5591 acb->acb_compressed = compressed_read; 5592 acb->acb_encrypted = encrypted_read; 5593 acb->acb_noauth = noauth_read; 5594 acb->acb_nobuf = no_buf; 5595 if (*arc_flags & ARC_FLAG_WAIT) { 5596 acb->acb_wait = B_TRUE; 5597 mutex_init(&acb->acb_wait_lock, NULL, 5598 MUTEX_DEFAULT, NULL); 5599 cv_init(&acb->acb_wait_cv, NULL, 5600 CV_DEFAULT, NULL); 5601 } 5602 acb->acb_zb = *zb; 5603 if (pio != NULL) { 5604 acb->acb_zio_dummy = zio_null(pio, 5605 spa, NULL, NULL, NULL, zio_flags); 5606 } 5607 acb->acb_zio_head = head_zio; 5608 acb->acb_next = hdr->b_l1hdr.b_acb; 5609 hdr->b_l1hdr.b_acb->acb_prev = acb; 5610 hdr->b_l1hdr.b_acb = acb; 5611 } 5612 mutex_exit(hash_lock); 5613 5614 ARCSTAT_BUMP(arcstat_iohits); 5615 ARCSTAT_CONDSTAT(!(*arc_flags & ARC_FLAG_PREFETCH), 5616 demand, prefetch, is_data, data, metadata, iohits); 5617 5618 if (*arc_flags & ARC_FLAG_WAIT) { 5619 mutex_enter(&acb->acb_wait_lock); 5620 while (acb->acb_wait) { 5621 cv_wait(&acb->acb_wait_cv, 5622 &acb->acb_wait_lock); 5623 } 5624 rc = acb->acb_wait_error; 5625 mutex_exit(&acb->acb_wait_lock); 5626 mutex_destroy(&acb->acb_wait_lock); 5627 cv_destroy(&acb->acb_wait_cv); 5628 kmem_free(acb, sizeof (arc_callback_t)); 5629 } 5630 goto out; 5631 } 5632 5633 ASSERT(hdr->b_l1hdr.b_state == arc_mru || 5634 hdr->b_l1hdr.b_state == arc_mfu || 5635 hdr->b_l1hdr.b_state == arc_uncached); 5636 5637 DTRACE_PROBE1(arc__hit, arc_buf_hdr_t *, hdr); 5638 arc_access(hdr, *arc_flags, B_TRUE); 5639 5640 if (done && !no_buf) { 5641 ASSERT(!embedded_bp || !BP_IS_HOLE(bp)); 5642 5643 /* Get a buf with the desired data in it. */ 5644 rc = arc_buf_alloc_impl(hdr, spa, zb, private, 5645 encrypted_read, compressed_read, noauth_read, 5646 B_TRUE, &buf); 5647 if (rc == ECKSUM) { 5648 /* 5649 * Convert authentication and decryption errors 5650 * to EIO (and generate an ereport if needed) 5651 * before leaving the ARC. 5652 */ 5653 rc = SET_ERROR(EIO); 5654 if ((zio_flags & ZIO_FLAG_SPECULATIVE) == 0) { 5655 spa_log_error(spa, zb, hdr->b_birth); 5656 (void) zfs_ereport_post( 5657 FM_EREPORT_ZFS_AUTHENTICATION, 5658 spa, NULL, zb, NULL, 0); 5659 } 5660 } 5661 if (rc != 0) { 5662 arc_buf_destroy_impl(buf); 5663 buf = NULL; 5664 (void) remove_reference(hdr, private); 5665 } 5666 5667 /* assert any errors weren't due to unloaded keys */ 5668 ASSERT((zio_flags & ZIO_FLAG_SPECULATIVE) || 5669 rc != EACCES); 5670 } 5671 mutex_exit(hash_lock); 5672 ARCSTAT_BUMP(arcstat_hits); 5673 ARCSTAT_CONDSTAT(!(*arc_flags & ARC_FLAG_PREFETCH), 5674 demand, prefetch, is_data, data, metadata, hits); 5675 *arc_flags |= ARC_FLAG_CACHED; 5676 goto done; 5677 } else { 5678 uint64_t lsize = BP_GET_LSIZE(bp); 5679 uint64_t psize = BP_GET_PSIZE(bp); 5680 arc_callback_t *acb; 5681 vdev_t *vd = NULL; 5682 uint64_t addr = 0; 5683 boolean_t devw = B_FALSE; 5684 uint64_t size; 5685 abd_t *hdr_abd; 5686 int alloc_flags = encrypted_read ? ARC_HDR_ALLOC_RDATA : 0; 5687 arc_buf_contents_t type = BP_GET_BUFC_TYPE(bp); 5688 5689 if (*arc_flags & ARC_FLAG_CACHED_ONLY) { 5690 if (hash_lock != NULL) 5691 mutex_exit(hash_lock); 5692 rc = SET_ERROR(ENOENT); 5693 goto done; 5694 } 5695 5696 if (hdr == NULL) { 5697 /* 5698 * This block is not in the cache or it has 5699 * embedded data. 5700 */ 5701 arc_buf_hdr_t *exists = NULL; 5702 hdr = arc_hdr_alloc(guid, psize, lsize, 5703 BP_IS_PROTECTED(bp), BP_GET_COMPRESS(bp), 0, type); 5704 5705 if (!embedded_bp) { 5706 hdr->b_dva = *BP_IDENTITY(bp); 5707 hdr->b_birth = BP_GET_BIRTH(bp); 5708 exists = buf_hash_insert(hdr, &hash_lock); 5709 } 5710 if (exists != NULL) { 5711 /* somebody beat us to the hash insert */ 5712 mutex_exit(hash_lock); 5713 buf_discard_identity(hdr); 5714 arc_hdr_destroy(hdr); 5715 goto top; /* restart the IO request */ 5716 } 5717 } else { 5718 /* 5719 * This block is in the ghost cache or encrypted data 5720 * was requested and we didn't have it. If it was 5721 * L2-only (and thus didn't have an L1 hdr), 5722 * we realloc the header to add an L1 hdr. 5723 */ 5724 if (!HDR_HAS_L1HDR(hdr)) { 5725 hdr = arc_hdr_realloc(hdr, hdr_l2only_cache, 5726 hdr_full_cache); 5727 } 5728 5729 if (GHOST_STATE(hdr->b_l1hdr.b_state)) { 5730 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 5731 ASSERT(!HDR_HAS_RABD(hdr)); 5732 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 5733 ASSERT0(zfs_refcount_count( 5734 &hdr->b_l1hdr.b_refcnt)); 5735 ASSERT3P(hdr->b_l1hdr.b_buf, ==, NULL); 5736 #ifdef ZFS_DEBUG 5737 ASSERT3P(hdr->b_l1hdr.b_freeze_cksum, ==, NULL); 5738 #endif 5739 } else if (HDR_IO_IN_PROGRESS(hdr)) { 5740 /* 5741 * If this header already had an IO in progress 5742 * and we are performing another IO to fetch 5743 * encrypted data we must wait until the first 5744 * IO completes so as not to confuse 5745 * arc_read_done(). This should be very rare 5746 * and so the performance impact shouldn't 5747 * matter. 5748 */ 5749 arc_callback_t *acb = kmem_zalloc( 5750 sizeof (arc_callback_t), KM_SLEEP); 5751 acb->acb_wait = B_TRUE; 5752 mutex_init(&acb->acb_wait_lock, NULL, 5753 MUTEX_DEFAULT, NULL); 5754 cv_init(&acb->acb_wait_cv, NULL, CV_DEFAULT, 5755 NULL); 5756 acb->acb_zio_head = 5757 hdr->b_l1hdr.b_acb->acb_zio_head; 5758 acb->acb_next = hdr->b_l1hdr.b_acb; 5759 hdr->b_l1hdr.b_acb->acb_prev = acb; 5760 hdr->b_l1hdr.b_acb = acb; 5761 mutex_exit(hash_lock); 5762 mutex_enter(&acb->acb_wait_lock); 5763 while (acb->acb_wait) { 5764 cv_wait(&acb->acb_wait_cv, 5765 &acb->acb_wait_lock); 5766 } 5767 mutex_exit(&acb->acb_wait_lock); 5768 mutex_destroy(&acb->acb_wait_lock); 5769 cv_destroy(&acb->acb_wait_cv); 5770 kmem_free(acb, sizeof (arc_callback_t)); 5771 goto top; 5772 } 5773 } 5774 if (*arc_flags & ARC_FLAG_UNCACHED) { 5775 arc_hdr_set_flags(hdr, ARC_FLAG_UNCACHED); 5776 if (!encrypted_read) 5777 alloc_flags |= ARC_HDR_ALLOC_LINEAR; 5778 } 5779 5780 /* 5781 * Take additional reference for IO_IN_PROGRESS. It stops 5782 * arc_access() from putting this header without any buffers 5783 * and so other references but obviously nonevictable onto 5784 * the evictable list of MRU or MFU state. 5785 */ 5786 add_reference(hdr, hdr); 5787 if (!embedded_bp) 5788 arc_access(hdr, *arc_flags, B_FALSE); 5789 arc_hdr_set_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 5790 arc_hdr_alloc_abd(hdr, alloc_flags); 5791 if (encrypted_read) { 5792 ASSERT(HDR_HAS_RABD(hdr)); 5793 size = HDR_GET_PSIZE(hdr); 5794 hdr_abd = hdr->b_crypt_hdr.b_rabd; 5795 zio_flags |= ZIO_FLAG_RAW; 5796 } else { 5797 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 5798 size = arc_hdr_size(hdr); 5799 hdr_abd = hdr->b_l1hdr.b_pabd; 5800 5801 if (arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF) { 5802 zio_flags |= ZIO_FLAG_RAW_COMPRESS; 5803 } 5804 5805 /* 5806 * For authenticated bp's, we do not ask the ZIO layer 5807 * to authenticate them since this will cause the entire 5808 * IO to fail if the key isn't loaded. Instead, we 5809 * defer authentication until arc_buf_fill(), which will 5810 * verify the data when the key is available. 5811 */ 5812 if (BP_IS_AUTHENTICATED(bp)) 5813 zio_flags |= ZIO_FLAG_RAW_ENCRYPT; 5814 } 5815 5816 if (BP_IS_AUTHENTICATED(bp)) 5817 arc_hdr_set_flags(hdr, ARC_FLAG_NOAUTH); 5818 if (BP_GET_LEVEL(bp) > 0) 5819 arc_hdr_set_flags(hdr, ARC_FLAG_INDIRECT); 5820 ASSERT(!GHOST_STATE(hdr->b_l1hdr.b_state)); 5821 5822 acb = kmem_zalloc(sizeof (arc_callback_t), KM_SLEEP); 5823 acb->acb_done = done; 5824 acb->acb_private = private; 5825 acb->acb_compressed = compressed_read; 5826 acb->acb_encrypted = encrypted_read; 5827 acb->acb_noauth = noauth_read; 5828 acb->acb_zb = *zb; 5829 5830 ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); 5831 hdr->b_l1hdr.b_acb = acb; 5832 5833 if (HDR_HAS_L2HDR(hdr) && 5834 (vd = hdr->b_l2hdr.b_dev->l2ad_vdev) != NULL) { 5835 devw = hdr->b_l2hdr.b_dev->l2ad_writing; 5836 addr = hdr->b_l2hdr.b_daddr; 5837 /* 5838 * Lock out L2ARC device removal. 5839 */ 5840 if (vdev_is_dead(vd) || 5841 !spa_config_tryenter(spa, SCL_L2ARC, vd, RW_READER)) 5842 vd = NULL; 5843 } 5844 5845 /* 5846 * We count both async reads and scrub IOs as asynchronous so 5847 * that both can be upgraded in the event of a cache hit while 5848 * the read IO is still in-flight. 5849 */ 5850 if (priority == ZIO_PRIORITY_ASYNC_READ || 5851 priority == ZIO_PRIORITY_SCRUB) 5852 arc_hdr_set_flags(hdr, ARC_FLAG_PRIO_ASYNC_READ); 5853 else 5854 arc_hdr_clear_flags(hdr, ARC_FLAG_PRIO_ASYNC_READ); 5855 5856 /* 5857 * At this point, we have a level 1 cache miss or a blkptr 5858 * with embedded data. Try again in L2ARC if possible. 5859 */ 5860 ASSERT3U(HDR_GET_LSIZE(hdr), ==, lsize); 5861 5862 /* 5863 * Skip ARC stat bump for block pointers with embedded 5864 * data. The data are read from the blkptr itself via 5865 * decode_embedded_bp_compressed(). 5866 */ 5867 if (!embedded_bp) { 5868 DTRACE_PROBE4(arc__miss, arc_buf_hdr_t *, hdr, 5869 blkptr_t *, bp, uint64_t, lsize, 5870 zbookmark_phys_t *, zb); 5871 ARCSTAT_BUMP(arcstat_misses); 5872 ARCSTAT_CONDSTAT(!(*arc_flags & ARC_FLAG_PREFETCH), 5873 demand, prefetch, !HDR_ISTYPE_METADATA(hdr), data, 5874 metadata, misses); 5875 zfs_racct_read(size, 1); 5876 } 5877 5878 /* Check if the spa even has l2 configured */ 5879 const boolean_t spa_has_l2 = l2arc_ndev != 0 && 5880 spa->spa_l2cache.sav_count > 0; 5881 5882 if (vd != NULL && spa_has_l2 && !(l2arc_norw && devw)) { 5883 /* 5884 * Read from the L2ARC if the following are true: 5885 * 1. The L2ARC vdev was previously cached. 5886 * 2. This buffer still has L2ARC metadata. 5887 * 3. This buffer isn't currently writing to the L2ARC. 5888 * 4. The L2ARC entry wasn't evicted, which may 5889 * also have invalidated the vdev. 5890 */ 5891 if (HDR_HAS_L2HDR(hdr) && 5892 !HDR_L2_WRITING(hdr) && !HDR_L2_EVICTED(hdr)) { 5893 l2arc_read_callback_t *cb; 5894 abd_t *abd; 5895 uint64_t asize; 5896 5897 DTRACE_PROBE1(l2arc__hit, arc_buf_hdr_t *, hdr); 5898 ARCSTAT_BUMP(arcstat_l2_hits); 5899 hdr->b_l2hdr.b_hits++; 5900 5901 cb = kmem_zalloc(sizeof (l2arc_read_callback_t), 5902 KM_SLEEP); 5903 cb->l2rcb_hdr = hdr; 5904 cb->l2rcb_bp = *bp; 5905 cb->l2rcb_zb = *zb; 5906 cb->l2rcb_flags = zio_flags; 5907 5908 /* 5909 * When Compressed ARC is disabled, but the 5910 * L2ARC block is compressed, arc_hdr_size() 5911 * will have returned LSIZE rather than PSIZE. 5912 */ 5913 if (HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 5914 !HDR_COMPRESSION_ENABLED(hdr) && 5915 HDR_GET_PSIZE(hdr) != 0) { 5916 size = HDR_GET_PSIZE(hdr); 5917 } 5918 5919 asize = vdev_psize_to_asize(vd, size); 5920 if (asize != size) { 5921 abd = abd_alloc_for_io(asize, 5922 HDR_ISTYPE_METADATA(hdr)); 5923 cb->l2rcb_abd = abd; 5924 } else { 5925 abd = hdr_abd; 5926 } 5927 5928 ASSERT(addr >= VDEV_LABEL_START_SIZE && 5929 addr + asize <= vd->vdev_psize - 5930 VDEV_LABEL_END_SIZE); 5931 5932 /* 5933 * l2arc read. The SCL_L2ARC lock will be 5934 * released by l2arc_read_done(). 5935 * Issue a null zio if the underlying buffer 5936 * was squashed to zero size by compression. 5937 */ 5938 ASSERT3U(arc_hdr_get_compress(hdr), !=, 5939 ZIO_COMPRESS_EMPTY); 5940 rzio = zio_read_phys(pio, vd, addr, 5941 asize, abd, 5942 ZIO_CHECKSUM_OFF, 5943 l2arc_read_done, cb, priority, 5944 zio_flags | ZIO_FLAG_CANFAIL | 5945 ZIO_FLAG_DONT_PROPAGATE | 5946 ZIO_FLAG_DONT_RETRY, B_FALSE); 5947 acb->acb_zio_head = rzio; 5948 5949 if (hash_lock != NULL) 5950 mutex_exit(hash_lock); 5951 5952 DTRACE_PROBE2(l2arc__read, vdev_t *, vd, 5953 zio_t *, rzio); 5954 ARCSTAT_INCR(arcstat_l2_read_bytes, 5955 HDR_GET_PSIZE(hdr)); 5956 5957 if (*arc_flags & ARC_FLAG_NOWAIT) { 5958 zio_nowait(rzio); 5959 goto out; 5960 } 5961 5962 ASSERT(*arc_flags & ARC_FLAG_WAIT); 5963 if (zio_wait(rzio) == 0) 5964 goto out; 5965 5966 /* l2arc read error; goto zio_read() */ 5967 if (hash_lock != NULL) 5968 mutex_enter(hash_lock); 5969 } else { 5970 DTRACE_PROBE1(l2arc__miss, 5971 arc_buf_hdr_t *, hdr); 5972 ARCSTAT_BUMP(arcstat_l2_misses); 5973 if (HDR_L2_WRITING(hdr)) 5974 ARCSTAT_BUMP(arcstat_l2_rw_clash); 5975 spa_config_exit(spa, SCL_L2ARC, vd); 5976 } 5977 } else { 5978 if (vd != NULL) 5979 spa_config_exit(spa, SCL_L2ARC, vd); 5980 5981 /* 5982 * Only a spa with l2 should contribute to l2 5983 * miss stats. (Including the case of having a 5984 * faulted cache device - that's also a miss.) 5985 */ 5986 if (spa_has_l2) { 5987 /* 5988 * Skip ARC stat bump for block pointers with 5989 * embedded data. The data are read from the 5990 * blkptr itself via 5991 * decode_embedded_bp_compressed(). 5992 */ 5993 if (!embedded_bp) { 5994 DTRACE_PROBE1(l2arc__miss, 5995 arc_buf_hdr_t *, hdr); 5996 ARCSTAT_BUMP(arcstat_l2_misses); 5997 } 5998 } 5999 } 6000 6001 rzio = zio_read(pio, spa, bp, hdr_abd, size, 6002 arc_read_done, hdr, priority, zio_flags, zb); 6003 acb->acb_zio_head = rzio; 6004 6005 if (hash_lock != NULL) 6006 mutex_exit(hash_lock); 6007 6008 if (*arc_flags & ARC_FLAG_WAIT) { 6009 rc = zio_wait(rzio); 6010 goto out; 6011 } 6012 6013 ASSERT(*arc_flags & ARC_FLAG_NOWAIT); 6014 zio_nowait(rzio); 6015 } 6016 6017 out: 6018 /* embedded bps don't actually go to disk */ 6019 if (!embedded_bp) 6020 spa_read_history_add(spa, zb, *arc_flags); 6021 spl_fstrans_unmark(cookie); 6022 return (rc); 6023 6024 done: 6025 if (done) 6026 done(NULL, zb, bp, buf, private); 6027 if (pio && rc != 0) { 6028 zio_t *zio = zio_null(pio, spa, NULL, NULL, NULL, zio_flags); 6029 zio->io_error = rc; 6030 zio_nowait(zio); 6031 } 6032 goto out; 6033 } 6034 6035 arc_prune_t * 6036 arc_add_prune_callback(arc_prune_func_t *func, void *private) 6037 { 6038 arc_prune_t *p; 6039 6040 p = kmem_alloc(sizeof (*p), KM_SLEEP); 6041 p->p_pfunc = func; 6042 p->p_private = private; 6043 list_link_init(&p->p_node); 6044 zfs_refcount_create(&p->p_refcnt); 6045 6046 mutex_enter(&arc_prune_mtx); 6047 zfs_refcount_add(&p->p_refcnt, &arc_prune_list); 6048 list_insert_head(&arc_prune_list, p); 6049 mutex_exit(&arc_prune_mtx); 6050 6051 return (p); 6052 } 6053 6054 void 6055 arc_remove_prune_callback(arc_prune_t *p) 6056 { 6057 boolean_t wait = B_FALSE; 6058 mutex_enter(&arc_prune_mtx); 6059 list_remove(&arc_prune_list, p); 6060 if (zfs_refcount_remove(&p->p_refcnt, &arc_prune_list) > 0) 6061 wait = B_TRUE; 6062 mutex_exit(&arc_prune_mtx); 6063 6064 /* wait for arc_prune_task to finish */ 6065 if (wait) 6066 taskq_wait_outstanding(arc_prune_taskq, 0); 6067 ASSERT0(zfs_refcount_count(&p->p_refcnt)); 6068 zfs_refcount_destroy(&p->p_refcnt); 6069 kmem_free(p, sizeof (*p)); 6070 } 6071 6072 /* 6073 * Helper function for arc_prune_async() it is responsible for safely 6074 * handling the execution of a registered arc_prune_func_t. 6075 */ 6076 static void 6077 arc_prune_task(void *ptr) 6078 { 6079 arc_prune_t *ap = (arc_prune_t *)ptr; 6080 arc_prune_func_t *func = ap->p_pfunc; 6081 6082 if (func != NULL) 6083 func(ap->p_adjust, ap->p_private); 6084 6085 (void) zfs_refcount_remove(&ap->p_refcnt, func); 6086 } 6087 6088 /* 6089 * Notify registered consumers they must drop holds on a portion of the ARC 6090 * buffers they reference. This provides a mechanism to ensure the ARC can 6091 * honor the metadata limit and reclaim otherwise pinned ARC buffers. 6092 * 6093 * This operation is performed asynchronously so it may be safely called 6094 * in the context of the arc_reclaim_thread(). A reference is taken here 6095 * for each registered arc_prune_t and the arc_prune_task() is responsible 6096 * for releasing it once the registered arc_prune_func_t has completed. 6097 */ 6098 static void 6099 arc_prune_async(uint64_t adjust) 6100 { 6101 arc_prune_t *ap; 6102 6103 mutex_enter(&arc_prune_mtx); 6104 for (ap = list_head(&arc_prune_list); ap != NULL; 6105 ap = list_next(&arc_prune_list, ap)) { 6106 6107 if (zfs_refcount_count(&ap->p_refcnt) >= 2) 6108 continue; 6109 6110 zfs_refcount_add(&ap->p_refcnt, ap->p_pfunc); 6111 ap->p_adjust = adjust; 6112 if (taskq_dispatch(arc_prune_taskq, arc_prune_task, 6113 ap, TQ_SLEEP) == TASKQID_INVALID) { 6114 (void) zfs_refcount_remove(&ap->p_refcnt, ap->p_pfunc); 6115 continue; 6116 } 6117 ARCSTAT_BUMP(arcstat_prune); 6118 } 6119 mutex_exit(&arc_prune_mtx); 6120 } 6121 6122 /* 6123 * Notify the arc that a block was freed, and thus will never be used again. 6124 */ 6125 void 6126 arc_freed(spa_t *spa, const blkptr_t *bp) 6127 { 6128 arc_buf_hdr_t *hdr; 6129 kmutex_t *hash_lock; 6130 uint64_t guid = spa_load_guid(spa); 6131 6132 ASSERT(!BP_IS_EMBEDDED(bp)); 6133 6134 hdr = buf_hash_find(guid, bp, &hash_lock); 6135 if (hdr == NULL) 6136 return; 6137 6138 /* 6139 * We might be trying to free a block that is still doing I/O 6140 * (i.e. prefetch) or has some other reference (i.e. a dedup-ed, 6141 * dmu_sync-ed block). A block may also have a reference if it is 6142 * part of a dedup-ed, dmu_synced write. The dmu_sync() function would 6143 * have written the new block to its final resting place on disk but 6144 * without the dedup flag set. This would have left the hdr in the MRU 6145 * state and discoverable. When the txg finally syncs it detects that 6146 * the block was overridden in open context and issues an override I/O. 6147 * Since this is a dedup block, the override I/O will determine if the 6148 * block is already in the DDT. If so, then it will replace the io_bp 6149 * with the bp from the DDT and allow the I/O to finish. When the I/O 6150 * reaches the done callback, dbuf_write_override_done, it will 6151 * check to see if the io_bp and io_bp_override are identical. 6152 * If they are not, then it indicates that the bp was replaced with 6153 * the bp in the DDT and the override bp is freed. This allows 6154 * us to arrive here with a reference on a block that is being 6155 * freed. So if we have an I/O in progress, or a reference to 6156 * this hdr, then we don't destroy the hdr. 6157 */ 6158 if (!HDR_HAS_L1HDR(hdr) || 6159 zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)) { 6160 arc_change_state(arc_anon, hdr); 6161 arc_hdr_destroy(hdr); 6162 mutex_exit(hash_lock); 6163 } else { 6164 mutex_exit(hash_lock); 6165 } 6166 6167 } 6168 6169 /* 6170 * Release this buffer from the cache, making it an anonymous buffer. This 6171 * must be done after a read and prior to modifying the buffer contents. 6172 * If the buffer has more than one reference, we must make 6173 * a new hdr for the buffer. 6174 */ 6175 void 6176 arc_release(arc_buf_t *buf, const void *tag) 6177 { 6178 arc_buf_hdr_t *hdr = buf->b_hdr; 6179 6180 /* 6181 * It would be nice to assert that if its DMU metadata (level > 6182 * 0 || it's the dnode file), then it must be syncing context. 6183 * But we don't know that information at this level. 6184 */ 6185 6186 ASSERT(HDR_HAS_L1HDR(hdr)); 6187 6188 /* 6189 * We don't grab the hash lock prior to this check, because if 6190 * the buffer's header is in the arc_anon state, it won't be 6191 * linked into the hash table. 6192 */ 6193 if (hdr->b_l1hdr.b_state == arc_anon) { 6194 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 6195 ASSERT(!HDR_IN_HASH_TABLE(hdr)); 6196 ASSERT(!HDR_HAS_L2HDR(hdr)); 6197 6198 ASSERT3P(hdr->b_l1hdr.b_buf, ==, buf); 6199 ASSERT(ARC_BUF_LAST(buf)); 6200 ASSERT3S(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt), ==, 1); 6201 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 6202 6203 hdr->b_l1hdr.b_arc_access = 0; 6204 6205 /* 6206 * If the buf is being overridden then it may already 6207 * have a hdr that is not empty. 6208 */ 6209 buf_discard_identity(hdr); 6210 arc_buf_thaw(buf); 6211 6212 return; 6213 } 6214 6215 kmutex_t *hash_lock = HDR_LOCK(hdr); 6216 mutex_enter(hash_lock); 6217 6218 /* 6219 * This assignment is only valid as long as the hash_lock is 6220 * held, we must be careful not to reference state or the 6221 * b_state field after dropping the lock. 6222 */ 6223 arc_state_t *state = hdr->b_l1hdr.b_state; 6224 ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); 6225 ASSERT3P(state, !=, arc_anon); 6226 6227 /* this buffer is not on any list */ 6228 ASSERT3S(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt), >, 0); 6229 6230 if (HDR_HAS_L2HDR(hdr)) { 6231 mutex_enter(&hdr->b_l2hdr.b_dev->l2ad_mtx); 6232 6233 /* 6234 * We have to recheck this conditional again now that 6235 * we're holding the l2ad_mtx to prevent a race with 6236 * another thread which might be concurrently calling 6237 * l2arc_evict(). In that case, l2arc_evict() might have 6238 * destroyed the header's L2 portion as we were waiting 6239 * to acquire the l2ad_mtx. 6240 */ 6241 if (HDR_HAS_L2HDR(hdr)) 6242 arc_hdr_l2hdr_destroy(hdr); 6243 6244 mutex_exit(&hdr->b_l2hdr.b_dev->l2ad_mtx); 6245 } 6246 6247 /* 6248 * Do we have more than one buf? 6249 */ 6250 if (hdr->b_l1hdr.b_buf != buf || !ARC_BUF_LAST(buf)) { 6251 arc_buf_hdr_t *nhdr; 6252 uint64_t spa = hdr->b_spa; 6253 uint64_t psize = HDR_GET_PSIZE(hdr); 6254 uint64_t lsize = HDR_GET_LSIZE(hdr); 6255 boolean_t protected = HDR_PROTECTED(hdr); 6256 enum zio_compress compress = arc_hdr_get_compress(hdr); 6257 arc_buf_contents_t type = arc_buf_type(hdr); 6258 VERIFY3U(hdr->b_type, ==, type); 6259 6260 ASSERT(hdr->b_l1hdr.b_buf != buf || buf->b_next != NULL); 6261 VERIFY3S(remove_reference(hdr, tag), >, 0); 6262 6263 if (ARC_BUF_SHARED(buf) && !ARC_BUF_COMPRESSED(buf)) { 6264 ASSERT3P(hdr->b_l1hdr.b_buf, !=, buf); 6265 ASSERT(ARC_BUF_LAST(buf)); 6266 } 6267 6268 /* 6269 * Pull the data off of this hdr and attach it to 6270 * a new anonymous hdr. Also find the last buffer 6271 * in the hdr's buffer list. 6272 */ 6273 arc_buf_t *lastbuf = arc_buf_remove(hdr, buf); 6274 ASSERT3P(lastbuf, !=, NULL); 6275 6276 /* 6277 * If the current arc_buf_t and the hdr are sharing their data 6278 * buffer, then we must stop sharing that block. 6279 */ 6280 if (ARC_BUF_SHARED(buf)) { 6281 ASSERT3P(hdr->b_l1hdr.b_buf, !=, buf); 6282 ASSERT(!arc_buf_is_shared(lastbuf)); 6283 6284 /* 6285 * First, sever the block sharing relationship between 6286 * buf and the arc_buf_hdr_t. 6287 */ 6288 arc_unshare_buf(hdr, buf); 6289 6290 /* 6291 * Now we need to recreate the hdr's b_pabd. Since we 6292 * have lastbuf handy, we try to share with it, but if 6293 * we can't then we allocate a new b_pabd and copy the 6294 * data from buf into it. 6295 */ 6296 if (arc_can_share(hdr, lastbuf)) { 6297 arc_share_buf(hdr, lastbuf); 6298 } else { 6299 arc_hdr_alloc_abd(hdr, 0); 6300 abd_copy_from_buf(hdr->b_l1hdr.b_pabd, 6301 buf->b_data, psize); 6302 } 6303 VERIFY3P(lastbuf->b_data, !=, NULL); 6304 } else if (HDR_SHARED_DATA(hdr)) { 6305 /* 6306 * Uncompressed shared buffers are always at the end 6307 * of the list. Compressed buffers don't have the 6308 * same requirements. This makes it hard to 6309 * simply assert that the lastbuf is shared so 6310 * we rely on the hdr's compression flags to determine 6311 * if we have a compressed, shared buffer. 6312 */ 6313 ASSERT(arc_buf_is_shared(lastbuf) || 6314 arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF); 6315 ASSERT(!arc_buf_is_shared(buf)); 6316 } 6317 6318 ASSERT(hdr->b_l1hdr.b_pabd != NULL || HDR_HAS_RABD(hdr)); 6319 ASSERT3P(state, !=, arc_l2c_only); 6320 6321 (void) zfs_refcount_remove_many(&state->arcs_size[type], 6322 arc_buf_size(buf), buf); 6323 6324 if (zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)) { 6325 ASSERT3P(state, !=, arc_l2c_only); 6326 (void) zfs_refcount_remove_many( 6327 &state->arcs_esize[type], 6328 arc_buf_size(buf), buf); 6329 } 6330 6331 arc_cksum_verify(buf); 6332 arc_buf_unwatch(buf); 6333 6334 /* if this is the last uncompressed buf free the checksum */ 6335 if (!arc_hdr_has_uncompressed_buf(hdr)) 6336 arc_cksum_free(hdr); 6337 6338 mutex_exit(hash_lock); 6339 6340 nhdr = arc_hdr_alloc(spa, psize, lsize, protected, 6341 compress, hdr->b_complevel, type); 6342 ASSERT3P(nhdr->b_l1hdr.b_buf, ==, NULL); 6343 ASSERT0(zfs_refcount_count(&nhdr->b_l1hdr.b_refcnt)); 6344 VERIFY3U(nhdr->b_type, ==, type); 6345 ASSERT(!HDR_SHARED_DATA(nhdr)); 6346 6347 nhdr->b_l1hdr.b_buf = buf; 6348 (void) zfs_refcount_add(&nhdr->b_l1hdr.b_refcnt, tag); 6349 buf->b_hdr = nhdr; 6350 6351 (void) zfs_refcount_add_many(&arc_anon->arcs_size[type], 6352 arc_buf_size(buf), buf); 6353 } else { 6354 ASSERT(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt) == 1); 6355 /* protected by hash lock, or hdr is on arc_anon */ 6356 ASSERT(!multilist_link_active(&hdr->b_l1hdr.b_arc_node)); 6357 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 6358 hdr->b_l1hdr.b_mru_hits = 0; 6359 hdr->b_l1hdr.b_mru_ghost_hits = 0; 6360 hdr->b_l1hdr.b_mfu_hits = 0; 6361 hdr->b_l1hdr.b_mfu_ghost_hits = 0; 6362 arc_change_state(arc_anon, hdr); 6363 hdr->b_l1hdr.b_arc_access = 0; 6364 6365 mutex_exit(hash_lock); 6366 buf_discard_identity(hdr); 6367 arc_buf_thaw(buf); 6368 } 6369 } 6370 6371 int 6372 arc_released(arc_buf_t *buf) 6373 { 6374 return (buf->b_data != NULL && 6375 buf->b_hdr->b_l1hdr.b_state == arc_anon); 6376 } 6377 6378 #ifdef ZFS_DEBUG 6379 int 6380 arc_referenced(arc_buf_t *buf) 6381 { 6382 return (zfs_refcount_count(&buf->b_hdr->b_l1hdr.b_refcnt)); 6383 } 6384 #endif 6385 6386 static void 6387 arc_write_ready(zio_t *zio) 6388 { 6389 arc_write_callback_t *callback = zio->io_private; 6390 arc_buf_t *buf = callback->awcb_buf; 6391 arc_buf_hdr_t *hdr = buf->b_hdr; 6392 blkptr_t *bp = zio->io_bp; 6393 uint64_t psize = BP_IS_HOLE(bp) ? 0 : BP_GET_PSIZE(bp); 6394 fstrans_cookie_t cookie = spl_fstrans_mark(); 6395 6396 ASSERT(HDR_HAS_L1HDR(hdr)); 6397 ASSERT(!zfs_refcount_is_zero(&buf->b_hdr->b_l1hdr.b_refcnt)); 6398 ASSERT3P(hdr->b_l1hdr.b_buf, !=, NULL); 6399 6400 /* 6401 * If we're reexecuting this zio because the pool suspended, then 6402 * cleanup any state that was previously set the first time the 6403 * callback was invoked. 6404 */ 6405 if (zio->io_flags & ZIO_FLAG_REEXECUTED) { 6406 arc_cksum_free(hdr); 6407 arc_buf_unwatch(buf); 6408 if (hdr->b_l1hdr.b_pabd != NULL) { 6409 if (ARC_BUF_SHARED(buf)) { 6410 arc_unshare_buf(hdr, buf); 6411 } else { 6412 ASSERT(!arc_buf_is_shared(buf)); 6413 arc_hdr_free_abd(hdr, B_FALSE); 6414 } 6415 } 6416 6417 if (HDR_HAS_RABD(hdr)) 6418 arc_hdr_free_abd(hdr, B_TRUE); 6419 } 6420 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 6421 ASSERT(!HDR_HAS_RABD(hdr)); 6422 ASSERT(!HDR_SHARED_DATA(hdr)); 6423 ASSERT(!arc_buf_is_shared(buf)); 6424 6425 callback->awcb_ready(zio, buf, callback->awcb_private); 6426 6427 if (HDR_IO_IN_PROGRESS(hdr)) { 6428 ASSERT(zio->io_flags & ZIO_FLAG_REEXECUTED); 6429 } else { 6430 arc_hdr_set_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 6431 add_reference(hdr, hdr); /* For IO_IN_PROGRESS. */ 6432 } 6433 6434 if (BP_IS_PROTECTED(bp)) { 6435 /* ZIL blocks are written through zio_rewrite */ 6436 ASSERT3U(BP_GET_TYPE(bp), !=, DMU_OT_INTENT_LOG); 6437 6438 if (BP_SHOULD_BYTESWAP(bp)) { 6439 if (BP_GET_LEVEL(bp) > 0) { 6440 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_UINT64; 6441 } else { 6442 hdr->b_l1hdr.b_byteswap = 6443 DMU_OT_BYTESWAP(BP_GET_TYPE(bp)); 6444 } 6445 } else { 6446 hdr->b_l1hdr.b_byteswap = DMU_BSWAP_NUMFUNCS; 6447 } 6448 6449 arc_hdr_set_flags(hdr, ARC_FLAG_PROTECTED); 6450 hdr->b_crypt_hdr.b_ot = BP_GET_TYPE(bp); 6451 hdr->b_crypt_hdr.b_dsobj = zio->io_bookmark.zb_objset; 6452 zio_crypt_decode_params_bp(bp, hdr->b_crypt_hdr.b_salt, 6453 hdr->b_crypt_hdr.b_iv); 6454 zio_crypt_decode_mac_bp(bp, hdr->b_crypt_hdr.b_mac); 6455 } else { 6456 arc_hdr_clear_flags(hdr, ARC_FLAG_PROTECTED); 6457 } 6458 6459 /* 6460 * If this block was written for raw encryption but the zio layer 6461 * ended up only authenticating it, adjust the buffer flags now. 6462 */ 6463 if (BP_IS_AUTHENTICATED(bp) && ARC_BUF_ENCRYPTED(buf)) { 6464 arc_hdr_set_flags(hdr, ARC_FLAG_NOAUTH); 6465 buf->b_flags &= ~ARC_BUF_FLAG_ENCRYPTED; 6466 if (BP_GET_COMPRESS(bp) == ZIO_COMPRESS_OFF) 6467 buf->b_flags &= ~ARC_BUF_FLAG_COMPRESSED; 6468 } else if (BP_IS_HOLE(bp) && ARC_BUF_ENCRYPTED(buf)) { 6469 buf->b_flags &= ~ARC_BUF_FLAG_ENCRYPTED; 6470 buf->b_flags &= ~ARC_BUF_FLAG_COMPRESSED; 6471 } 6472 6473 /* this must be done after the buffer flags are adjusted */ 6474 arc_cksum_compute(buf); 6475 6476 enum zio_compress compress; 6477 if (BP_IS_HOLE(bp) || BP_IS_EMBEDDED(bp)) { 6478 compress = ZIO_COMPRESS_OFF; 6479 } else { 6480 ASSERT3U(HDR_GET_LSIZE(hdr), ==, BP_GET_LSIZE(bp)); 6481 compress = BP_GET_COMPRESS(bp); 6482 } 6483 HDR_SET_PSIZE(hdr, psize); 6484 arc_hdr_set_compress(hdr, compress); 6485 hdr->b_complevel = zio->io_prop.zp_complevel; 6486 6487 if (zio->io_error != 0 || psize == 0) 6488 goto out; 6489 6490 /* 6491 * Fill the hdr with data. If the buffer is encrypted we have no choice 6492 * but to copy the data into b_radb. If the hdr is compressed, the data 6493 * we want is available from the zio, otherwise we can take it from 6494 * the buf. 6495 * 6496 * We might be able to share the buf's data with the hdr here. However, 6497 * doing so would cause the ARC to be full of linear ABDs if we write a 6498 * lot of shareable data. As a compromise, we check whether scattered 6499 * ABDs are allowed, and assume that if they are then the user wants 6500 * the ARC to be primarily filled with them regardless of the data being 6501 * written. Therefore, if they're allowed then we allocate one and copy 6502 * the data into it; otherwise, we share the data directly if we can. 6503 */ 6504 if (ARC_BUF_ENCRYPTED(buf)) { 6505 ASSERT3U(psize, >, 0); 6506 ASSERT(ARC_BUF_COMPRESSED(buf)); 6507 arc_hdr_alloc_abd(hdr, ARC_HDR_ALLOC_RDATA | 6508 ARC_HDR_USE_RESERVE); 6509 abd_copy(hdr->b_crypt_hdr.b_rabd, zio->io_abd, psize); 6510 } else if (!(HDR_UNCACHED(hdr) || 6511 abd_size_alloc_linear(arc_buf_size(buf))) || 6512 !arc_can_share(hdr, buf)) { 6513 /* 6514 * Ideally, we would always copy the io_abd into b_pabd, but the 6515 * user may have disabled compressed ARC, thus we must check the 6516 * hdr's compression setting rather than the io_bp's. 6517 */ 6518 if (BP_IS_ENCRYPTED(bp)) { 6519 ASSERT3U(psize, >, 0); 6520 arc_hdr_alloc_abd(hdr, ARC_HDR_ALLOC_RDATA | 6521 ARC_HDR_USE_RESERVE); 6522 abd_copy(hdr->b_crypt_hdr.b_rabd, zio->io_abd, psize); 6523 } else if (arc_hdr_get_compress(hdr) != ZIO_COMPRESS_OFF && 6524 !ARC_BUF_COMPRESSED(buf)) { 6525 ASSERT3U(psize, >, 0); 6526 arc_hdr_alloc_abd(hdr, ARC_HDR_USE_RESERVE); 6527 abd_copy(hdr->b_l1hdr.b_pabd, zio->io_abd, psize); 6528 } else { 6529 ASSERT3U(zio->io_orig_size, ==, arc_hdr_size(hdr)); 6530 arc_hdr_alloc_abd(hdr, ARC_HDR_USE_RESERVE); 6531 abd_copy_from_buf(hdr->b_l1hdr.b_pabd, buf->b_data, 6532 arc_buf_size(buf)); 6533 } 6534 } else { 6535 ASSERT3P(buf->b_data, ==, abd_to_buf(zio->io_orig_abd)); 6536 ASSERT3U(zio->io_orig_size, ==, arc_buf_size(buf)); 6537 ASSERT3P(hdr->b_l1hdr.b_buf, ==, buf); 6538 ASSERT(ARC_BUF_LAST(buf)); 6539 6540 arc_share_buf(hdr, buf); 6541 } 6542 6543 out: 6544 arc_hdr_verify(hdr, bp); 6545 spl_fstrans_unmark(cookie); 6546 } 6547 6548 static void 6549 arc_write_children_ready(zio_t *zio) 6550 { 6551 arc_write_callback_t *callback = zio->io_private; 6552 arc_buf_t *buf = callback->awcb_buf; 6553 6554 callback->awcb_children_ready(zio, buf, callback->awcb_private); 6555 } 6556 6557 static void 6558 arc_write_done(zio_t *zio) 6559 { 6560 arc_write_callback_t *callback = zio->io_private; 6561 arc_buf_t *buf = callback->awcb_buf; 6562 arc_buf_hdr_t *hdr = buf->b_hdr; 6563 6564 ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); 6565 6566 if (zio->io_error == 0) { 6567 arc_hdr_verify(hdr, zio->io_bp); 6568 6569 if (BP_IS_HOLE(zio->io_bp) || BP_IS_EMBEDDED(zio->io_bp)) { 6570 buf_discard_identity(hdr); 6571 } else { 6572 hdr->b_dva = *BP_IDENTITY(zio->io_bp); 6573 hdr->b_birth = BP_GET_BIRTH(zio->io_bp); 6574 } 6575 } else { 6576 ASSERT(HDR_EMPTY(hdr)); 6577 } 6578 6579 /* 6580 * If the block to be written was all-zero or compressed enough to be 6581 * embedded in the BP, no write was performed so there will be no 6582 * dva/birth/checksum. The buffer must therefore remain anonymous 6583 * (and uncached). 6584 */ 6585 if (!HDR_EMPTY(hdr)) { 6586 arc_buf_hdr_t *exists; 6587 kmutex_t *hash_lock; 6588 6589 ASSERT3U(zio->io_error, ==, 0); 6590 6591 arc_cksum_verify(buf); 6592 6593 exists = buf_hash_insert(hdr, &hash_lock); 6594 if (exists != NULL) { 6595 /* 6596 * This can only happen if we overwrite for 6597 * sync-to-convergence, because we remove 6598 * buffers from the hash table when we arc_free(). 6599 */ 6600 if (zio->io_flags & ZIO_FLAG_IO_REWRITE) { 6601 if (!BP_EQUAL(&zio->io_bp_orig, zio->io_bp)) 6602 panic("bad overwrite, hdr=%p exists=%p", 6603 (void *)hdr, (void *)exists); 6604 ASSERT(zfs_refcount_is_zero( 6605 &exists->b_l1hdr.b_refcnt)); 6606 arc_change_state(arc_anon, exists); 6607 arc_hdr_destroy(exists); 6608 mutex_exit(hash_lock); 6609 exists = buf_hash_insert(hdr, &hash_lock); 6610 ASSERT3P(exists, ==, NULL); 6611 } else if (zio->io_flags & ZIO_FLAG_NOPWRITE) { 6612 /* nopwrite */ 6613 ASSERT(zio->io_prop.zp_nopwrite); 6614 if (!BP_EQUAL(&zio->io_bp_orig, zio->io_bp)) 6615 panic("bad nopwrite, hdr=%p exists=%p", 6616 (void *)hdr, (void *)exists); 6617 } else { 6618 /* Dedup */ 6619 ASSERT3P(hdr->b_l1hdr.b_buf, !=, NULL); 6620 ASSERT(ARC_BUF_LAST(hdr->b_l1hdr.b_buf)); 6621 ASSERT(hdr->b_l1hdr.b_state == arc_anon); 6622 ASSERT(BP_GET_DEDUP(zio->io_bp)); 6623 ASSERT(BP_GET_LEVEL(zio->io_bp) == 0); 6624 } 6625 } 6626 arc_hdr_clear_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 6627 VERIFY3S(remove_reference(hdr, hdr), >, 0); 6628 /* if it's not anon, we are doing a scrub */ 6629 if (exists == NULL && hdr->b_l1hdr.b_state == arc_anon) 6630 arc_access(hdr, 0, B_FALSE); 6631 mutex_exit(hash_lock); 6632 } else { 6633 arc_hdr_clear_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); 6634 VERIFY3S(remove_reference(hdr, hdr), >, 0); 6635 } 6636 6637 callback->awcb_done(zio, buf, callback->awcb_private); 6638 6639 abd_free(zio->io_abd); 6640 kmem_free(callback, sizeof (arc_write_callback_t)); 6641 } 6642 6643 zio_t * 6644 arc_write(zio_t *pio, spa_t *spa, uint64_t txg, 6645 blkptr_t *bp, arc_buf_t *buf, boolean_t uncached, boolean_t l2arc, 6646 const zio_prop_t *zp, arc_write_done_func_t *ready, 6647 arc_write_done_func_t *children_ready, arc_write_done_func_t *done, 6648 void *private, zio_priority_t priority, int zio_flags, 6649 const zbookmark_phys_t *zb) 6650 { 6651 arc_buf_hdr_t *hdr = buf->b_hdr; 6652 arc_write_callback_t *callback; 6653 zio_t *zio; 6654 zio_prop_t localprop = *zp; 6655 6656 ASSERT3P(ready, !=, NULL); 6657 ASSERT3P(done, !=, NULL); 6658 ASSERT(!HDR_IO_ERROR(hdr)); 6659 ASSERT(!HDR_IO_IN_PROGRESS(hdr)); 6660 ASSERT3P(hdr->b_l1hdr.b_acb, ==, NULL); 6661 ASSERT3P(hdr->b_l1hdr.b_buf, !=, NULL); 6662 if (uncached) 6663 arc_hdr_set_flags(hdr, ARC_FLAG_UNCACHED); 6664 else if (l2arc) 6665 arc_hdr_set_flags(hdr, ARC_FLAG_L2CACHE); 6666 6667 if (ARC_BUF_ENCRYPTED(buf)) { 6668 ASSERT(ARC_BUF_COMPRESSED(buf)); 6669 localprop.zp_encrypt = B_TRUE; 6670 localprop.zp_compress = HDR_GET_COMPRESS(hdr); 6671 localprop.zp_complevel = hdr->b_complevel; 6672 localprop.zp_byteorder = 6673 (hdr->b_l1hdr.b_byteswap == DMU_BSWAP_NUMFUNCS) ? 6674 ZFS_HOST_BYTEORDER : !ZFS_HOST_BYTEORDER; 6675 memcpy(localprop.zp_salt, hdr->b_crypt_hdr.b_salt, 6676 ZIO_DATA_SALT_LEN); 6677 memcpy(localprop.zp_iv, hdr->b_crypt_hdr.b_iv, 6678 ZIO_DATA_IV_LEN); 6679 memcpy(localprop.zp_mac, hdr->b_crypt_hdr.b_mac, 6680 ZIO_DATA_MAC_LEN); 6681 if (DMU_OT_IS_ENCRYPTED(localprop.zp_type)) { 6682 localprop.zp_nopwrite = B_FALSE; 6683 localprop.zp_copies = 6684 MIN(localprop.zp_copies, SPA_DVAS_PER_BP - 1); 6685 } 6686 zio_flags |= ZIO_FLAG_RAW; 6687 } else if (ARC_BUF_COMPRESSED(buf)) { 6688 ASSERT3U(HDR_GET_LSIZE(hdr), !=, arc_buf_size(buf)); 6689 localprop.zp_compress = HDR_GET_COMPRESS(hdr); 6690 localprop.zp_complevel = hdr->b_complevel; 6691 zio_flags |= ZIO_FLAG_RAW_COMPRESS; 6692 } 6693 callback = kmem_zalloc(sizeof (arc_write_callback_t), KM_SLEEP); 6694 callback->awcb_ready = ready; 6695 callback->awcb_children_ready = children_ready; 6696 callback->awcb_done = done; 6697 callback->awcb_private = private; 6698 callback->awcb_buf = buf; 6699 6700 /* 6701 * The hdr's b_pabd is now stale, free it now. A new data block 6702 * will be allocated when the zio pipeline calls arc_write_ready(). 6703 */ 6704 if (hdr->b_l1hdr.b_pabd != NULL) { 6705 /* 6706 * If the buf is currently sharing the data block with 6707 * the hdr then we need to break that relationship here. 6708 * The hdr will remain with a NULL data pointer and the 6709 * buf will take sole ownership of the block. 6710 */ 6711 if (ARC_BUF_SHARED(buf)) { 6712 arc_unshare_buf(hdr, buf); 6713 } else { 6714 ASSERT(!arc_buf_is_shared(buf)); 6715 arc_hdr_free_abd(hdr, B_FALSE); 6716 } 6717 VERIFY3P(buf->b_data, !=, NULL); 6718 } 6719 6720 if (HDR_HAS_RABD(hdr)) 6721 arc_hdr_free_abd(hdr, B_TRUE); 6722 6723 if (!(zio_flags & ZIO_FLAG_RAW)) 6724 arc_hdr_set_compress(hdr, ZIO_COMPRESS_OFF); 6725 6726 ASSERT(!arc_buf_is_shared(buf)); 6727 ASSERT3P(hdr->b_l1hdr.b_pabd, ==, NULL); 6728 6729 zio = zio_write(pio, spa, txg, bp, 6730 abd_get_from_buf(buf->b_data, HDR_GET_LSIZE(hdr)), 6731 HDR_GET_LSIZE(hdr), arc_buf_size(buf), &localprop, arc_write_ready, 6732 (children_ready != NULL) ? arc_write_children_ready : NULL, 6733 arc_write_done, callback, priority, zio_flags, zb); 6734 6735 return (zio); 6736 } 6737 6738 void 6739 arc_tempreserve_clear(uint64_t reserve) 6740 { 6741 atomic_add_64(&arc_tempreserve, -reserve); 6742 ASSERT((int64_t)arc_tempreserve >= 0); 6743 } 6744 6745 int 6746 arc_tempreserve_space(spa_t *spa, uint64_t reserve, uint64_t txg) 6747 { 6748 int error; 6749 uint64_t anon_size; 6750 6751 if (!arc_no_grow && 6752 reserve > arc_c/4 && 6753 reserve * 4 > (2ULL << SPA_MAXBLOCKSHIFT)) 6754 arc_c = MIN(arc_c_max, reserve * 4); 6755 6756 /* 6757 * Throttle when the calculated memory footprint for the TXG 6758 * exceeds the target ARC size. 6759 */ 6760 if (reserve > arc_c) { 6761 DMU_TX_STAT_BUMP(dmu_tx_memory_reserve); 6762 return (SET_ERROR(ERESTART)); 6763 } 6764 6765 /* 6766 * Don't count loaned bufs as in flight dirty data to prevent long 6767 * network delays from blocking transactions that are ready to be 6768 * assigned to a txg. 6769 */ 6770 6771 /* assert that it has not wrapped around */ 6772 ASSERT3S(atomic_add_64_nv(&arc_loaned_bytes, 0), >=, 0); 6773 6774 anon_size = MAX((int64_t) 6775 (zfs_refcount_count(&arc_anon->arcs_size[ARC_BUFC_DATA]) + 6776 zfs_refcount_count(&arc_anon->arcs_size[ARC_BUFC_METADATA]) - 6777 arc_loaned_bytes), 0); 6778 6779 /* 6780 * Writes will, almost always, require additional memory allocations 6781 * in order to compress/encrypt/etc the data. We therefore need to 6782 * make sure that there is sufficient available memory for this. 6783 */ 6784 error = arc_memory_throttle(spa, reserve, txg); 6785 if (error != 0) 6786 return (error); 6787 6788 /* 6789 * Throttle writes when the amount of dirty data in the cache 6790 * gets too large. We try to keep the cache less than half full 6791 * of dirty blocks so that our sync times don't grow too large. 6792 * 6793 * In the case of one pool being built on another pool, we want 6794 * to make sure we don't end up throttling the lower (backing) 6795 * pool when the upper pool is the majority contributor to dirty 6796 * data. To insure we make forward progress during throttling, we 6797 * also check the current pool's net dirty data and only throttle 6798 * if it exceeds zfs_arc_pool_dirty_percent of the anonymous dirty 6799 * data in the cache. 6800 * 6801 * Note: if two requests come in concurrently, we might let them 6802 * both succeed, when one of them should fail. Not a huge deal. 6803 */ 6804 uint64_t total_dirty = reserve + arc_tempreserve + anon_size; 6805 uint64_t spa_dirty_anon = spa_dirty_data(spa); 6806 uint64_t rarc_c = arc_warm ? arc_c : arc_c_max; 6807 if (total_dirty > rarc_c * zfs_arc_dirty_limit_percent / 100 && 6808 anon_size > rarc_c * zfs_arc_anon_limit_percent / 100 && 6809 spa_dirty_anon > anon_size * zfs_arc_pool_dirty_percent / 100) { 6810 #ifdef ZFS_DEBUG 6811 uint64_t meta_esize = zfs_refcount_count( 6812 &arc_anon->arcs_esize[ARC_BUFC_METADATA]); 6813 uint64_t data_esize = 6814 zfs_refcount_count(&arc_anon->arcs_esize[ARC_BUFC_DATA]); 6815 dprintf("failing, arc_tempreserve=%lluK anon_meta=%lluK " 6816 "anon_data=%lluK tempreserve=%lluK rarc_c=%lluK\n", 6817 (u_longlong_t)arc_tempreserve >> 10, 6818 (u_longlong_t)meta_esize >> 10, 6819 (u_longlong_t)data_esize >> 10, 6820 (u_longlong_t)reserve >> 10, 6821 (u_longlong_t)rarc_c >> 10); 6822 #endif 6823 DMU_TX_STAT_BUMP(dmu_tx_dirty_throttle); 6824 return (SET_ERROR(ERESTART)); 6825 } 6826 atomic_add_64(&arc_tempreserve, reserve); 6827 return (0); 6828 } 6829 6830 static void 6831 arc_kstat_update_state(arc_state_t *state, kstat_named_t *size, 6832 kstat_named_t *data, kstat_named_t *metadata, 6833 kstat_named_t *evict_data, kstat_named_t *evict_metadata) 6834 { 6835 data->value.ui64 = 6836 zfs_refcount_count(&state->arcs_size[ARC_BUFC_DATA]); 6837 metadata->value.ui64 = 6838 zfs_refcount_count(&state->arcs_size[ARC_BUFC_METADATA]); 6839 size->value.ui64 = data->value.ui64 + metadata->value.ui64; 6840 evict_data->value.ui64 = 6841 zfs_refcount_count(&state->arcs_esize[ARC_BUFC_DATA]); 6842 evict_metadata->value.ui64 = 6843 zfs_refcount_count(&state->arcs_esize[ARC_BUFC_METADATA]); 6844 } 6845 6846 static int 6847 arc_kstat_update(kstat_t *ksp, int rw) 6848 { 6849 arc_stats_t *as = ksp->ks_data; 6850 6851 if (rw == KSTAT_WRITE) 6852 return (SET_ERROR(EACCES)); 6853 6854 as->arcstat_hits.value.ui64 = 6855 wmsum_value(&arc_sums.arcstat_hits); 6856 as->arcstat_iohits.value.ui64 = 6857 wmsum_value(&arc_sums.arcstat_iohits); 6858 as->arcstat_misses.value.ui64 = 6859 wmsum_value(&arc_sums.arcstat_misses); 6860 as->arcstat_demand_data_hits.value.ui64 = 6861 wmsum_value(&arc_sums.arcstat_demand_data_hits); 6862 as->arcstat_demand_data_iohits.value.ui64 = 6863 wmsum_value(&arc_sums.arcstat_demand_data_iohits); 6864 as->arcstat_demand_data_misses.value.ui64 = 6865 wmsum_value(&arc_sums.arcstat_demand_data_misses); 6866 as->arcstat_demand_metadata_hits.value.ui64 = 6867 wmsum_value(&arc_sums.arcstat_demand_metadata_hits); 6868 as->arcstat_demand_metadata_iohits.value.ui64 = 6869 wmsum_value(&arc_sums.arcstat_demand_metadata_iohits); 6870 as->arcstat_demand_metadata_misses.value.ui64 = 6871 wmsum_value(&arc_sums.arcstat_demand_metadata_misses); 6872 as->arcstat_prefetch_data_hits.value.ui64 = 6873 wmsum_value(&arc_sums.arcstat_prefetch_data_hits); 6874 as->arcstat_prefetch_data_iohits.value.ui64 = 6875 wmsum_value(&arc_sums.arcstat_prefetch_data_iohits); 6876 as->arcstat_prefetch_data_misses.value.ui64 = 6877 wmsum_value(&arc_sums.arcstat_prefetch_data_misses); 6878 as->arcstat_prefetch_metadata_hits.value.ui64 = 6879 wmsum_value(&arc_sums.arcstat_prefetch_metadata_hits); 6880 as->arcstat_prefetch_metadata_iohits.value.ui64 = 6881 wmsum_value(&arc_sums.arcstat_prefetch_metadata_iohits); 6882 as->arcstat_prefetch_metadata_misses.value.ui64 = 6883 wmsum_value(&arc_sums.arcstat_prefetch_metadata_misses); 6884 as->arcstat_mru_hits.value.ui64 = 6885 wmsum_value(&arc_sums.arcstat_mru_hits); 6886 as->arcstat_mru_ghost_hits.value.ui64 = 6887 wmsum_value(&arc_sums.arcstat_mru_ghost_hits); 6888 as->arcstat_mfu_hits.value.ui64 = 6889 wmsum_value(&arc_sums.arcstat_mfu_hits); 6890 as->arcstat_mfu_ghost_hits.value.ui64 = 6891 wmsum_value(&arc_sums.arcstat_mfu_ghost_hits); 6892 as->arcstat_uncached_hits.value.ui64 = 6893 wmsum_value(&arc_sums.arcstat_uncached_hits); 6894 as->arcstat_deleted.value.ui64 = 6895 wmsum_value(&arc_sums.arcstat_deleted); 6896 as->arcstat_mutex_miss.value.ui64 = 6897 wmsum_value(&arc_sums.arcstat_mutex_miss); 6898 as->arcstat_access_skip.value.ui64 = 6899 wmsum_value(&arc_sums.arcstat_access_skip); 6900 as->arcstat_evict_skip.value.ui64 = 6901 wmsum_value(&arc_sums.arcstat_evict_skip); 6902 as->arcstat_evict_not_enough.value.ui64 = 6903 wmsum_value(&arc_sums.arcstat_evict_not_enough); 6904 as->arcstat_evict_l2_cached.value.ui64 = 6905 wmsum_value(&arc_sums.arcstat_evict_l2_cached); 6906 as->arcstat_evict_l2_eligible.value.ui64 = 6907 wmsum_value(&arc_sums.arcstat_evict_l2_eligible); 6908 as->arcstat_evict_l2_eligible_mfu.value.ui64 = 6909 wmsum_value(&arc_sums.arcstat_evict_l2_eligible_mfu); 6910 as->arcstat_evict_l2_eligible_mru.value.ui64 = 6911 wmsum_value(&arc_sums.arcstat_evict_l2_eligible_mru); 6912 as->arcstat_evict_l2_ineligible.value.ui64 = 6913 wmsum_value(&arc_sums.arcstat_evict_l2_ineligible); 6914 as->arcstat_evict_l2_skip.value.ui64 = 6915 wmsum_value(&arc_sums.arcstat_evict_l2_skip); 6916 as->arcstat_hash_collisions.value.ui64 = 6917 wmsum_value(&arc_sums.arcstat_hash_collisions); 6918 as->arcstat_hash_chains.value.ui64 = 6919 wmsum_value(&arc_sums.arcstat_hash_chains); 6920 as->arcstat_size.value.ui64 = 6921 aggsum_value(&arc_sums.arcstat_size); 6922 as->arcstat_compressed_size.value.ui64 = 6923 wmsum_value(&arc_sums.arcstat_compressed_size); 6924 as->arcstat_uncompressed_size.value.ui64 = 6925 wmsum_value(&arc_sums.arcstat_uncompressed_size); 6926 as->arcstat_overhead_size.value.ui64 = 6927 wmsum_value(&arc_sums.arcstat_overhead_size); 6928 as->arcstat_hdr_size.value.ui64 = 6929 wmsum_value(&arc_sums.arcstat_hdr_size); 6930 as->arcstat_data_size.value.ui64 = 6931 wmsum_value(&arc_sums.arcstat_data_size); 6932 as->arcstat_metadata_size.value.ui64 = 6933 wmsum_value(&arc_sums.arcstat_metadata_size); 6934 as->arcstat_dbuf_size.value.ui64 = 6935 wmsum_value(&arc_sums.arcstat_dbuf_size); 6936 #if defined(COMPAT_FREEBSD11) 6937 as->arcstat_other_size.value.ui64 = 6938 wmsum_value(&arc_sums.arcstat_bonus_size) + 6939 wmsum_value(&arc_sums.arcstat_dnode_size) + 6940 wmsum_value(&arc_sums.arcstat_dbuf_size); 6941 #endif 6942 6943 arc_kstat_update_state(arc_anon, 6944 &as->arcstat_anon_size, 6945 &as->arcstat_anon_data, 6946 &as->arcstat_anon_metadata, 6947 &as->arcstat_anon_evictable_data, 6948 &as->arcstat_anon_evictable_metadata); 6949 arc_kstat_update_state(arc_mru, 6950 &as->arcstat_mru_size, 6951 &as->arcstat_mru_data, 6952 &as->arcstat_mru_metadata, 6953 &as->arcstat_mru_evictable_data, 6954 &as->arcstat_mru_evictable_metadata); 6955 arc_kstat_update_state(arc_mru_ghost, 6956 &as->arcstat_mru_ghost_size, 6957 &as->arcstat_mru_ghost_data, 6958 &as->arcstat_mru_ghost_metadata, 6959 &as->arcstat_mru_ghost_evictable_data, 6960 &as->arcstat_mru_ghost_evictable_metadata); 6961 arc_kstat_update_state(arc_mfu, 6962 &as->arcstat_mfu_size, 6963 &as->arcstat_mfu_data, 6964 &as->arcstat_mfu_metadata, 6965 &as->arcstat_mfu_evictable_data, 6966 &as->arcstat_mfu_evictable_metadata); 6967 arc_kstat_update_state(arc_mfu_ghost, 6968 &as->arcstat_mfu_ghost_size, 6969 &as->arcstat_mfu_ghost_data, 6970 &as->arcstat_mfu_ghost_metadata, 6971 &as->arcstat_mfu_ghost_evictable_data, 6972 &as->arcstat_mfu_ghost_evictable_metadata); 6973 arc_kstat_update_state(arc_uncached, 6974 &as->arcstat_uncached_size, 6975 &as->arcstat_uncached_data, 6976 &as->arcstat_uncached_metadata, 6977 &as->arcstat_uncached_evictable_data, 6978 &as->arcstat_uncached_evictable_metadata); 6979 6980 as->arcstat_dnode_size.value.ui64 = 6981 wmsum_value(&arc_sums.arcstat_dnode_size); 6982 as->arcstat_bonus_size.value.ui64 = 6983 wmsum_value(&arc_sums.arcstat_bonus_size); 6984 as->arcstat_l2_hits.value.ui64 = 6985 wmsum_value(&arc_sums.arcstat_l2_hits); 6986 as->arcstat_l2_misses.value.ui64 = 6987 wmsum_value(&arc_sums.arcstat_l2_misses); 6988 as->arcstat_l2_prefetch_asize.value.ui64 = 6989 wmsum_value(&arc_sums.arcstat_l2_prefetch_asize); 6990 as->arcstat_l2_mru_asize.value.ui64 = 6991 wmsum_value(&arc_sums.arcstat_l2_mru_asize); 6992 as->arcstat_l2_mfu_asize.value.ui64 = 6993 wmsum_value(&arc_sums.arcstat_l2_mfu_asize); 6994 as->arcstat_l2_bufc_data_asize.value.ui64 = 6995 wmsum_value(&arc_sums.arcstat_l2_bufc_data_asize); 6996 as->arcstat_l2_bufc_metadata_asize.value.ui64 = 6997 wmsum_value(&arc_sums.arcstat_l2_bufc_metadata_asize); 6998 as->arcstat_l2_feeds.value.ui64 = 6999 wmsum_value(&arc_sums.arcstat_l2_feeds); 7000 as->arcstat_l2_rw_clash.value.ui64 = 7001 wmsum_value(&arc_sums.arcstat_l2_rw_clash); 7002 as->arcstat_l2_read_bytes.value.ui64 = 7003 wmsum_value(&arc_sums.arcstat_l2_read_bytes); 7004 as->arcstat_l2_write_bytes.value.ui64 = 7005 wmsum_value(&arc_sums.arcstat_l2_write_bytes); 7006 as->arcstat_l2_writes_sent.value.ui64 = 7007 wmsum_value(&arc_sums.arcstat_l2_writes_sent); 7008 as->arcstat_l2_writes_done.value.ui64 = 7009 wmsum_value(&arc_sums.arcstat_l2_writes_done); 7010 as->arcstat_l2_writes_error.value.ui64 = 7011 wmsum_value(&arc_sums.arcstat_l2_writes_error); 7012 as->arcstat_l2_writes_lock_retry.value.ui64 = 7013 wmsum_value(&arc_sums.arcstat_l2_writes_lock_retry); 7014 as->arcstat_l2_evict_lock_retry.value.ui64 = 7015 wmsum_value(&arc_sums.arcstat_l2_evict_lock_retry); 7016 as->arcstat_l2_evict_reading.value.ui64 = 7017 wmsum_value(&arc_sums.arcstat_l2_evict_reading); 7018 as->arcstat_l2_evict_l1cached.value.ui64 = 7019 wmsum_value(&arc_sums.arcstat_l2_evict_l1cached); 7020 as->arcstat_l2_free_on_write.value.ui64 = 7021 wmsum_value(&arc_sums.arcstat_l2_free_on_write); 7022 as->arcstat_l2_abort_lowmem.value.ui64 = 7023 wmsum_value(&arc_sums.arcstat_l2_abort_lowmem); 7024 as->arcstat_l2_cksum_bad.value.ui64 = 7025 wmsum_value(&arc_sums.arcstat_l2_cksum_bad); 7026 as->arcstat_l2_io_error.value.ui64 = 7027 wmsum_value(&arc_sums.arcstat_l2_io_error); 7028 as->arcstat_l2_lsize.value.ui64 = 7029 wmsum_value(&arc_sums.arcstat_l2_lsize); 7030 as->arcstat_l2_psize.value.ui64 = 7031 wmsum_value(&arc_sums.arcstat_l2_psize); 7032 as->arcstat_l2_hdr_size.value.ui64 = 7033 aggsum_value(&arc_sums.arcstat_l2_hdr_size); 7034 as->arcstat_l2_log_blk_writes.value.ui64 = 7035 wmsum_value(&arc_sums.arcstat_l2_log_blk_writes); 7036 as->arcstat_l2_log_blk_asize.value.ui64 = 7037 wmsum_value(&arc_sums.arcstat_l2_log_blk_asize); 7038 as->arcstat_l2_log_blk_count.value.ui64 = 7039 wmsum_value(&arc_sums.arcstat_l2_log_blk_count); 7040 as->arcstat_l2_rebuild_success.value.ui64 = 7041 wmsum_value(&arc_sums.arcstat_l2_rebuild_success); 7042 as->arcstat_l2_rebuild_abort_unsupported.value.ui64 = 7043 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_unsupported); 7044 as->arcstat_l2_rebuild_abort_io_errors.value.ui64 = 7045 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_io_errors); 7046 as->arcstat_l2_rebuild_abort_dh_errors.value.ui64 = 7047 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_dh_errors); 7048 as->arcstat_l2_rebuild_abort_cksum_lb_errors.value.ui64 = 7049 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_cksum_lb_errors); 7050 as->arcstat_l2_rebuild_abort_lowmem.value.ui64 = 7051 wmsum_value(&arc_sums.arcstat_l2_rebuild_abort_lowmem); 7052 as->arcstat_l2_rebuild_size.value.ui64 = 7053 wmsum_value(&arc_sums.arcstat_l2_rebuild_size); 7054 as->arcstat_l2_rebuild_asize.value.ui64 = 7055 wmsum_value(&arc_sums.arcstat_l2_rebuild_asize); 7056 as->arcstat_l2_rebuild_bufs.value.ui64 = 7057 wmsum_value(&arc_sums.arcstat_l2_rebuild_bufs); 7058 as->arcstat_l2_rebuild_bufs_precached.value.ui64 = 7059 wmsum_value(&arc_sums.arcstat_l2_rebuild_bufs_precached); 7060 as->arcstat_l2_rebuild_log_blks.value.ui64 = 7061 wmsum_value(&arc_sums.arcstat_l2_rebuild_log_blks); 7062 as->arcstat_memory_throttle_count.value.ui64 = 7063 wmsum_value(&arc_sums.arcstat_memory_throttle_count); 7064 as->arcstat_memory_direct_count.value.ui64 = 7065 wmsum_value(&arc_sums.arcstat_memory_direct_count); 7066 as->arcstat_memory_indirect_count.value.ui64 = 7067 wmsum_value(&arc_sums.arcstat_memory_indirect_count); 7068 7069 as->arcstat_memory_all_bytes.value.ui64 = 7070 arc_all_memory(); 7071 as->arcstat_memory_free_bytes.value.ui64 = 7072 arc_free_memory(); 7073 as->arcstat_memory_available_bytes.value.i64 = 7074 arc_available_memory(); 7075 7076 as->arcstat_prune.value.ui64 = 7077 wmsum_value(&arc_sums.arcstat_prune); 7078 as->arcstat_meta_used.value.ui64 = 7079 wmsum_value(&arc_sums.arcstat_meta_used); 7080 as->arcstat_async_upgrade_sync.value.ui64 = 7081 wmsum_value(&arc_sums.arcstat_async_upgrade_sync); 7082 as->arcstat_predictive_prefetch.value.ui64 = 7083 wmsum_value(&arc_sums.arcstat_predictive_prefetch); 7084 as->arcstat_demand_hit_predictive_prefetch.value.ui64 = 7085 wmsum_value(&arc_sums.arcstat_demand_hit_predictive_prefetch); 7086 as->arcstat_demand_iohit_predictive_prefetch.value.ui64 = 7087 wmsum_value(&arc_sums.arcstat_demand_iohit_predictive_prefetch); 7088 as->arcstat_prescient_prefetch.value.ui64 = 7089 wmsum_value(&arc_sums.arcstat_prescient_prefetch); 7090 as->arcstat_demand_hit_prescient_prefetch.value.ui64 = 7091 wmsum_value(&arc_sums.arcstat_demand_hit_prescient_prefetch); 7092 as->arcstat_demand_iohit_prescient_prefetch.value.ui64 = 7093 wmsum_value(&arc_sums.arcstat_demand_iohit_prescient_prefetch); 7094 as->arcstat_raw_size.value.ui64 = 7095 wmsum_value(&arc_sums.arcstat_raw_size); 7096 as->arcstat_cached_only_in_progress.value.ui64 = 7097 wmsum_value(&arc_sums.arcstat_cached_only_in_progress); 7098 as->arcstat_abd_chunk_waste_size.value.ui64 = 7099 wmsum_value(&arc_sums.arcstat_abd_chunk_waste_size); 7100 7101 return (0); 7102 } 7103 7104 /* 7105 * This function *must* return indices evenly distributed between all 7106 * sublists of the multilist. This is needed due to how the ARC eviction 7107 * code is laid out; arc_evict_state() assumes ARC buffers are evenly 7108 * distributed between all sublists and uses this assumption when 7109 * deciding which sublist to evict from and how much to evict from it. 7110 */ 7111 static unsigned int 7112 arc_state_multilist_index_func(multilist_t *ml, void *obj) 7113 { 7114 arc_buf_hdr_t *hdr = obj; 7115 7116 /* 7117 * We rely on b_dva to generate evenly distributed index 7118 * numbers using buf_hash below. So, as an added precaution, 7119 * let's make sure we never add empty buffers to the arc lists. 7120 */ 7121 ASSERT(!HDR_EMPTY(hdr)); 7122 7123 /* 7124 * The assumption here, is the hash value for a given 7125 * arc_buf_hdr_t will remain constant throughout its lifetime 7126 * (i.e. its b_spa, b_dva, and b_birth fields don't change). 7127 * Thus, we don't need to store the header's sublist index 7128 * on insertion, as this index can be recalculated on removal. 7129 * 7130 * Also, the low order bits of the hash value are thought to be 7131 * distributed evenly. Otherwise, in the case that the multilist 7132 * has a power of two number of sublists, each sublists' usage 7133 * would not be evenly distributed. In this context full 64bit 7134 * division would be a waste of time, so limit it to 32 bits. 7135 */ 7136 return ((unsigned int)buf_hash(hdr->b_spa, &hdr->b_dva, hdr->b_birth) % 7137 multilist_get_num_sublists(ml)); 7138 } 7139 7140 static unsigned int 7141 arc_state_l2c_multilist_index_func(multilist_t *ml, void *obj) 7142 { 7143 panic("Header %p insert into arc_l2c_only %p", obj, ml); 7144 } 7145 7146 #define WARN_IF_TUNING_IGNORED(tuning, value, do_warn) do { \ 7147 if ((do_warn) && (tuning) && ((tuning) != (value))) { \ 7148 cmn_err(CE_WARN, \ 7149 "ignoring tunable %s (using %llu instead)", \ 7150 (#tuning), (u_longlong_t)(value)); \ 7151 } \ 7152 } while (0) 7153 7154 /* 7155 * Called during module initialization and periodically thereafter to 7156 * apply reasonable changes to the exposed performance tunings. Can also be 7157 * called explicitly by param_set_arc_*() functions when ARC tunables are 7158 * updated manually. Non-zero zfs_* values which differ from the currently set 7159 * values will be applied. 7160 */ 7161 void 7162 arc_tuning_update(boolean_t verbose) 7163 { 7164 uint64_t allmem = arc_all_memory(); 7165 7166 /* Valid range: 32M - <arc_c_max> */ 7167 if ((zfs_arc_min) && (zfs_arc_min != arc_c_min) && 7168 (zfs_arc_min >= 2ULL << SPA_MAXBLOCKSHIFT) && 7169 (zfs_arc_min <= arc_c_max)) { 7170 arc_c_min = zfs_arc_min; 7171 arc_c = MAX(arc_c, arc_c_min); 7172 } 7173 WARN_IF_TUNING_IGNORED(zfs_arc_min, arc_c_min, verbose); 7174 7175 /* Valid range: 64M - <all physical memory> */ 7176 if ((zfs_arc_max) && (zfs_arc_max != arc_c_max) && 7177 (zfs_arc_max >= MIN_ARC_MAX) && (zfs_arc_max < allmem) && 7178 (zfs_arc_max > arc_c_min)) { 7179 arc_c_max = zfs_arc_max; 7180 arc_c = MIN(arc_c, arc_c_max); 7181 if (arc_dnode_limit > arc_c_max) 7182 arc_dnode_limit = arc_c_max; 7183 } 7184 WARN_IF_TUNING_IGNORED(zfs_arc_max, arc_c_max, verbose); 7185 7186 /* Valid range: 0 - <all physical memory> */ 7187 arc_dnode_limit = zfs_arc_dnode_limit ? zfs_arc_dnode_limit : 7188 MIN(zfs_arc_dnode_limit_percent, 100) * arc_c_max / 100; 7189 WARN_IF_TUNING_IGNORED(zfs_arc_dnode_limit, arc_dnode_limit, verbose); 7190 7191 /* Valid range: 1 - N */ 7192 if (zfs_arc_grow_retry) 7193 arc_grow_retry = zfs_arc_grow_retry; 7194 7195 /* Valid range: 1 - N */ 7196 if (zfs_arc_shrink_shift) { 7197 arc_shrink_shift = zfs_arc_shrink_shift; 7198 arc_no_grow_shift = MIN(arc_no_grow_shift, arc_shrink_shift -1); 7199 } 7200 7201 /* Valid range: 1 - N ms */ 7202 if (zfs_arc_min_prefetch_ms) 7203 arc_min_prefetch_ms = zfs_arc_min_prefetch_ms; 7204 7205 /* Valid range: 1 - N ms */ 7206 if (zfs_arc_min_prescient_prefetch_ms) { 7207 arc_min_prescient_prefetch_ms = 7208 zfs_arc_min_prescient_prefetch_ms; 7209 } 7210 7211 /* Valid range: 0 - 100 */ 7212 if (zfs_arc_lotsfree_percent <= 100) 7213 arc_lotsfree_percent = zfs_arc_lotsfree_percent; 7214 WARN_IF_TUNING_IGNORED(zfs_arc_lotsfree_percent, arc_lotsfree_percent, 7215 verbose); 7216 7217 /* Valid range: 0 - <all physical memory> */ 7218 if ((zfs_arc_sys_free) && (zfs_arc_sys_free != arc_sys_free)) 7219 arc_sys_free = MIN(zfs_arc_sys_free, allmem); 7220 WARN_IF_TUNING_IGNORED(zfs_arc_sys_free, arc_sys_free, verbose); 7221 } 7222 7223 static void 7224 arc_state_multilist_init(multilist_t *ml, 7225 multilist_sublist_index_func_t *index_func, int *maxcountp) 7226 { 7227 multilist_create(ml, sizeof (arc_buf_hdr_t), 7228 offsetof(arc_buf_hdr_t, b_l1hdr.b_arc_node), index_func); 7229 *maxcountp = MAX(*maxcountp, multilist_get_num_sublists(ml)); 7230 } 7231 7232 static void 7233 arc_state_init(void) 7234 { 7235 int num_sublists = 0; 7236 7237 arc_state_multilist_init(&arc_mru->arcs_list[ARC_BUFC_METADATA], 7238 arc_state_multilist_index_func, &num_sublists); 7239 arc_state_multilist_init(&arc_mru->arcs_list[ARC_BUFC_DATA], 7240 arc_state_multilist_index_func, &num_sublists); 7241 arc_state_multilist_init(&arc_mru_ghost->arcs_list[ARC_BUFC_METADATA], 7242 arc_state_multilist_index_func, &num_sublists); 7243 arc_state_multilist_init(&arc_mru_ghost->arcs_list[ARC_BUFC_DATA], 7244 arc_state_multilist_index_func, &num_sublists); 7245 arc_state_multilist_init(&arc_mfu->arcs_list[ARC_BUFC_METADATA], 7246 arc_state_multilist_index_func, &num_sublists); 7247 arc_state_multilist_init(&arc_mfu->arcs_list[ARC_BUFC_DATA], 7248 arc_state_multilist_index_func, &num_sublists); 7249 arc_state_multilist_init(&arc_mfu_ghost->arcs_list[ARC_BUFC_METADATA], 7250 arc_state_multilist_index_func, &num_sublists); 7251 arc_state_multilist_init(&arc_mfu_ghost->arcs_list[ARC_BUFC_DATA], 7252 arc_state_multilist_index_func, &num_sublists); 7253 arc_state_multilist_init(&arc_uncached->arcs_list[ARC_BUFC_METADATA], 7254 arc_state_multilist_index_func, &num_sublists); 7255 arc_state_multilist_init(&arc_uncached->arcs_list[ARC_BUFC_DATA], 7256 arc_state_multilist_index_func, &num_sublists); 7257 7258 /* 7259 * L2 headers should never be on the L2 state list since they don't 7260 * have L1 headers allocated. Special index function asserts that. 7261 */ 7262 arc_state_multilist_init(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA], 7263 arc_state_l2c_multilist_index_func, &num_sublists); 7264 arc_state_multilist_init(&arc_l2c_only->arcs_list[ARC_BUFC_DATA], 7265 arc_state_l2c_multilist_index_func, &num_sublists); 7266 7267 /* 7268 * Keep track of the number of markers needed to reclaim buffers from 7269 * any ARC state. The markers will be pre-allocated so as to minimize 7270 * the number of memory allocations performed by the eviction thread. 7271 */ 7272 arc_state_evict_marker_count = num_sublists; 7273 7274 zfs_refcount_create(&arc_anon->arcs_esize[ARC_BUFC_METADATA]); 7275 zfs_refcount_create(&arc_anon->arcs_esize[ARC_BUFC_DATA]); 7276 zfs_refcount_create(&arc_mru->arcs_esize[ARC_BUFC_METADATA]); 7277 zfs_refcount_create(&arc_mru->arcs_esize[ARC_BUFC_DATA]); 7278 zfs_refcount_create(&arc_mru_ghost->arcs_esize[ARC_BUFC_METADATA]); 7279 zfs_refcount_create(&arc_mru_ghost->arcs_esize[ARC_BUFC_DATA]); 7280 zfs_refcount_create(&arc_mfu->arcs_esize[ARC_BUFC_METADATA]); 7281 zfs_refcount_create(&arc_mfu->arcs_esize[ARC_BUFC_DATA]); 7282 zfs_refcount_create(&arc_mfu_ghost->arcs_esize[ARC_BUFC_METADATA]); 7283 zfs_refcount_create(&arc_mfu_ghost->arcs_esize[ARC_BUFC_DATA]); 7284 zfs_refcount_create(&arc_l2c_only->arcs_esize[ARC_BUFC_METADATA]); 7285 zfs_refcount_create(&arc_l2c_only->arcs_esize[ARC_BUFC_DATA]); 7286 zfs_refcount_create(&arc_uncached->arcs_esize[ARC_BUFC_METADATA]); 7287 zfs_refcount_create(&arc_uncached->arcs_esize[ARC_BUFC_DATA]); 7288 7289 zfs_refcount_create(&arc_anon->arcs_size[ARC_BUFC_DATA]); 7290 zfs_refcount_create(&arc_anon->arcs_size[ARC_BUFC_METADATA]); 7291 zfs_refcount_create(&arc_mru->arcs_size[ARC_BUFC_DATA]); 7292 zfs_refcount_create(&arc_mru->arcs_size[ARC_BUFC_METADATA]); 7293 zfs_refcount_create(&arc_mru_ghost->arcs_size[ARC_BUFC_DATA]); 7294 zfs_refcount_create(&arc_mru_ghost->arcs_size[ARC_BUFC_METADATA]); 7295 zfs_refcount_create(&arc_mfu->arcs_size[ARC_BUFC_DATA]); 7296 zfs_refcount_create(&arc_mfu->arcs_size[ARC_BUFC_METADATA]); 7297 zfs_refcount_create(&arc_mfu_ghost->arcs_size[ARC_BUFC_DATA]); 7298 zfs_refcount_create(&arc_mfu_ghost->arcs_size[ARC_BUFC_METADATA]); 7299 zfs_refcount_create(&arc_l2c_only->arcs_size[ARC_BUFC_DATA]); 7300 zfs_refcount_create(&arc_l2c_only->arcs_size[ARC_BUFC_METADATA]); 7301 zfs_refcount_create(&arc_uncached->arcs_size[ARC_BUFC_DATA]); 7302 zfs_refcount_create(&arc_uncached->arcs_size[ARC_BUFC_METADATA]); 7303 7304 wmsum_init(&arc_mru_ghost->arcs_hits[ARC_BUFC_DATA], 0); 7305 wmsum_init(&arc_mru_ghost->arcs_hits[ARC_BUFC_METADATA], 0); 7306 wmsum_init(&arc_mfu_ghost->arcs_hits[ARC_BUFC_DATA], 0); 7307 wmsum_init(&arc_mfu_ghost->arcs_hits[ARC_BUFC_METADATA], 0); 7308 7309 wmsum_init(&arc_sums.arcstat_hits, 0); 7310 wmsum_init(&arc_sums.arcstat_iohits, 0); 7311 wmsum_init(&arc_sums.arcstat_misses, 0); 7312 wmsum_init(&arc_sums.arcstat_demand_data_hits, 0); 7313 wmsum_init(&arc_sums.arcstat_demand_data_iohits, 0); 7314 wmsum_init(&arc_sums.arcstat_demand_data_misses, 0); 7315 wmsum_init(&arc_sums.arcstat_demand_metadata_hits, 0); 7316 wmsum_init(&arc_sums.arcstat_demand_metadata_iohits, 0); 7317 wmsum_init(&arc_sums.arcstat_demand_metadata_misses, 0); 7318 wmsum_init(&arc_sums.arcstat_prefetch_data_hits, 0); 7319 wmsum_init(&arc_sums.arcstat_prefetch_data_iohits, 0); 7320 wmsum_init(&arc_sums.arcstat_prefetch_data_misses, 0); 7321 wmsum_init(&arc_sums.arcstat_prefetch_metadata_hits, 0); 7322 wmsum_init(&arc_sums.arcstat_prefetch_metadata_iohits, 0); 7323 wmsum_init(&arc_sums.arcstat_prefetch_metadata_misses, 0); 7324 wmsum_init(&arc_sums.arcstat_mru_hits, 0); 7325 wmsum_init(&arc_sums.arcstat_mru_ghost_hits, 0); 7326 wmsum_init(&arc_sums.arcstat_mfu_hits, 0); 7327 wmsum_init(&arc_sums.arcstat_mfu_ghost_hits, 0); 7328 wmsum_init(&arc_sums.arcstat_uncached_hits, 0); 7329 wmsum_init(&arc_sums.arcstat_deleted, 0); 7330 wmsum_init(&arc_sums.arcstat_mutex_miss, 0); 7331 wmsum_init(&arc_sums.arcstat_access_skip, 0); 7332 wmsum_init(&arc_sums.arcstat_evict_skip, 0); 7333 wmsum_init(&arc_sums.arcstat_evict_not_enough, 0); 7334 wmsum_init(&arc_sums.arcstat_evict_l2_cached, 0); 7335 wmsum_init(&arc_sums.arcstat_evict_l2_eligible, 0); 7336 wmsum_init(&arc_sums.arcstat_evict_l2_eligible_mfu, 0); 7337 wmsum_init(&arc_sums.arcstat_evict_l2_eligible_mru, 0); 7338 wmsum_init(&arc_sums.arcstat_evict_l2_ineligible, 0); 7339 wmsum_init(&arc_sums.arcstat_evict_l2_skip, 0); 7340 wmsum_init(&arc_sums.arcstat_hash_collisions, 0); 7341 wmsum_init(&arc_sums.arcstat_hash_chains, 0); 7342 aggsum_init(&arc_sums.arcstat_size, 0); 7343 wmsum_init(&arc_sums.arcstat_compressed_size, 0); 7344 wmsum_init(&arc_sums.arcstat_uncompressed_size, 0); 7345 wmsum_init(&arc_sums.arcstat_overhead_size, 0); 7346 wmsum_init(&arc_sums.arcstat_hdr_size, 0); 7347 wmsum_init(&arc_sums.arcstat_data_size, 0); 7348 wmsum_init(&arc_sums.arcstat_metadata_size, 0); 7349 wmsum_init(&arc_sums.arcstat_dbuf_size, 0); 7350 wmsum_init(&arc_sums.arcstat_dnode_size, 0); 7351 wmsum_init(&arc_sums.arcstat_bonus_size, 0); 7352 wmsum_init(&arc_sums.arcstat_l2_hits, 0); 7353 wmsum_init(&arc_sums.arcstat_l2_misses, 0); 7354 wmsum_init(&arc_sums.arcstat_l2_prefetch_asize, 0); 7355 wmsum_init(&arc_sums.arcstat_l2_mru_asize, 0); 7356 wmsum_init(&arc_sums.arcstat_l2_mfu_asize, 0); 7357 wmsum_init(&arc_sums.arcstat_l2_bufc_data_asize, 0); 7358 wmsum_init(&arc_sums.arcstat_l2_bufc_metadata_asize, 0); 7359 wmsum_init(&arc_sums.arcstat_l2_feeds, 0); 7360 wmsum_init(&arc_sums.arcstat_l2_rw_clash, 0); 7361 wmsum_init(&arc_sums.arcstat_l2_read_bytes, 0); 7362 wmsum_init(&arc_sums.arcstat_l2_write_bytes, 0); 7363 wmsum_init(&arc_sums.arcstat_l2_writes_sent, 0); 7364 wmsum_init(&arc_sums.arcstat_l2_writes_done, 0); 7365 wmsum_init(&arc_sums.arcstat_l2_writes_error, 0); 7366 wmsum_init(&arc_sums.arcstat_l2_writes_lock_retry, 0); 7367 wmsum_init(&arc_sums.arcstat_l2_evict_lock_retry, 0); 7368 wmsum_init(&arc_sums.arcstat_l2_evict_reading, 0); 7369 wmsum_init(&arc_sums.arcstat_l2_evict_l1cached, 0); 7370 wmsum_init(&arc_sums.arcstat_l2_free_on_write, 0); 7371 wmsum_init(&arc_sums.arcstat_l2_abort_lowmem, 0); 7372 wmsum_init(&arc_sums.arcstat_l2_cksum_bad, 0); 7373 wmsum_init(&arc_sums.arcstat_l2_io_error, 0); 7374 wmsum_init(&arc_sums.arcstat_l2_lsize, 0); 7375 wmsum_init(&arc_sums.arcstat_l2_psize, 0); 7376 aggsum_init(&arc_sums.arcstat_l2_hdr_size, 0); 7377 wmsum_init(&arc_sums.arcstat_l2_log_blk_writes, 0); 7378 wmsum_init(&arc_sums.arcstat_l2_log_blk_asize, 0); 7379 wmsum_init(&arc_sums.arcstat_l2_log_blk_count, 0); 7380 wmsum_init(&arc_sums.arcstat_l2_rebuild_success, 0); 7381 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_unsupported, 0); 7382 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_io_errors, 0); 7383 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_dh_errors, 0); 7384 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_cksum_lb_errors, 0); 7385 wmsum_init(&arc_sums.arcstat_l2_rebuild_abort_lowmem, 0); 7386 wmsum_init(&arc_sums.arcstat_l2_rebuild_size, 0); 7387 wmsum_init(&arc_sums.arcstat_l2_rebuild_asize, 0); 7388 wmsum_init(&arc_sums.arcstat_l2_rebuild_bufs, 0); 7389 wmsum_init(&arc_sums.arcstat_l2_rebuild_bufs_precached, 0); 7390 wmsum_init(&arc_sums.arcstat_l2_rebuild_log_blks, 0); 7391 wmsum_init(&arc_sums.arcstat_memory_throttle_count, 0); 7392 wmsum_init(&arc_sums.arcstat_memory_direct_count, 0); 7393 wmsum_init(&arc_sums.arcstat_memory_indirect_count, 0); 7394 wmsum_init(&arc_sums.arcstat_prune, 0); 7395 wmsum_init(&arc_sums.arcstat_meta_used, 0); 7396 wmsum_init(&arc_sums.arcstat_async_upgrade_sync, 0); 7397 wmsum_init(&arc_sums.arcstat_predictive_prefetch, 0); 7398 wmsum_init(&arc_sums.arcstat_demand_hit_predictive_prefetch, 0); 7399 wmsum_init(&arc_sums.arcstat_demand_iohit_predictive_prefetch, 0); 7400 wmsum_init(&arc_sums.arcstat_prescient_prefetch, 0); 7401 wmsum_init(&arc_sums.arcstat_demand_hit_prescient_prefetch, 0); 7402 wmsum_init(&arc_sums.arcstat_demand_iohit_prescient_prefetch, 0); 7403 wmsum_init(&arc_sums.arcstat_raw_size, 0); 7404 wmsum_init(&arc_sums.arcstat_cached_only_in_progress, 0); 7405 wmsum_init(&arc_sums.arcstat_abd_chunk_waste_size, 0); 7406 7407 arc_anon->arcs_state = ARC_STATE_ANON; 7408 arc_mru->arcs_state = ARC_STATE_MRU; 7409 arc_mru_ghost->arcs_state = ARC_STATE_MRU_GHOST; 7410 arc_mfu->arcs_state = ARC_STATE_MFU; 7411 arc_mfu_ghost->arcs_state = ARC_STATE_MFU_GHOST; 7412 arc_l2c_only->arcs_state = ARC_STATE_L2C_ONLY; 7413 arc_uncached->arcs_state = ARC_STATE_UNCACHED; 7414 } 7415 7416 static void 7417 arc_state_fini(void) 7418 { 7419 zfs_refcount_destroy(&arc_anon->arcs_esize[ARC_BUFC_METADATA]); 7420 zfs_refcount_destroy(&arc_anon->arcs_esize[ARC_BUFC_DATA]); 7421 zfs_refcount_destroy(&arc_mru->arcs_esize[ARC_BUFC_METADATA]); 7422 zfs_refcount_destroy(&arc_mru->arcs_esize[ARC_BUFC_DATA]); 7423 zfs_refcount_destroy(&arc_mru_ghost->arcs_esize[ARC_BUFC_METADATA]); 7424 zfs_refcount_destroy(&arc_mru_ghost->arcs_esize[ARC_BUFC_DATA]); 7425 zfs_refcount_destroy(&arc_mfu->arcs_esize[ARC_BUFC_METADATA]); 7426 zfs_refcount_destroy(&arc_mfu->arcs_esize[ARC_BUFC_DATA]); 7427 zfs_refcount_destroy(&arc_mfu_ghost->arcs_esize[ARC_BUFC_METADATA]); 7428 zfs_refcount_destroy(&arc_mfu_ghost->arcs_esize[ARC_BUFC_DATA]); 7429 zfs_refcount_destroy(&arc_l2c_only->arcs_esize[ARC_BUFC_METADATA]); 7430 zfs_refcount_destroy(&arc_l2c_only->arcs_esize[ARC_BUFC_DATA]); 7431 zfs_refcount_destroy(&arc_uncached->arcs_esize[ARC_BUFC_METADATA]); 7432 zfs_refcount_destroy(&arc_uncached->arcs_esize[ARC_BUFC_DATA]); 7433 7434 zfs_refcount_destroy(&arc_anon->arcs_size[ARC_BUFC_DATA]); 7435 zfs_refcount_destroy(&arc_anon->arcs_size[ARC_BUFC_METADATA]); 7436 zfs_refcount_destroy(&arc_mru->arcs_size[ARC_BUFC_DATA]); 7437 zfs_refcount_destroy(&arc_mru->arcs_size[ARC_BUFC_METADATA]); 7438 zfs_refcount_destroy(&arc_mru_ghost->arcs_size[ARC_BUFC_DATA]); 7439 zfs_refcount_destroy(&arc_mru_ghost->arcs_size[ARC_BUFC_METADATA]); 7440 zfs_refcount_destroy(&arc_mfu->arcs_size[ARC_BUFC_DATA]); 7441 zfs_refcount_destroy(&arc_mfu->arcs_size[ARC_BUFC_METADATA]); 7442 zfs_refcount_destroy(&arc_mfu_ghost->arcs_size[ARC_BUFC_DATA]); 7443 zfs_refcount_destroy(&arc_mfu_ghost->arcs_size[ARC_BUFC_METADATA]); 7444 zfs_refcount_destroy(&arc_l2c_only->arcs_size[ARC_BUFC_DATA]); 7445 zfs_refcount_destroy(&arc_l2c_only->arcs_size[ARC_BUFC_METADATA]); 7446 zfs_refcount_destroy(&arc_uncached->arcs_size[ARC_BUFC_DATA]); 7447 zfs_refcount_destroy(&arc_uncached->arcs_size[ARC_BUFC_METADATA]); 7448 7449 multilist_destroy(&arc_mru->arcs_list[ARC_BUFC_METADATA]); 7450 multilist_destroy(&arc_mru_ghost->arcs_list[ARC_BUFC_METADATA]); 7451 multilist_destroy(&arc_mfu->arcs_list[ARC_BUFC_METADATA]); 7452 multilist_destroy(&arc_mfu_ghost->arcs_list[ARC_BUFC_METADATA]); 7453 multilist_destroy(&arc_mru->arcs_list[ARC_BUFC_DATA]); 7454 multilist_destroy(&arc_mru_ghost->arcs_list[ARC_BUFC_DATA]); 7455 multilist_destroy(&arc_mfu->arcs_list[ARC_BUFC_DATA]); 7456 multilist_destroy(&arc_mfu_ghost->arcs_list[ARC_BUFC_DATA]); 7457 multilist_destroy(&arc_l2c_only->arcs_list[ARC_BUFC_METADATA]); 7458 multilist_destroy(&arc_l2c_only->arcs_list[ARC_BUFC_DATA]); 7459 multilist_destroy(&arc_uncached->arcs_list[ARC_BUFC_METADATA]); 7460 multilist_destroy(&arc_uncached->arcs_list[ARC_BUFC_DATA]); 7461 7462 wmsum_fini(&arc_mru_ghost->arcs_hits[ARC_BUFC_DATA]); 7463 wmsum_fini(&arc_mru_ghost->arcs_hits[ARC_BUFC_METADATA]); 7464 wmsum_fini(&arc_mfu_ghost->arcs_hits[ARC_BUFC_DATA]); 7465 wmsum_fini(&arc_mfu_ghost->arcs_hits[ARC_BUFC_METADATA]); 7466 7467 wmsum_fini(&arc_sums.arcstat_hits); 7468 wmsum_fini(&arc_sums.arcstat_iohits); 7469 wmsum_fini(&arc_sums.arcstat_misses); 7470 wmsum_fini(&arc_sums.arcstat_demand_data_hits); 7471 wmsum_fini(&arc_sums.arcstat_demand_data_iohits); 7472 wmsum_fini(&arc_sums.arcstat_demand_data_misses); 7473 wmsum_fini(&arc_sums.arcstat_demand_metadata_hits); 7474 wmsum_fini(&arc_sums.arcstat_demand_metadata_iohits); 7475 wmsum_fini(&arc_sums.arcstat_demand_metadata_misses); 7476 wmsum_fini(&arc_sums.arcstat_prefetch_data_hits); 7477 wmsum_fini(&arc_sums.arcstat_prefetch_data_iohits); 7478 wmsum_fini(&arc_sums.arcstat_prefetch_data_misses); 7479 wmsum_fini(&arc_sums.arcstat_prefetch_metadata_hits); 7480 wmsum_fini(&arc_sums.arcstat_prefetch_metadata_iohits); 7481 wmsum_fini(&arc_sums.arcstat_prefetch_metadata_misses); 7482 wmsum_fini(&arc_sums.arcstat_mru_hits); 7483 wmsum_fini(&arc_sums.arcstat_mru_ghost_hits); 7484 wmsum_fini(&arc_sums.arcstat_mfu_hits); 7485 wmsum_fini(&arc_sums.arcstat_mfu_ghost_hits); 7486 wmsum_fini(&arc_sums.arcstat_uncached_hits); 7487 wmsum_fini(&arc_sums.arcstat_deleted); 7488 wmsum_fini(&arc_sums.arcstat_mutex_miss); 7489 wmsum_fini(&arc_sums.arcstat_access_skip); 7490 wmsum_fini(&arc_sums.arcstat_evict_skip); 7491 wmsum_fini(&arc_sums.arcstat_evict_not_enough); 7492 wmsum_fini(&arc_sums.arcstat_evict_l2_cached); 7493 wmsum_fini(&arc_sums.arcstat_evict_l2_eligible); 7494 wmsum_fini(&arc_sums.arcstat_evict_l2_eligible_mfu); 7495 wmsum_fini(&arc_sums.arcstat_evict_l2_eligible_mru); 7496 wmsum_fini(&arc_sums.arcstat_evict_l2_ineligible); 7497 wmsum_fini(&arc_sums.arcstat_evict_l2_skip); 7498 wmsum_fini(&arc_sums.arcstat_hash_collisions); 7499 wmsum_fini(&arc_sums.arcstat_hash_chains); 7500 aggsum_fini(&arc_sums.arcstat_size); 7501 wmsum_fini(&arc_sums.arcstat_compressed_size); 7502 wmsum_fini(&arc_sums.arcstat_uncompressed_size); 7503 wmsum_fini(&arc_sums.arcstat_overhead_size); 7504 wmsum_fini(&arc_sums.arcstat_hdr_size); 7505 wmsum_fini(&arc_sums.arcstat_data_size); 7506 wmsum_fini(&arc_sums.arcstat_metadata_size); 7507 wmsum_fini(&arc_sums.arcstat_dbuf_size); 7508 wmsum_fini(&arc_sums.arcstat_dnode_size); 7509 wmsum_fini(&arc_sums.arcstat_bonus_size); 7510 wmsum_fini(&arc_sums.arcstat_l2_hits); 7511 wmsum_fini(&arc_sums.arcstat_l2_misses); 7512 wmsum_fini(&arc_sums.arcstat_l2_prefetch_asize); 7513 wmsum_fini(&arc_sums.arcstat_l2_mru_asize); 7514 wmsum_fini(&arc_sums.arcstat_l2_mfu_asize); 7515 wmsum_fini(&arc_sums.arcstat_l2_bufc_data_asize); 7516 wmsum_fini(&arc_sums.arcstat_l2_bufc_metadata_asize); 7517 wmsum_fini(&arc_sums.arcstat_l2_feeds); 7518 wmsum_fini(&arc_sums.arcstat_l2_rw_clash); 7519 wmsum_fini(&arc_sums.arcstat_l2_read_bytes); 7520 wmsum_fini(&arc_sums.arcstat_l2_write_bytes); 7521 wmsum_fini(&arc_sums.arcstat_l2_writes_sent); 7522 wmsum_fini(&arc_sums.arcstat_l2_writes_done); 7523 wmsum_fini(&arc_sums.arcstat_l2_writes_error); 7524 wmsum_fini(&arc_sums.arcstat_l2_writes_lock_retry); 7525 wmsum_fini(&arc_sums.arcstat_l2_evict_lock_retry); 7526 wmsum_fini(&arc_sums.arcstat_l2_evict_reading); 7527 wmsum_fini(&arc_sums.arcstat_l2_evict_l1cached); 7528 wmsum_fini(&arc_sums.arcstat_l2_free_on_write); 7529 wmsum_fini(&arc_sums.arcstat_l2_abort_lowmem); 7530 wmsum_fini(&arc_sums.arcstat_l2_cksum_bad); 7531 wmsum_fini(&arc_sums.arcstat_l2_io_error); 7532 wmsum_fini(&arc_sums.arcstat_l2_lsize); 7533 wmsum_fini(&arc_sums.arcstat_l2_psize); 7534 aggsum_fini(&arc_sums.arcstat_l2_hdr_size); 7535 wmsum_fini(&arc_sums.arcstat_l2_log_blk_writes); 7536 wmsum_fini(&arc_sums.arcstat_l2_log_blk_asize); 7537 wmsum_fini(&arc_sums.arcstat_l2_log_blk_count); 7538 wmsum_fini(&arc_sums.arcstat_l2_rebuild_success); 7539 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_unsupported); 7540 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_io_errors); 7541 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_dh_errors); 7542 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_cksum_lb_errors); 7543 wmsum_fini(&arc_sums.arcstat_l2_rebuild_abort_lowmem); 7544 wmsum_fini(&arc_sums.arcstat_l2_rebuild_size); 7545 wmsum_fini(&arc_sums.arcstat_l2_rebuild_asize); 7546 wmsum_fini(&arc_sums.arcstat_l2_rebuild_bufs); 7547 wmsum_fini(&arc_sums.arcstat_l2_rebuild_bufs_precached); 7548 wmsum_fini(&arc_sums.arcstat_l2_rebuild_log_blks); 7549 wmsum_fini(&arc_sums.arcstat_memory_throttle_count); 7550 wmsum_fini(&arc_sums.arcstat_memory_direct_count); 7551 wmsum_fini(&arc_sums.arcstat_memory_indirect_count); 7552 wmsum_fini(&arc_sums.arcstat_prune); 7553 wmsum_fini(&arc_sums.arcstat_meta_used); 7554 wmsum_fini(&arc_sums.arcstat_async_upgrade_sync); 7555 wmsum_fini(&arc_sums.arcstat_predictive_prefetch); 7556 wmsum_fini(&arc_sums.arcstat_demand_hit_predictive_prefetch); 7557 wmsum_fini(&arc_sums.arcstat_demand_iohit_predictive_prefetch); 7558 wmsum_fini(&arc_sums.arcstat_prescient_prefetch); 7559 wmsum_fini(&arc_sums.arcstat_demand_hit_prescient_prefetch); 7560 wmsum_fini(&arc_sums.arcstat_demand_iohit_prescient_prefetch); 7561 wmsum_fini(&arc_sums.arcstat_raw_size); 7562 wmsum_fini(&arc_sums.arcstat_cached_only_in_progress); 7563 wmsum_fini(&arc_sums.arcstat_abd_chunk_waste_size); 7564 } 7565 7566 uint64_t 7567 arc_target_bytes(void) 7568 { 7569 return (arc_c); 7570 } 7571 7572 void 7573 arc_set_limits(uint64_t allmem) 7574 { 7575 /* Set min cache to 1/32 of all memory, or 32MB, whichever is more. */ 7576 arc_c_min = MAX(allmem / 32, 2ULL << SPA_MAXBLOCKSHIFT); 7577 7578 /* How to set default max varies by platform. */ 7579 arc_c_max = arc_default_max(arc_c_min, allmem); 7580 } 7581 void 7582 arc_init(void) 7583 { 7584 uint64_t percent, allmem = arc_all_memory(); 7585 mutex_init(&arc_evict_lock, NULL, MUTEX_DEFAULT, NULL); 7586 list_create(&arc_evict_waiters, sizeof (arc_evict_waiter_t), 7587 offsetof(arc_evict_waiter_t, aew_node)); 7588 7589 arc_min_prefetch_ms = 1000; 7590 arc_min_prescient_prefetch_ms = 6000; 7591 7592 #if defined(_KERNEL) 7593 arc_lowmem_init(); 7594 #endif 7595 7596 arc_set_limits(allmem); 7597 7598 #ifdef _KERNEL 7599 /* 7600 * If zfs_arc_max is non-zero at init, meaning it was set in the kernel 7601 * environment before the module was loaded, don't block setting the 7602 * maximum because it is less than arc_c_min, instead, reset arc_c_min 7603 * to a lower value. 7604 * zfs_arc_min will be handled by arc_tuning_update(). 7605 */ 7606 if (zfs_arc_max != 0 && zfs_arc_max >= MIN_ARC_MAX && 7607 zfs_arc_max < allmem) { 7608 arc_c_max = zfs_arc_max; 7609 if (arc_c_min >= arc_c_max) { 7610 arc_c_min = MAX(zfs_arc_max / 2, 7611 2ULL << SPA_MAXBLOCKSHIFT); 7612 } 7613 } 7614 #else 7615 /* 7616 * In userland, there's only the memory pressure that we artificially 7617 * create (see arc_available_memory()). Don't let arc_c get too 7618 * small, because it can cause transactions to be larger than 7619 * arc_c, causing arc_tempreserve_space() to fail. 7620 */ 7621 arc_c_min = MAX(arc_c_max / 2, 2ULL << SPA_MAXBLOCKSHIFT); 7622 #endif 7623 7624 arc_c = arc_c_min; 7625 /* 7626 * 32-bit fixed point fractions of metadata from total ARC size, 7627 * MRU data from all data and MRU metadata from all metadata. 7628 */ 7629 arc_meta = (1ULL << 32) / 4; /* Metadata is 25% of arc_c. */ 7630 arc_pd = (1ULL << 32) / 2; /* Data MRU is 50% of data. */ 7631 arc_pm = (1ULL << 32) / 2; /* Metadata MRU is 50% of metadata. */ 7632 7633 percent = MIN(zfs_arc_dnode_limit_percent, 100); 7634 arc_dnode_limit = arc_c_max * percent / 100; 7635 7636 /* Apply user specified tunings */ 7637 arc_tuning_update(B_TRUE); 7638 7639 /* if kmem_flags are set, lets try to use less memory */ 7640 if (kmem_debugging()) 7641 arc_c = arc_c / 2; 7642 if (arc_c < arc_c_min) 7643 arc_c = arc_c_min; 7644 7645 arc_register_hotplug(); 7646 7647 arc_state_init(); 7648 7649 buf_init(); 7650 7651 list_create(&arc_prune_list, sizeof (arc_prune_t), 7652 offsetof(arc_prune_t, p_node)); 7653 mutex_init(&arc_prune_mtx, NULL, MUTEX_DEFAULT, NULL); 7654 7655 arc_prune_taskq = taskq_create("arc_prune", zfs_arc_prune_task_threads, 7656 defclsyspri, 100, INT_MAX, TASKQ_PREPOPULATE | TASKQ_DYNAMIC); 7657 7658 arc_ksp = kstat_create("zfs", 0, "arcstats", "misc", KSTAT_TYPE_NAMED, 7659 sizeof (arc_stats) / sizeof (kstat_named_t), KSTAT_FLAG_VIRTUAL); 7660 7661 if (arc_ksp != NULL) { 7662 arc_ksp->ks_data = &arc_stats; 7663 arc_ksp->ks_update = arc_kstat_update; 7664 kstat_install(arc_ksp); 7665 } 7666 7667 arc_state_evict_markers = 7668 arc_state_alloc_markers(arc_state_evict_marker_count); 7669 arc_evict_zthr = zthr_create_timer("arc_evict", 7670 arc_evict_cb_check, arc_evict_cb, NULL, SEC2NSEC(1), defclsyspri); 7671 arc_reap_zthr = zthr_create_timer("arc_reap", 7672 arc_reap_cb_check, arc_reap_cb, NULL, SEC2NSEC(1), minclsyspri); 7673 7674 arc_warm = B_FALSE; 7675 7676 /* 7677 * Calculate maximum amount of dirty data per pool. 7678 * 7679 * If it has been set by a module parameter, take that. 7680 * Otherwise, use a percentage of physical memory defined by 7681 * zfs_dirty_data_max_percent (default 10%) with a cap at 7682 * zfs_dirty_data_max_max (default 4G or 25% of physical memory). 7683 */ 7684 #ifdef __LP64__ 7685 if (zfs_dirty_data_max_max == 0) 7686 zfs_dirty_data_max_max = MIN(4ULL * 1024 * 1024 * 1024, 7687 allmem * zfs_dirty_data_max_max_percent / 100); 7688 #else 7689 if (zfs_dirty_data_max_max == 0) 7690 zfs_dirty_data_max_max = MIN(1ULL * 1024 * 1024 * 1024, 7691 allmem * zfs_dirty_data_max_max_percent / 100); 7692 #endif 7693 7694 if (zfs_dirty_data_max == 0) { 7695 zfs_dirty_data_max = allmem * 7696 zfs_dirty_data_max_percent / 100; 7697 zfs_dirty_data_max = MIN(zfs_dirty_data_max, 7698 zfs_dirty_data_max_max); 7699 } 7700 7701 if (zfs_wrlog_data_max == 0) { 7702 7703 /* 7704 * dp_wrlog_total is reduced for each txg at the end of 7705 * spa_sync(). However, dp_dirty_total is reduced every time 7706 * a block is written out. Thus under normal operation, 7707 * dp_wrlog_total could grow 2 times as big as 7708 * zfs_dirty_data_max. 7709 */ 7710 zfs_wrlog_data_max = zfs_dirty_data_max * 2; 7711 } 7712 } 7713 7714 void 7715 arc_fini(void) 7716 { 7717 arc_prune_t *p; 7718 7719 #ifdef _KERNEL 7720 arc_lowmem_fini(); 7721 #endif /* _KERNEL */ 7722 7723 /* Use B_TRUE to ensure *all* buffers are evicted */ 7724 arc_flush(NULL, B_TRUE); 7725 7726 if (arc_ksp != NULL) { 7727 kstat_delete(arc_ksp); 7728 arc_ksp = NULL; 7729 } 7730 7731 taskq_wait(arc_prune_taskq); 7732 taskq_destroy(arc_prune_taskq); 7733 7734 mutex_enter(&arc_prune_mtx); 7735 while ((p = list_remove_head(&arc_prune_list)) != NULL) { 7736 (void) zfs_refcount_remove(&p->p_refcnt, &arc_prune_list); 7737 zfs_refcount_destroy(&p->p_refcnt); 7738 kmem_free(p, sizeof (*p)); 7739 } 7740 mutex_exit(&arc_prune_mtx); 7741 7742 list_destroy(&arc_prune_list); 7743 mutex_destroy(&arc_prune_mtx); 7744 7745 (void) zthr_cancel(arc_evict_zthr); 7746 (void) zthr_cancel(arc_reap_zthr); 7747 arc_state_free_markers(arc_state_evict_markers, 7748 arc_state_evict_marker_count); 7749 7750 mutex_destroy(&arc_evict_lock); 7751 list_destroy(&arc_evict_waiters); 7752 7753 /* 7754 * Free any buffers that were tagged for destruction. This needs 7755 * to occur before arc_state_fini() runs and destroys the aggsum 7756 * values which are updated when freeing scatter ABDs. 7757 */ 7758 l2arc_do_free_on_write(); 7759 7760 /* 7761 * buf_fini() must proceed arc_state_fini() because buf_fin() may 7762 * trigger the release of kmem magazines, which can callback to 7763 * arc_space_return() which accesses aggsums freed in act_state_fini(). 7764 */ 7765 buf_fini(); 7766 arc_state_fini(); 7767 7768 arc_unregister_hotplug(); 7769 7770 /* 7771 * We destroy the zthrs after all the ARC state has been 7772 * torn down to avoid the case of them receiving any 7773 * wakeup() signals after they are destroyed. 7774 */ 7775 zthr_destroy(arc_evict_zthr); 7776 zthr_destroy(arc_reap_zthr); 7777 7778 ASSERT0(arc_loaned_bytes); 7779 } 7780 7781 /* 7782 * Level 2 ARC 7783 * 7784 * The level 2 ARC (L2ARC) is a cache layer in-between main memory and disk. 7785 * It uses dedicated storage devices to hold cached data, which are populated 7786 * using large infrequent writes. The main role of this cache is to boost 7787 * the performance of random read workloads. The intended L2ARC devices 7788 * include short-stroked disks, solid state disks, and other media with 7789 * substantially faster read latency than disk. 7790 * 7791 * +-----------------------+ 7792 * | ARC | 7793 * +-----------------------+ 7794 * | ^ ^ 7795 * | | | 7796 * l2arc_feed_thread() arc_read() 7797 * | | | 7798 * | l2arc read | 7799 * V | | 7800 * +---------------+ | 7801 * | L2ARC | | 7802 * +---------------+ | 7803 * | ^ | 7804 * l2arc_write() | | 7805 * | | | 7806 * V | | 7807 * +-------+ +-------+ 7808 * | vdev | | vdev | 7809 * | cache | | cache | 7810 * +-------+ +-------+ 7811 * +=========+ .-----. 7812 * : L2ARC : |-_____-| 7813 * : devices : | Disks | 7814 * +=========+ `-_____-' 7815 * 7816 * Read requests are satisfied from the following sources, in order: 7817 * 7818 * 1) ARC 7819 * 2) vdev cache of L2ARC devices 7820 * 3) L2ARC devices 7821 * 4) vdev cache of disks 7822 * 5) disks 7823 * 7824 * Some L2ARC device types exhibit extremely slow write performance. 7825 * To accommodate for this there are some significant differences between 7826 * the L2ARC and traditional cache design: 7827 * 7828 * 1. There is no eviction path from the ARC to the L2ARC. Evictions from 7829 * the ARC behave as usual, freeing buffers and placing headers on ghost 7830 * lists. The ARC does not send buffers to the L2ARC during eviction as 7831 * this would add inflated write latencies for all ARC memory pressure. 7832 * 7833 * 2. The L2ARC attempts to cache data from the ARC before it is evicted. 7834 * It does this by periodically scanning buffers from the eviction-end of 7835 * the MFU and MRU ARC lists, copying them to the L2ARC devices if they are 7836 * not already there. It scans until a headroom of buffers is satisfied, 7837 * which itself is a buffer for ARC eviction. If a compressible buffer is 7838 * found during scanning and selected for writing to an L2ARC device, we 7839 * temporarily boost scanning headroom during the next scan cycle to make 7840 * sure we adapt to compression effects (which might significantly reduce 7841 * the data volume we write to L2ARC). The thread that does this is 7842 * l2arc_feed_thread(), illustrated below; example sizes are included to 7843 * provide a better sense of ratio than this diagram: 7844 * 7845 * head --> tail 7846 * +---------------------+----------+ 7847 * ARC_mfu |:::::#:::::::::::::::|o#o###o###|-->. # already on L2ARC 7848 * +---------------------+----------+ | o L2ARC eligible 7849 * ARC_mru |:#:::::::::::::::::::|#o#ooo####|-->| : ARC buffer 7850 * +---------------------+----------+ | 7851 * 15.9 Gbytes ^ 32 Mbytes | 7852 * headroom | 7853 * l2arc_feed_thread() 7854 * | 7855 * l2arc write hand <--[oooo]--' 7856 * | 8 Mbyte 7857 * | write max 7858 * V 7859 * +==============================+ 7860 * L2ARC dev |####|#|###|###| |####| ... | 7861 * +==============================+ 7862 * 32 Gbytes 7863 * 7864 * 3. If an ARC buffer is copied to the L2ARC but then hit instead of 7865 * evicted, then the L2ARC has cached a buffer much sooner than it probably 7866 * needed to, potentially wasting L2ARC device bandwidth and storage. It is 7867 * safe to say that this is an uncommon case, since buffers at the end of 7868 * the ARC lists have moved there due to inactivity. 7869 * 7870 * 4. If the ARC evicts faster than the L2ARC can maintain a headroom, 7871 * then the L2ARC simply misses copying some buffers. This serves as a 7872 * pressure valve to prevent heavy read workloads from both stalling the ARC 7873 * with waits and clogging the L2ARC with writes. This also helps prevent 7874 * the potential for the L2ARC to churn if it attempts to cache content too 7875 * quickly, such as during backups of the entire pool. 7876 * 7877 * 5. After system boot and before the ARC has filled main memory, there are 7878 * no evictions from the ARC and so the tails of the ARC_mfu and ARC_mru 7879 * lists can remain mostly static. Instead of searching from tail of these 7880 * lists as pictured, the l2arc_feed_thread() will search from the list heads 7881 * for eligible buffers, greatly increasing its chance of finding them. 7882 * 7883 * The L2ARC device write speed is also boosted during this time so that 7884 * the L2ARC warms up faster. Since there have been no ARC evictions yet, 7885 * there are no L2ARC reads, and no fear of degrading read performance 7886 * through increased writes. 7887 * 7888 * 6. Writes to the L2ARC devices are grouped and sent in-sequence, so that 7889 * the vdev queue can aggregate them into larger and fewer writes. Each 7890 * device is written to in a rotor fashion, sweeping writes through 7891 * available space then repeating. 7892 * 7893 * 7. The L2ARC does not store dirty content. It never needs to flush 7894 * write buffers back to disk based storage. 7895 * 7896 * 8. If an ARC buffer is written (and dirtied) which also exists in the 7897 * L2ARC, the now stale L2ARC buffer is immediately dropped. 7898 * 7899 * The performance of the L2ARC can be tweaked by a number of tunables, which 7900 * may be necessary for different workloads: 7901 * 7902 * l2arc_write_max max write bytes per interval 7903 * l2arc_write_boost extra write bytes during device warmup 7904 * l2arc_noprefetch skip caching prefetched buffers 7905 * l2arc_headroom number of max device writes to precache 7906 * l2arc_headroom_boost when we find compressed buffers during ARC 7907 * scanning, we multiply headroom by this 7908 * percentage factor for the next scan cycle, 7909 * since more compressed buffers are likely to 7910 * be present 7911 * l2arc_feed_secs seconds between L2ARC writing 7912 * 7913 * Tunables may be removed or added as future performance improvements are 7914 * integrated, and also may become zpool properties. 7915 * 7916 * There are three key functions that control how the L2ARC warms up: 7917 * 7918 * l2arc_write_eligible() check if a buffer is eligible to cache 7919 * l2arc_write_size() calculate how much to write 7920 * l2arc_write_interval() calculate sleep delay between writes 7921 * 7922 * These three functions determine what to write, how much, and how quickly 7923 * to send writes. 7924 * 7925 * L2ARC persistence: 7926 * 7927 * When writing buffers to L2ARC, we periodically add some metadata to 7928 * make sure we can pick them up after reboot, thus dramatically reducing 7929 * the impact that any downtime has on the performance of storage systems 7930 * with large caches. 7931 * 7932 * The implementation works fairly simply by integrating the following two 7933 * modifications: 7934 * 7935 * *) When writing to the L2ARC, we occasionally write a "l2arc log block", 7936 * which is an additional piece of metadata which describes what's been 7937 * written. This allows us to rebuild the arc_buf_hdr_t structures of the 7938 * main ARC buffers. There are 2 linked-lists of log blocks headed by 7939 * dh_start_lbps[2]. We alternate which chain we append to, so they are 7940 * time-wise and offset-wise interleaved, but that is an optimization rather 7941 * than for correctness. The log block also includes a pointer to the 7942 * previous block in its chain. 7943 * 7944 * *) We reserve SPA_MINBLOCKSIZE of space at the start of each L2ARC device 7945 * for our header bookkeeping purposes. This contains a device header, 7946 * which contains our top-level reference structures. We update it each 7947 * time we write a new log block, so that we're able to locate it in the 7948 * L2ARC device. If this write results in an inconsistent device header 7949 * (e.g. due to power failure), we detect this by verifying the header's 7950 * checksum and simply fail to reconstruct the L2ARC after reboot. 7951 * 7952 * Implementation diagram: 7953 * 7954 * +=== L2ARC device (not to scale) ======================================+ 7955 * | ___two newest log block pointers__.__________ | 7956 * | / \dh_start_lbps[1] | 7957 * | / \ \dh_start_lbps[0]| 7958 * |.___/__. V V | 7959 * ||L2 dev|....|lb |bufs |lb |bufs |lb |bufs |lb |bufs |lb |---(empty)---| 7960 * || hdr| ^ /^ /^ / / | 7961 * |+------+ ...--\-------/ \-----/--\------/ / | 7962 * | \--------------/ \--------------/ | 7963 * +======================================================================+ 7964 * 7965 * As can be seen on the diagram, rather than using a simple linked list, 7966 * we use a pair of linked lists with alternating elements. This is a 7967 * performance enhancement due to the fact that we only find out the 7968 * address of the next log block access once the current block has been 7969 * completely read in. Obviously, this hurts performance, because we'd be 7970 * keeping the device's I/O queue at only a 1 operation deep, thus 7971 * incurring a large amount of I/O round-trip latency. Having two lists 7972 * allows us to fetch two log blocks ahead of where we are currently 7973 * rebuilding L2ARC buffers. 7974 * 7975 * On-device data structures: 7976 * 7977 * L2ARC device header: l2arc_dev_hdr_phys_t 7978 * L2ARC log block: l2arc_log_blk_phys_t 7979 * 7980 * L2ARC reconstruction: 7981 * 7982 * When writing data, we simply write in the standard rotary fashion, 7983 * evicting buffers as we go and simply writing new data over them (writing 7984 * a new log block every now and then). This obviously means that once we 7985 * loop around the end of the device, we will start cutting into an already 7986 * committed log block (and its referenced data buffers), like so: 7987 * 7988 * current write head__ __old tail 7989 * \ / 7990 * V V 7991 * <--|bufs |lb |bufs |lb | |bufs |lb |bufs |lb |--> 7992 * ^ ^^^^^^^^^___________________________________ 7993 * | \ 7994 * <<nextwrite>> may overwrite this blk and/or its bufs --' 7995 * 7996 * When importing the pool, we detect this situation and use it to stop 7997 * our scanning process (see l2arc_rebuild). 7998 * 7999 * There is one significant caveat to consider when rebuilding ARC contents 8000 * from an L2ARC device: what about invalidated buffers? Given the above 8001 * construction, we cannot update blocks which we've already written to amend 8002 * them to remove buffers which were invalidated. Thus, during reconstruction, 8003 * we might be populating the cache with buffers for data that's not on the 8004 * main pool anymore, or may have been overwritten! 8005 * 8006 * As it turns out, this isn't a problem. Every arc_read request includes 8007 * both the DVA and, crucially, the birth TXG of the BP the caller is 8008 * looking for. So even if the cache were populated by completely rotten 8009 * blocks for data that had been long deleted and/or overwritten, we'll 8010 * never actually return bad data from the cache, since the DVA with the 8011 * birth TXG uniquely identify a block in space and time - once created, 8012 * a block is immutable on disk. The worst thing we have done is wasted 8013 * some time and memory at l2arc rebuild to reconstruct outdated ARC 8014 * entries that will get dropped from the l2arc as it is being updated 8015 * with new blocks. 8016 * 8017 * L2ARC buffers that have been evicted by l2arc_evict() ahead of the write 8018 * hand are not restored. This is done by saving the offset (in bytes) 8019 * l2arc_evict() has evicted to in the L2ARC device header and taking it 8020 * into account when restoring buffers. 8021 */ 8022 8023 static boolean_t 8024 l2arc_write_eligible(uint64_t spa_guid, arc_buf_hdr_t *hdr) 8025 { 8026 /* 8027 * A buffer is *not* eligible for the L2ARC if it: 8028 * 1. belongs to a different spa. 8029 * 2. is already cached on the L2ARC. 8030 * 3. has an I/O in progress (it may be an incomplete read). 8031 * 4. is flagged not eligible (zfs property). 8032 */ 8033 if (hdr->b_spa != spa_guid || HDR_HAS_L2HDR(hdr) || 8034 HDR_IO_IN_PROGRESS(hdr) || !HDR_L2CACHE(hdr)) 8035 return (B_FALSE); 8036 8037 return (B_TRUE); 8038 } 8039 8040 static uint64_t 8041 l2arc_write_size(l2arc_dev_t *dev) 8042 { 8043 uint64_t size; 8044 8045 /* 8046 * Make sure our globals have meaningful values in case the user 8047 * altered them. 8048 */ 8049 size = l2arc_write_max; 8050 if (size == 0) { 8051 cmn_err(CE_NOTE, "l2arc_write_max must be greater than zero, " 8052 "resetting it to the default (%d)", L2ARC_WRITE_SIZE); 8053 size = l2arc_write_max = L2ARC_WRITE_SIZE; 8054 } 8055 8056 if (arc_warm == B_FALSE) 8057 size += l2arc_write_boost; 8058 8059 /* We need to add in the worst case scenario of log block overhead. */ 8060 size += l2arc_log_blk_overhead(size, dev); 8061 if (dev->l2ad_vdev->vdev_has_trim && l2arc_trim_ahead > 0) { 8062 /* 8063 * Trim ahead of the write size 64MB or (l2arc_trim_ahead/100) 8064 * times the writesize, whichever is greater. 8065 */ 8066 size += MAX(64 * 1024 * 1024, 8067 (size * l2arc_trim_ahead) / 100); 8068 } 8069 8070 /* 8071 * Make sure the write size does not exceed the size of the cache 8072 * device. This is important in l2arc_evict(), otherwise infinite 8073 * iteration can occur. 8074 */ 8075 size = MIN(size, (dev->l2ad_end - dev->l2ad_start) / 4); 8076 8077 size = P2ROUNDUP(size, 1ULL << dev->l2ad_vdev->vdev_ashift); 8078 8079 return (size); 8080 8081 } 8082 8083 static clock_t 8084 l2arc_write_interval(clock_t began, uint64_t wanted, uint64_t wrote) 8085 { 8086 clock_t interval, next, now; 8087 8088 /* 8089 * If the ARC lists are busy, increase our write rate; if the 8090 * lists are stale, idle back. This is achieved by checking 8091 * how much we previously wrote - if it was more than half of 8092 * what we wanted, schedule the next write much sooner. 8093 */ 8094 if (l2arc_feed_again && wrote > (wanted / 2)) 8095 interval = (hz * l2arc_feed_min_ms) / 1000; 8096 else 8097 interval = hz * l2arc_feed_secs; 8098 8099 now = ddi_get_lbolt(); 8100 next = MAX(now, MIN(now + interval, began + interval)); 8101 8102 return (next); 8103 } 8104 8105 /* 8106 * Cycle through L2ARC devices. This is how L2ARC load balances. 8107 * If a device is returned, this also returns holding the spa config lock. 8108 */ 8109 static l2arc_dev_t * 8110 l2arc_dev_get_next(void) 8111 { 8112 l2arc_dev_t *first, *next = NULL; 8113 8114 /* 8115 * Lock out the removal of spas (spa_namespace_lock), then removal 8116 * of cache devices (l2arc_dev_mtx). Once a device has been selected, 8117 * both locks will be dropped and a spa config lock held instead. 8118 */ 8119 mutex_enter(&spa_namespace_lock); 8120 mutex_enter(&l2arc_dev_mtx); 8121 8122 /* if there are no vdevs, there is nothing to do */ 8123 if (l2arc_ndev == 0) 8124 goto out; 8125 8126 first = NULL; 8127 next = l2arc_dev_last; 8128 do { 8129 /* loop around the list looking for a non-faulted vdev */ 8130 if (next == NULL) { 8131 next = list_head(l2arc_dev_list); 8132 } else { 8133 next = list_next(l2arc_dev_list, next); 8134 if (next == NULL) 8135 next = list_head(l2arc_dev_list); 8136 } 8137 8138 /* if we have come back to the start, bail out */ 8139 if (first == NULL) 8140 first = next; 8141 else if (next == first) 8142 break; 8143 8144 ASSERT3P(next, !=, NULL); 8145 } while (vdev_is_dead(next->l2ad_vdev) || next->l2ad_rebuild || 8146 next->l2ad_trim_all); 8147 8148 /* if we were unable to find any usable vdevs, return NULL */ 8149 if (vdev_is_dead(next->l2ad_vdev) || next->l2ad_rebuild || 8150 next->l2ad_trim_all) 8151 next = NULL; 8152 8153 l2arc_dev_last = next; 8154 8155 out: 8156 mutex_exit(&l2arc_dev_mtx); 8157 8158 /* 8159 * Grab the config lock to prevent the 'next' device from being 8160 * removed while we are writing to it. 8161 */ 8162 if (next != NULL) 8163 spa_config_enter(next->l2ad_spa, SCL_L2ARC, next, RW_READER); 8164 mutex_exit(&spa_namespace_lock); 8165 8166 return (next); 8167 } 8168 8169 /* 8170 * Free buffers that were tagged for destruction. 8171 */ 8172 static void 8173 l2arc_do_free_on_write(void) 8174 { 8175 l2arc_data_free_t *df; 8176 8177 mutex_enter(&l2arc_free_on_write_mtx); 8178 while ((df = list_remove_head(l2arc_free_on_write)) != NULL) { 8179 ASSERT3P(df->l2df_abd, !=, NULL); 8180 abd_free(df->l2df_abd); 8181 kmem_free(df, sizeof (l2arc_data_free_t)); 8182 } 8183 mutex_exit(&l2arc_free_on_write_mtx); 8184 } 8185 8186 /* 8187 * A write to a cache device has completed. Update all headers to allow 8188 * reads from these buffers to begin. 8189 */ 8190 static void 8191 l2arc_write_done(zio_t *zio) 8192 { 8193 l2arc_write_callback_t *cb; 8194 l2arc_lb_abd_buf_t *abd_buf; 8195 l2arc_lb_ptr_buf_t *lb_ptr_buf; 8196 l2arc_dev_t *dev; 8197 l2arc_dev_hdr_phys_t *l2dhdr; 8198 list_t *buflist; 8199 arc_buf_hdr_t *head, *hdr, *hdr_prev; 8200 kmutex_t *hash_lock; 8201 int64_t bytes_dropped = 0; 8202 8203 cb = zio->io_private; 8204 ASSERT3P(cb, !=, NULL); 8205 dev = cb->l2wcb_dev; 8206 l2dhdr = dev->l2ad_dev_hdr; 8207 ASSERT3P(dev, !=, NULL); 8208 head = cb->l2wcb_head; 8209 ASSERT3P(head, !=, NULL); 8210 buflist = &dev->l2ad_buflist; 8211 ASSERT3P(buflist, !=, NULL); 8212 DTRACE_PROBE2(l2arc__iodone, zio_t *, zio, 8213 l2arc_write_callback_t *, cb); 8214 8215 /* 8216 * All writes completed, or an error was hit. 8217 */ 8218 top: 8219 mutex_enter(&dev->l2ad_mtx); 8220 for (hdr = list_prev(buflist, head); hdr; hdr = hdr_prev) { 8221 hdr_prev = list_prev(buflist, hdr); 8222 8223 hash_lock = HDR_LOCK(hdr); 8224 8225 /* 8226 * We cannot use mutex_enter or else we can deadlock 8227 * with l2arc_write_buffers (due to swapping the order 8228 * the hash lock and l2ad_mtx are taken). 8229 */ 8230 if (!mutex_tryenter(hash_lock)) { 8231 /* 8232 * Missed the hash lock. We must retry so we 8233 * don't leave the ARC_FLAG_L2_WRITING bit set. 8234 */ 8235 ARCSTAT_BUMP(arcstat_l2_writes_lock_retry); 8236 8237 /* 8238 * We don't want to rescan the headers we've 8239 * already marked as having been written out, so 8240 * we reinsert the head node so we can pick up 8241 * where we left off. 8242 */ 8243 list_remove(buflist, head); 8244 list_insert_after(buflist, hdr, head); 8245 8246 mutex_exit(&dev->l2ad_mtx); 8247 8248 /* 8249 * We wait for the hash lock to become available 8250 * to try and prevent busy waiting, and increase 8251 * the chance we'll be able to acquire the lock 8252 * the next time around. 8253 */ 8254 mutex_enter(hash_lock); 8255 mutex_exit(hash_lock); 8256 goto top; 8257 } 8258 8259 /* 8260 * We could not have been moved into the arc_l2c_only 8261 * state while in-flight due to our ARC_FLAG_L2_WRITING 8262 * bit being set. Let's just ensure that's being enforced. 8263 */ 8264 ASSERT(HDR_HAS_L1HDR(hdr)); 8265 8266 /* 8267 * Skipped - drop L2ARC entry and mark the header as no 8268 * longer L2 eligibile. 8269 */ 8270 if (zio->io_error != 0) { 8271 /* 8272 * Error - drop L2ARC entry. 8273 */ 8274 list_remove(buflist, hdr); 8275 arc_hdr_clear_flags(hdr, ARC_FLAG_HAS_L2HDR); 8276 8277 uint64_t psize = HDR_GET_PSIZE(hdr); 8278 l2arc_hdr_arcstats_decrement(hdr); 8279 8280 bytes_dropped += 8281 vdev_psize_to_asize(dev->l2ad_vdev, psize); 8282 (void) zfs_refcount_remove_many(&dev->l2ad_alloc, 8283 arc_hdr_size(hdr), hdr); 8284 } 8285 8286 /* 8287 * Allow ARC to begin reads and ghost list evictions to 8288 * this L2ARC entry. 8289 */ 8290 arc_hdr_clear_flags(hdr, ARC_FLAG_L2_WRITING); 8291 8292 mutex_exit(hash_lock); 8293 } 8294 8295 /* 8296 * Free the allocated abd buffers for writing the log blocks. 8297 * If the zio failed reclaim the allocated space and remove the 8298 * pointers to these log blocks from the log block pointer list 8299 * of the L2ARC device. 8300 */ 8301 while ((abd_buf = list_remove_tail(&cb->l2wcb_abd_list)) != NULL) { 8302 abd_free(abd_buf->abd); 8303 zio_buf_free(abd_buf, sizeof (*abd_buf)); 8304 if (zio->io_error != 0) { 8305 lb_ptr_buf = list_remove_head(&dev->l2ad_lbptr_list); 8306 /* 8307 * L2BLK_GET_PSIZE returns aligned size for log 8308 * blocks. 8309 */ 8310 uint64_t asize = 8311 L2BLK_GET_PSIZE((lb_ptr_buf->lb_ptr)->lbp_prop); 8312 bytes_dropped += asize; 8313 ARCSTAT_INCR(arcstat_l2_log_blk_asize, -asize); 8314 ARCSTAT_BUMPDOWN(arcstat_l2_log_blk_count); 8315 zfs_refcount_remove_many(&dev->l2ad_lb_asize, asize, 8316 lb_ptr_buf); 8317 (void) zfs_refcount_remove(&dev->l2ad_lb_count, 8318 lb_ptr_buf); 8319 kmem_free(lb_ptr_buf->lb_ptr, 8320 sizeof (l2arc_log_blkptr_t)); 8321 kmem_free(lb_ptr_buf, sizeof (l2arc_lb_ptr_buf_t)); 8322 } 8323 } 8324 list_destroy(&cb->l2wcb_abd_list); 8325 8326 if (zio->io_error != 0) { 8327 ARCSTAT_BUMP(arcstat_l2_writes_error); 8328 8329 /* 8330 * Restore the lbps array in the header to its previous state. 8331 * If the list of log block pointers is empty, zero out the 8332 * log block pointers in the device header. 8333 */ 8334 lb_ptr_buf = list_head(&dev->l2ad_lbptr_list); 8335 for (int i = 0; i < 2; i++) { 8336 if (lb_ptr_buf == NULL) { 8337 /* 8338 * If the list is empty zero out the device 8339 * header. Otherwise zero out the second log 8340 * block pointer in the header. 8341 */ 8342 if (i == 0) { 8343 memset(l2dhdr, 0, 8344 dev->l2ad_dev_hdr_asize); 8345 } else { 8346 memset(&l2dhdr->dh_start_lbps[i], 0, 8347 sizeof (l2arc_log_blkptr_t)); 8348 } 8349 break; 8350 } 8351 memcpy(&l2dhdr->dh_start_lbps[i], lb_ptr_buf->lb_ptr, 8352 sizeof (l2arc_log_blkptr_t)); 8353 lb_ptr_buf = list_next(&dev->l2ad_lbptr_list, 8354 lb_ptr_buf); 8355 } 8356 } 8357 8358 ARCSTAT_BUMP(arcstat_l2_writes_done); 8359 list_remove(buflist, head); 8360 ASSERT(!HDR_HAS_L1HDR(head)); 8361 kmem_cache_free(hdr_l2only_cache, head); 8362 mutex_exit(&dev->l2ad_mtx); 8363 8364 ASSERT(dev->l2ad_vdev != NULL); 8365 vdev_space_update(dev->l2ad_vdev, -bytes_dropped, 0, 0); 8366 8367 l2arc_do_free_on_write(); 8368 8369 kmem_free(cb, sizeof (l2arc_write_callback_t)); 8370 } 8371 8372 static int 8373 l2arc_untransform(zio_t *zio, l2arc_read_callback_t *cb) 8374 { 8375 int ret; 8376 spa_t *spa = zio->io_spa; 8377 arc_buf_hdr_t *hdr = cb->l2rcb_hdr; 8378 blkptr_t *bp = zio->io_bp; 8379 uint8_t salt[ZIO_DATA_SALT_LEN]; 8380 uint8_t iv[ZIO_DATA_IV_LEN]; 8381 uint8_t mac[ZIO_DATA_MAC_LEN]; 8382 boolean_t no_crypt = B_FALSE; 8383 8384 /* 8385 * ZIL data is never be written to the L2ARC, so we don't need 8386 * special handling for its unique MAC storage. 8387 */ 8388 ASSERT3U(BP_GET_TYPE(bp), !=, DMU_OT_INTENT_LOG); 8389 ASSERT(MUTEX_HELD(HDR_LOCK(hdr))); 8390 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 8391 8392 /* 8393 * If the data was encrypted, decrypt it now. Note that 8394 * we must check the bp here and not the hdr, since the 8395 * hdr does not have its encryption parameters updated 8396 * until arc_read_done(). 8397 */ 8398 if (BP_IS_ENCRYPTED(bp)) { 8399 abd_t *eabd = arc_get_data_abd(hdr, arc_hdr_size(hdr), hdr, 8400 ARC_HDR_USE_RESERVE); 8401 8402 zio_crypt_decode_params_bp(bp, salt, iv); 8403 zio_crypt_decode_mac_bp(bp, mac); 8404 8405 ret = spa_do_crypt_abd(B_FALSE, spa, &cb->l2rcb_zb, 8406 BP_GET_TYPE(bp), BP_GET_DEDUP(bp), BP_SHOULD_BYTESWAP(bp), 8407 salt, iv, mac, HDR_GET_PSIZE(hdr), eabd, 8408 hdr->b_l1hdr.b_pabd, &no_crypt); 8409 if (ret != 0) { 8410 arc_free_data_abd(hdr, eabd, arc_hdr_size(hdr), hdr); 8411 goto error; 8412 } 8413 8414 /* 8415 * If we actually performed decryption, replace b_pabd 8416 * with the decrypted data. Otherwise we can just throw 8417 * our decryption buffer away. 8418 */ 8419 if (!no_crypt) { 8420 arc_free_data_abd(hdr, hdr->b_l1hdr.b_pabd, 8421 arc_hdr_size(hdr), hdr); 8422 hdr->b_l1hdr.b_pabd = eabd; 8423 zio->io_abd = eabd; 8424 } else { 8425 arc_free_data_abd(hdr, eabd, arc_hdr_size(hdr), hdr); 8426 } 8427 } 8428 8429 /* 8430 * If the L2ARC block was compressed, but ARC compression 8431 * is disabled we decompress the data into a new buffer and 8432 * replace the existing data. 8433 */ 8434 if (HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 8435 !HDR_COMPRESSION_ENABLED(hdr)) { 8436 abd_t *cabd = arc_get_data_abd(hdr, arc_hdr_size(hdr), hdr, 8437 ARC_HDR_USE_RESERVE); 8438 void *tmp = abd_borrow_buf(cabd, arc_hdr_size(hdr)); 8439 8440 ret = zio_decompress_data(HDR_GET_COMPRESS(hdr), 8441 hdr->b_l1hdr.b_pabd, tmp, HDR_GET_PSIZE(hdr), 8442 HDR_GET_LSIZE(hdr), &hdr->b_complevel); 8443 if (ret != 0) { 8444 abd_return_buf_copy(cabd, tmp, arc_hdr_size(hdr)); 8445 arc_free_data_abd(hdr, cabd, arc_hdr_size(hdr), hdr); 8446 goto error; 8447 } 8448 8449 abd_return_buf_copy(cabd, tmp, arc_hdr_size(hdr)); 8450 arc_free_data_abd(hdr, hdr->b_l1hdr.b_pabd, 8451 arc_hdr_size(hdr), hdr); 8452 hdr->b_l1hdr.b_pabd = cabd; 8453 zio->io_abd = cabd; 8454 zio->io_size = HDR_GET_LSIZE(hdr); 8455 } 8456 8457 return (0); 8458 8459 error: 8460 return (ret); 8461 } 8462 8463 8464 /* 8465 * A read to a cache device completed. Validate buffer contents before 8466 * handing over to the regular ARC routines. 8467 */ 8468 static void 8469 l2arc_read_done(zio_t *zio) 8470 { 8471 int tfm_error = 0; 8472 l2arc_read_callback_t *cb = zio->io_private; 8473 arc_buf_hdr_t *hdr; 8474 kmutex_t *hash_lock; 8475 boolean_t valid_cksum; 8476 boolean_t using_rdata = (BP_IS_ENCRYPTED(&cb->l2rcb_bp) && 8477 (cb->l2rcb_flags & ZIO_FLAG_RAW_ENCRYPT)); 8478 8479 ASSERT3P(zio->io_vd, !=, NULL); 8480 ASSERT(zio->io_flags & ZIO_FLAG_DONT_PROPAGATE); 8481 8482 spa_config_exit(zio->io_spa, SCL_L2ARC, zio->io_vd); 8483 8484 ASSERT3P(cb, !=, NULL); 8485 hdr = cb->l2rcb_hdr; 8486 ASSERT3P(hdr, !=, NULL); 8487 8488 hash_lock = HDR_LOCK(hdr); 8489 mutex_enter(hash_lock); 8490 ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); 8491 8492 /* 8493 * If the data was read into a temporary buffer, 8494 * move it and free the buffer. 8495 */ 8496 if (cb->l2rcb_abd != NULL) { 8497 ASSERT3U(arc_hdr_size(hdr), <, zio->io_size); 8498 if (zio->io_error == 0) { 8499 if (using_rdata) { 8500 abd_copy(hdr->b_crypt_hdr.b_rabd, 8501 cb->l2rcb_abd, arc_hdr_size(hdr)); 8502 } else { 8503 abd_copy(hdr->b_l1hdr.b_pabd, 8504 cb->l2rcb_abd, arc_hdr_size(hdr)); 8505 } 8506 } 8507 8508 /* 8509 * The following must be done regardless of whether 8510 * there was an error: 8511 * - free the temporary buffer 8512 * - point zio to the real ARC buffer 8513 * - set zio size accordingly 8514 * These are required because zio is either re-used for 8515 * an I/O of the block in the case of the error 8516 * or the zio is passed to arc_read_done() and it 8517 * needs real data. 8518 */ 8519 abd_free(cb->l2rcb_abd); 8520 zio->io_size = zio->io_orig_size = arc_hdr_size(hdr); 8521 8522 if (using_rdata) { 8523 ASSERT(HDR_HAS_RABD(hdr)); 8524 zio->io_abd = zio->io_orig_abd = 8525 hdr->b_crypt_hdr.b_rabd; 8526 } else { 8527 ASSERT3P(hdr->b_l1hdr.b_pabd, !=, NULL); 8528 zio->io_abd = zio->io_orig_abd = hdr->b_l1hdr.b_pabd; 8529 } 8530 } 8531 8532 ASSERT3P(zio->io_abd, !=, NULL); 8533 8534 /* 8535 * Check this survived the L2ARC journey. 8536 */ 8537 ASSERT(zio->io_abd == hdr->b_l1hdr.b_pabd || 8538 (HDR_HAS_RABD(hdr) && zio->io_abd == hdr->b_crypt_hdr.b_rabd)); 8539 zio->io_bp_copy = cb->l2rcb_bp; /* XXX fix in L2ARC 2.0 */ 8540 zio->io_bp = &zio->io_bp_copy; /* XXX fix in L2ARC 2.0 */ 8541 zio->io_prop.zp_complevel = hdr->b_complevel; 8542 8543 valid_cksum = arc_cksum_is_equal(hdr, zio); 8544 8545 /* 8546 * b_rabd will always match the data as it exists on disk if it is 8547 * being used. Therefore if we are reading into b_rabd we do not 8548 * attempt to untransform the data. 8549 */ 8550 if (valid_cksum && !using_rdata) 8551 tfm_error = l2arc_untransform(zio, cb); 8552 8553 if (valid_cksum && tfm_error == 0 && zio->io_error == 0 && 8554 !HDR_L2_EVICTED(hdr)) { 8555 mutex_exit(hash_lock); 8556 zio->io_private = hdr; 8557 arc_read_done(zio); 8558 } else { 8559 /* 8560 * Buffer didn't survive caching. Increment stats and 8561 * reissue to the original storage device. 8562 */ 8563 if (zio->io_error != 0) { 8564 ARCSTAT_BUMP(arcstat_l2_io_error); 8565 } else { 8566 zio->io_error = SET_ERROR(EIO); 8567 } 8568 if (!valid_cksum || tfm_error != 0) 8569 ARCSTAT_BUMP(arcstat_l2_cksum_bad); 8570 8571 /* 8572 * If there's no waiter, issue an async i/o to the primary 8573 * storage now. If there *is* a waiter, the caller must 8574 * issue the i/o in a context where it's OK to block. 8575 */ 8576 if (zio->io_waiter == NULL) { 8577 zio_t *pio = zio_unique_parent(zio); 8578 void *abd = (using_rdata) ? 8579 hdr->b_crypt_hdr.b_rabd : hdr->b_l1hdr.b_pabd; 8580 8581 ASSERT(!pio || pio->io_child_type == ZIO_CHILD_LOGICAL); 8582 8583 zio = zio_read(pio, zio->io_spa, zio->io_bp, 8584 abd, zio->io_size, arc_read_done, 8585 hdr, zio->io_priority, cb->l2rcb_flags, 8586 &cb->l2rcb_zb); 8587 8588 /* 8589 * Original ZIO will be freed, so we need to update 8590 * ARC header with the new ZIO pointer to be used 8591 * by zio_change_priority() in arc_read(). 8592 */ 8593 for (struct arc_callback *acb = hdr->b_l1hdr.b_acb; 8594 acb != NULL; acb = acb->acb_next) 8595 acb->acb_zio_head = zio; 8596 8597 mutex_exit(hash_lock); 8598 zio_nowait(zio); 8599 } else { 8600 mutex_exit(hash_lock); 8601 } 8602 } 8603 8604 kmem_free(cb, sizeof (l2arc_read_callback_t)); 8605 } 8606 8607 /* 8608 * This is the list priority from which the L2ARC will search for pages to 8609 * cache. This is used within loops (0..3) to cycle through lists in the 8610 * desired order. This order can have a significant effect on cache 8611 * performance. 8612 * 8613 * Currently the metadata lists are hit first, MFU then MRU, followed by 8614 * the data lists. This function returns a locked list, and also returns 8615 * the lock pointer. 8616 */ 8617 static multilist_sublist_t * 8618 l2arc_sublist_lock(int list_num) 8619 { 8620 multilist_t *ml = NULL; 8621 unsigned int idx; 8622 8623 ASSERT(list_num >= 0 && list_num < L2ARC_FEED_TYPES); 8624 8625 switch (list_num) { 8626 case 0: 8627 ml = &arc_mfu->arcs_list[ARC_BUFC_METADATA]; 8628 break; 8629 case 1: 8630 ml = &arc_mru->arcs_list[ARC_BUFC_METADATA]; 8631 break; 8632 case 2: 8633 ml = &arc_mfu->arcs_list[ARC_BUFC_DATA]; 8634 break; 8635 case 3: 8636 ml = &arc_mru->arcs_list[ARC_BUFC_DATA]; 8637 break; 8638 default: 8639 return (NULL); 8640 } 8641 8642 /* 8643 * Return a randomly-selected sublist. This is acceptable 8644 * because the caller feeds only a little bit of data for each 8645 * call (8MB). Subsequent calls will result in different 8646 * sublists being selected. 8647 */ 8648 idx = multilist_get_random_index(ml); 8649 return (multilist_sublist_lock_idx(ml, idx)); 8650 } 8651 8652 /* 8653 * Calculates the maximum overhead of L2ARC metadata log blocks for a given 8654 * L2ARC write size. l2arc_evict and l2arc_write_size need to include this 8655 * overhead in processing to make sure there is enough headroom available 8656 * when writing buffers. 8657 */ 8658 static inline uint64_t 8659 l2arc_log_blk_overhead(uint64_t write_sz, l2arc_dev_t *dev) 8660 { 8661 if (dev->l2ad_log_entries == 0) { 8662 return (0); 8663 } else { 8664 uint64_t log_entries = write_sz >> SPA_MINBLOCKSHIFT; 8665 8666 uint64_t log_blocks = (log_entries + 8667 dev->l2ad_log_entries - 1) / 8668 dev->l2ad_log_entries; 8669 8670 return (vdev_psize_to_asize(dev->l2ad_vdev, 8671 sizeof (l2arc_log_blk_phys_t)) * log_blocks); 8672 } 8673 } 8674 8675 /* 8676 * Evict buffers from the device write hand to the distance specified in 8677 * bytes. This distance may span populated buffers, it may span nothing. 8678 * This is clearing a region on the L2ARC device ready for writing. 8679 * If the 'all' boolean is set, every buffer is evicted. 8680 */ 8681 static void 8682 l2arc_evict(l2arc_dev_t *dev, uint64_t distance, boolean_t all) 8683 { 8684 list_t *buflist; 8685 arc_buf_hdr_t *hdr, *hdr_prev; 8686 kmutex_t *hash_lock; 8687 uint64_t taddr; 8688 l2arc_lb_ptr_buf_t *lb_ptr_buf, *lb_ptr_buf_prev; 8689 vdev_t *vd = dev->l2ad_vdev; 8690 boolean_t rerun; 8691 8692 buflist = &dev->l2ad_buflist; 8693 8694 top: 8695 rerun = B_FALSE; 8696 if (dev->l2ad_hand + distance > dev->l2ad_end) { 8697 /* 8698 * When there is no space to accommodate upcoming writes, 8699 * evict to the end. Then bump the write and evict hands 8700 * to the start and iterate. This iteration does not 8701 * happen indefinitely as we make sure in 8702 * l2arc_write_size() that when the write hand is reset, 8703 * the write size does not exceed the end of the device. 8704 */ 8705 rerun = B_TRUE; 8706 taddr = dev->l2ad_end; 8707 } else { 8708 taddr = dev->l2ad_hand + distance; 8709 } 8710 DTRACE_PROBE4(l2arc__evict, l2arc_dev_t *, dev, list_t *, buflist, 8711 uint64_t, taddr, boolean_t, all); 8712 8713 if (!all) { 8714 /* 8715 * This check has to be placed after deciding whether to 8716 * iterate (rerun). 8717 */ 8718 if (dev->l2ad_first) { 8719 /* 8720 * This is the first sweep through the device. There is 8721 * nothing to evict. We have already trimmmed the 8722 * whole device. 8723 */ 8724 goto out; 8725 } else { 8726 /* 8727 * Trim the space to be evicted. 8728 */ 8729 if (vd->vdev_has_trim && dev->l2ad_evict < taddr && 8730 l2arc_trim_ahead > 0) { 8731 /* 8732 * We have to drop the spa_config lock because 8733 * vdev_trim_range() will acquire it. 8734 * l2ad_evict already accounts for the label 8735 * size. To prevent vdev_trim_ranges() from 8736 * adding it again, we subtract it from 8737 * l2ad_evict. 8738 */ 8739 spa_config_exit(dev->l2ad_spa, SCL_L2ARC, dev); 8740 vdev_trim_simple(vd, 8741 dev->l2ad_evict - VDEV_LABEL_START_SIZE, 8742 taddr - dev->l2ad_evict); 8743 spa_config_enter(dev->l2ad_spa, SCL_L2ARC, dev, 8744 RW_READER); 8745 } 8746 8747 /* 8748 * When rebuilding L2ARC we retrieve the evict hand 8749 * from the header of the device. Of note, l2arc_evict() 8750 * does not actually delete buffers from the cache 8751 * device, but trimming may do so depending on the 8752 * hardware implementation. Thus keeping track of the 8753 * evict hand is useful. 8754 */ 8755 dev->l2ad_evict = MAX(dev->l2ad_evict, taddr); 8756 } 8757 } 8758 8759 retry: 8760 mutex_enter(&dev->l2ad_mtx); 8761 /* 8762 * We have to account for evicted log blocks. Run vdev_space_update() 8763 * on log blocks whose offset (in bytes) is before the evicted offset 8764 * (in bytes) by searching in the list of pointers to log blocks 8765 * present in the L2ARC device. 8766 */ 8767 for (lb_ptr_buf = list_tail(&dev->l2ad_lbptr_list); lb_ptr_buf; 8768 lb_ptr_buf = lb_ptr_buf_prev) { 8769 8770 lb_ptr_buf_prev = list_prev(&dev->l2ad_lbptr_list, lb_ptr_buf); 8771 8772 /* L2BLK_GET_PSIZE returns aligned size for log blocks */ 8773 uint64_t asize = L2BLK_GET_PSIZE( 8774 (lb_ptr_buf->lb_ptr)->lbp_prop); 8775 8776 /* 8777 * We don't worry about log blocks left behind (ie 8778 * lbp_payload_start < l2ad_hand) because l2arc_write_buffers() 8779 * will never write more than l2arc_evict() evicts. 8780 */ 8781 if (!all && l2arc_log_blkptr_valid(dev, lb_ptr_buf->lb_ptr)) { 8782 break; 8783 } else { 8784 vdev_space_update(vd, -asize, 0, 0); 8785 ARCSTAT_INCR(arcstat_l2_log_blk_asize, -asize); 8786 ARCSTAT_BUMPDOWN(arcstat_l2_log_blk_count); 8787 zfs_refcount_remove_many(&dev->l2ad_lb_asize, asize, 8788 lb_ptr_buf); 8789 (void) zfs_refcount_remove(&dev->l2ad_lb_count, 8790 lb_ptr_buf); 8791 list_remove(&dev->l2ad_lbptr_list, lb_ptr_buf); 8792 kmem_free(lb_ptr_buf->lb_ptr, 8793 sizeof (l2arc_log_blkptr_t)); 8794 kmem_free(lb_ptr_buf, sizeof (l2arc_lb_ptr_buf_t)); 8795 } 8796 } 8797 8798 for (hdr = list_tail(buflist); hdr; hdr = hdr_prev) { 8799 hdr_prev = list_prev(buflist, hdr); 8800 8801 ASSERT(!HDR_EMPTY(hdr)); 8802 hash_lock = HDR_LOCK(hdr); 8803 8804 /* 8805 * We cannot use mutex_enter or else we can deadlock 8806 * with l2arc_write_buffers (due to swapping the order 8807 * the hash lock and l2ad_mtx are taken). 8808 */ 8809 if (!mutex_tryenter(hash_lock)) { 8810 /* 8811 * Missed the hash lock. Retry. 8812 */ 8813 ARCSTAT_BUMP(arcstat_l2_evict_lock_retry); 8814 mutex_exit(&dev->l2ad_mtx); 8815 mutex_enter(hash_lock); 8816 mutex_exit(hash_lock); 8817 goto retry; 8818 } 8819 8820 /* 8821 * A header can't be on this list if it doesn't have L2 header. 8822 */ 8823 ASSERT(HDR_HAS_L2HDR(hdr)); 8824 8825 /* Ensure this header has finished being written. */ 8826 ASSERT(!HDR_L2_WRITING(hdr)); 8827 ASSERT(!HDR_L2_WRITE_HEAD(hdr)); 8828 8829 if (!all && (hdr->b_l2hdr.b_daddr >= dev->l2ad_evict || 8830 hdr->b_l2hdr.b_daddr < dev->l2ad_hand)) { 8831 /* 8832 * We've evicted to the target address, 8833 * or the end of the device. 8834 */ 8835 mutex_exit(hash_lock); 8836 break; 8837 } 8838 8839 if (!HDR_HAS_L1HDR(hdr)) { 8840 ASSERT(!HDR_L2_READING(hdr)); 8841 /* 8842 * This doesn't exist in the ARC. Destroy. 8843 * arc_hdr_destroy() will call list_remove() 8844 * and decrement arcstat_l2_lsize. 8845 */ 8846 arc_change_state(arc_anon, hdr); 8847 arc_hdr_destroy(hdr); 8848 } else { 8849 ASSERT(hdr->b_l1hdr.b_state != arc_l2c_only); 8850 ARCSTAT_BUMP(arcstat_l2_evict_l1cached); 8851 /* 8852 * Invalidate issued or about to be issued 8853 * reads, since we may be about to write 8854 * over this location. 8855 */ 8856 if (HDR_L2_READING(hdr)) { 8857 ARCSTAT_BUMP(arcstat_l2_evict_reading); 8858 arc_hdr_set_flags(hdr, ARC_FLAG_L2_EVICTED); 8859 } 8860 8861 arc_hdr_l2hdr_destroy(hdr); 8862 } 8863 mutex_exit(hash_lock); 8864 } 8865 mutex_exit(&dev->l2ad_mtx); 8866 8867 out: 8868 /* 8869 * We need to check if we evict all buffers, otherwise we may iterate 8870 * unnecessarily. 8871 */ 8872 if (!all && rerun) { 8873 /* 8874 * Bump device hand to the device start if it is approaching the 8875 * end. l2arc_evict() has already evicted ahead for this case. 8876 */ 8877 dev->l2ad_hand = dev->l2ad_start; 8878 dev->l2ad_evict = dev->l2ad_start; 8879 dev->l2ad_first = B_FALSE; 8880 goto top; 8881 } 8882 8883 if (!all) { 8884 /* 8885 * In case of cache device removal (all) the following 8886 * assertions may be violated without functional consequences 8887 * as the device is about to be removed. 8888 */ 8889 ASSERT3U(dev->l2ad_hand + distance, <, dev->l2ad_end); 8890 if (!dev->l2ad_first) 8891 ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); 8892 } 8893 } 8894 8895 /* 8896 * Handle any abd transforms that might be required for writing to the L2ARC. 8897 * If successful, this function will always return an abd with the data 8898 * transformed as it is on disk in a new abd of asize bytes. 8899 */ 8900 static int 8901 l2arc_apply_transforms(spa_t *spa, arc_buf_hdr_t *hdr, uint64_t asize, 8902 abd_t **abd_out) 8903 { 8904 int ret; 8905 abd_t *cabd = NULL, *eabd = NULL, *to_write = hdr->b_l1hdr.b_pabd; 8906 enum zio_compress compress = HDR_GET_COMPRESS(hdr); 8907 uint64_t psize = HDR_GET_PSIZE(hdr); 8908 uint64_t size = arc_hdr_size(hdr); 8909 boolean_t ismd = HDR_ISTYPE_METADATA(hdr); 8910 boolean_t bswap = (hdr->b_l1hdr.b_byteswap != DMU_BSWAP_NUMFUNCS); 8911 dsl_crypto_key_t *dck = NULL; 8912 uint8_t mac[ZIO_DATA_MAC_LEN] = { 0 }; 8913 boolean_t no_crypt = B_FALSE; 8914 8915 ASSERT((HDR_GET_COMPRESS(hdr) != ZIO_COMPRESS_OFF && 8916 !HDR_COMPRESSION_ENABLED(hdr)) || 8917 HDR_ENCRYPTED(hdr) || HDR_SHARED_DATA(hdr) || psize != asize); 8918 ASSERT3U(psize, <=, asize); 8919 8920 /* 8921 * If this data simply needs its own buffer, we simply allocate it 8922 * and copy the data. This may be done to eliminate a dependency on a 8923 * shared buffer or to reallocate the buffer to match asize. 8924 */ 8925 if (HDR_HAS_RABD(hdr)) { 8926 ASSERT3U(asize, >, psize); 8927 to_write = abd_alloc_for_io(asize, ismd); 8928 abd_copy(to_write, hdr->b_crypt_hdr.b_rabd, psize); 8929 abd_zero_off(to_write, psize, asize - psize); 8930 goto out; 8931 } 8932 8933 if ((compress == ZIO_COMPRESS_OFF || HDR_COMPRESSION_ENABLED(hdr)) && 8934 !HDR_ENCRYPTED(hdr)) { 8935 ASSERT3U(size, ==, psize); 8936 to_write = abd_alloc_for_io(asize, ismd); 8937 abd_copy(to_write, hdr->b_l1hdr.b_pabd, size); 8938 if (asize > size) 8939 abd_zero_off(to_write, size, asize - size); 8940 goto out; 8941 } 8942 8943 if (compress != ZIO_COMPRESS_OFF && !HDR_COMPRESSION_ENABLED(hdr)) { 8944 size_t bufsize = MAX(size, asize); 8945 void *buf = zio_buf_alloc(bufsize); 8946 uint64_t csize = zio_compress_data(compress, to_write, &buf, 8947 size, hdr->b_complevel); 8948 if (csize > psize) { 8949 /* 8950 * We can't re-compress the block into the original 8951 * psize. Even if it fits into asize, it does not 8952 * matter, since checksum will never match on read. 8953 */ 8954 zio_buf_free(buf, bufsize); 8955 return (SET_ERROR(EIO)); 8956 } 8957 if (asize > csize) 8958 memset((char *)buf + csize, 0, asize - csize); 8959 to_write = cabd = abd_get_from_buf(buf, bufsize); 8960 abd_take_ownership_of_buf(cabd, B_TRUE); 8961 } 8962 8963 if (HDR_ENCRYPTED(hdr)) { 8964 eabd = abd_alloc_for_io(asize, ismd); 8965 8966 /* 8967 * If the dataset was disowned before the buffer 8968 * made it to this point, the key to re-encrypt 8969 * it won't be available. In this case we simply 8970 * won't write the buffer to the L2ARC. 8971 */ 8972 ret = spa_keystore_lookup_key(spa, hdr->b_crypt_hdr.b_dsobj, 8973 FTAG, &dck); 8974 if (ret != 0) 8975 goto error; 8976 8977 ret = zio_do_crypt_abd(B_TRUE, &dck->dck_key, 8978 hdr->b_crypt_hdr.b_ot, bswap, hdr->b_crypt_hdr.b_salt, 8979 hdr->b_crypt_hdr.b_iv, mac, psize, to_write, eabd, 8980 &no_crypt); 8981 if (ret != 0) 8982 goto error; 8983 8984 if (no_crypt) 8985 abd_copy(eabd, to_write, psize); 8986 8987 if (psize != asize) 8988 abd_zero_off(eabd, psize, asize - psize); 8989 8990 /* assert that the MAC we got here matches the one we saved */ 8991 ASSERT0(memcmp(mac, hdr->b_crypt_hdr.b_mac, ZIO_DATA_MAC_LEN)); 8992 spa_keystore_dsl_key_rele(spa, dck, FTAG); 8993 8994 if (to_write == cabd) 8995 abd_free(cabd); 8996 8997 to_write = eabd; 8998 } 8999 9000 out: 9001 ASSERT3P(to_write, !=, hdr->b_l1hdr.b_pabd); 9002 *abd_out = to_write; 9003 return (0); 9004 9005 error: 9006 if (dck != NULL) 9007 spa_keystore_dsl_key_rele(spa, dck, FTAG); 9008 if (cabd != NULL) 9009 abd_free(cabd); 9010 if (eabd != NULL) 9011 abd_free(eabd); 9012 9013 *abd_out = NULL; 9014 return (ret); 9015 } 9016 9017 static void 9018 l2arc_blk_fetch_done(zio_t *zio) 9019 { 9020 l2arc_read_callback_t *cb; 9021 9022 cb = zio->io_private; 9023 if (cb->l2rcb_abd != NULL) 9024 abd_free(cb->l2rcb_abd); 9025 kmem_free(cb, sizeof (l2arc_read_callback_t)); 9026 } 9027 9028 /* 9029 * Find and write ARC buffers to the L2ARC device. 9030 * 9031 * An ARC_FLAG_L2_WRITING flag is set so that the L2ARC buffers are not valid 9032 * for reading until they have completed writing. 9033 * The headroom_boost is an in-out parameter used to maintain headroom boost 9034 * state between calls to this function. 9035 * 9036 * Returns the number of bytes actually written (which may be smaller than 9037 * the delta by which the device hand has changed due to alignment and the 9038 * writing of log blocks). 9039 */ 9040 static uint64_t 9041 l2arc_write_buffers(spa_t *spa, l2arc_dev_t *dev, uint64_t target_sz) 9042 { 9043 arc_buf_hdr_t *hdr, *head, *marker; 9044 uint64_t write_asize, write_psize, headroom; 9045 boolean_t full, from_head = !arc_warm; 9046 l2arc_write_callback_t *cb = NULL; 9047 zio_t *pio, *wzio; 9048 uint64_t guid = spa_load_guid(spa); 9049 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 9050 9051 ASSERT3P(dev->l2ad_vdev, !=, NULL); 9052 9053 pio = NULL; 9054 write_asize = write_psize = 0; 9055 full = B_FALSE; 9056 head = kmem_cache_alloc(hdr_l2only_cache, KM_PUSHPAGE); 9057 arc_hdr_set_flags(head, ARC_FLAG_L2_WRITE_HEAD | ARC_FLAG_HAS_L2HDR); 9058 marker = arc_state_alloc_marker(); 9059 9060 /* 9061 * Copy buffers for L2ARC writing. 9062 */ 9063 for (int pass = 0; pass < L2ARC_FEED_TYPES; pass++) { 9064 /* 9065 * If pass == 1 or 3, we cache MRU metadata and data 9066 * respectively. 9067 */ 9068 if (l2arc_mfuonly) { 9069 if (pass == 1 || pass == 3) 9070 continue; 9071 } 9072 9073 uint64_t passed_sz = 0; 9074 headroom = target_sz * l2arc_headroom; 9075 if (zfs_compressed_arc_enabled) 9076 headroom = (headroom * l2arc_headroom_boost) / 100; 9077 9078 /* 9079 * Until the ARC is warm and starts to evict, read from the 9080 * head of the ARC lists rather than the tail. 9081 */ 9082 multilist_sublist_t *mls = l2arc_sublist_lock(pass); 9083 ASSERT3P(mls, !=, NULL); 9084 if (from_head) 9085 hdr = multilist_sublist_head(mls); 9086 else 9087 hdr = multilist_sublist_tail(mls); 9088 9089 while (hdr != NULL) { 9090 kmutex_t *hash_lock; 9091 abd_t *to_write = NULL; 9092 9093 hash_lock = HDR_LOCK(hdr); 9094 if (!mutex_tryenter(hash_lock)) { 9095 skip: 9096 /* Skip this buffer rather than waiting. */ 9097 if (from_head) 9098 hdr = multilist_sublist_next(mls, hdr); 9099 else 9100 hdr = multilist_sublist_prev(mls, hdr); 9101 continue; 9102 } 9103 9104 passed_sz += HDR_GET_LSIZE(hdr); 9105 if (l2arc_headroom != 0 && passed_sz > headroom) { 9106 /* 9107 * Searched too far. 9108 */ 9109 mutex_exit(hash_lock); 9110 break; 9111 } 9112 9113 if (!l2arc_write_eligible(guid, hdr)) { 9114 mutex_exit(hash_lock); 9115 goto skip; 9116 } 9117 9118 ASSERT(HDR_HAS_L1HDR(hdr)); 9119 ASSERT3U(HDR_GET_PSIZE(hdr), >, 0); 9120 ASSERT3U(arc_hdr_size(hdr), >, 0); 9121 ASSERT(hdr->b_l1hdr.b_pabd != NULL || 9122 HDR_HAS_RABD(hdr)); 9123 uint64_t psize = HDR_GET_PSIZE(hdr); 9124 uint64_t asize = vdev_psize_to_asize(dev->l2ad_vdev, 9125 psize); 9126 9127 /* 9128 * If the allocated size of this buffer plus the max 9129 * size for the pending log block exceeds the evicted 9130 * target size, terminate writing buffers for this run. 9131 */ 9132 if (write_asize + asize + 9133 sizeof (l2arc_log_blk_phys_t) > target_sz) { 9134 full = B_TRUE; 9135 mutex_exit(hash_lock); 9136 break; 9137 } 9138 9139 /* 9140 * We should not sleep with sublist lock held or it 9141 * may block ARC eviction. Insert a marker to save 9142 * the position and drop the lock. 9143 */ 9144 if (from_head) { 9145 multilist_sublist_insert_after(mls, hdr, 9146 marker); 9147 } else { 9148 multilist_sublist_insert_before(mls, hdr, 9149 marker); 9150 } 9151 multilist_sublist_unlock(mls); 9152 9153 /* 9154 * If this header has b_rabd, we can use this since it 9155 * must always match the data exactly as it exists on 9156 * disk. Otherwise, the L2ARC can normally use the 9157 * hdr's data, but if we're sharing data between the 9158 * hdr and one of its bufs, L2ARC needs its own copy of 9159 * the data so that the ZIO below can't race with the 9160 * buf consumer. To ensure that this copy will be 9161 * available for the lifetime of the ZIO and be cleaned 9162 * up afterwards, we add it to the l2arc_free_on_write 9163 * queue. If we need to apply any transforms to the 9164 * data (compression, encryption) we will also need the 9165 * extra buffer. 9166 */ 9167 if (HDR_HAS_RABD(hdr) && psize == asize) { 9168 to_write = hdr->b_crypt_hdr.b_rabd; 9169 } else if ((HDR_COMPRESSION_ENABLED(hdr) || 9170 HDR_GET_COMPRESS(hdr) == ZIO_COMPRESS_OFF) && 9171 !HDR_ENCRYPTED(hdr) && !HDR_SHARED_DATA(hdr) && 9172 psize == asize) { 9173 to_write = hdr->b_l1hdr.b_pabd; 9174 } else { 9175 int ret; 9176 arc_buf_contents_t type = arc_buf_type(hdr); 9177 9178 ret = l2arc_apply_transforms(spa, hdr, asize, 9179 &to_write); 9180 if (ret != 0) { 9181 arc_hdr_clear_flags(hdr, 9182 ARC_FLAG_L2CACHE); 9183 mutex_exit(hash_lock); 9184 goto next; 9185 } 9186 9187 l2arc_free_abd_on_write(to_write, asize, type); 9188 } 9189 9190 hdr->b_l2hdr.b_dev = dev; 9191 hdr->b_l2hdr.b_daddr = dev->l2ad_hand; 9192 hdr->b_l2hdr.b_hits = 0; 9193 hdr->b_l2hdr.b_arcs_state = 9194 hdr->b_l1hdr.b_state->arcs_state; 9195 mutex_enter(&dev->l2ad_mtx); 9196 if (pio == NULL) { 9197 /* 9198 * Insert a dummy header on the buflist so 9199 * l2arc_write_done() can find where the 9200 * write buffers begin without searching. 9201 */ 9202 list_insert_head(&dev->l2ad_buflist, head); 9203 } 9204 list_insert_head(&dev->l2ad_buflist, hdr); 9205 mutex_exit(&dev->l2ad_mtx); 9206 arc_hdr_set_flags(hdr, ARC_FLAG_HAS_L2HDR | 9207 ARC_FLAG_L2_WRITING); 9208 9209 (void) zfs_refcount_add_many(&dev->l2ad_alloc, 9210 arc_hdr_size(hdr), hdr); 9211 l2arc_hdr_arcstats_increment(hdr); 9212 9213 boolean_t commit = l2arc_log_blk_insert(dev, hdr); 9214 mutex_exit(hash_lock); 9215 9216 if (pio == NULL) { 9217 cb = kmem_alloc( 9218 sizeof (l2arc_write_callback_t), KM_SLEEP); 9219 cb->l2wcb_dev = dev; 9220 cb->l2wcb_head = head; 9221 list_create(&cb->l2wcb_abd_list, 9222 sizeof (l2arc_lb_abd_buf_t), 9223 offsetof(l2arc_lb_abd_buf_t, node)); 9224 pio = zio_root(spa, l2arc_write_done, cb, 9225 ZIO_FLAG_CANFAIL); 9226 } 9227 9228 wzio = zio_write_phys(pio, dev->l2ad_vdev, 9229 dev->l2ad_hand, asize, to_write, 9230 ZIO_CHECKSUM_OFF, NULL, hdr, 9231 ZIO_PRIORITY_ASYNC_WRITE, 9232 ZIO_FLAG_CANFAIL, B_FALSE); 9233 9234 DTRACE_PROBE2(l2arc__write, vdev_t *, dev->l2ad_vdev, 9235 zio_t *, wzio); 9236 zio_nowait(wzio); 9237 9238 write_psize += psize; 9239 write_asize += asize; 9240 dev->l2ad_hand += asize; 9241 vdev_space_update(dev->l2ad_vdev, asize, 0, 0); 9242 9243 if (commit) { 9244 /* l2ad_hand will be adjusted inside. */ 9245 write_asize += 9246 l2arc_log_blk_commit(dev, pio, cb); 9247 } 9248 9249 next: 9250 multilist_sublist_lock(mls); 9251 if (from_head) 9252 hdr = multilist_sublist_next(mls, marker); 9253 else 9254 hdr = multilist_sublist_prev(mls, marker); 9255 multilist_sublist_remove(mls, marker); 9256 } 9257 9258 multilist_sublist_unlock(mls); 9259 9260 if (full == B_TRUE) 9261 break; 9262 } 9263 9264 arc_state_free_marker(marker); 9265 9266 /* No buffers selected for writing? */ 9267 if (pio == NULL) { 9268 ASSERT0(write_psize); 9269 ASSERT(!HDR_HAS_L1HDR(head)); 9270 kmem_cache_free(hdr_l2only_cache, head); 9271 9272 /* 9273 * Although we did not write any buffers l2ad_evict may 9274 * have advanced. 9275 */ 9276 if (dev->l2ad_evict != l2dhdr->dh_evict) 9277 l2arc_dev_hdr_update(dev); 9278 9279 return (0); 9280 } 9281 9282 if (!dev->l2ad_first) 9283 ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); 9284 9285 ASSERT3U(write_asize, <=, target_sz); 9286 ARCSTAT_BUMP(arcstat_l2_writes_sent); 9287 ARCSTAT_INCR(arcstat_l2_write_bytes, write_psize); 9288 9289 dev->l2ad_writing = B_TRUE; 9290 (void) zio_wait(pio); 9291 dev->l2ad_writing = B_FALSE; 9292 9293 /* 9294 * Update the device header after the zio completes as 9295 * l2arc_write_done() may have updated the memory holding the log block 9296 * pointers in the device header. 9297 */ 9298 l2arc_dev_hdr_update(dev); 9299 9300 return (write_asize); 9301 } 9302 9303 static boolean_t 9304 l2arc_hdr_limit_reached(void) 9305 { 9306 int64_t s = aggsum_upper_bound(&arc_sums.arcstat_l2_hdr_size); 9307 9308 return (arc_reclaim_needed() || 9309 (s > (arc_warm ? arc_c : arc_c_max) * l2arc_meta_percent / 100)); 9310 } 9311 9312 /* 9313 * This thread feeds the L2ARC at regular intervals. This is the beating 9314 * heart of the L2ARC. 9315 */ 9316 static __attribute__((noreturn)) void 9317 l2arc_feed_thread(void *unused) 9318 { 9319 (void) unused; 9320 callb_cpr_t cpr; 9321 l2arc_dev_t *dev; 9322 spa_t *spa; 9323 uint64_t size, wrote; 9324 clock_t begin, next = ddi_get_lbolt(); 9325 fstrans_cookie_t cookie; 9326 9327 CALLB_CPR_INIT(&cpr, &l2arc_feed_thr_lock, callb_generic_cpr, FTAG); 9328 9329 mutex_enter(&l2arc_feed_thr_lock); 9330 9331 cookie = spl_fstrans_mark(); 9332 while (l2arc_thread_exit == 0) { 9333 CALLB_CPR_SAFE_BEGIN(&cpr); 9334 (void) cv_timedwait_idle(&l2arc_feed_thr_cv, 9335 &l2arc_feed_thr_lock, next); 9336 CALLB_CPR_SAFE_END(&cpr, &l2arc_feed_thr_lock); 9337 next = ddi_get_lbolt() + hz; 9338 9339 /* 9340 * Quick check for L2ARC devices. 9341 */ 9342 mutex_enter(&l2arc_dev_mtx); 9343 if (l2arc_ndev == 0) { 9344 mutex_exit(&l2arc_dev_mtx); 9345 continue; 9346 } 9347 mutex_exit(&l2arc_dev_mtx); 9348 begin = ddi_get_lbolt(); 9349 9350 /* 9351 * This selects the next l2arc device to write to, and in 9352 * doing so the next spa to feed from: dev->l2ad_spa. This 9353 * will return NULL if there are now no l2arc devices or if 9354 * they are all faulted. 9355 * 9356 * If a device is returned, its spa's config lock is also 9357 * held to prevent device removal. l2arc_dev_get_next() 9358 * will grab and release l2arc_dev_mtx. 9359 */ 9360 if ((dev = l2arc_dev_get_next()) == NULL) 9361 continue; 9362 9363 spa = dev->l2ad_spa; 9364 ASSERT3P(spa, !=, NULL); 9365 9366 /* 9367 * If the pool is read-only then force the feed thread to 9368 * sleep a little longer. 9369 */ 9370 if (!spa_writeable(spa)) { 9371 next = ddi_get_lbolt() + 5 * l2arc_feed_secs * hz; 9372 spa_config_exit(spa, SCL_L2ARC, dev); 9373 continue; 9374 } 9375 9376 /* 9377 * Avoid contributing to memory pressure. 9378 */ 9379 if (l2arc_hdr_limit_reached()) { 9380 ARCSTAT_BUMP(arcstat_l2_abort_lowmem); 9381 spa_config_exit(spa, SCL_L2ARC, dev); 9382 continue; 9383 } 9384 9385 ARCSTAT_BUMP(arcstat_l2_feeds); 9386 9387 size = l2arc_write_size(dev); 9388 9389 /* 9390 * Evict L2ARC buffers that will be overwritten. 9391 */ 9392 l2arc_evict(dev, size, B_FALSE); 9393 9394 /* 9395 * Write ARC buffers. 9396 */ 9397 wrote = l2arc_write_buffers(spa, dev, size); 9398 9399 /* 9400 * Calculate interval between writes. 9401 */ 9402 next = l2arc_write_interval(begin, size, wrote); 9403 spa_config_exit(spa, SCL_L2ARC, dev); 9404 } 9405 spl_fstrans_unmark(cookie); 9406 9407 l2arc_thread_exit = 0; 9408 cv_broadcast(&l2arc_feed_thr_cv); 9409 CALLB_CPR_EXIT(&cpr); /* drops l2arc_feed_thr_lock */ 9410 thread_exit(); 9411 } 9412 9413 boolean_t 9414 l2arc_vdev_present(vdev_t *vd) 9415 { 9416 return (l2arc_vdev_get(vd) != NULL); 9417 } 9418 9419 /* 9420 * Returns the l2arc_dev_t associated with a particular vdev_t or NULL if 9421 * the vdev_t isn't an L2ARC device. 9422 */ 9423 l2arc_dev_t * 9424 l2arc_vdev_get(vdev_t *vd) 9425 { 9426 l2arc_dev_t *dev; 9427 9428 mutex_enter(&l2arc_dev_mtx); 9429 for (dev = list_head(l2arc_dev_list); dev != NULL; 9430 dev = list_next(l2arc_dev_list, dev)) { 9431 if (dev->l2ad_vdev == vd) 9432 break; 9433 } 9434 mutex_exit(&l2arc_dev_mtx); 9435 9436 return (dev); 9437 } 9438 9439 static void 9440 l2arc_rebuild_dev(l2arc_dev_t *dev, boolean_t reopen) 9441 { 9442 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 9443 uint64_t l2dhdr_asize = dev->l2ad_dev_hdr_asize; 9444 spa_t *spa = dev->l2ad_spa; 9445 9446 /* 9447 * The L2ARC has to hold at least the payload of one log block for 9448 * them to be restored (persistent L2ARC). The payload of a log block 9449 * depends on the amount of its log entries. We always write log blocks 9450 * with 1022 entries. How many of them are committed or restored depends 9451 * on the size of the L2ARC device. Thus the maximum payload of 9452 * one log block is 1022 * SPA_MAXBLOCKSIZE = 16GB. If the L2ARC device 9453 * is less than that, we reduce the amount of committed and restored 9454 * log entries per block so as to enable persistence. 9455 */ 9456 if (dev->l2ad_end < l2arc_rebuild_blocks_min_l2size) { 9457 dev->l2ad_log_entries = 0; 9458 } else { 9459 dev->l2ad_log_entries = MIN((dev->l2ad_end - 9460 dev->l2ad_start) >> SPA_MAXBLOCKSHIFT, 9461 L2ARC_LOG_BLK_MAX_ENTRIES); 9462 } 9463 9464 /* 9465 * Read the device header, if an error is returned do not rebuild L2ARC. 9466 */ 9467 if (l2arc_dev_hdr_read(dev) == 0 && dev->l2ad_log_entries > 0) { 9468 /* 9469 * If we are onlining a cache device (vdev_reopen) that was 9470 * still present (l2arc_vdev_present()) and rebuild is enabled, 9471 * we should evict all ARC buffers and pointers to log blocks 9472 * and reclaim their space before restoring its contents to 9473 * L2ARC. 9474 */ 9475 if (reopen) { 9476 if (!l2arc_rebuild_enabled) { 9477 return; 9478 } else { 9479 l2arc_evict(dev, 0, B_TRUE); 9480 /* start a new log block */ 9481 dev->l2ad_log_ent_idx = 0; 9482 dev->l2ad_log_blk_payload_asize = 0; 9483 dev->l2ad_log_blk_payload_start = 0; 9484 } 9485 } 9486 /* 9487 * Just mark the device as pending for a rebuild. We won't 9488 * be starting a rebuild in line here as it would block pool 9489 * import. Instead spa_load_impl will hand that off to an 9490 * async task which will call l2arc_spa_rebuild_start. 9491 */ 9492 dev->l2ad_rebuild = B_TRUE; 9493 } else if (spa_writeable(spa)) { 9494 /* 9495 * In this case TRIM the whole device if l2arc_trim_ahead > 0, 9496 * otherwise create a new header. We zero out the memory holding 9497 * the header to reset dh_start_lbps. If we TRIM the whole 9498 * device the new header will be written by 9499 * vdev_trim_l2arc_thread() at the end of the TRIM to update the 9500 * trim_state in the header too. When reading the header, if 9501 * trim_state is not VDEV_TRIM_COMPLETE and l2arc_trim_ahead > 0 9502 * we opt to TRIM the whole device again. 9503 */ 9504 if (l2arc_trim_ahead > 0) { 9505 dev->l2ad_trim_all = B_TRUE; 9506 } else { 9507 memset(l2dhdr, 0, l2dhdr_asize); 9508 l2arc_dev_hdr_update(dev); 9509 } 9510 } 9511 } 9512 9513 /* 9514 * Add a vdev for use by the L2ARC. By this point the spa has already 9515 * validated the vdev and opened it. 9516 */ 9517 void 9518 l2arc_add_vdev(spa_t *spa, vdev_t *vd) 9519 { 9520 l2arc_dev_t *adddev; 9521 uint64_t l2dhdr_asize; 9522 9523 ASSERT(!l2arc_vdev_present(vd)); 9524 9525 /* 9526 * Create a new l2arc device entry. 9527 */ 9528 adddev = vmem_zalloc(sizeof (l2arc_dev_t), KM_SLEEP); 9529 adddev->l2ad_spa = spa; 9530 adddev->l2ad_vdev = vd; 9531 /* leave extra size for an l2arc device header */ 9532 l2dhdr_asize = adddev->l2ad_dev_hdr_asize = 9533 MAX(sizeof (*adddev->l2ad_dev_hdr), 1 << vd->vdev_ashift); 9534 adddev->l2ad_start = VDEV_LABEL_START_SIZE + l2dhdr_asize; 9535 adddev->l2ad_end = VDEV_LABEL_START_SIZE + vdev_get_min_asize(vd); 9536 ASSERT3U(adddev->l2ad_start, <, adddev->l2ad_end); 9537 adddev->l2ad_hand = adddev->l2ad_start; 9538 adddev->l2ad_evict = adddev->l2ad_start; 9539 adddev->l2ad_first = B_TRUE; 9540 adddev->l2ad_writing = B_FALSE; 9541 adddev->l2ad_trim_all = B_FALSE; 9542 list_link_init(&adddev->l2ad_node); 9543 adddev->l2ad_dev_hdr = kmem_zalloc(l2dhdr_asize, KM_SLEEP); 9544 9545 mutex_init(&adddev->l2ad_mtx, NULL, MUTEX_DEFAULT, NULL); 9546 /* 9547 * This is a list of all ARC buffers that are still valid on the 9548 * device. 9549 */ 9550 list_create(&adddev->l2ad_buflist, sizeof (arc_buf_hdr_t), 9551 offsetof(arc_buf_hdr_t, b_l2hdr.b_l2node)); 9552 9553 /* 9554 * This is a list of pointers to log blocks that are still present 9555 * on the device. 9556 */ 9557 list_create(&adddev->l2ad_lbptr_list, sizeof (l2arc_lb_ptr_buf_t), 9558 offsetof(l2arc_lb_ptr_buf_t, node)); 9559 9560 vdev_space_update(vd, 0, 0, adddev->l2ad_end - adddev->l2ad_hand); 9561 zfs_refcount_create(&adddev->l2ad_alloc); 9562 zfs_refcount_create(&adddev->l2ad_lb_asize); 9563 zfs_refcount_create(&adddev->l2ad_lb_count); 9564 9565 /* 9566 * Decide if dev is eligible for L2ARC rebuild or whole device 9567 * trimming. This has to happen before the device is added in the 9568 * cache device list and l2arc_dev_mtx is released. Otherwise 9569 * l2arc_feed_thread() might already start writing on the 9570 * device. 9571 */ 9572 l2arc_rebuild_dev(adddev, B_FALSE); 9573 9574 /* 9575 * Add device to global list 9576 */ 9577 mutex_enter(&l2arc_dev_mtx); 9578 list_insert_head(l2arc_dev_list, adddev); 9579 atomic_inc_64(&l2arc_ndev); 9580 mutex_exit(&l2arc_dev_mtx); 9581 } 9582 9583 /* 9584 * Decide if a vdev is eligible for L2ARC rebuild, called from vdev_reopen() 9585 * in case of onlining a cache device. 9586 */ 9587 void 9588 l2arc_rebuild_vdev(vdev_t *vd, boolean_t reopen) 9589 { 9590 l2arc_dev_t *dev = NULL; 9591 9592 dev = l2arc_vdev_get(vd); 9593 ASSERT3P(dev, !=, NULL); 9594 9595 /* 9596 * In contrast to l2arc_add_vdev() we do not have to worry about 9597 * l2arc_feed_thread() invalidating previous content when onlining a 9598 * cache device. The device parameters (l2ad*) are not cleared when 9599 * offlining the device and writing new buffers will not invalidate 9600 * all previous content. In worst case only buffers that have not had 9601 * their log block written to the device will be lost. 9602 * When onlining the cache device (ie offline->online without exporting 9603 * the pool in between) this happens: 9604 * vdev_reopen() -> vdev_open() -> l2arc_rebuild_vdev() 9605 * | | 9606 * vdev_is_dead() = B_FALSE l2ad_rebuild = B_TRUE 9607 * During the time where vdev_is_dead = B_FALSE and until l2ad_rebuild 9608 * is set to B_TRUE we might write additional buffers to the device. 9609 */ 9610 l2arc_rebuild_dev(dev, reopen); 9611 } 9612 9613 /* 9614 * Remove a vdev from the L2ARC. 9615 */ 9616 void 9617 l2arc_remove_vdev(vdev_t *vd) 9618 { 9619 l2arc_dev_t *remdev = NULL; 9620 9621 /* 9622 * Find the device by vdev 9623 */ 9624 remdev = l2arc_vdev_get(vd); 9625 ASSERT3P(remdev, !=, NULL); 9626 9627 /* 9628 * Cancel any ongoing or scheduled rebuild. 9629 */ 9630 mutex_enter(&l2arc_rebuild_thr_lock); 9631 if (remdev->l2ad_rebuild_began == B_TRUE) { 9632 remdev->l2ad_rebuild_cancel = B_TRUE; 9633 while (remdev->l2ad_rebuild == B_TRUE) 9634 cv_wait(&l2arc_rebuild_thr_cv, &l2arc_rebuild_thr_lock); 9635 } 9636 mutex_exit(&l2arc_rebuild_thr_lock); 9637 9638 /* 9639 * Remove device from global list 9640 */ 9641 mutex_enter(&l2arc_dev_mtx); 9642 list_remove(l2arc_dev_list, remdev); 9643 l2arc_dev_last = NULL; /* may have been invalidated */ 9644 atomic_dec_64(&l2arc_ndev); 9645 mutex_exit(&l2arc_dev_mtx); 9646 9647 /* 9648 * Clear all buflists and ARC references. L2ARC device flush. 9649 */ 9650 l2arc_evict(remdev, 0, B_TRUE); 9651 list_destroy(&remdev->l2ad_buflist); 9652 ASSERT(list_is_empty(&remdev->l2ad_lbptr_list)); 9653 list_destroy(&remdev->l2ad_lbptr_list); 9654 mutex_destroy(&remdev->l2ad_mtx); 9655 zfs_refcount_destroy(&remdev->l2ad_alloc); 9656 zfs_refcount_destroy(&remdev->l2ad_lb_asize); 9657 zfs_refcount_destroy(&remdev->l2ad_lb_count); 9658 kmem_free(remdev->l2ad_dev_hdr, remdev->l2ad_dev_hdr_asize); 9659 vmem_free(remdev, sizeof (l2arc_dev_t)); 9660 } 9661 9662 void 9663 l2arc_init(void) 9664 { 9665 l2arc_thread_exit = 0; 9666 l2arc_ndev = 0; 9667 9668 mutex_init(&l2arc_feed_thr_lock, NULL, MUTEX_DEFAULT, NULL); 9669 cv_init(&l2arc_feed_thr_cv, NULL, CV_DEFAULT, NULL); 9670 mutex_init(&l2arc_rebuild_thr_lock, NULL, MUTEX_DEFAULT, NULL); 9671 cv_init(&l2arc_rebuild_thr_cv, NULL, CV_DEFAULT, NULL); 9672 mutex_init(&l2arc_dev_mtx, NULL, MUTEX_DEFAULT, NULL); 9673 mutex_init(&l2arc_free_on_write_mtx, NULL, MUTEX_DEFAULT, NULL); 9674 9675 l2arc_dev_list = &L2ARC_dev_list; 9676 l2arc_free_on_write = &L2ARC_free_on_write; 9677 list_create(l2arc_dev_list, sizeof (l2arc_dev_t), 9678 offsetof(l2arc_dev_t, l2ad_node)); 9679 list_create(l2arc_free_on_write, sizeof (l2arc_data_free_t), 9680 offsetof(l2arc_data_free_t, l2df_list_node)); 9681 } 9682 9683 void 9684 l2arc_fini(void) 9685 { 9686 mutex_destroy(&l2arc_feed_thr_lock); 9687 cv_destroy(&l2arc_feed_thr_cv); 9688 mutex_destroy(&l2arc_rebuild_thr_lock); 9689 cv_destroy(&l2arc_rebuild_thr_cv); 9690 mutex_destroy(&l2arc_dev_mtx); 9691 mutex_destroy(&l2arc_free_on_write_mtx); 9692 9693 list_destroy(l2arc_dev_list); 9694 list_destroy(l2arc_free_on_write); 9695 } 9696 9697 void 9698 l2arc_start(void) 9699 { 9700 if (!(spa_mode_global & SPA_MODE_WRITE)) 9701 return; 9702 9703 (void) thread_create(NULL, 0, l2arc_feed_thread, NULL, 0, &p0, 9704 TS_RUN, defclsyspri); 9705 } 9706 9707 void 9708 l2arc_stop(void) 9709 { 9710 if (!(spa_mode_global & SPA_MODE_WRITE)) 9711 return; 9712 9713 mutex_enter(&l2arc_feed_thr_lock); 9714 cv_signal(&l2arc_feed_thr_cv); /* kick thread out of startup */ 9715 l2arc_thread_exit = 1; 9716 while (l2arc_thread_exit != 0) 9717 cv_wait(&l2arc_feed_thr_cv, &l2arc_feed_thr_lock); 9718 mutex_exit(&l2arc_feed_thr_lock); 9719 } 9720 9721 /* 9722 * Punches out rebuild threads for the L2ARC devices in a spa. This should 9723 * be called after pool import from the spa async thread, since starting 9724 * these threads directly from spa_import() will make them part of the 9725 * "zpool import" context and delay process exit (and thus pool import). 9726 */ 9727 void 9728 l2arc_spa_rebuild_start(spa_t *spa) 9729 { 9730 ASSERT(MUTEX_HELD(&spa_namespace_lock)); 9731 9732 /* 9733 * Locate the spa's l2arc devices and kick off rebuild threads. 9734 */ 9735 for (int i = 0; i < spa->spa_l2cache.sav_count; i++) { 9736 l2arc_dev_t *dev = 9737 l2arc_vdev_get(spa->spa_l2cache.sav_vdevs[i]); 9738 if (dev == NULL) { 9739 /* Don't attempt a rebuild if the vdev is UNAVAIL */ 9740 continue; 9741 } 9742 mutex_enter(&l2arc_rebuild_thr_lock); 9743 if (dev->l2ad_rebuild && !dev->l2ad_rebuild_cancel) { 9744 dev->l2ad_rebuild_began = B_TRUE; 9745 (void) thread_create(NULL, 0, l2arc_dev_rebuild_thread, 9746 dev, 0, &p0, TS_RUN, minclsyspri); 9747 } 9748 mutex_exit(&l2arc_rebuild_thr_lock); 9749 } 9750 } 9751 9752 /* 9753 * Main entry point for L2ARC rebuilding. 9754 */ 9755 static __attribute__((noreturn)) void 9756 l2arc_dev_rebuild_thread(void *arg) 9757 { 9758 l2arc_dev_t *dev = arg; 9759 9760 VERIFY(!dev->l2ad_rebuild_cancel); 9761 VERIFY(dev->l2ad_rebuild); 9762 (void) l2arc_rebuild(dev); 9763 mutex_enter(&l2arc_rebuild_thr_lock); 9764 dev->l2ad_rebuild_began = B_FALSE; 9765 dev->l2ad_rebuild = B_FALSE; 9766 mutex_exit(&l2arc_rebuild_thr_lock); 9767 9768 thread_exit(); 9769 } 9770 9771 /* 9772 * This function implements the actual L2ARC metadata rebuild. It: 9773 * starts reading the log block chain and restores each block's contents 9774 * to memory (reconstructing arc_buf_hdr_t's). 9775 * 9776 * Operation stops under any of the following conditions: 9777 * 9778 * 1) We reach the end of the log block chain. 9779 * 2) We encounter *any* error condition (cksum errors, io errors) 9780 */ 9781 static int 9782 l2arc_rebuild(l2arc_dev_t *dev) 9783 { 9784 vdev_t *vd = dev->l2ad_vdev; 9785 spa_t *spa = vd->vdev_spa; 9786 int err = 0; 9787 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 9788 l2arc_log_blk_phys_t *this_lb, *next_lb; 9789 zio_t *this_io = NULL, *next_io = NULL; 9790 l2arc_log_blkptr_t lbps[2]; 9791 l2arc_lb_ptr_buf_t *lb_ptr_buf; 9792 boolean_t lock_held; 9793 9794 this_lb = vmem_zalloc(sizeof (*this_lb), KM_SLEEP); 9795 next_lb = vmem_zalloc(sizeof (*next_lb), KM_SLEEP); 9796 9797 /* 9798 * We prevent device removal while issuing reads to the device, 9799 * then during the rebuilding phases we drop this lock again so 9800 * that a spa_unload or device remove can be initiated - this is 9801 * safe, because the spa will signal us to stop before removing 9802 * our device and wait for us to stop. 9803 */ 9804 spa_config_enter(spa, SCL_L2ARC, vd, RW_READER); 9805 lock_held = B_TRUE; 9806 9807 /* 9808 * Retrieve the persistent L2ARC device state. 9809 * L2BLK_GET_PSIZE returns aligned size for log blocks. 9810 */ 9811 dev->l2ad_evict = MAX(l2dhdr->dh_evict, dev->l2ad_start); 9812 dev->l2ad_hand = MAX(l2dhdr->dh_start_lbps[0].lbp_daddr + 9813 L2BLK_GET_PSIZE((&l2dhdr->dh_start_lbps[0])->lbp_prop), 9814 dev->l2ad_start); 9815 dev->l2ad_first = !!(l2dhdr->dh_flags & L2ARC_DEV_HDR_EVICT_FIRST); 9816 9817 vd->vdev_trim_action_time = l2dhdr->dh_trim_action_time; 9818 vd->vdev_trim_state = l2dhdr->dh_trim_state; 9819 9820 /* 9821 * In case the zfs module parameter l2arc_rebuild_enabled is false 9822 * we do not start the rebuild process. 9823 */ 9824 if (!l2arc_rebuild_enabled) 9825 goto out; 9826 9827 /* Prepare the rebuild process */ 9828 memcpy(lbps, l2dhdr->dh_start_lbps, sizeof (lbps)); 9829 9830 /* Start the rebuild process */ 9831 for (;;) { 9832 if (!l2arc_log_blkptr_valid(dev, &lbps[0])) 9833 break; 9834 9835 if ((err = l2arc_log_blk_read(dev, &lbps[0], &lbps[1], 9836 this_lb, next_lb, this_io, &next_io)) != 0) 9837 goto out; 9838 9839 /* 9840 * Our memory pressure valve. If the system is running low 9841 * on memory, rather than swamping memory with new ARC buf 9842 * hdrs, we opt not to rebuild the L2ARC. At this point, 9843 * however, we have already set up our L2ARC dev to chain in 9844 * new metadata log blocks, so the user may choose to offline/ 9845 * online the L2ARC dev at a later time (or re-import the pool) 9846 * to reconstruct it (when there's less memory pressure). 9847 */ 9848 if (l2arc_hdr_limit_reached()) { 9849 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_lowmem); 9850 cmn_err(CE_NOTE, "System running low on memory, " 9851 "aborting L2ARC rebuild."); 9852 err = SET_ERROR(ENOMEM); 9853 goto out; 9854 } 9855 9856 spa_config_exit(spa, SCL_L2ARC, vd); 9857 lock_held = B_FALSE; 9858 9859 /* 9860 * Now that we know that the next_lb checks out alright, we 9861 * can start reconstruction from this log block. 9862 * L2BLK_GET_PSIZE returns aligned size for log blocks. 9863 */ 9864 uint64_t asize = L2BLK_GET_PSIZE((&lbps[0])->lbp_prop); 9865 l2arc_log_blk_restore(dev, this_lb, asize); 9866 9867 /* 9868 * log block restored, include its pointer in the list of 9869 * pointers to log blocks present in the L2ARC device. 9870 */ 9871 lb_ptr_buf = kmem_zalloc(sizeof (l2arc_lb_ptr_buf_t), KM_SLEEP); 9872 lb_ptr_buf->lb_ptr = kmem_zalloc(sizeof (l2arc_log_blkptr_t), 9873 KM_SLEEP); 9874 memcpy(lb_ptr_buf->lb_ptr, &lbps[0], 9875 sizeof (l2arc_log_blkptr_t)); 9876 mutex_enter(&dev->l2ad_mtx); 9877 list_insert_tail(&dev->l2ad_lbptr_list, lb_ptr_buf); 9878 ARCSTAT_INCR(arcstat_l2_log_blk_asize, asize); 9879 ARCSTAT_BUMP(arcstat_l2_log_blk_count); 9880 zfs_refcount_add_many(&dev->l2ad_lb_asize, asize, lb_ptr_buf); 9881 zfs_refcount_add(&dev->l2ad_lb_count, lb_ptr_buf); 9882 mutex_exit(&dev->l2ad_mtx); 9883 vdev_space_update(vd, asize, 0, 0); 9884 9885 /* 9886 * Protection against loops of log blocks: 9887 * 9888 * l2ad_hand l2ad_evict 9889 * V V 9890 * l2ad_start |=======================================| l2ad_end 9891 * -----|||----|||---|||----||| 9892 * (3) (2) (1) (0) 9893 * ---|||---|||----|||---||| 9894 * (7) (6) (5) (4) 9895 * 9896 * In this situation the pointer of log block (4) passes 9897 * l2arc_log_blkptr_valid() but the log block should not be 9898 * restored as it is overwritten by the payload of log block 9899 * (0). Only log blocks (0)-(3) should be restored. We check 9900 * whether l2ad_evict lies in between the payload starting 9901 * offset of the next log block (lbps[1].lbp_payload_start) 9902 * and the payload starting offset of the present log block 9903 * (lbps[0].lbp_payload_start). If true and this isn't the 9904 * first pass, we are looping from the beginning and we should 9905 * stop. 9906 */ 9907 if (l2arc_range_check_overlap(lbps[1].lbp_payload_start, 9908 lbps[0].lbp_payload_start, dev->l2ad_evict) && 9909 !dev->l2ad_first) 9910 goto out; 9911 9912 kpreempt(KPREEMPT_SYNC); 9913 for (;;) { 9914 mutex_enter(&l2arc_rebuild_thr_lock); 9915 if (dev->l2ad_rebuild_cancel) { 9916 dev->l2ad_rebuild = B_FALSE; 9917 cv_signal(&l2arc_rebuild_thr_cv); 9918 mutex_exit(&l2arc_rebuild_thr_lock); 9919 err = SET_ERROR(ECANCELED); 9920 goto out; 9921 } 9922 mutex_exit(&l2arc_rebuild_thr_lock); 9923 if (spa_config_tryenter(spa, SCL_L2ARC, vd, 9924 RW_READER)) { 9925 lock_held = B_TRUE; 9926 break; 9927 } 9928 /* 9929 * L2ARC config lock held by somebody in writer, 9930 * possibly due to them trying to remove us. They'll 9931 * likely to want us to shut down, so after a little 9932 * delay, we check l2ad_rebuild_cancel and retry 9933 * the lock again. 9934 */ 9935 delay(1); 9936 } 9937 9938 /* 9939 * Continue with the next log block. 9940 */ 9941 lbps[0] = lbps[1]; 9942 lbps[1] = this_lb->lb_prev_lbp; 9943 PTR_SWAP(this_lb, next_lb); 9944 this_io = next_io; 9945 next_io = NULL; 9946 } 9947 9948 if (this_io != NULL) 9949 l2arc_log_blk_fetch_abort(this_io); 9950 out: 9951 if (next_io != NULL) 9952 l2arc_log_blk_fetch_abort(next_io); 9953 vmem_free(this_lb, sizeof (*this_lb)); 9954 vmem_free(next_lb, sizeof (*next_lb)); 9955 9956 if (!l2arc_rebuild_enabled) { 9957 spa_history_log_internal(spa, "L2ARC rebuild", NULL, 9958 "disabled"); 9959 } else if (err == 0 && zfs_refcount_count(&dev->l2ad_lb_count) > 0) { 9960 ARCSTAT_BUMP(arcstat_l2_rebuild_success); 9961 spa_history_log_internal(spa, "L2ARC rebuild", NULL, 9962 "successful, restored %llu blocks", 9963 (u_longlong_t)zfs_refcount_count(&dev->l2ad_lb_count)); 9964 } else if (err == 0 && zfs_refcount_count(&dev->l2ad_lb_count) == 0) { 9965 /* 9966 * No error but also nothing restored, meaning the lbps array 9967 * in the device header points to invalid/non-present log 9968 * blocks. Reset the header. 9969 */ 9970 spa_history_log_internal(spa, "L2ARC rebuild", NULL, 9971 "no valid log blocks"); 9972 memset(l2dhdr, 0, dev->l2ad_dev_hdr_asize); 9973 l2arc_dev_hdr_update(dev); 9974 } else if (err == ECANCELED) { 9975 /* 9976 * In case the rebuild was canceled do not log to spa history 9977 * log as the pool may be in the process of being removed. 9978 */ 9979 zfs_dbgmsg("L2ARC rebuild aborted, restored %llu blocks", 9980 (u_longlong_t)zfs_refcount_count(&dev->l2ad_lb_count)); 9981 } else if (err != 0) { 9982 spa_history_log_internal(spa, "L2ARC rebuild", NULL, 9983 "aborted, restored %llu blocks", 9984 (u_longlong_t)zfs_refcount_count(&dev->l2ad_lb_count)); 9985 } 9986 9987 if (lock_held) 9988 spa_config_exit(spa, SCL_L2ARC, vd); 9989 9990 return (err); 9991 } 9992 9993 /* 9994 * Attempts to read the device header on the provided L2ARC device and writes 9995 * it to `hdr'. On success, this function returns 0, otherwise the appropriate 9996 * error code is returned. 9997 */ 9998 static int 9999 l2arc_dev_hdr_read(l2arc_dev_t *dev) 10000 { 10001 int err; 10002 uint64_t guid; 10003 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 10004 const uint64_t l2dhdr_asize = dev->l2ad_dev_hdr_asize; 10005 abd_t *abd; 10006 10007 guid = spa_guid(dev->l2ad_vdev->vdev_spa); 10008 10009 abd = abd_get_from_buf(l2dhdr, l2dhdr_asize); 10010 10011 err = zio_wait(zio_read_phys(NULL, dev->l2ad_vdev, 10012 VDEV_LABEL_START_SIZE, l2dhdr_asize, abd, 10013 ZIO_CHECKSUM_LABEL, NULL, NULL, ZIO_PRIORITY_SYNC_READ, 10014 ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY | 10015 ZIO_FLAG_SPECULATIVE, B_FALSE)); 10016 10017 abd_free(abd); 10018 10019 if (err != 0) { 10020 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_dh_errors); 10021 zfs_dbgmsg("L2ARC IO error (%d) while reading device header, " 10022 "vdev guid: %llu", err, 10023 (u_longlong_t)dev->l2ad_vdev->vdev_guid); 10024 return (err); 10025 } 10026 10027 if (l2dhdr->dh_magic == BSWAP_64(L2ARC_DEV_HDR_MAGIC)) 10028 byteswap_uint64_array(l2dhdr, sizeof (*l2dhdr)); 10029 10030 if (l2dhdr->dh_magic != L2ARC_DEV_HDR_MAGIC || 10031 l2dhdr->dh_spa_guid != guid || 10032 l2dhdr->dh_vdev_guid != dev->l2ad_vdev->vdev_guid || 10033 l2dhdr->dh_version != L2ARC_PERSISTENT_VERSION || 10034 l2dhdr->dh_log_entries != dev->l2ad_log_entries || 10035 l2dhdr->dh_end != dev->l2ad_end || 10036 !l2arc_range_check_overlap(dev->l2ad_start, dev->l2ad_end, 10037 l2dhdr->dh_evict) || 10038 (l2dhdr->dh_trim_state != VDEV_TRIM_COMPLETE && 10039 l2arc_trim_ahead > 0)) { 10040 /* 10041 * Attempt to rebuild a device containing no actual dev hdr 10042 * or containing a header from some other pool or from another 10043 * version of persistent L2ARC. 10044 */ 10045 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_unsupported); 10046 return (SET_ERROR(ENOTSUP)); 10047 } 10048 10049 return (0); 10050 } 10051 10052 /* 10053 * Reads L2ARC log blocks from storage and validates their contents. 10054 * 10055 * This function implements a simple fetcher to make sure that while 10056 * we're processing one buffer the L2ARC is already fetching the next 10057 * one in the chain. 10058 * 10059 * The arguments this_lp and next_lp point to the current and next log block 10060 * address in the block chain. Similarly, this_lb and next_lb hold the 10061 * l2arc_log_blk_phys_t's of the current and next L2ARC blk. 10062 * 10063 * The `this_io' and `next_io' arguments are used for block fetching. 10064 * When issuing the first blk IO during rebuild, you should pass NULL for 10065 * `this_io'. This function will then issue a sync IO to read the block and 10066 * also issue an async IO to fetch the next block in the block chain. The 10067 * fetched IO is returned in `next_io'. On subsequent calls to this 10068 * function, pass the value returned in `next_io' from the previous call 10069 * as `this_io' and a fresh `next_io' pointer to hold the next fetch IO. 10070 * Prior to the call, you should initialize your `next_io' pointer to be 10071 * NULL. If no fetch IO was issued, the pointer is left set at NULL. 10072 * 10073 * On success, this function returns 0, otherwise it returns an appropriate 10074 * error code. On error the fetching IO is aborted and cleared before 10075 * returning from this function. Therefore, if we return `success', the 10076 * caller can assume that we have taken care of cleanup of fetch IOs. 10077 */ 10078 static int 10079 l2arc_log_blk_read(l2arc_dev_t *dev, 10080 const l2arc_log_blkptr_t *this_lbp, const l2arc_log_blkptr_t *next_lbp, 10081 l2arc_log_blk_phys_t *this_lb, l2arc_log_blk_phys_t *next_lb, 10082 zio_t *this_io, zio_t **next_io) 10083 { 10084 int err = 0; 10085 zio_cksum_t cksum; 10086 abd_t *abd = NULL; 10087 uint64_t asize; 10088 10089 ASSERT(this_lbp != NULL && next_lbp != NULL); 10090 ASSERT(this_lb != NULL && next_lb != NULL); 10091 ASSERT(next_io != NULL && *next_io == NULL); 10092 ASSERT(l2arc_log_blkptr_valid(dev, this_lbp)); 10093 10094 /* 10095 * Check to see if we have issued the IO for this log block in a 10096 * previous run. If not, this is the first call, so issue it now. 10097 */ 10098 if (this_io == NULL) { 10099 this_io = l2arc_log_blk_fetch(dev->l2ad_vdev, this_lbp, 10100 this_lb); 10101 } 10102 10103 /* 10104 * Peek to see if we can start issuing the next IO immediately. 10105 */ 10106 if (l2arc_log_blkptr_valid(dev, next_lbp)) { 10107 /* 10108 * Start issuing IO for the next log block early - this 10109 * should help keep the L2ARC device busy while we 10110 * decompress and restore this log block. 10111 */ 10112 *next_io = l2arc_log_blk_fetch(dev->l2ad_vdev, next_lbp, 10113 next_lb); 10114 } 10115 10116 /* Wait for the IO to read this log block to complete */ 10117 if ((err = zio_wait(this_io)) != 0) { 10118 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_io_errors); 10119 zfs_dbgmsg("L2ARC IO error (%d) while reading log block, " 10120 "offset: %llu, vdev guid: %llu", err, 10121 (u_longlong_t)this_lbp->lbp_daddr, 10122 (u_longlong_t)dev->l2ad_vdev->vdev_guid); 10123 goto cleanup; 10124 } 10125 10126 /* 10127 * Make sure the buffer checks out. 10128 * L2BLK_GET_PSIZE returns aligned size for log blocks. 10129 */ 10130 asize = L2BLK_GET_PSIZE((this_lbp)->lbp_prop); 10131 fletcher_4_native(this_lb, asize, NULL, &cksum); 10132 if (!ZIO_CHECKSUM_EQUAL(cksum, this_lbp->lbp_cksum)) { 10133 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_cksum_lb_errors); 10134 zfs_dbgmsg("L2ARC log block cksum failed, offset: %llu, " 10135 "vdev guid: %llu, l2ad_hand: %llu, l2ad_evict: %llu", 10136 (u_longlong_t)this_lbp->lbp_daddr, 10137 (u_longlong_t)dev->l2ad_vdev->vdev_guid, 10138 (u_longlong_t)dev->l2ad_hand, 10139 (u_longlong_t)dev->l2ad_evict); 10140 err = SET_ERROR(ECKSUM); 10141 goto cleanup; 10142 } 10143 10144 /* Now we can take our time decoding this buffer */ 10145 switch (L2BLK_GET_COMPRESS((this_lbp)->lbp_prop)) { 10146 case ZIO_COMPRESS_OFF: 10147 break; 10148 case ZIO_COMPRESS_LZ4: 10149 abd = abd_alloc_for_io(asize, B_TRUE); 10150 abd_copy_from_buf_off(abd, this_lb, 0, asize); 10151 if ((err = zio_decompress_data( 10152 L2BLK_GET_COMPRESS((this_lbp)->lbp_prop), 10153 abd, this_lb, asize, sizeof (*this_lb), NULL)) != 0) { 10154 err = SET_ERROR(EINVAL); 10155 goto cleanup; 10156 } 10157 break; 10158 default: 10159 err = SET_ERROR(EINVAL); 10160 goto cleanup; 10161 } 10162 if (this_lb->lb_magic == BSWAP_64(L2ARC_LOG_BLK_MAGIC)) 10163 byteswap_uint64_array(this_lb, sizeof (*this_lb)); 10164 if (this_lb->lb_magic != L2ARC_LOG_BLK_MAGIC) { 10165 err = SET_ERROR(EINVAL); 10166 goto cleanup; 10167 } 10168 cleanup: 10169 /* Abort an in-flight fetch I/O in case of error */ 10170 if (err != 0 && *next_io != NULL) { 10171 l2arc_log_blk_fetch_abort(*next_io); 10172 *next_io = NULL; 10173 } 10174 if (abd != NULL) 10175 abd_free(abd); 10176 return (err); 10177 } 10178 10179 /* 10180 * Restores the payload of a log block to ARC. This creates empty ARC hdr 10181 * entries which only contain an l2arc hdr, essentially restoring the 10182 * buffers to their L2ARC evicted state. This function also updates space 10183 * usage on the L2ARC vdev to make sure it tracks restored buffers. 10184 */ 10185 static void 10186 l2arc_log_blk_restore(l2arc_dev_t *dev, const l2arc_log_blk_phys_t *lb, 10187 uint64_t lb_asize) 10188 { 10189 uint64_t size = 0, asize = 0; 10190 uint64_t log_entries = dev->l2ad_log_entries; 10191 10192 /* 10193 * Usually arc_adapt() is called only for data, not headers, but 10194 * since we may allocate significant amount of memory here, let ARC 10195 * grow its arc_c. 10196 */ 10197 arc_adapt(log_entries * HDR_L2ONLY_SIZE); 10198 10199 for (int i = log_entries - 1; i >= 0; i--) { 10200 /* 10201 * Restore goes in the reverse temporal direction to preserve 10202 * correct temporal ordering of buffers in the l2ad_buflist. 10203 * l2arc_hdr_restore also does a list_insert_tail instead of 10204 * list_insert_head on the l2ad_buflist: 10205 * 10206 * LIST l2ad_buflist LIST 10207 * HEAD <------ (time) ------ TAIL 10208 * direction +-----+-----+-----+-----+-----+ direction 10209 * of l2arc <== | buf | buf | buf | buf | buf | ===> of rebuild 10210 * fill +-----+-----+-----+-----+-----+ 10211 * ^ ^ 10212 * | | 10213 * | | 10214 * l2arc_feed_thread l2arc_rebuild 10215 * will place new bufs here restores bufs here 10216 * 10217 * During l2arc_rebuild() the device is not used by 10218 * l2arc_feed_thread() as dev->l2ad_rebuild is set to true. 10219 */ 10220 size += L2BLK_GET_LSIZE((&lb->lb_entries[i])->le_prop); 10221 asize += vdev_psize_to_asize(dev->l2ad_vdev, 10222 L2BLK_GET_PSIZE((&lb->lb_entries[i])->le_prop)); 10223 l2arc_hdr_restore(&lb->lb_entries[i], dev); 10224 } 10225 10226 /* 10227 * Record rebuild stats: 10228 * size Logical size of restored buffers in the L2ARC 10229 * asize Aligned size of restored buffers in the L2ARC 10230 */ 10231 ARCSTAT_INCR(arcstat_l2_rebuild_size, size); 10232 ARCSTAT_INCR(arcstat_l2_rebuild_asize, asize); 10233 ARCSTAT_INCR(arcstat_l2_rebuild_bufs, log_entries); 10234 ARCSTAT_F_AVG(arcstat_l2_log_blk_avg_asize, lb_asize); 10235 ARCSTAT_F_AVG(arcstat_l2_data_to_meta_ratio, asize / lb_asize); 10236 ARCSTAT_BUMP(arcstat_l2_rebuild_log_blks); 10237 } 10238 10239 /* 10240 * Restores a single ARC buf hdr from a log entry. The ARC buffer is put 10241 * into a state indicating that it has been evicted to L2ARC. 10242 */ 10243 static void 10244 l2arc_hdr_restore(const l2arc_log_ent_phys_t *le, l2arc_dev_t *dev) 10245 { 10246 arc_buf_hdr_t *hdr, *exists; 10247 kmutex_t *hash_lock; 10248 arc_buf_contents_t type = L2BLK_GET_TYPE((le)->le_prop); 10249 uint64_t asize; 10250 10251 /* 10252 * Do all the allocation before grabbing any locks, this lets us 10253 * sleep if memory is full and we don't have to deal with failed 10254 * allocations. 10255 */ 10256 hdr = arc_buf_alloc_l2only(L2BLK_GET_LSIZE((le)->le_prop), type, 10257 dev, le->le_dva, le->le_daddr, 10258 L2BLK_GET_PSIZE((le)->le_prop), le->le_birth, 10259 L2BLK_GET_COMPRESS((le)->le_prop), le->le_complevel, 10260 L2BLK_GET_PROTECTED((le)->le_prop), 10261 L2BLK_GET_PREFETCH((le)->le_prop), 10262 L2BLK_GET_STATE((le)->le_prop)); 10263 asize = vdev_psize_to_asize(dev->l2ad_vdev, 10264 L2BLK_GET_PSIZE((le)->le_prop)); 10265 10266 /* 10267 * vdev_space_update() has to be called before arc_hdr_destroy() to 10268 * avoid underflow since the latter also calls vdev_space_update(). 10269 */ 10270 l2arc_hdr_arcstats_increment(hdr); 10271 vdev_space_update(dev->l2ad_vdev, asize, 0, 0); 10272 10273 mutex_enter(&dev->l2ad_mtx); 10274 list_insert_tail(&dev->l2ad_buflist, hdr); 10275 (void) zfs_refcount_add_many(&dev->l2ad_alloc, arc_hdr_size(hdr), hdr); 10276 mutex_exit(&dev->l2ad_mtx); 10277 10278 exists = buf_hash_insert(hdr, &hash_lock); 10279 if (exists) { 10280 /* Buffer was already cached, no need to restore it. */ 10281 arc_hdr_destroy(hdr); 10282 /* 10283 * If the buffer is already cached, check whether it has 10284 * L2ARC metadata. If not, enter them and update the flag. 10285 * This is important is case of onlining a cache device, since 10286 * we previously evicted all L2ARC metadata from ARC. 10287 */ 10288 if (!HDR_HAS_L2HDR(exists)) { 10289 arc_hdr_set_flags(exists, ARC_FLAG_HAS_L2HDR); 10290 exists->b_l2hdr.b_dev = dev; 10291 exists->b_l2hdr.b_daddr = le->le_daddr; 10292 exists->b_l2hdr.b_arcs_state = 10293 L2BLK_GET_STATE((le)->le_prop); 10294 mutex_enter(&dev->l2ad_mtx); 10295 list_insert_tail(&dev->l2ad_buflist, exists); 10296 (void) zfs_refcount_add_many(&dev->l2ad_alloc, 10297 arc_hdr_size(exists), exists); 10298 mutex_exit(&dev->l2ad_mtx); 10299 l2arc_hdr_arcstats_increment(exists); 10300 vdev_space_update(dev->l2ad_vdev, asize, 0, 0); 10301 } 10302 ARCSTAT_BUMP(arcstat_l2_rebuild_bufs_precached); 10303 } 10304 10305 mutex_exit(hash_lock); 10306 } 10307 10308 /* 10309 * Starts an asynchronous read IO to read a log block. This is used in log 10310 * block reconstruction to start reading the next block before we are done 10311 * decoding and reconstructing the current block, to keep the l2arc device 10312 * nice and hot with read IO to process. 10313 * The returned zio will contain a newly allocated memory buffers for the IO 10314 * data which should then be freed by the caller once the zio is no longer 10315 * needed (i.e. due to it having completed). If you wish to abort this 10316 * zio, you should do so using l2arc_log_blk_fetch_abort, which takes 10317 * care of disposing of the allocated buffers correctly. 10318 */ 10319 static zio_t * 10320 l2arc_log_blk_fetch(vdev_t *vd, const l2arc_log_blkptr_t *lbp, 10321 l2arc_log_blk_phys_t *lb) 10322 { 10323 uint32_t asize; 10324 zio_t *pio; 10325 l2arc_read_callback_t *cb; 10326 10327 /* L2BLK_GET_PSIZE returns aligned size for log blocks */ 10328 asize = L2BLK_GET_PSIZE((lbp)->lbp_prop); 10329 ASSERT(asize <= sizeof (l2arc_log_blk_phys_t)); 10330 10331 cb = kmem_zalloc(sizeof (l2arc_read_callback_t), KM_SLEEP); 10332 cb->l2rcb_abd = abd_get_from_buf(lb, asize); 10333 pio = zio_root(vd->vdev_spa, l2arc_blk_fetch_done, cb, 10334 ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY); 10335 (void) zio_nowait(zio_read_phys(pio, vd, lbp->lbp_daddr, asize, 10336 cb->l2rcb_abd, ZIO_CHECKSUM_OFF, NULL, NULL, 10337 ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL | 10338 ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY, B_FALSE)); 10339 10340 return (pio); 10341 } 10342 10343 /* 10344 * Aborts a zio returned from l2arc_log_blk_fetch and frees the data 10345 * buffers allocated for it. 10346 */ 10347 static void 10348 l2arc_log_blk_fetch_abort(zio_t *zio) 10349 { 10350 (void) zio_wait(zio); 10351 } 10352 10353 /* 10354 * Creates a zio to update the device header on an l2arc device. 10355 */ 10356 void 10357 l2arc_dev_hdr_update(l2arc_dev_t *dev) 10358 { 10359 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 10360 const uint64_t l2dhdr_asize = dev->l2ad_dev_hdr_asize; 10361 abd_t *abd; 10362 int err; 10363 10364 VERIFY(spa_config_held(dev->l2ad_spa, SCL_STATE_ALL, RW_READER)); 10365 10366 l2dhdr->dh_magic = L2ARC_DEV_HDR_MAGIC; 10367 l2dhdr->dh_version = L2ARC_PERSISTENT_VERSION; 10368 l2dhdr->dh_spa_guid = spa_guid(dev->l2ad_vdev->vdev_spa); 10369 l2dhdr->dh_vdev_guid = dev->l2ad_vdev->vdev_guid; 10370 l2dhdr->dh_log_entries = dev->l2ad_log_entries; 10371 l2dhdr->dh_evict = dev->l2ad_evict; 10372 l2dhdr->dh_start = dev->l2ad_start; 10373 l2dhdr->dh_end = dev->l2ad_end; 10374 l2dhdr->dh_lb_asize = zfs_refcount_count(&dev->l2ad_lb_asize); 10375 l2dhdr->dh_lb_count = zfs_refcount_count(&dev->l2ad_lb_count); 10376 l2dhdr->dh_flags = 0; 10377 l2dhdr->dh_trim_action_time = dev->l2ad_vdev->vdev_trim_action_time; 10378 l2dhdr->dh_trim_state = dev->l2ad_vdev->vdev_trim_state; 10379 if (dev->l2ad_first) 10380 l2dhdr->dh_flags |= L2ARC_DEV_HDR_EVICT_FIRST; 10381 10382 abd = abd_get_from_buf(l2dhdr, l2dhdr_asize); 10383 10384 err = zio_wait(zio_write_phys(NULL, dev->l2ad_vdev, 10385 VDEV_LABEL_START_SIZE, l2dhdr_asize, abd, ZIO_CHECKSUM_LABEL, NULL, 10386 NULL, ZIO_PRIORITY_ASYNC_WRITE, ZIO_FLAG_CANFAIL, B_FALSE)); 10387 10388 abd_free(abd); 10389 10390 if (err != 0) { 10391 zfs_dbgmsg("L2ARC IO error (%d) while writing device header, " 10392 "vdev guid: %llu", err, 10393 (u_longlong_t)dev->l2ad_vdev->vdev_guid); 10394 } 10395 } 10396 10397 /* 10398 * Commits a log block to the L2ARC device. This routine is invoked from 10399 * l2arc_write_buffers when the log block fills up. 10400 * This function allocates some memory to temporarily hold the serialized 10401 * buffer to be written. This is then released in l2arc_write_done. 10402 */ 10403 static uint64_t 10404 l2arc_log_blk_commit(l2arc_dev_t *dev, zio_t *pio, l2arc_write_callback_t *cb) 10405 { 10406 l2arc_log_blk_phys_t *lb = &dev->l2ad_log_blk; 10407 l2arc_dev_hdr_phys_t *l2dhdr = dev->l2ad_dev_hdr; 10408 uint64_t psize, asize; 10409 zio_t *wzio; 10410 l2arc_lb_abd_buf_t *abd_buf; 10411 uint8_t *tmpbuf = NULL; 10412 l2arc_lb_ptr_buf_t *lb_ptr_buf; 10413 10414 VERIFY3S(dev->l2ad_log_ent_idx, ==, dev->l2ad_log_entries); 10415 10416 abd_buf = zio_buf_alloc(sizeof (*abd_buf)); 10417 abd_buf->abd = abd_get_from_buf(lb, sizeof (*lb)); 10418 lb_ptr_buf = kmem_zalloc(sizeof (l2arc_lb_ptr_buf_t), KM_SLEEP); 10419 lb_ptr_buf->lb_ptr = kmem_zalloc(sizeof (l2arc_log_blkptr_t), KM_SLEEP); 10420 10421 /* link the buffer into the block chain */ 10422 lb->lb_prev_lbp = l2dhdr->dh_start_lbps[1]; 10423 lb->lb_magic = L2ARC_LOG_BLK_MAGIC; 10424 10425 /* 10426 * l2arc_log_blk_commit() may be called multiple times during a single 10427 * l2arc_write_buffers() call. Save the allocated abd buffers in a list 10428 * so we can free them in l2arc_write_done() later on. 10429 */ 10430 list_insert_tail(&cb->l2wcb_abd_list, abd_buf); 10431 10432 /* try to compress the buffer */ 10433 psize = zio_compress_data(ZIO_COMPRESS_LZ4, 10434 abd_buf->abd, (void **) &tmpbuf, sizeof (*lb), 0); 10435 10436 /* a log block is never entirely zero */ 10437 ASSERT(psize != 0); 10438 asize = vdev_psize_to_asize(dev->l2ad_vdev, psize); 10439 ASSERT(asize <= sizeof (*lb)); 10440 10441 /* 10442 * Update the start log block pointer in the device header to point 10443 * to the log block we're about to write. 10444 */ 10445 l2dhdr->dh_start_lbps[1] = l2dhdr->dh_start_lbps[0]; 10446 l2dhdr->dh_start_lbps[0].lbp_daddr = dev->l2ad_hand; 10447 l2dhdr->dh_start_lbps[0].lbp_payload_asize = 10448 dev->l2ad_log_blk_payload_asize; 10449 l2dhdr->dh_start_lbps[0].lbp_payload_start = 10450 dev->l2ad_log_blk_payload_start; 10451 L2BLK_SET_LSIZE( 10452 (&l2dhdr->dh_start_lbps[0])->lbp_prop, sizeof (*lb)); 10453 L2BLK_SET_PSIZE( 10454 (&l2dhdr->dh_start_lbps[0])->lbp_prop, asize); 10455 L2BLK_SET_CHECKSUM( 10456 (&l2dhdr->dh_start_lbps[0])->lbp_prop, 10457 ZIO_CHECKSUM_FLETCHER_4); 10458 if (asize < sizeof (*lb)) { 10459 /* compression succeeded */ 10460 memset(tmpbuf + psize, 0, asize - psize); 10461 L2BLK_SET_COMPRESS( 10462 (&l2dhdr->dh_start_lbps[0])->lbp_prop, 10463 ZIO_COMPRESS_LZ4); 10464 } else { 10465 /* compression failed */ 10466 memcpy(tmpbuf, lb, sizeof (*lb)); 10467 L2BLK_SET_COMPRESS( 10468 (&l2dhdr->dh_start_lbps[0])->lbp_prop, 10469 ZIO_COMPRESS_OFF); 10470 } 10471 10472 /* checksum what we're about to write */ 10473 fletcher_4_native(tmpbuf, asize, NULL, 10474 &l2dhdr->dh_start_lbps[0].lbp_cksum); 10475 10476 abd_free(abd_buf->abd); 10477 10478 /* perform the write itself */ 10479 abd_buf->abd = abd_get_from_buf(tmpbuf, sizeof (*lb)); 10480 abd_take_ownership_of_buf(abd_buf->abd, B_TRUE); 10481 wzio = zio_write_phys(pio, dev->l2ad_vdev, dev->l2ad_hand, 10482 asize, abd_buf->abd, ZIO_CHECKSUM_OFF, NULL, NULL, 10483 ZIO_PRIORITY_ASYNC_WRITE, ZIO_FLAG_CANFAIL, B_FALSE); 10484 DTRACE_PROBE2(l2arc__write, vdev_t *, dev->l2ad_vdev, zio_t *, wzio); 10485 (void) zio_nowait(wzio); 10486 10487 dev->l2ad_hand += asize; 10488 /* 10489 * Include the committed log block's pointer in the list of pointers 10490 * to log blocks present in the L2ARC device. 10491 */ 10492 memcpy(lb_ptr_buf->lb_ptr, &l2dhdr->dh_start_lbps[0], 10493 sizeof (l2arc_log_blkptr_t)); 10494 mutex_enter(&dev->l2ad_mtx); 10495 list_insert_head(&dev->l2ad_lbptr_list, lb_ptr_buf); 10496 ARCSTAT_INCR(arcstat_l2_log_blk_asize, asize); 10497 ARCSTAT_BUMP(arcstat_l2_log_blk_count); 10498 zfs_refcount_add_many(&dev->l2ad_lb_asize, asize, lb_ptr_buf); 10499 zfs_refcount_add(&dev->l2ad_lb_count, lb_ptr_buf); 10500 mutex_exit(&dev->l2ad_mtx); 10501 vdev_space_update(dev->l2ad_vdev, asize, 0, 0); 10502 10503 /* bump the kstats */ 10504 ARCSTAT_INCR(arcstat_l2_write_bytes, asize); 10505 ARCSTAT_BUMP(arcstat_l2_log_blk_writes); 10506 ARCSTAT_F_AVG(arcstat_l2_log_blk_avg_asize, asize); 10507 ARCSTAT_F_AVG(arcstat_l2_data_to_meta_ratio, 10508 dev->l2ad_log_blk_payload_asize / asize); 10509 10510 /* start a new log block */ 10511 dev->l2ad_log_ent_idx = 0; 10512 dev->l2ad_log_blk_payload_asize = 0; 10513 dev->l2ad_log_blk_payload_start = 0; 10514 10515 return (asize); 10516 } 10517 10518 /* 10519 * Validates an L2ARC log block address to make sure that it can be read 10520 * from the provided L2ARC device. 10521 */ 10522 boolean_t 10523 l2arc_log_blkptr_valid(l2arc_dev_t *dev, const l2arc_log_blkptr_t *lbp) 10524 { 10525 /* L2BLK_GET_PSIZE returns aligned size for log blocks */ 10526 uint64_t asize = L2BLK_GET_PSIZE((lbp)->lbp_prop); 10527 uint64_t end = lbp->lbp_daddr + asize - 1; 10528 uint64_t start = lbp->lbp_payload_start; 10529 boolean_t evicted = B_FALSE; 10530 10531 /* 10532 * A log block is valid if all of the following conditions are true: 10533 * - it fits entirely (including its payload) between l2ad_start and 10534 * l2ad_end 10535 * - it has a valid size 10536 * - neither the log block itself nor part of its payload was evicted 10537 * by l2arc_evict(): 10538 * 10539 * l2ad_hand l2ad_evict 10540 * | | lbp_daddr 10541 * | start | | end 10542 * | | | | | 10543 * V V V V V 10544 * l2ad_start ============================================ l2ad_end 10545 * --------------------------|||| 10546 * ^ ^ 10547 * | log block 10548 * payload 10549 */ 10550 10551 evicted = 10552 l2arc_range_check_overlap(start, end, dev->l2ad_hand) || 10553 l2arc_range_check_overlap(start, end, dev->l2ad_evict) || 10554 l2arc_range_check_overlap(dev->l2ad_hand, dev->l2ad_evict, start) || 10555 l2arc_range_check_overlap(dev->l2ad_hand, dev->l2ad_evict, end); 10556 10557 return (start >= dev->l2ad_start && end <= dev->l2ad_end && 10558 asize > 0 && asize <= sizeof (l2arc_log_blk_phys_t) && 10559 (!evicted || dev->l2ad_first)); 10560 } 10561 10562 /* 10563 * Inserts ARC buffer header `hdr' into the current L2ARC log block on 10564 * the device. The buffer being inserted must be present in L2ARC. 10565 * Returns B_TRUE if the L2ARC log block is full and needs to be committed 10566 * to L2ARC, or B_FALSE if it still has room for more ARC buffers. 10567 */ 10568 static boolean_t 10569 l2arc_log_blk_insert(l2arc_dev_t *dev, const arc_buf_hdr_t *hdr) 10570 { 10571 l2arc_log_blk_phys_t *lb = &dev->l2ad_log_blk; 10572 l2arc_log_ent_phys_t *le; 10573 10574 if (dev->l2ad_log_entries == 0) 10575 return (B_FALSE); 10576 10577 int index = dev->l2ad_log_ent_idx++; 10578 10579 ASSERT3S(index, <, dev->l2ad_log_entries); 10580 ASSERT(HDR_HAS_L2HDR(hdr)); 10581 10582 le = &lb->lb_entries[index]; 10583 memset(le, 0, sizeof (*le)); 10584 le->le_dva = hdr->b_dva; 10585 le->le_birth = hdr->b_birth; 10586 le->le_daddr = hdr->b_l2hdr.b_daddr; 10587 if (index == 0) 10588 dev->l2ad_log_blk_payload_start = le->le_daddr; 10589 L2BLK_SET_LSIZE((le)->le_prop, HDR_GET_LSIZE(hdr)); 10590 L2BLK_SET_PSIZE((le)->le_prop, HDR_GET_PSIZE(hdr)); 10591 L2BLK_SET_COMPRESS((le)->le_prop, HDR_GET_COMPRESS(hdr)); 10592 le->le_complevel = hdr->b_complevel; 10593 L2BLK_SET_TYPE((le)->le_prop, hdr->b_type); 10594 L2BLK_SET_PROTECTED((le)->le_prop, !!(HDR_PROTECTED(hdr))); 10595 L2BLK_SET_PREFETCH((le)->le_prop, !!(HDR_PREFETCH(hdr))); 10596 L2BLK_SET_STATE((le)->le_prop, hdr->b_l2hdr.b_arcs_state); 10597 10598 dev->l2ad_log_blk_payload_asize += vdev_psize_to_asize(dev->l2ad_vdev, 10599 HDR_GET_PSIZE(hdr)); 10600 10601 return (dev->l2ad_log_ent_idx == dev->l2ad_log_entries); 10602 } 10603 10604 /* 10605 * Checks whether a given L2ARC device address sits in a time-sequential 10606 * range. The trick here is that the L2ARC is a rotary buffer, so we can't 10607 * just do a range comparison, we need to handle the situation in which the 10608 * range wraps around the end of the L2ARC device. Arguments: 10609 * bottom -- Lower end of the range to check (written to earlier). 10610 * top -- Upper end of the range to check (written to later). 10611 * check -- The address for which we want to determine if it sits in 10612 * between the top and bottom. 10613 * 10614 * The 3-way conditional below represents the following cases: 10615 * 10616 * bottom < top : Sequentially ordered case: 10617 * <check>--------+-------------------+ 10618 * | (overlap here?) | 10619 * L2ARC dev V V 10620 * |---------------<bottom>============<top>--------------| 10621 * 10622 * bottom > top: Looped-around case: 10623 * <check>--------+------------------+ 10624 * | (overlap here?) | 10625 * L2ARC dev V V 10626 * |===============<top>---------------<bottom>===========| 10627 * ^ ^ 10628 * | (or here?) | 10629 * +---------------+---------<check> 10630 * 10631 * top == bottom : Just a single address comparison. 10632 */ 10633 boolean_t 10634 l2arc_range_check_overlap(uint64_t bottom, uint64_t top, uint64_t check) 10635 { 10636 if (bottom < top) 10637 return (bottom <= check && check <= top); 10638 else if (bottom > top) 10639 return (check <= top || bottom <= check); 10640 else 10641 return (check == top); 10642 } 10643 10644 EXPORT_SYMBOL(arc_buf_size); 10645 EXPORT_SYMBOL(arc_write); 10646 EXPORT_SYMBOL(arc_read); 10647 EXPORT_SYMBOL(arc_buf_info); 10648 EXPORT_SYMBOL(arc_getbuf_func); 10649 EXPORT_SYMBOL(arc_add_prune_callback); 10650 EXPORT_SYMBOL(arc_remove_prune_callback); 10651 10652 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, min, param_set_arc_min, 10653 spl_param_get_u64, ZMOD_RW, "Minimum ARC size in bytes"); 10654 10655 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, max, param_set_arc_max, 10656 spl_param_get_u64, ZMOD_RW, "Maximum ARC size in bytes"); 10657 10658 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, meta_balance, UINT, ZMOD_RW, 10659 "Balance between metadata and data on ghost hits."); 10660 10661 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, grow_retry, param_set_arc_int, 10662 param_get_uint, ZMOD_RW, "Seconds before growing ARC size"); 10663 10664 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, shrink_shift, param_set_arc_int, 10665 param_get_uint, ZMOD_RW, "log2(fraction of ARC to reclaim)"); 10666 10667 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, pc_percent, UINT, ZMOD_RW, 10668 "Percent of pagecache to reclaim ARC to"); 10669 10670 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, average_blocksize, UINT, ZMOD_RD, 10671 "Target average block size"); 10672 10673 ZFS_MODULE_PARAM(zfs, zfs_, compressed_arc_enabled, INT, ZMOD_RW, 10674 "Disable compressed ARC buffers"); 10675 10676 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, min_prefetch_ms, param_set_arc_int, 10677 param_get_uint, ZMOD_RW, "Min life of prefetch block in ms"); 10678 10679 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, min_prescient_prefetch_ms, 10680 param_set_arc_int, param_get_uint, ZMOD_RW, 10681 "Min life of prescient prefetched block in ms"); 10682 10683 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, write_max, U64, ZMOD_RW, 10684 "Max write bytes per interval"); 10685 10686 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, write_boost, U64, ZMOD_RW, 10687 "Extra write bytes during device warmup"); 10688 10689 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, headroom, U64, ZMOD_RW, 10690 "Number of max device writes to precache"); 10691 10692 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, headroom_boost, U64, ZMOD_RW, 10693 "Compressed l2arc_headroom multiplier"); 10694 10695 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, trim_ahead, U64, ZMOD_RW, 10696 "TRIM ahead L2ARC write size multiplier"); 10697 10698 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, feed_secs, U64, ZMOD_RW, 10699 "Seconds between L2ARC writing"); 10700 10701 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, feed_min_ms, U64, ZMOD_RW, 10702 "Min feed interval in milliseconds"); 10703 10704 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, noprefetch, INT, ZMOD_RW, 10705 "Skip caching prefetched buffers"); 10706 10707 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, feed_again, INT, ZMOD_RW, 10708 "Turbo L2ARC warmup"); 10709 10710 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, norw, INT, ZMOD_RW, 10711 "No reads during writes"); 10712 10713 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, meta_percent, UINT, ZMOD_RW, 10714 "Percent of ARC size allowed for L2ARC-only headers"); 10715 10716 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, rebuild_enabled, INT, ZMOD_RW, 10717 "Rebuild the L2ARC when importing a pool"); 10718 10719 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, rebuild_blocks_min_l2size, U64, ZMOD_RW, 10720 "Min size in bytes to write rebuild log blocks in L2ARC"); 10721 10722 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, mfuonly, INT, ZMOD_RW, 10723 "Cache only MFU data from ARC into L2ARC"); 10724 10725 ZFS_MODULE_PARAM(zfs_l2arc, l2arc_, exclude_special, INT, ZMOD_RW, 10726 "Exclude dbufs on special vdevs from being cached to L2ARC if set."); 10727 10728 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, lotsfree_percent, param_set_arc_int, 10729 param_get_uint, ZMOD_RW, "System free memory I/O throttle in bytes"); 10730 10731 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, sys_free, param_set_arc_u64, 10732 spl_param_get_u64, ZMOD_RW, "System free memory target size in bytes"); 10733 10734 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, dnode_limit, param_set_arc_u64, 10735 spl_param_get_u64, ZMOD_RW, "Minimum bytes of dnodes in ARC"); 10736 10737 ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, dnode_limit_percent, 10738 param_set_arc_int, param_get_uint, ZMOD_RW, 10739 "Percent of ARC meta buffers for dnodes"); 10740 10741 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, dnode_reduce_percent, UINT, ZMOD_RW, 10742 "Percentage of excess dnodes to try to unpin"); 10743 10744 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, eviction_pct, UINT, ZMOD_RW, 10745 "When full, ARC allocation waits for eviction of this % of alloc size"); 10746 10747 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, evict_batch_limit, UINT, ZMOD_RW, 10748 "The number of headers to evict per sublist before moving to the next"); 10749 10750 ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, prune_task_threads, INT, ZMOD_RW, 10751 "Number of arc_prune threads"); 10752