1 /* 2 * CDDL HEADER START 3 * 4 * This file and its contents are supplied under the terms of the 5 * Common Development and Distribution License ("CDDL"), version 1.0. 6 * You may only use this file in accordance with the terms of version 7 * 1.0 of the CDDL. 8 * 9 * A full copy of the text of the CDDL should have accompanied this 10 * source. A copy of the CDDL is also available via the Internet at 11 * http://www.illumos.org/license/CDDL. 12 * 13 * CDDL HEADER END 14 */ 15 16 /* 17 * Copyright (c) 2017, 2020 by Delphix. All rights reserved. 18 */ 19 20 /* 21 * ZTHR Infrastructure 22 * =================== 23 * 24 * ZTHR threads are used for isolated operations that span multiple txgs 25 * within a SPA. They generally exist from SPA creation/loading and until 26 * the SPA is exported/destroyed. The ideal requirements for an operation 27 * to be modeled with a zthr are the following: 28 * 29 * 1] The operation needs to run over multiple txgs. 30 * 2] There is be a single point of reference in memory or on disk that 31 * indicates whether the operation should run/is running or has 32 * stopped. 33 * 34 * If the operation satisfies the above then the following rules guarantee 35 * a certain level of correctness: 36 * 37 * 1] Any thread EXCEPT the zthr changes the work indicator from stopped 38 * to running but not the opposite. 39 * 2] Only the zthr can change the work indicator from running to stopped 40 * (e.g. when it is done) but not the opposite. 41 * 42 * This way a normal zthr cycle should go like this: 43 * 44 * 1] An external thread changes the work indicator from stopped to 45 * running and wakes up the zthr. 46 * 2] The zthr wakes up, checks the indicator and starts working. 47 * 3] When the zthr is done, it changes the indicator to stopped, allowing 48 * a new cycle to start. 49 * 50 * Besides being awakened by other threads, a zthr can be configured 51 * during creation to wakeup on its own after a specified interval 52 * [see zthr_create_timer()]. 53 * 54 * Note: ZTHR threads are NOT a replacement for generic threads! Please 55 * ensure that they fit your use-case well before using them. 56 * 57 * == ZTHR creation 58 * 59 * Every zthr needs three inputs to start running: 60 * 61 * 1] A user-defined checker function (checkfunc) that decides whether 62 * the zthr should start working or go to sleep. The function should 63 * return TRUE when the zthr needs to work or FALSE to let it sleep, 64 * and should adhere to the following signature: 65 * boolean_t checkfunc_name(void *args, zthr_t *t); 66 * 67 * 2] A user-defined ZTHR function (func) which the zthr executes when 68 * it is not sleeping. The function should adhere to the following 69 * signature type: 70 * void func_name(void *args, zthr_t *t); 71 * 72 * 3] A void args pointer that will be passed to checkfunc and func 73 * implicitly by the infrastructure. 74 * 75 * The reason why the above API needs two different functions, 76 * instead of one that both checks and does the work, has to do with 77 * the zthr's internal state lock (zthr_state_lock) and the allowed 78 * cancellation windows. We want to hold the zthr_state_lock while 79 * running checkfunc but not while running func. This way the zthr 80 * can be cancelled while doing work and not while checking for work. 81 * 82 * To start a zthr: 83 * zthr_t *zthr_pointer = zthr_create(checkfunc, func, args); 84 * or 85 * zthr_t *zthr_pointer = zthr_create_timer(checkfunc, func, 86 * args, max_sleep); 87 * 88 * After that you should be able to wakeup, cancel, and resume the 89 * zthr from another thread using the zthr_pointer. 90 * 91 * NOTE: ZTHR threads could potentially wake up spuriously and the 92 * user should take this into account when writing a checkfunc. 93 * [see ZTHR state transitions] 94 * 95 * == ZTHR wakeup 96 * 97 * ZTHR wakeup should be used when new work is added for the zthr. The 98 * sleeping zthr will wakeup, see that it has more work to complete 99 * and proceed. This can be invoked from open or syncing context. 100 * 101 * To wakeup a zthr: 102 * zthr_wakeup(zthr_t *t) 103 * 104 * == ZTHR cancellation and resumption 105 * 106 * ZTHR threads must be cancelled when their SPA is being exported 107 * or when they need to be paused so they don't interfere with other 108 * operations. 109 * 110 * To cancel a zthr: 111 * zthr_cancel(zthr_pointer); 112 * 113 * To resume it: 114 * zthr_resume(zthr_pointer); 115 * 116 * ZTHR cancel and resume should be invoked in open context during the 117 * lifecycle of the pool as it is imported, exported or destroyed. 118 * 119 * A zthr will implicitly check if it has received a cancellation 120 * signal every time func returns and every time it wakes up [see 121 * ZTHR state transitions below]. 122 * 123 * At times, waiting for the zthr's func to finish its job may take 124 * time. This may be very time-consuming for some operations that 125 * need to cancel the SPA's zthrs (e.g spa_export). For this scenario 126 * the user can explicitly make their ZTHR function aware of incoming 127 * cancellation signals using zthr_iscancelled(). A common pattern for 128 * that looks like this: 129 * 130 * int 131 * func_name(void *args, zthr_t *t) 132 * { 133 * ... <unpack args> ... 134 * while (!work_done && !zthr_iscancelled(t)) { 135 * ... <do more work> ... 136 * } 137 * } 138 * 139 * == ZTHR cleanup 140 * 141 * Cancelling a zthr doesn't clean up its metadata (internal locks, 142 * function pointers to func and checkfunc, etc..). This is because 143 * we want to keep them around in case we want to resume the execution 144 * of the zthr later. Similarly for zthrs that exit themselves. 145 * 146 * To completely cleanup a zthr, cancel it first to ensure that it 147 * is not running and then use zthr_destroy(). 148 * 149 * == ZTHR state transitions 150 * 151 * zthr creation 152 * + 153 * | 154 * | woke up 155 * | +--------------+ sleep 156 * | | ^ 157 * | | | 158 * | | | FALSE 159 * | | | 160 * v v FALSE + 161 * cancelled? +---------> checkfunc? 162 * + ^ + 163 * | | | 164 * | | | TRUE 165 * | | | 166 * | | func returned v 167 * | +---------------+ func 168 * | 169 * | TRUE 170 * | 171 * v 172 * zthr stopped running 173 * 174 * == Implementation of ZTHR requests 175 * 176 * ZTHR cancel and resume are requests on a zthr to change its 177 * internal state. These requests are serialized using the 178 * zthr_request_lock, while changes in its internal state are 179 * protected by the zthr_state_lock. A request will first acquire 180 * the zthr_request_lock and then immediately acquire the 181 * zthr_state_lock. We do this so that incoming requests are 182 * serialized using the request lock, while still allowing us 183 * to use the state lock for thread communication via zthr_cv. 184 * 185 * ZTHR wakeup broadcasts to zthr_cv, causing sleeping threads 186 * to wakeup. It acquires the zthr_state_lock but not the 187 * zthr_request_lock, so that a wakeup on a zthr in the middle 188 * of being cancelled will not block. 189 */ 190 191 #include <sys/zfs_context.h> 192 #include <sys/zthr.h> 193 194 struct zthr { 195 /* running thread doing the work */ 196 kthread_t *zthr_thread; 197 198 /* lock protecting internal data & invariants */ 199 kmutex_t zthr_state_lock; 200 201 /* mutex that serializes external requests */ 202 kmutex_t zthr_request_lock; 203 204 /* notification mechanism for requests */ 205 kcondvar_t zthr_cv; 206 207 /* flag set to true if we are canceling the zthr */ 208 boolean_t zthr_cancel; 209 210 /* flag set to true if we are waiting for the zthr to finish */ 211 boolean_t zthr_haswaiters; 212 kcondvar_t zthr_wait_cv; 213 /* 214 * maximum amount of time that the zthr is spent sleeping; 215 * if this is 0, the thread doesn't wake up until it gets 216 * signaled. 217 */ 218 hrtime_t zthr_sleep_timeout; 219 220 /* consumer-provided callbacks & data */ 221 zthr_checkfunc_t *zthr_checkfunc; 222 zthr_func_t *zthr_func; 223 void *zthr_arg; 224 }; 225 226 static void 227 zthr_procedure(void *arg) 228 { 229 zthr_t *t = arg; 230 231 mutex_enter(&t->zthr_state_lock); 232 ASSERT3P(t->zthr_thread, ==, curthread); 233 234 while (!t->zthr_cancel) { 235 if (t->zthr_checkfunc(t->zthr_arg, t)) { 236 mutex_exit(&t->zthr_state_lock); 237 t->zthr_func(t->zthr_arg, t); 238 mutex_enter(&t->zthr_state_lock); 239 } else { 240 /* 241 * cv_wait_sig() is used instead of cv_wait() in 242 * order to prevent this process from incorrectly 243 * contributing to the system load average when idle. 244 */ 245 if (t->zthr_sleep_timeout == 0) { 246 cv_wait_sig(&t->zthr_cv, &t->zthr_state_lock); 247 } else { 248 (void) cv_timedwait_sig_hires(&t->zthr_cv, 249 &t->zthr_state_lock, t->zthr_sleep_timeout, 250 MSEC2NSEC(1), 0); 251 } 252 } 253 if (t->zthr_haswaiters) { 254 t->zthr_haswaiters = B_FALSE; 255 cv_broadcast(&t->zthr_wait_cv); 256 } 257 } 258 259 /* 260 * Clear out the kernel thread metadata and notify the 261 * zthr_cancel() thread that we've stopped running. 262 */ 263 t->zthr_thread = NULL; 264 t->zthr_cancel = B_FALSE; 265 cv_broadcast(&t->zthr_cv); 266 267 mutex_exit(&t->zthr_state_lock); 268 thread_exit(); 269 } 270 271 zthr_t * 272 zthr_create(const char *zthr_name, zthr_checkfunc_t *checkfunc, 273 zthr_func_t *func, void *arg) 274 { 275 return (zthr_create_timer(zthr_name, checkfunc, 276 func, arg, (hrtime_t)0)); 277 } 278 279 /* 280 * Create a zthr with specified maximum sleep time. If the time 281 * in sleeping state exceeds max_sleep, a wakeup(do the check and 282 * start working if required) will be triggered. 283 */ 284 zthr_t * 285 zthr_create_timer(const char *zthr_name, zthr_checkfunc_t *checkfunc, 286 zthr_func_t *func, void *arg, hrtime_t max_sleep) 287 { 288 zthr_t *t = kmem_zalloc(sizeof (*t), KM_SLEEP); 289 mutex_init(&t->zthr_state_lock, NULL, MUTEX_DEFAULT, NULL); 290 mutex_init(&t->zthr_request_lock, NULL, MUTEX_DEFAULT, NULL); 291 cv_init(&t->zthr_cv, NULL, CV_DEFAULT, NULL); 292 cv_init(&t->zthr_wait_cv, NULL, CV_DEFAULT, NULL); 293 294 mutex_enter(&t->zthr_state_lock); 295 t->zthr_checkfunc = checkfunc; 296 t->zthr_func = func; 297 t->zthr_arg = arg; 298 t->zthr_sleep_timeout = max_sleep; 299 300 t->zthr_thread = thread_create_named(zthr_name, NULL, 0, 301 zthr_procedure, t, 0, &p0, TS_RUN, minclsyspri); 302 303 mutex_exit(&t->zthr_state_lock); 304 305 return (t); 306 } 307 308 void 309 zthr_destroy(zthr_t *t) 310 { 311 ASSERT(!MUTEX_HELD(&t->zthr_state_lock)); 312 ASSERT(!MUTEX_HELD(&t->zthr_request_lock)); 313 VERIFY3P(t->zthr_thread, ==, NULL); 314 mutex_destroy(&t->zthr_request_lock); 315 mutex_destroy(&t->zthr_state_lock); 316 cv_destroy(&t->zthr_cv); 317 cv_destroy(&t->zthr_wait_cv); 318 kmem_free(t, sizeof (*t)); 319 } 320 321 /* 322 * Wake up the zthr if it is sleeping. If the thread has been cancelled 323 * or is in the process of being cancelled, this is a no-op. 324 */ 325 void 326 zthr_wakeup(zthr_t *t) 327 { 328 mutex_enter(&t->zthr_state_lock); 329 330 /* 331 * There are 5 states that we can find the zthr when issuing 332 * this broadcast: 333 * 334 * [1] The common case of the thread being asleep, at which 335 * point the broadcast will wake it up. 336 * [2] The thread has been cancelled. Waking up a cancelled 337 * thread is a no-op. Any work that is still left to be 338 * done should be handled the next time the thread is 339 * resumed. 340 * [3] The thread is doing work and is already up, so this 341 * is basically a no-op. 342 * [4] The thread was just created/resumed, in which case the 343 * behavior is similar to [3]. 344 * [5] The thread is in the middle of being cancelled, which 345 * will be a no-op. 346 */ 347 cv_broadcast(&t->zthr_cv); 348 349 mutex_exit(&t->zthr_state_lock); 350 } 351 352 /* 353 * Sends a cancel request to the zthr and blocks until the zthr is 354 * cancelled. If the zthr is not running (e.g. has been cancelled 355 * already), this is a no-op. Note that this function should not be 356 * called from syncing context as it could deadlock with the zthr_func. 357 */ 358 void 359 zthr_cancel(zthr_t *t) 360 { 361 mutex_enter(&t->zthr_request_lock); 362 mutex_enter(&t->zthr_state_lock); 363 364 /* 365 * Since we are holding the zthr_state_lock at this point 366 * we can find the state in one of the following 4 states: 367 * 368 * [1] The thread has already been cancelled, therefore 369 * there is nothing for us to do. 370 * [2] The thread is sleeping so we set the flag, broadcast 371 * the CV and wait for it to exit. 372 * [3] The thread is doing work, in which case we just set 373 * the flag and wait for it to finish. 374 * [4] The thread was just created/resumed, in which case 375 * the behavior is similar to [3]. 376 * 377 * Since requests are serialized, by the time that we get 378 * control back we expect that the zthr is cancelled and 379 * not running anymore. 380 */ 381 if (t->zthr_thread != NULL) { 382 t->zthr_cancel = B_TRUE; 383 384 /* broadcast in case the zthr is sleeping */ 385 cv_broadcast(&t->zthr_cv); 386 387 while (t->zthr_thread != NULL) 388 cv_wait(&t->zthr_cv, &t->zthr_state_lock); 389 390 ASSERT(!t->zthr_cancel); 391 } 392 393 mutex_exit(&t->zthr_state_lock); 394 mutex_exit(&t->zthr_request_lock); 395 } 396 397 /* 398 * Sends a resume request to the supplied zthr. If the zthr is already 399 * running this is a no-op. Note that this function should not be 400 * called from syncing context as it could deadlock with the zthr_func. 401 */ 402 void 403 zthr_resume(zthr_t *t) 404 { 405 mutex_enter(&t->zthr_request_lock); 406 mutex_enter(&t->zthr_state_lock); 407 408 ASSERT3P(&t->zthr_checkfunc, !=, NULL); 409 ASSERT3P(&t->zthr_func, !=, NULL); 410 ASSERT(!t->zthr_cancel); 411 ASSERT(!t->zthr_haswaiters); 412 413 /* 414 * There are 4 states that we find the zthr in at this point 415 * given the locks that we hold: 416 * 417 * [1] The zthr was cancelled, so we spawn a new thread for 418 * the zthr (common case). 419 * [2] The zthr is running at which point this is a no-op. 420 * [3] The zthr is sleeping at which point this is a no-op. 421 * [4] The zthr was just spawned at which point this is a 422 * no-op. 423 */ 424 if (t->zthr_thread == NULL) { 425 t->zthr_thread = thread_create(NULL, 0, zthr_procedure, t, 426 0, &p0, TS_RUN, minclsyspri); 427 } 428 429 mutex_exit(&t->zthr_state_lock); 430 mutex_exit(&t->zthr_request_lock); 431 } 432 433 /* 434 * This function is intended to be used by the zthr itself 435 * (specifically the zthr_func callback provided) to check 436 * if another thread has signaled it to stop running before 437 * doing some expensive operation. 438 * 439 * returns TRUE if we are in the middle of trying to cancel 440 * this thread. 441 * 442 * returns FALSE otherwise. 443 */ 444 boolean_t 445 zthr_iscancelled(zthr_t *t) 446 { 447 ASSERT3P(t->zthr_thread, ==, curthread); 448 449 /* 450 * The majority of the functions here grab zthr_request_lock 451 * first and then zthr_state_lock. This function only grabs 452 * the zthr_state_lock. That is because this function should 453 * only be called from the zthr_func to check if someone has 454 * issued a zthr_cancel() on the thread. If there is a zthr_cancel() 455 * happening concurrently, attempting to grab the request lock 456 * here would result in a deadlock. 457 * 458 * By grabbing only the zthr_state_lock this function is allowed 459 * to run concurrently with a zthr_cancel() request. 460 */ 461 mutex_enter(&t->zthr_state_lock); 462 boolean_t cancelled = t->zthr_cancel; 463 mutex_exit(&t->zthr_state_lock); 464 return (cancelled); 465 } 466 467 /* 468 * Wait for the zthr to finish its current function. Similar to 469 * zthr_iscancelled, you can use zthr_has_waiters to have the zthr_func end 470 * early. Unlike zthr_cancel, the thread is not destroyed. If the zthr was 471 * sleeping or cancelled, return immediately. 472 */ 473 void 474 zthr_wait_cycle_done(zthr_t *t) 475 { 476 mutex_enter(&t->zthr_state_lock); 477 478 /* 479 * Since we are holding the zthr_state_lock at this point 480 * we can find the state in one of the following 5 states: 481 * 482 * [1] The thread has already cancelled, therefore 483 * there is nothing for us to do. 484 * [2] The thread is sleeping so we set the flag, broadcast 485 * the CV and wait for it to exit. 486 * [3] The thread is doing work, in which case we just set 487 * the flag and wait for it to finish. 488 * [4] The thread was just created/resumed, in which case 489 * the behavior is similar to [3]. 490 * [5] The thread is the middle of being cancelled, which is 491 * similar to [3]. We'll wait for the cancel, which is 492 * waiting for the zthr func. 493 * 494 * Since requests are serialized, by the time that we get 495 * control back we expect that the zthr has completed it's 496 * zthr_func. 497 */ 498 if (t->zthr_thread != NULL) { 499 t->zthr_haswaiters = B_TRUE; 500 501 /* broadcast in case the zthr is sleeping */ 502 cv_broadcast(&t->zthr_cv); 503 504 while ((t->zthr_haswaiters) && (t->zthr_thread != NULL)) 505 cv_wait(&t->zthr_wait_cv, &t->zthr_state_lock); 506 507 ASSERT(!t->zthr_haswaiters); 508 } 509 510 mutex_exit(&t->zthr_state_lock); 511 } 512 513 /* 514 * This function is intended to be used by the zthr itself 515 * to check if another thread is waiting on it to finish 516 * 517 * returns TRUE if we have been asked to finish. 518 * 519 * returns FALSE otherwise. 520 */ 521 boolean_t 522 zthr_has_waiters(zthr_t *t) 523 { 524 ASSERT3P(t->zthr_thread, ==, curthread); 525 526 mutex_enter(&t->zthr_state_lock); 527 528 /* 529 * Similarly to zthr_iscancelled(), we only grab the 530 * zthr_state_lock so that the zthr itself can use this 531 * to check for the request. 532 */ 533 boolean_t has_waiters = t->zthr_haswaiters; 534 mutex_exit(&t->zthr_state_lock); 535 return (has_waiters); 536 } 537