1 /* 2 * CDDL HEADER START 3 * 4 * The contents of this file are subject to the terms of the 5 * Common Development and Distribution License, Version 1.0 only 6 * (the "License"). You may not use this file except in compliance 7 * with the License. 8 * 9 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 10 * or http://www.opensolaris.org/os/licensing. 11 * See the License for the specific language governing permissions 12 * and limitations under the License. 13 * 14 * When distributing Covered Code, include this CDDL HEADER in each 15 * file and include the License file at usr/src/OPENSOLARIS.LICENSE. 16 * If applicable, add the following below this CDDL HEADER, with the 17 * fields enclosed by brackets "[]" replaced with your own identifying 18 * information: Portions Copyright [yyyy] [name of copyright owner] 19 * 20 * CDDL HEADER END 21 */ 22 /* 23 * Copyright 2006 Sun Microsystems, Inc. All rights reserved. 24 * Use is subject to license terms. 25 */ 26 27 #pragma ident "%Z%%M% %I% %E% SMI" 28 29 /* 30 * Page Retire - Big Theory Statement. 31 * 32 * This file handles removing sections of faulty memory from use when the 33 * user land FMA Diagnosis Engine requests that a page be removed or when 34 * a CE or UE is detected by the hardware. 35 * 36 * In the bad old days, the kernel side of Page Retire did a lot of the work 37 * on its own. Now, with the DE keeping track of errors, the kernel side is 38 * rather simple minded on most platforms. 39 * 40 * Errors are all reflected to the DE, and after digesting the error and 41 * looking at all previously reported errors, the DE decides what should 42 * be done about the current error. If the DE wants a particular page to 43 * be retired, then the kernel page retire code is invoked via an ioctl. 44 * On non-FMA platforms, the ue_drain and ce_drain paths ends up calling 45 * page retire to handle the error. Since page retire is just a simple 46 * mechanism it doesn't need to differentiate between the different callers. 47 * 48 * The p_toxic field in the page_t is used to indicate which errors have 49 * occurred and what action has been taken on a given page. Because errors are 50 * reported without regard to the locked state of a page, no locks are used 51 * to SET the error bits in p_toxic. However, in order to clear the error 52 * bits, the page_t must be held exclusively locked. 53 * 54 * When page_retire() is called, it must be able to acquire locks, sleep, etc. 55 * It must not be called from high-level interrupt context. 56 * 57 * Depending on how the requested page is being used at the time of the retire 58 * request (and on the availability of sufficient system resources), the page 59 * may be retired immediately, or just marked for retirement later. For 60 * example, locked pages are marked, while free pages are retired. Multiple 61 * requests may be made to retire the same page, although there is no need 62 * to: once the p_toxic flags are set, the page will be retired as soon as it 63 * can be exclusively locked. 64 * 65 * The retire mechanism is driven centrally out of page_unlock(). To expedite 66 * the retirement of pages, further requests for SE_SHARED locks are denied 67 * as long as a page retirement is pending. In addition, as long as pages are 68 * pending retirement a background thread runs periodically trying to retire 69 * those pages. Pages which could not be retired while the system is running 70 * are scrubbed prior to rebooting to avoid latent errors on the next boot. 71 * 72 * UE pages without persistent errors are scrubbed and returned to service. 73 * Recidivist pages, as well as FMA-directed requests for retirement, result 74 * in the page being taken out of service. Once the decision is made to take 75 * a page out of service, the page is cleared, hashed onto the retired_pages 76 * vnode, marked as retired, and it is unlocked. No other requesters (except 77 * for unretire) are allowed to lock retired pages. 78 * 79 * The public routines return (sadly) 0 if they worked and a non-zero error 80 * value if something went wrong. This is done for the ioctl side of the 81 * world to allow errors to be reflected all the way out to user land. The 82 * non-zero values are explained in comments atop each function. 83 */ 84 85 /* 86 * Things to fix: 87 * 88 * 1. Cleanup SE_EWANTED. Since we're aggressive about trying to retire 89 * pages, we can use page_retire_pp() to replace SE_EWANTED and all 90 * the special delete_memory_thread() code just goes away. 91 * 92 * 2. Trying to retire non-relocatable kvp pages may result in a 93 * quagmire. This is because seg_kmem() no longer keeps its pages locked, 94 * and calls page_lookup() in the free path; since kvp pages are modified 95 * and don't have a usable backing store, page_retire() can't do anything 96 * with them, and we'll keep denying the lock to seg_kmem_free() in a 97 * vicious cycle. To prevent that, we don't deny locks to kvp pages, and 98 * hence only call page_retire_pp() from page_unlock() in the free path. 99 * Since most kernel pages are indefinitely held anyway, and don't 100 * participate in I/O, this is of little consequence. 101 * 102 * 3. Low memory situations will be interesting. If we don't have 103 * enough memory for page_relocate() to succeed, we won't be able to 104 * retire dirty pages; nobody will be able to push them out to disk 105 * either, since we aggressively deny the page lock. We could change 106 * fsflush so it can recognize this situation, grab the lock, and push 107 * the page out, where we'll catch it in the free path and retire it. 108 * 109 * 4. Beware of places that have code like this in them: 110 * 111 * if (! page_tryupgrade(pp)) { 112 * page_unlock(pp); 113 * while (! page_lock(pp, SE_EXCL, NULL, P_RECLAIM)) { 114 * / *NOTHING* / 115 * } 116 * } 117 * page_free(pp); 118 * 119 * The problem is that pp can change identity right after the 120 * page_unlock() call. In particular, page_retire() can step in 121 * there, change pp's identity, and hash pp onto the retired_vnode. 122 * 123 * Of course, other functions besides page_retire() can have the 124 * same effect. A kmem reader can waltz by, set up a mapping to the 125 * page, and then unlock the page. Page_free() will then go castors 126 * up. So if anybody is doing this, it's already a bug. 127 * 128 * 5. mdboot()'s call into page_retire_hunt() should probably be 129 * moved lower. Where the call is made now, we can get into trouble 130 * by scrubbing a kernel page that is then accessed later. 131 */ 132 133 #include <sys/types.h> 134 #include <sys/param.h> 135 #include <sys/systm.h> 136 #include <sys/mman.h> 137 #include <sys/vnode.h> 138 #include <sys/cmn_err.h> 139 #include <sys/ksynch.h> 140 #include <sys/thread.h> 141 #include <sys/disp.h> 142 #include <sys/ontrap.h> 143 #include <sys/vmsystm.h> 144 #include <sys/mem_config.h> 145 #include <sys/atomic.h> 146 #include <sys/callb.h> 147 #include <vm/page.h> 148 #include <vm/vm_dep.h> 149 #include <vm/as.h> 150 #include <vm/hat.h> 151 152 /* 153 * vnode for all pages which are retired from the VM system; 154 */ 155 vnode_t *retired_pages; 156 157 /* 158 * Background thread that wakes up periodically to try to retire pending 159 * pages. This prevents threads from becoming blocked indefinitely in 160 * page_lookup() or some other routine should the page(s) they are waiting 161 * on become eligible for social security. 162 */ 163 static void page_retire_thread(void); 164 static kthread_t *pr_thread_id; 165 static kcondvar_t pr_cv; 166 static kmutex_t pr_thread_mutex; 167 static clock_t pr_thread_shortwait; 168 static clock_t pr_thread_longwait; 169 170 /* 171 * Make a list of all of the pages that have been marked for retirement 172 * but are not yet retired. At system shutdown, we will scrub all of the 173 * pages in the list in case there are outstanding UEs. Then, we 174 * cross-check this list against the number of pages that are yet to be 175 * retired, and if we find inconsistencies, we scan every page_t in the 176 * whole system looking for any pages that need to be scrubbed for UEs. 177 * The background thread also uses this queue to determine which pages 178 * it should keep trying to retire. 179 */ 180 #ifdef DEBUG 181 #define PR_PENDING_QMAX 32 182 #else /* DEBUG */ 183 #define PR_PENDING_QMAX 256 184 #endif /* DEBUG */ 185 page_t *pr_pending_q[PR_PENDING_QMAX]; 186 kmutex_t pr_q_mutex; 187 188 /* 189 * Page retire global kstats 190 */ 191 struct page_retire_kstat { 192 kstat_named_t pr_retired; 193 kstat_named_t pr_requested; 194 kstat_named_t pr_requested_free; 195 kstat_named_t pr_enqueue_fail; 196 kstat_named_t pr_dequeue_fail; 197 kstat_named_t pr_pending; 198 kstat_named_t pr_failed; 199 kstat_named_t pr_failed_kernel; 200 kstat_named_t pr_limit; 201 kstat_named_t pr_limit_exceeded; 202 kstat_named_t pr_fma; 203 kstat_named_t pr_mce; 204 kstat_named_t pr_ue; 205 kstat_named_t pr_ue_cleared_retire; 206 kstat_named_t pr_ue_cleared_free; 207 kstat_named_t pr_ue_persistent; 208 kstat_named_t pr_unretired; 209 }; 210 211 static struct page_retire_kstat page_retire_kstat = { 212 { "pages_retired", KSTAT_DATA_UINT64}, 213 { "pages_retire_request", KSTAT_DATA_UINT64}, 214 { "pages_retire_request_free", KSTAT_DATA_UINT64}, 215 { "pages_notenqueued", KSTAT_DATA_UINT64}, 216 { "pages_notdequeued", KSTAT_DATA_UINT64}, 217 { "pages_pending", KSTAT_DATA_UINT64}, 218 { "pages_deferred", KSTAT_DATA_UINT64}, 219 { "pages_deferred_kernel", KSTAT_DATA_UINT64}, 220 { "pages_limit", KSTAT_DATA_UINT64}, 221 { "pages_limit_exceeded", KSTAT_DATA_UINT64}, 222 { "pages_fma", KSTAT_DATA_UINT64}, 223 { "pages_multiple_ce", KSTAT_DATA_UINT64}, 224 { "pages_ue", KSTAT_DATA_UINT64}, 225 { "pages_ue_cleared_retired", KSTAT_DATA_UINT64}, 226 { "pages_ue_cleared_freed", KSTAT_DATA_UINT64}, 227 { "pages_ue_persistent", KSTAT_DATA_UINT64}, 228 { "pages_unretired", KSTAT_DATA_UINT64}, 229 }; 230 231 static kstat_t *page_retire_ksp = NULL; 232 233 #define PR_INCR_KSTAT(stat) \ 234 atomic_add_64(&(page_retire_kstat.stat.value.ui64), 1) 235 #define PR_DECR_KSTAT(stat) \ 236 atomic_add_64(&(page_retire_kstat.stat.value.ui64), -1) 237 238 #define PR_KSTAT_RETIRED_CE (page_retire_kstat.pr_mce.value.ui64) 239 #define PR_KSTAT_RETIRED_FMA (page_retire_kstat.pr_fma.value.ui64) 240 #define PR_KSTAT_RETIRED_NOTUE (PR_KSTAT_RETIRED_CE + PR_KSTAT_RETIRED_FMA) 241 #define PR_KSTAT_PENDING (page_retire_kstat.pr_pending.value.ui64) 242 #define PR_KSTAT_EQFAIL (page_retire_kstat.pr_enqueue_fail.value.ui64) 243 #define PR_KSTAT_DQFAIL (page_retire_kstat.pr_dequeue_fail.value.ui64) 244 245 /* 246 * Limit the number of multiple CE page retires. 247 * The default is 0.1% of physmem, or 1 in 1000 pages. This is set in 248 * basis points, where 100 basis points equals one percent. 249 */ 250 #define MCE_BPT 10 251 uint64_t max_pages_retired_bps = MCE_BPT; 252 #define PAGE_RETIRE_LIMIT ((physmem * max_pages_retired_bps) / 10000) 253 254 /* 255 * Control over the verbosity of page retirement. 256 * 257 * When set to zero (the default), no messages will be printed. 258 * When set to one, summary messages will be printed. 259 * When set > one, all messages will be printed. 260 * 261 * A value of one will trigger detailed messages for retirement operations, 262 * and is intended as a platform tunable for processors where FMA's DE does 263 * not run (e.g., spitfire). Values > one are intended for debugging only. 264 */ 265 int page_retire_messages = 0; 266 267 /* 268 * Control whether or not we return scrubbed UE pages to service. 269 * By default we do not since FMA wants to run its diagnostics first 270 * and then ask us to unretire the page if it passes. Non-FMA platforms 271 * may set this to zero so we will only retire recidivist pages. It should 272 * not be changed by the user. 273 */ 274 int page_retire_first_ue = 1; 275 276 /* 277 * Master enable for page retire. This prevents a CE or UE early in boot 278 * from trying to retire a page before page_retire_init() has finished 279 * setting things up. This is internal only and is not a tunable! 280 */ 281 static int pr_enable = 0; 282 283 extern struct vnode kvp; 284 285 #ifdef DEBUG 286 struct page_retire_debug { 287 int prd_dup1; 288 int prd_dup2; 289 int prd_qdup; 290 int prd_noaction; 291 int prd_queued; 292 int prd_notqueued; 293 int prd_dequeue; 294 int prd_top; 295 int prd_locked; 296 int prd_reloc; 297 int prd_relocfail; 298 int prd_mod; 299 int prd_mod_late; 300 int prd_kern; 301 int prd_free; 302 int prd_noreclaim; 303 int prd_hashout; 304 int prd_fma; 305 int prd_uescrubbed; 306 int prd_uenotscrubbed; 307 int prd_mce; 308 int prd_prlocked; 309 int prd_prnotlocked; 310 int prd_prretired; 311 int prd_ulocked; 312 int prd_unotretired; 313 int prd_udestroy; 314 int prd_uhashout; 315 int prd_uunretired; 316 int prd_unotlocked; 317 int prd_checkhit; 318 int prd_checkmiss_pend; 319 int prd_checkmiss_noerr; 320 int prd_tctop; 321 int prd_tclocked; 322 int prd_hunt; 323 int prd_dohunt; 324 int prd_earlyhunt; 325 int prd_latehunt; 326 int prd_nofreedemote; 327 int prd_nodemote; 328 int prd_demoted; 329 } pr_debug; 330 331 #define PR_DEBUG(foo) ((pr_debug.foo)++) 332 333 /* 334 * A type histogram. We record the incidence of the various toxic 335 * flag combinations along with the interesting page attributes. The 336 * goal is to get as many combinations as we can while driving all 337 * pr_debug values nonzero (indicating we've exercised all possible 338 * code paths across all possible page types). Not all combinations 339 * will make sense -- e.g. PRT_MOD|PRT_KERNEL. 340 * 341 * pr_type offset bit encoding (when examining with a debugger): 342 * 343 * PRT_NAMED - 0x4 344 * PRT_KERNEL - 0x8 345 * PRT_FREE - 0x10 346 * PRT_MOD - 0x20 347 * PRT_FMA - 0x0 348 * PRT_MCE - 0x40 349 * PRT_UE - 0x80 350 */ 351 352 #define PRT_NAMED 0x01 353 #define PRT_KERNEL 0x02 354 #define PRT_FREE 0x04 355 #define PRT_MOD 0x08 356 #define PRT_FMA 0x00 /* yes, this is not a mistake */ 357 #define PRT_MCE 0x10 358 #define PRT_UE 0x20 359 #define PRT_ALL 0x3F 360 361 int pr_types[PRT_ALL+1]; 362 363 #define PR_TYPES(pp) { \ 364 int whichtype = 0; \ 365 if (pp->p_vnode) \ 366 whichtype |= PRT_NAMED; \ 367 if (PP_ISKVP(pp)) \ 368 whichtype |= PRT_KERNEL; \ 369 if (PP_ISFREE(pp)) \ 370 whichtype |= PRT_FREE; \ 371 if (hat_ismod(pp)) \ 372 whichtype |= PRT_MOD; \ 373 if (pp->p_toxic & PR_UE) \ 374 whichtype |= PRT_UE; \ 375 if (pp->p_toxic & PR_MCE) \ 376 whichtype |= PRT_MCE; \ 377 pr_types[whichtype]++; \ 378 } 379 380 int recl_calls; 381 int recl_mtbf = 3; 382 int reloc_calls; 383 int reloc_mtbf = 7; 384 int pr_calls; 385 int pr_mtbf = 15; 386 387 #define MTBF(v, f) (((++(v)) & (f)) != (f)) 388 389 #else /* DEBUG */ 390 391 #define PR_DEBUG(foo) /* nothing */ 392 #define PR_TYPES(foo) /* nothing */ 393 #define MTBF(v, f) (1) 394 395 #endif /* DEBUG */ 396 397 /* 398 * page_retire_done() - completion processing 399 * 400 * Used by the page_retire code for common completion processing. 401 * It keeps track of how many times a given result has happened, 402 * and writes out an occasional message. 403 * 404 * May be called with a NULL pp (PRD_INVALID_PA case). 405 */ 406 #define PRD_INVALID_KEY -1 407 #define PRD_SUCCESS 0 408 #define PRD_PENDING 1 409 #define PRD_FAILED 2 410 #define PRD_DUPLICATE 3 411 #define PRD_INVALID_PA 4 412 #define PRD_LIMIT 5 413 #define PRD_UE_SCRUBBED 6 414 #define PRD_UNR_SUCCESS 7 415 #define PRD_UNR_CANTLOCK 8 416 #define PRD_UNR_NOT 9 417 418 typedef struct page_retire_op { 419 int pr_key; /* one of the PRD_* defines from above */ 420 int pr_count; /* How many times this has happened */ 421 int pr_retval; /* return value */ 422 int pr_msglvl; /* message level - when to print */ 423 char *pr_message; /* Cryptic message for field service */ 424 } page_retire_op_t; 425 426 static page_retire_op_t page_retire_ops[] = { 427 /* key count retval msglvl message */ 428 {PRD_SUCCESS, 0, 0, 1, 429 "Page 0x%08x.%08x removed from service"}, 430 {PRD_PENDING, 0, EAGAIN, 2, 431 "Page 0x%08x.%08x will be retired on free"}, 432 {PRD_FAILED, 0, EAGAIN, 0, NULL}, 433 {PRD_DUPLICATE, 0, EIO, 2, 434 "Page 0x%08x.%08x already retired or pending"}, 435 {PRD_INVALID_PA, 0, EINVAL, 2, 436 "PA 0x%08x.%08x is not a relocatable page"}, 437 {PRD_LIMIT, 0, 0, 1, 438 "Page 0x%08x.%08x not retired due to limit exceeded"}, 439 {PRD_UE_SCRUBBED, 0, 0, 1, 440 "Previously reported error on page 0x%08x.%08x cleared"}, 441 {PRD_UNR_SUCCESS, 0, 0, 1, 442 "Page 0x%08x.%08x returned to service"}, 443 {PRD_UNR_CANTLOCK, 0, EAGAIN, 2, 444 "Page 0x%08x.%08x could not be unretired"}, 445 {PRD_UNR_NOT, 0, EIO, 2, 446 "Page 0x%08x.%08x is not retired"}, 447 {PRD_INVALID_KEY, 0, 0, 0, NULL} /* MUST BE LAST! */ 448 }; 449 450 /* 451 * print a message if page_retire_messages is true. 452 */ 453 #define PR_MESSAGE(debuglvl, msglvl, msg, pa) \ 454 { \ 455 uint64_t p = (uint64_t)pa; \ 456 if (page_retire_messages >= msglvl && msg != NULL) { \ 457 cmn_err(debuglvl, msg, \ 458 (uint32_t)(p >> 32), (uint32_t)p); \ 459 } \ 460 } 461 462 /* 463 * Note that multiple bits may be set in a single settoxic operation. 464 * May be called without the page locked. 465 */ 466 void 467 page_settoxic(page_t *pp, uchar_t bits) 468 { 469 atomic_or_8(&pp->p_toxic, bits); 470 } 471 472 /* 473 * Note that multiple bits may cleared in a single clrtoxic operation. 474 * Must be called with the page exclusively locked to prevent races which 475 * may attempt to retire a page without any toxic bits set. 476 */ 477 void 478 page_clrtoxic(page_t *pp, uchar_t bits) 479 { 480 ASSERT(PAGE_EXCL(pp)); 481 atomic_and_8(&pp->p_toxic, ~bits); 482 } 483 484 /* 485 * Prints any page retire messages to the user, and decides what 486 * error code is appropriate for the condition reported. 487 */ 488 static int 489 page_retire_done(page_t *pp, int code) 490 { 491 page_retire_op_t *prop; 492 uint64_t pa = 0; 493 int i; 494 495 if (pp != NULL) { 496 pa = mmu_ptob((uint64_t)pp->p_pagenum); 497 } 498 499 prop = NULL; 500 for (i = 0; page_retire_ops[i].pr_key != PRD_INVALID_KEY; i++) { 501 if (page_retire_ops[i].pr_key == code) { 502 prop = &page_retire_ops[i]; 503 break; 504 } 505 } 506 507 #ifdef DEBUG 508 if (page_retire_ops[i].pr_key == PRD_INVALID_KEY) { 509 cmn_err(CE_PANIC, "page_retire_done: Invalid opcode %d", code); 510 } 511 #endif 512 513 ASSERT(prop->pr_key == code); 514 515 prop->pr_count++; 516 517 PR_MESSAGE(CE_NOTE, prop->pr_msglvl, prop->pr_message, pa); 518 if (pp != NULL) { 519 page_settoxic(pp, PR_MSG); 520 } 521 522 return (prop->pr_retval); 523 } 524 525 /* 526 * On a reboot, our friend mdboot() wants to clear up any PP_PR_REQ() pages 527 * that we were not able to retire. On large machines, walking the complete 528 * page_t array and looking at every page_t takes too long. So, as a page is 529 * marked toxic, we track it using a list that can be processed at reboot 530 * time. page_retire_enqueue() will do its best to try to avoid duplicate 531 * entries, but if we get too many errors at once the queue can overflow, 532 * in which case we will end up walking every page_t as a last resort. 533 * The background thread also makes use of this queue to find which pages 534 * are pending retirement. 535 */ 536 static void 537 page_retire_enqueue(page_t *pp) 538 { 539 int nslot = -1; 540 int i; 541 542 mutex_enter(&pr_q_mutex); 543 544 /* 545 * Check to make sure retire hasn't already dequeued it. 546 * In the meantime if the page was cleaned up, no need 547 * to enqueue it. 548 */ 549 if (PP_RETIRED(pp) || pp->p_toxic == 0) { 550 mutex_exit(&pr_q_mutex); 551 PR_DEBUG(prd_noaction); 552 return; 553 } 554 555 for (i = 0; i < PR_PENDING_QMAX; i++) { 556 if (pr_pending_q[i] == pp) { 557 mutex_exit(&pr_q_mutex); 558 PR_DEBUG(prd_qdup); 559 return; 560 } else if (nslot == -1 && pr_pending_q[i] == NULL) { 561 nslot = i; 562 } 563 } 564 565 PR_INCR_KSTAT(pr_pending); 566 567 if (nslot != -1) { 568 pr_pending_q[nslot] = pp; 569 PR_DEBUG(prd_queued); 570 } else { 571 PR_INCR_KSTAT(pr_enqueue_fail); 572 PR_DEBUG(prd_notqueued); 573 } 574 mutex_exit(&pr_q_mutex); 575 } 576 577 static void 578 page_retire_dequeue(page_t *pp) 579 { 580 int i; 581 582 mutex_enter(&pr_q_mutex); 583 584 for (i = 0; i < PR_PENDING_QMAX; i++) { 585 if (pr_pending_q[i] == pp) { 586 pr_pending_q[i] = NULL; 587 break; 588 } 589 } 590 591 if (i == PR_PENDING_QMAX) { 592 PR_INCR_KSTAT(pr_dequeue_fail); 593 } 594 595 PR_DECR_KSTAT(pr_pending); 596 PR_DEBUG(prd_dequeue); 597 598 mutex_exit(&pr_q_mutex); 599 } 600 601 /* 602 * Act like page_destroy(), but instead of freeing the page, hash it onto 603 * the retired_pages vnode, and mark it retired. 604 * 605 * For fun, we try to scrub the page until it's squeaky clean. 606 * availrmem is adjusted here. 607 */ 608 static void 609 page_retire_destroy(page_t *pp) 610 { 611 u_offset_t off = (u_offset_t)((uintptr_t)pp); 612 613 ASSERT(PAGE_EXCL(pp)); 614 ASSERT(!PP_ISFREE(pp)); 615 ASSERT(pp->p_szc == 0); 616 ASSERT(!hat_page_is_mapped(pp)); 617 ASSERT(!pp->p_vnode); 618 619 page_clr_all_props(pp); 620 pagescrub(pp, 0, MMU_PAGESIZE); 621 622 pp->p_next = NULL; 623 pp->p_prev = NULL; 624 if (page_hashin(pp, retired_pages, off, NULL) == 0) { 625 cmn_err(CE_PANIC, "retired page %p hashin failed", (void *)pp); 626 } 627 628 page_settoxic(pp, PR_RETIRED); 629 page_clrtoxic(pp, PR_BUSY); 630 page_retire_dequeue(pp); 631 PR_INCR_KSTAT(pr_retired); 632 633 if (pp->p_toxic & PR_FMA) { 634 PR_INCR_KSTAT(pr_fma); 635 } else if (pp->p_toxic & PR_UE) { 636 PR_INCR_KSTAT(pr_ue); 637 } else { 638 PR_INCR_KSTAT(pr_mce); 639 } 640 641 mutex_enter(&freemem_lock); 642 availrmem--; 643 mutex_exit(&freemem_lock); 644 645 page_unlock(pp); 646 } 647 648 /* 649 * Check whether the number of pages which have been retired already exceeds 650 * the maximum allowable percentage of memory which may be retired. 651 * 652 * Returns 1 if the limit has been exceeded. 653 */ 654 static int 655 page_retire_limit(void) 656 { 657 if (PR_KSTAT_RETIRED_NOTUE >= (uint64_t)PAGE_RETIRE_LIMIT) { 658 PR_INCR_KSTAT(pr_limit_exceeded); 659 return (1); 660 } 661 662 return (0); 663 } 664 665 #define MSG_DM "Data Mismatch occurred at PA 0x%08x.%08x" \ 666 "[ 0x%x != 0x%x ] while attempting to clear previously " \ 667 "reported error; page removed from service" 668 669 #define MSG_UE "Uncorrectable Error occurred at PA 0x%08x.%08x while " \ 670 "attempting to clear previously reported error; page removed " \ 671 "from service" 672 673 /* 674 * Attempt to clear a UE from a page. 675 * Returns 1 if the error has been successfully cleared. 676 */ 677 static int 678 page_clear_transient_ue(page_t *pp) 679 { 680 caddr_t kaddr; 681 uint8_t rb, wb; 682 uint64_t pa; 683 uint32_t pa_hi, pa_lo; 684 on_trap_data_t otd; 685 int errors = 0; 686 int i; 687 688 ASSERT(PAGE_EXCL(pp)); 689 ASSERT(PP_PR_REQ(pp)); 690 ASSERT(pp->p_szc == 0); 691 ASSERT(!hat_page_is_mapped(pp)); 692 693 /* 694 * Clear the page and attempt to clear the UE. If we trap 695 * on the next access to the page, we know the UE has recurred. 696 */ 697 pagescrub(pp, 0, PAGESIZE); 698 699 /* 700 * Map the page and write a bunch of bit patterns to compare 701 * what we wrote with what we read back. This isn't a perfect 702 * test but it should be good enough to catch most of the 703 * recurring UEs. If this fails to catch a recurrent UE, we'll 704 * retire the page the next time we see a UE on the page. 705 */ 706 kaddr = ppmapin(pp, PROT_READ|PROT_WRITE, (caddr_t)-1); 707 708 pa = ptob((uint64_t)page_pptonum(pp)); 709 pa_hi = (uint32_t)(pa >> 32); 710 pa_lo = (uint32_t)pa; 711 712 /* 713 * Fill the page with each (0x00 - 0xFF] bit pattern, flushing 714 * the cache in between reading and writing. We do this under 715 * on_trap() protection to avoid recursion. 716 */ 717 if (on_trap(&otd, OT_DATA_EC)) { 718 PR_MESSAGE(CE_WARN, 1, MSG_UE, pa); 719 errors = 1; 720 } else { 721 for (wb = 0xff; wb > 0; wb--) { 722 for (i = 0; i < PAGESIZE; i++) { 723 kaddr[i] = wb; 724 } 725 726 sync_data_memory(kaddr, PAGESIZE); 727 728 for (i = 0; i < PAGESIZE; i++) { 729 rb = kaddr[i]; 730 if (rb != wb) { 731 /* 732 * We had a mismatch without a trap. 733 * Uh-oh. Something is really wrong 734 * with this system. 735 */ 736 if (page_retire_messages) { 737 cmn_err(CE_WARN, MSG_DM, 738 pa_hi, pa_lo, rb, wb); 739 } 740 errors = 1; 741 goto out; /* double break */ 742 } 743 } 744 } 745 } 746 out: 747 no_trap(); 748 ppmapout(kaddr); 749 750 return (errors ? 0 : 1); 751 } 752 753 /* 754 * Try to clear a page_t with a single UE. If the UE was transient, it is 755 * returned to service, and we return 1. Otherwise we return 0 meaning 756 * that further processing is required to retire the page. 757 */ 758 static int 759 page_retire_transient_ue(page_t *pp) 760 { 761 ASSERT(PAGE_EXCL(pp)); 762 ASSERT(!hat_page_is_mapped(pp)); 763 764 /* 765 * If this page is a repeat offender, retire him under the 766 * "two strikes and you're out" rule. The caller is responsible 767 * for scrubbing the page to try to clear the error. 768 */ 769 if (pp->p_toxic & PR_UE_SCRUBBED) { 770 PR_INCR_KSTAT(pr_ue_persistent); 771 return (0); 772 } 773 774 if (page_clear_transient_ue(pp)) { 775 /* 776 * We set the PR_SCRUBBED_UE bit; if we ever see this 777 * page again, we will retire it, no questions asked. 778 */ 779 page_settoxic(pp, PR_UE_SCRUBBED); 780 781 if (page_retire_first_ue) { 782 PR_INCR_KSTAT(pr_ue_cleared_retire); 783 return (0); 784 } else { 785 PR_INCR_KSTAT(pr_ue_cleared_free); 786 787 page_clrtoxic(pp, PR_UE | PR_MCE | PR_MSG | PR_BUSY); 788 page_retire_dequeue(pp); 789 790 /* LINTED: CONSTCOND */ 791 VN_DISPOSE(pp, B_FREE, 1, kcred); 792 return (1); 793 } 794 } 795 796 PR_INCR_KSTAT(pr_ue_persistent); 797 return (0); 798 } 799 800 /* 801 * Update the statistics dynamically when our kstat is read. 802 */ 803 static int 804 page_retire_kstat_update(kstat_t *ksp, int rw) 805 { 806 struct page_retire_kstat *pr; 807 808 if (ksp == NULL) 809 return (EINVAL); 810 811 switch (rw) { 812 813 case KSTAT_READ: 814 pr = (struct page_retire_kstat *)ksp->ks_data; 815 ASSERT(pr == &page_retire_kstat); 816 pr->pr_limit.value.ui64 = PAGE_RETIRE_LIMIT; 817 return (0); 818 819 case KSTAT_WRITE: 820 return (EACCES); 821 822 default: 823 return (EINVAL); 824 } 825 /*NOTREACHED*/ 826 } 827 828 /* 829 * Initialize the page retire mechanism: 830 * 831 * - Establish the correctable error retire limit. 832 * - Initialize locks. 833 * - Build the retired_pages vnode. 834 * - Set up the kstats. 835 * - Fire off the background thread. 836 * - Tell page_tryretire() it's OK to start retiring pages. 837 */ 838 void 839 page_retire_init(void) 840 { 841 const fs_operation_def_t retired_vnodeops_template[] = {NULL, NULL}; 842 struct vnodeops *vops; 843 844 const uint_t page_retire_ndata = 845 sizeof (page_retire_kstat) / sizeof (kstat_named_t); 846 847 ASSERT(page_retire_ksp == NULL); 848 849 if (max_pages_retired_bps <= 0) { 850 max_pages_retired_bps = MCE_BPT; 851 } 852 853 mutex_init(&pr_q_mutex, NULL, MUTEX_DEFAULT, NULL); 854 855 retired_pages = vn_alloc(KM_SLEEP); 856 if (vn_make_ops("retired_pages", retired_vnodeops_template, &vops)) { 857 cmn_err(CE_PANIC, 858 "page_retired_init: can't make retired vnodeops"); 859 } 860 vn_setops(retired_pages, vops); 861 862 if ((page_retire_ksp = kstat_create("unix", 0, "page_retire", 863 "misc", KSTAT_TYPE_NAMED, page_retire_ndata, 864 KSTAT_FLAG_VIRTUAL)) == NULL) { 865 cmn_err(CE_WARN, "kstat_create for page_retire failed"); 866 } else { 867 page_retire_ksp->ks_data = (void *)&page_retire_kstat; 868 page_retire_ksp->ks_update = page_retire_kstat_update; 869 kstat_install(page_retire_ksp); 870 } 871 872 pr_thread_shortwait = 23 * hz; 873 pr_thread_longwait = 1201 * hz; 874 mutex_init(&pr_thread_mutex, NULL, MUTEX_DEFAULT, NULL); 875 cv_init(&pr_cv, NULL, CV_DEFAULT, NULL); 876 pr_thread_id = thread_create(NULL, 0, page_retire_thread, NULL, 0, &p0, 877 TS_RUN, minclsyspri); 878 879 pr_enable = 1; 880 } 881 882 /* 883 * page_retire_hunt() callback for the retire thread. 884 */ 885 static void 886 page_retire_thread_cb(page_t *pp) 887 { 888 PR_DEBUG(prd_tctop); 889 if (!PP_ISKVP(pp) && page_trylock(pp, SE_EXCL)) { 890 PR_DEBUG(prd_tclocked); 891 page_unlock(pp); 892 } 893 } 894 895 /* 896 * page_retire_hunt() callback for mdboot(). 897 * 898 * It is necessary to scrub any failing pages prior to reboot in order to 899 * prevent a latent error trap from occurring on the next boot. 900 */ 901 void 902 page_retire_mdboot_cb(page_t *pp) 903 { 904 /* 905 * Don't scrub the kernel, since we might still need it, unless 906 * we have UEs on the page, in which case we have nothing to lose. 907 */ 908 if (!PP_ISKVP(pp) || PP_TOXIC(pp)) { 909 pp->p_selock = -1; /* pacify ASSERTs */ 910 PP_CLRFREE(pp); 911 pagescrub(pp, 0, PAGESIZE); 912 pp->p_selock = 0; 913 } 914 pp->p_toxic = 0; 915 } 916 917 /* 918 * Hunt down any pages in the system that have not yet been retired, invoking 919 * the provided callback function on each of them. 920 */ 921 void 922 page_retire_hunt(void (*callback)(page_t *)) 923 { 924 page_t *pp; 925 page_t *first; 926 uint64_t tbr, found; 927 int i; 928 929 PR_DEBUG(prd_hunt); 930 931 if (PR_KSTAT_PENDING == 0) { 932 return; 933 } 934 935 PR_DEBUG(prd_dohunt); 936 937 found = 0; 938 mutex_enter(&pr_q_mutex); 939 940 tbr = PR_KSTAT_PENDING; 941 942 for (i = 0; i < PR_PENDING_QMAX; i++) { 943 if ((pp = pr_pending_q[i]) != NULL) { 944 mutex_exit(&pr_q_mutex); 945 callback(pp); 946 mutex_enter(&pr_q_mutex); 947 found++; 948 } 949 } 950 951 if (PR_KSTAT_EQFAIL == PR_KSTAT_DQFAIL && found == tbr) { 952 mutex_exit(&pr_q_mutex); 953 PR_DEBUG(prd_earlyhunt); 954 return; 955 } 956 mutex_exit(&pr_q_mutex); 957 958 PR_DEBUG(prd_latehunt); 959 960 /* 961 * We've lost track of a page somewhere. Hunt it down. 962 */ 963 memsegs_lock(0); 964 pp = first = page_first(); 965 do { 966 if (PP_PR_REQ(pp)) { 967 callback(pp); 968 if (++found == tbr) { 969 break; /* got 'em all */ 970 } 971 } 972 } while ((pp = page_next(pp)) != first); 973 memsegs_unlock(0); 974 } 975 976 /* 977 * The page_retire_thread loops forever, looking to see if there are 978 * pages still waiting to be retired. 979 */ 980 static void 981 page_retire_thread(void) 982 { 983 callb_cpr_t c; 984 985 CALLB_CPR_INIT(&c, &pr_thread_mutex, callb_generic_cpr, "page_retire"); 986 987 mutex_enter(&pr_thread_mutex); 988 for (;;) { 989 if (pr_enable && PR_KSTAT_PENDING) { 990 /* 991 * Sigh. It's SO broken how we have to try to shake 992 * loose the holder of the page. Since we have no 993 * idea who or what has it locked, we go bang on 994 * every door in the city to try to locate it. 995 */ 996 kmem_reap(); 997 seg_preap(); 998 page_retire_hunt(page_retire_thread_cb); 999 CALLB_CPR_SAFE_BEGIN(&c); 1000 (void) cv_timedwait(&pr_cv, &pr_thread_mutex, 1001 lbolt + pr_thread_shortwait); 1002 CALLB_CPR_SAFE_END(&c, &pr_thread_mutex); 1003 } else { 1004 CALLB_CPR_SAFE_BEGIN(&c); 1005 (void) cv_timedwait(&pr_cv, &pr_thread_mutex, 1006 lbolt + pr_thread_longwait); 1007 CALLB_CPR_SAFE_END(&c, &pr_thread_mutex); 1008 } 1009 } 1010 /*NOTREACHED*/ 1011 } 1012 1013 /* 1014 * page_retire_pp() decides what to do with a failing page. 1015 * 1016 * When we get a free page (e.g. the scrubber or in the free path) life is 1017 * nice because the page is clean and marked free -- those always retire 1018 * nicely. From there we go by order of difficulty. If the page has data, 1019 * we attempt to relocate its contents to a suitable replacement page. If 1020 * that does not succeed, we look to see if it is clean. If after all of 1021 * this we have a clean, unmapped page (which we usually do!), we retire it. 1022 * If the page is not clean, we still process it regardless on a UE; for 1023 * CEs or FMA requests, we fail leaving the page in service. The page will 1024 * eventually be tried again later. We always return with the page unlocked 1025 * since we are called from page_unlock(). 1026 * 1027 * We don't call panic or do anything fancy down in here. Our boss the DE 1028 * gets paid handsomely to do his job of figuring out what to do when errors 1029 * occur. We just do what he tells us to do. 1030 */ 1031 static int 1032 page_retire_pp(page_t *pp) 1033 { 1034 int toxic; 1035 1036 ASSERT(PAGE_EXCL(pp)); 1037 ASSERT(pp->p_iolock_state == 0); 1038 ASSERT(pp->p_szc == 0); 1039 1040 PR_DEBUG(prd_top); 1041 PR_TYPES(pp); 1042 1043 toxic = pp->p_toxic; 1044 ASSERT(toxic & PR_REASONS); 1045 1046 if ((toxic & (PR_FMA | PR_MCE)) && !(toxic & PR_UE) && 1047 page_retire_limit()) { 1048 page_clrtoxic(pp, PR_FMA | PR_MCE | PR_MSG | PR_BUSY); 1049 page_retire_dequeue(pp); 1050 page_unlock(pp); 1051 return (page_retire_done(pp, PRD_LIMIT)); 1052 } 1053 1054 if (PP_ISFREE(pp)) { 1055 int dbgnoreclaim = MTBF(recl_calls, recl_mtbf) == 0; 1056 1057 PR_DEBUG(prd_free); 1058 1059 if (dbgnoreclaim || !page_reclaim(pp, NULL)) { 1060 PR_DEBUG(prd_noreclaim); 1061 PR_INCR_KSTAT(pr_failed); 1062 /* 1063 * page_reclaim() returns with `pp' unlocked when 1064 * it fails. 1065 */ 1066 if (dbgnoreclaim) 1067 page_unlock(pp); 1068 return (page_retire_done(pp, PRD_FAILED)); 1069 } 1070 } 1071 ASSERT(!PP_ISFREE(pp)); 1072 1073 if ((toxic & PR_UE) == 0 && pp->p_vnode && !PP_ISNORELOCKERNEL(pp) && 1074 MTBF(reloc_calls, reloc_mtbf)) { 1075 page_t *newpp; 1076 spgcnt_t count; 1077 1078 /* 1079 * If we can relocate the page, great! newpp will go 1080 * on without us, and everything is fine. Regardless 1081 * of whether the relocation succeeds, we are still 1082 * going to take `pp' around back and shoot it. 1083 */ 1084 newpp = NULL; 1085 if (page_relocate(&pp, &newpp, 0, 0, &count, NULL) == 0) { 1086 PR_DEBUG(prd_reloc); 1087 page_unlock(newpp); 1088 ASSERT(hat_page_getattr(pp, P_MOD) == 0); 1089 } else { 1090 PR_DEBUG(prd_relocfail); 1091 } 1092 } 1093 1094 if (hat_ismod(pp)) { 1095 PR_DEBUG(prd_mod); 1096 PR_INCR_KSTAT(pr_failed); 1097 page_unlock(pp); 1098 return (page_retire_done(pp, PRD_FAILED)); 1099 } 1100 1101 if (PP_ISKVP(pp)) { 1102 PR_DEBUG(prd_kern); 1103 PR_INCR_KSTAT(pr_failed_kernel); 1104 page_unlock(pp); 1105 return (page_retire_done(pp, PRD_FAILED)); 1106 } 1107 1108 if (pp->p_lckcnt || pp->p_cowcnt) { 1109 PR_DEBUG(prd_locked); 1110 PR_INCR_KSTAT(pr_failed); 1111 page_unlock(pp); 1112 return (page_retire_done(pp, PRD_FAILED)); 1113 } 1114 1115 (void) hat_pageunload(pp, HAT_FORCE_PGUNLOAD); 1116 ASSERT(!hat_page_is_mapped(pp)); 1117 1118 /* 1119 * If the page is modified, and was not relocated; we can't 1120 * retire it without dropping data on the floor. We have to 1121 * recheck after unloading since the dirty bit could have been 1122 * set since we last checked. 1123 */ 1124 if (hat_ismod(pp)) { 1125 PR_DEBUG(prd_mod_late); 1126 PR_INCR_KSTAT(pr_failed); 1127 page_unlock(pp); 1128 return (page_retire_done(pp, PRD_FAILED)); 1129 } 1130 1131 if (pp->p_vnode) { 1132 PR_DEBUG(prd_hashout); 1133 page_hashout(pp, NULL); 1134 } 1135 ASSERT(!pp->p_vnode); 1136 1137 /* 1138 * The problem page is locked, demoted, unmapped, not free, 1139 * hashed out, and not COW or mlocked (whew!). 1140 * 1141 * Now we select our ammunition, take it around back, and shoot it. 1142 */ 1143 if (toxic & PR_UE) { 1144 if (page_retire_transient_ue(pp)) { 1145 PR_DEBUG(prd_uescrubbed); 1146 return (page_retire_done(pp, PRD_UE_SCRUBBED)); 1147 } else { 1148 PR_DEBUG(prd_uenotscrubbed); 1149 page_retire_destroy(pp); 1150 return (page_retire_done(pp, PRD_SUCCESS)); 1151 } 1152 } else if (toxic & PR_FMA) { 1153 PR_DEBUG(prd_fma); 1154 page_retire_destroy(pp); 1155 return (page_retire_done(pp, PRD_SUCCESS)); 1156 } else if (toxic & PR_MCE) { 1157 PR_DEBUG(prd_mce); 1158 page_retire_destroy(pp); 1159 return (page_retire_done(pp, PRD_SUCCESS)); 1160 } 1161 panic("page_retire_pp: bad toxic flags %d", toxic); 1162 /*NOTREACHED*/ 1163 } 1164 1165 /* 1166 * Try to retire a page when we stumble onto it in the page lock routines. 1167 */ 1168 void 1169 page_tryretire(page_t *pp) 1170 { 1171 ASSERT(PAGE_EXCL(pp)); 1172 1173 if (!pr_enable) { 1174 page_unlock(pp); 1175 return; 1176 } 1177 1178 /* 1179 * If the page is a big page, try to break it up. 1180 * 1181 * If there are other bad pages besides `pp', they will be 1182 * recursively retired for us thanks to a bit of magic. 1183 * If the page is a small page with errors, try to retire it. 1184 */ 1185 if (pp->p_szc > 0) { 1186 if (PP_ISFREE(pp) && !page_try_demote_free_pages(pp)) { 1187 page_unlock(pp); 1188 PR_DEBUG(prd_nofreedemote); 1189 return; 1190 } else if (!page_try_demote_pages(pp)) { 1191 page_unlock(pp); 1192 PR_DEBUG(prd_nodemote); 1193 return; 1194 } 1195 PR_DEBUG(prd_demoted); 1196 page_unlock(pp); 1197 } else { 1198 (void) page_retire_pp(pp); 1199 } 1200 } 1201 1202 /* 1203 * page_retire() - the front door in to retire a page. 1204 * 1205 * Ideally, page_retire() would instantly retire the requested page. 1206 * Unfortunately, some pages are locked or otherwise tied up and cannot be 1207 * retired right away. To deal with that, bits are set in p_toxic of the 1208 * page_t. An attempt is made to lock the page; if the attempt is successful, 1209 * we instantly unlock the page counting on page_unlock() to notice p_toxic 1210 * is nonzero and to call back into page_retire_pp(). Success is determined 1211 * by looking to see whether the page has been retired once it has been 1212 * unlocked. 1213 * 1214 * Returns: 1215 * 1216 * - 0 on success, 1217 * - EINVAL when the PA is whacko, 1218 * - EIO if the page is already retired or already pending retirement, or 1219 * - EAGAIN if the page could not be _immediately_ retired but is pending. 1220 */ 1221 int 1222 page_retire(uint64_t pa, uchar_t reason) 1223 { 1224 page_t *pp; 1225 1226 ASSERT(reason & PR_REASONS); /* there must be a reason */ 1227 ASSERT(!(reason & ~PR_REASONS)); /* but no other bits */ 1228 1229 pp = page_numtopp_nolock(mmu_btop(pa)); 1230 if (pp == NULL) { 1231 PR_MESSAGE(CE_WARN, 1, "Cannot schedule clearing of error on" 1232 " page 0x%08x.%08x; page is not relocatable memory", pa); 1233 return (page_retire_done(pp, PRD_INVALID_PA)); 1234 } 1235 if (PP_RETIRED(pp)) { 1236 PR_DEBUG(prd_dup1); 1237 return (page_retire_done(pp, PRD_DUPLICATE)); 1238 } 1239 1240 if ((reason & PR_UE) && !PP_TOXIC(pp)) { 1241 PR_MESSAGE(CE_NOTE, 1, "Scheduling clearing of error on" 1242 " page 0x%08x.%08x", pa); 1243 } else if (PP_PR_REQ(pp)) { 1244 PR_DEBUG(prd_dup2); 1245 return (page_retire_done(pp, PRD_DUPLICATE)); 1246 } else { 1247 PR_MESSAGE(CE_NOTE, 1, "Scheduling removal of" 1248 " page 0x%08x.%08x", pa); 1249 } 1250 page_settoxic(pp, reason); 1251 page_retire_enqueue(pp); 1252 1253 /* 1254 * And now for some magic. 1255 * 1256 * We marked this page toxic up above. All there is left to do is 1257 * to try to lock the page and then unlock it. The page lock routines 1258 * will intercept the page and retire it if they can. If the page 1259 * cannot be locked, 's okay -- page_unlock() will eventually get it, 1260 * or the background thread, until then the lock routines will deny 1261 * further locks on it. 1262 */ 1263 if (MTBF(pr_calls, pr_mtbf) && page_trylock(pp, SE_EXCL)) { 1264 PR_DEBUG(prd_prlocked); 1265 page_unlock(pp); 1266 } else { 1267 PR_DEBUG(prd_prnotlocked); 1268 } 1269 1270 if (PP_RETIRED(pp)) { 1271 PR_DEBUG(prd_prretired); 1272 return (0); 1273 } else { 1274 cv_signal(&pr_cv); 1275 PR_INCR_KSTAT(pr_failed); 1276 1277 if (pp->p_toxic & PR_MSG) { 1278 return (page_retire_done(pp, PRD_FAILED)); 1279 } else { 1280 return (page_retire_done(pp, PRD_PENDING)); 1281 } 1282 } 1283 } 1284 1285 /* 1286 * Take a retired page off the retired-pages vnode and clear the toxic flags. 1287 * If "free" is nonzero, lock it and put it back on the freelist. If "free" 1288 * is zero, the caller already holds SE_EXCL lock so we simply unretire it 1289 * and don't do anything else with it. 1290 * 1291 * Any unretire messages are printed from this routine. 1292 * 1293 * Returns 0 if page pp was unretired; else an error code. 1294 */ 1295 int 1296 page_unretire_pp(page_t *pp, int free) 1297 { 1298 /* 1299 * To be retired, a page has to be hashed onto the retired_pages vnode 1300 * and have PR_RETIRED set in p_toxic. 1301 */ 1302 if (free == 0 || page_try_reclaim_lock(pp, SE_EXCL, SE_RETIRED)) { 1303 ASSERT(PAGE_EXCL(pp)); 1304 PR_DEBUG(prd_ulocked); 1305 if (!PP_RETIRED(pp)) { 1306 PR_DEBUG(prd_unotretired); 1307 page_unlock(pp); 1308 return (page_retire_done(pp, PRD_UNR_NOT)); 1309 } 1310 1311 PR_MESSAGE(CE_NOTE, 1, "unretiring retired" 1312 " page 0x%08x.%08x", mmu_ptob((uint64_t)pp->p_pagenum)); 1313 if (pp->p_toxic & PR_FMA) { 1314 PR_DECR_KSTAT(pr_fma); 1315 } else if (pp->p_toxic & PR_UE) { 1316 PR_DECR_KSTAT(pr_ue); 1317 } else { 1318 PR_DECR_KSTAT(pr_mce); 1319 } 1320 page_clrtoxic(pp, PR_ALLFLAGS); 1321 1322 if (free) { 1323 PR_DEBUG(prd_udestroy); 1324 page_destroy(pp, 0); 1325 } else { 1326 PR_DEBUG(prd_uhashout); 1327 page_hashout(pp, NULL); 1328 } 1329 1330 mutex_enter(&freemem_lock); 1331 availrmem++; 1332 mutex_exit(&freemem_lock); 1333 1334 PR_DEBUG(prd_uunretired); 1335 PR_DECR_KSTAT(pr_retired); 1336 PR_INCR_KSTAT(pr_unretired); 1337 return (page_retire_done(pp, PRD_UNR_SUCCESS)); 1338 } 1339 PR_DEBUG(prd_unotlocked); 1340 return (page_retire_done(pp, PRD_UNR_CANTLOCK)); 1341 } 1342 1343 /* 1344 * Return a page to service by moving it from the retired_pages vnode 1345 * onto the freelist. 1346 * 1347 * Called from mmioctl_page_retire() on behalf of the FMA DE. 1348 * 1349 * Returns: 1350 * 1351 * - 0 if the page is unretired, 1352 * - EAGAIN if the pp can not be locked, 1353 * - EINVAL if the PA is whacko, and 1354 * - EIO if the pp is not retired. 1355 */ 1356 int 1357 page_unretire(uint64_t pa) 1358 { 1359 page_t *pp; 1360 1361 pp = page_numtopp_nolock(mmu_btop(pa)); 1362 if (pp == NULL) { 1363 return (page_retire_done(pp, PRD_INVALID_PA)); 1364 } 1365 1366 return (page_unretire_pp(pp, 1)); 1367 } 1368 1369 /* 1370 * Test a page to see if it is retired. If errors is non-NULL, the toxic 1371 * bits of the page are returned. Returns 0 on success, error code on failure. 1372 */ 1373 int 1374 page_retire_check_pp(page_t *pp, uint64_t *errors) 1375 { 1376 int rc; 1377 1378 if (PP_RETIRED(pp)) { 1379 PR_DEBUG(prd_checkhit); 1380 rc = 0; 1381 } else if (PP_PR_REQ(pp)) { 1382 PR_DEBUG(prd_checkmiss_pend); 1383 rc = EAGAIN; 1384 } else { 1385 PR_DEBUG(prd_checkmiss_noerr); 1386 rc = EIO; 1387 } 1388 1389 /* 1390 * We have magically arranged the bit values returned to fmd(1M) 1391 * to line up with the FMA, MCE, and UE bits of the page_t. 1392 */ 1393 if (errors) { 1394 uint64_t toxic = (uint64_t)(pp->p_toxic & PR_ERRMASK); 1395 if (toxic & PR_UE_SCRUBBED) { 1396 toxic &= ~PR_UE_SCRUBBED; 1397 toxic |= PR_UE; 1398 } 1399 *errors = toxic; 1400 } 1401 1402 return (rc); 1403 } 1404 1405 /* 1406 * Test to see if the page_t for a given PA is retired, and return the 1407 * hardware errors we have seen on the page if requested. 1408 * 1409 * Called from mmioctl_page_retire on behalf of the FMA DE. 1410 * 1411 * Returns: 1412 * 1413 * - 0 if the page is retired, 1414 * - EIO if the page is not retired and has no errors, 1415 * - EAGAIN if the page is not retired but is pending; and 1416 * - EINVAL if the PA is whacko. 1417 */ 1418 int 1419 page_retire_check(uint64_t pa, uint64_t *errors) 1420 { 1421 page_t *pp; 1422 1423 if (errors) { 1424 *errors = 0; 1425 } 1426 1427 pp = page_numtopp_nolock(mmu_btop(pa)); 1428 if (pp == NULL) { 1429 return (page_retire_done(pp, PRD_INVALID_PA)); 1430 } 1431 1432 return (page_retire_check_pp(pp, errors)); 1433 } 1434 1435 /* 1436 * Page retire self-test. For now, it always returns 0. 1437 */ 1438 int 1439 page_retire_test(void) 1440 { 1441 page_t *first, *pp, *cpp, *cpp2, *lpp; 1442 1443 /* 1444 * Tests the corner case where a large page can't be retired 1445 * because one of the constituent pages is locked. We mark 1446 * one page to be retired and try to retire it, and mark the 1447 * other page to be retired but don't try to retire it, so 1448 * that page_unlock() in the failure path will recurse and try 1449 * to retire THAT page. This is the worst possible situation 1450 * we can get ourselves into. 1451 */ 1452 memsegs_lock(0); 1453 pp = first = page_first(); 1454 do { 1455 if (pp->p_szc && PP_PAGEROOT(pp) == pp) { 1456 cpp = pp + 1; 1457 lpp = PP_ISFREE(pp)? pp : pp + 2; 1458 cpp2 = pp + 3; 1459 if (!page_trylock(lpp, pp == lpp? SE_EXCL : SE_SHARED)) 1460 continue; 1461 if (!page_trylock(cpp, SE_EXCL)) { 1462 page_unlock(lpp); 1463 continue; 1464 } 1465 page_settoxic(cpp, PR_FMA | PR_BUSY); 1466 page_settoxic(cpp2, PR_FMA); 1467 page_tryretire(cpp); /* will fail */ 1468 page_unlock(lpp); 1469 (void) page_retire(cpp->p_pagenum, PR_FMA); 1470 (void) page_retire(cpp2->p_pagenum, PR_FMA); 1471 } 1472 } while ((pp = page_next(pp)) != first); 1473 memsegs_unlock(0); 1474 1475 return (0); 1476 } 1477