1 /* 2 * CDDL HEADER START 3 * 4 * The contents of this file are subject to the terms of the 5 * Common Development and Distribution License (the "License"). 6 * You may not use this file except in compliance with the License. 7 * 8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9 * or http://www.opensolaris.org/os/licensing. 10 * See the License for the specific language governing permissions 11 * and limitations under the License. 12 * 13 * When distributing Covered Code, include this CDDL HEADER in each 14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15 * If applicable, add the following below this CDDL HEADER, with the 16 * fields enclosed by brackets "[]" replaced with your own identifying 17 * information: Portions Copyright [yyyy] [name of copyright owner] 18 * 19 * CDDL HEADER END 20 */ 21 /* 22 * Copyright 2010 Sun Microsystems, Inc. All rights reserved. 23 * Use is subject to license terms. 24 * Copyright (c) 2016 by Delphix. All rights reserved. 25 */ 26 27 /* 28 * Page Retire - Big Theory Statement. 29 * 30 * This file handles removing sections of faulty memory from use when the 31 * user land FMA Diagnosis Engine requests that a page be removed or when 32 * a CE or UE is detected by the hardware. 33 * 34 * In the bad old days, the kernel side of Page Retire did a lot of the work 35 * on its own. Now, with the DE keeping track of errors, the kernel side is 36 * rather simple minded on most platforms. 37 * 38 * Errors are all reflected to the DE, and after digesting the error and 39 * looking at all previously reported errors, the DE decides what should 40 * be done about the current error. If the DE wants a particular page to 41 * be retired, then the kernel page retire code is invoked via an ioctl. 42 * On non-FMA platforms, the ue_drain and ce_drain paths ends up calling 43 * page retire to handle the error. Since page retire is just a simple 44 * mechanism it doesn't need to differentiate between the different callers. 45 * 46 * The p_toxic field in the page_t is used to indicate which errors have 47 * occurred and what action has been taken on a given page. Because errors are 48 * reported without regard to the locked state of a page, no locks are used 49 * to SET the error bits in p_toxic. However, in order to clear the error 50 * bits, the page_t must be held exclusively locked. 51 * 52 * When page_retire() is called, it must be able to acquire locks, sleep, etc. 53 * It must not be called from high-level interrupt context. 54 * 55 * Depending on how the requested page is being used at the time of the retire 56 * request (and on the availability of sufficient system resources), the page 57 * may be retired immediately, or just marked for retirement later. For 58 * example, locked pages are marked, while free pages are retired. Multiple 59 * requests may be made to retire the same page, although there is no need 60 * to: once the p_toxic flags are set, the page will be retired as soon as it 61 * can be exclusively locked. 62 * 63 * The retire mechanism is driven centrally out of page_unlock(). To expedite 64 * the retirement of pages, further requests for SE_SHARED locks are denied 65 * as long as a page retirement is pending. In addition, as long as pages are 66 * pending retirement a background thread runs periodically trying to retire 67 * those pages. Pages which could not be retired while the system is running 68 * are scrubbed prior to rebooting to avoid latent errors on the next boot. 69 * 70 * UE pages without persistent errors are scrubbed and returned to service. 71 * Recidivist pages, as well as FMA-directed requests for retirement, result 72 * in the page being taken out of service. Once the decision is made to take 73 * a page out of service, the page is cleared, hashed onto the retired_pages 74 * vnode, marked as retired, and it is unlocked. No other requesters (except 75 * for unretire) are allowed to lock retired pages. 76 * 77 * The public routines return (sadly) 0 if they worked and a non-zero error 78 * value if something went wrong. This is done for the ioctl side of the 79 * world to allow errors to be reflected all the way out to user land. The 80 * non-zero values are explained in comments atop each function. 81 */ 82 83 /* 84 * Things to fix: 85 * 86 * 1. Trying to retire non-relocatable kvp pages may result in a 87 * quagmire. This is because seg_kmem() no longer keeps its pages locked, 88 * and calls page_lookup() in the free path; since kvp pages are modified 89 * and don't have a usable backing store, page_retire() can't do anything 90 * with them, and we'll keep denying the lock to seg_kmem_free() in a 91 * vicious cycle. To prevent that, we don't deny locks to kvp pages, and 92 * hence only try to retire a page from page_unlock() in the free path. 93 * Since most kernel pages are indefinitely held anyway, and don't 94 * participate in I/O, this is of little consequence. 95 * 96 * 2. Low memory situations will be interesting. If we don't have 97 * enough memory for page_relocate() to succeed, we won't be able to 98 * retire dirty pages; nobody will be able to push them out to disk 99 * either, since we aggressively deny the page lock. We could change 100 * fsflush so it can recognize this situation, grab the lock, and push 101 * the page out, where we'll catch it in the free path and retire it. 102 * 103 * 3. Beware of places that have code like this in them: 104 * 105 * if (! page_tryupgrade(pp)) { 106 * page_unlock(pp); 107 * while (! page_lock(pp, SE_EXCL, NULL, P_RECLAIM)) { 108 * / *NOTHING* / 109 * } 110 * } 111 * page_free(pp); 112 * 113 * The problem is that pp can change identity right after the 114 * page_unlock() call. In particular, page_retire() can step in 115 * there, change pp's identity, and hash pp onto the retired_vnode. 116 * 117 * Of course, other functions besides page_retire() can have the 118 * same effect. A kmem reader can waltz by, set up a mapping to the 119 * page, and then unlock the page. Page_free() will then go castors 120 * up. So if anybody is doing this, it's already a bug. 121 * 122 * 4. mdboot()'s call into page_retire_mdboot() should probably be 123 * moved lower. Where the call is made now, we can get into trouble 124 * by scrubbing a kernel page that is then accessed later. 125 */ 126 127 #include <sys/types.h> 128 #include <sys/param.h> 129 #include <sys/systm.h> 130 #include <sys/mman.h> 131 #include <sys/vnode.h> 132 #include <sys/vfs_opreg.h> 133 #include <sys/cmn_err.h> 134 #include <sys/ksynch.h> 135 #include <sys/thread.h> 136 #include <sys/disp.h> 137 #include <sys/ontrap.h> 138 #include <sys/vmsystm.h> 139 #include <sys/mem_config.h> 140 #include <sys/atomic.h> 141 #include <sys/callb.h> 142 #include <sys/kobj.h> 143 #include <vm/page.h> 144 #include <vm/vm_dep.h> 145 #include <vm/as.h> 146 #include <vm/hat.h> 147 #include <vm/seg_kmem.h> 148 149 /* 150 * vnode for all pages which are retired from the VM system; 151 */ 152 vnode_t *retired_pages; 153 154 static int page_retire_pp_finish(page_t *, void *, uint_t); 155 156 /* 157 * Make a list of all of the pages that have been marked for retirement 158 * but are not yet retired. At system shutdown, we will scrub all of the 159 * pages in the list in case there are outstanding UEs. Then, we 160 * cross-check this list against the number of pages that are yet to be 161 * retired, and if we find inconsistencies, we scan every page_t in the 162 * whole system looking for any pages that need to be scrubbed for UEs. 163 * The background thread also uses this queue to determine which pages 164 * it should keep trying to retire. 165 */ 166 #ifdef DEBUG 167 #define PR_PENDING_QMAX 32 168 #else /* DEBUG */ 169 #define PR_PENDING_QMAX 256 170 #endif /* DEBUG */ 171 page_t *pr_pending_q[PR_PENDING_QMAX]; 172 kmutex_t pr_q_mutex; 173 174 /* 175 * Page retire global kstats 176 */ 177 struct page_retire_kstat { 178 kstat_named_t pr_retired; 179 kstat_named_t pr_requested; 180 kstat_named_t pr_requested_free; 181 kstat_named_t pr_enqueue_fail; 182 kstat_named_t pr_dequeue_fail; 183 kstat_named_t pr_pending; 184 kstat_named_t pr_pending_kas; 185 kstat_named_t pr_failed; 186 kstat_named_t pr_failed_kernel; 187 kstat_named_t pr_limit; 188 kstat_named_t pr_limit_exceeded; 189 kstat_named_t pr_fma; 190 kstat_named_t pr_mce; 191 kstat_named_t pr_ue; 192 kstat_named_t pr_ue_cleared_retire; 193 kstat_named_t pr_ue_cleared_free; 194 kstat_named_t pr_ue_persistent; 195 kstat_named_t pr_unretired; 196 }; 197 198 static struct page_retire_kstat page_retire_kstat = { 199 { "pages_retired", KSTAT_DATA_UINT64}, 200 { "pages_retire_request", KSTAT_DATA_UINT64}, 201 { "pages_retire_request_free", KSTAT_DATA_UINT64}, 202 { "pages_notenqueued", KSTAT_DATA_UINT64}, 203 { "pages_notdequeued", KSTAT_DATA_UINT64}, 204 { "pages_pending", KSTAT_DATA_UINT64}, 205 { "pages_pending_kas", KSTAT_DATA_UINT64}, 206 { "pages_deferred", KSTAT_DATA_UINT64}, 207 { "pages_deferred_kernel", KSTAT_DATA_UINT64}, 208 { "pages_limit", KSTAT_DATA_UINT64}, 209 { "pages_limit_exceeded", KSTAT_DATA_UINT64}, 210 { "pages_fma", KSTAT_DATA_UINT64}, 211 { "pages_multiple_ce", KSTAT_DATA_UINT64}, 212 { "pages_ue", KSTAT_DATA_UINT64}, 213 { "pages_ue_cleared_retired", KSTAT_DATA_UINT64}, 214 { "pages_ue_cleared_freed", KSTAT_DATA_UINT64}, 215 { "pages_ue_persistent", KSTAT_DATA_UINT64}, 216 { "pages_unretired", KSTAT_DATA_UINT64}, 217 }; 218 219 static kstat_t *page_retire_ksp = NULL; 220 221 #define PR_INCR_KSTAT(stat) \ 222 atomic_inc_64(&(page_retire_kstat.stat.value.ui64)) 223 #define PR_DECR_KSTAT(stat) \ 224 atomic_dec_64(&(page_retire_kstat.stat.value.ui64)) 225 226 #define PR_KSTAT_RETIRED_CE (page_retire_kstat.pr_mce.value.ui64) 227 #define PR_KSTAT_RETIRED_FMA (page_retire_kstat.pr_fma.value.ui64) 228 #define PR_KSTAT_RETIRED_NOTUE (PR_KSTAT_RETIRED_CE + PR_KSTAT_RETIRED_FMA) 229 #define PR_KSTAT_PENDING (page_retire_kstat.pr_pending.value.ui64) 230 #define PR_KSTAT_PENDING_KAS (page_retire_kstat.pr_pending_kas.value.ui64) 231 #define PR_KSTAT_EQFAIL (page_retire_kstat.pr_enqueue_fail.value.ui64) 232 #define PR_KSTAT_DQFAIL (page_retire_kstat.pr_dequeue_fail.value.ui64) 233 234 /* 235 * page retire kstats to list all retired pages 236 */ 237 static int pr_list_kstat_update(kstat_t *ksp, int rw); 238 static int pr_list_kstat_snapshot(kstat_t *ksp, void *buf, int rw); 239 kmutex_t pr_list_kstat_mutex; 240 241 /* 242 * Limit the number of multiple CE page retires. 243 * The default is 0.1% of physmem, or 1 in 1000 pages. This is set in 244 * basis points, where 100 basis points equals one percent. 245 */ 246 #define MCE_BPT 10 247 uint64_t max_pages_retired_bps = MCE_BPT; 248 #define PAGE_RETIRE_LIMIT ((physmem * max_pages_retired_bps) / 10000) 249 250 /* 251 * Control over the verbosity of page retirement. 252 * 253 * When set to zero (the default), no messages will be printed. 254 * When set to one, summary messages will be printed. 255 * When set > one, all messages will be printed. 256 * 257 * A value of one will trigger detailed messages for retirement operations, 258 * and is intended as a platform tunable for processors where FMA's DE does 259 * not run (e.g., spitfire). Values > one are intended for debugging only. 260 */ 261 int page_retire_messages = 0; 262 263 /* 264 * Control whether or not we return scrubbed UE pages to service. 265 * By default we do not since FMA wants to run its diagnostics first 266 * and then ask us to unretire the page if it passes. Non-FMA platforms 267 * may set this to zero so we will only retire recidivist pages. It should 268 * not be changed by the user. 269 */ 270 int page_retire_first_ue = 1; 271 272 /* 273 * Master enable for page retire. This prevents a CE or UE early in boot 274 * from trying to retire a page before page_retire_init() has finished 275 * setting things up. This is internal only and is not a tunable! 276 */ 277 static int pr_enable = 0; 278 279 static void (*memscrub_notify_func)(uint64_t); 280 281 #ifdef DEBUG 282 struct page_retire_debug { 283 int prd_dup1; 284 int prd_dup2; 285 int prd_qdup; 286 int prd_noaction; 287 int prd_queued; 288 int prd_notqueued; 289 int prd_dequeue; 290 int prd_top; 291 int prd_locked; 292 int prd_reloc; 293 int prd_relocfail; 294 int prd_mod; 295 int prd_mod_late; 296 int prd_kern; 297 int prd_free; 298 int prd_noreclaim; 299 int prd_hashout; 300 int prd_fma; 301 int prd_uescrubbed; 302 int prd_uenotscrubbed; 303 int prd_mce; 304 int prd_prlocked; 305 int prd_prnotlocked; 306 int prd_prretired; 307 int prd_ulocked; 308 int prd_unotretired; 309 int prd_udestroy; 310 int prd_uhashout; 311 int prd_uunretired; 312 int prd_unotlocked; 313 int prd_checkhit; 314 int prd_checkmiss_pend; 315 int prd_checkmiss_noerr; 316 int prd_tctop; 317 int prd_tclocked; 318 int prd_hunt; 319 int prd_dohunt; 320 int prd_earlyhunt; 321 int prd_latehunt; 322 int prd_nofreedemote; 323 int prd_nodemote; 324 int prd_demoted; 325 } pr_debug; 326 327 #define PR_DEBUG(foo) ((pr_debug.foo)++) 328 329 /* 330 * A type histogram. We record the incidence of the various toxic 331 * flag combinations along with the interesting page attributes. The 332 * goal is to get as many combinations as we can while driving all 333 * pr_debug values nonzero (indicating we've exercised all possible 334 * code paths across all possible page types). Not all combinations 335 * will make sense -- e.g. PRT_MOD|PRT_KERNEL. 336 * 337 * pr_type offset bit encoding (when examining with a debugger): 338 * 339 * PRT_NAMED - 0x4 340 * PRT_KERNEL - 0x8 341 * PRT_FREE - 0x10 342 * PRT_MOD - 0x20 343 * PRT_FMA - 0x0 344 * PRT_MCE - 0x40 345 * PRT_UE - 0x80 346 */ 347 348 #define PRT_NAMED 0x01 349 #define PRT_KERNEL 0x02 350 #define PRT_FREE 0x04 351 #define PRT_MOD 0x08 352 #define PRT_FMA 0x00 /* yes, this is not a mistake */ 353 #define PRT_MCE 0x10 354 #define PRT_UE 0x20 355 #define PRT_ALL 0x3F 356 357 int pr_types[PRT_ALL+1]; 358 359 #define PR_TYPES(pp) { \ 360 int whichtype = 0; \ 361 if (pp->p_vnode) \ 362 whichtype |= PRT_NAMED; \ 363 if (PP_ISKAS(pp)) \ 364 whichtype |= PRT_KERNEL; \ 365 if (PP_ISFREE(pp)) \ 366 whichtype |= PRT_FREE; \ 367 if (hat_ismod(pp)) \ 368 whichtype |= PRT_MOD; \ 369 if (pp->p_toxic & PR_UE) \ 370 whichtype |= PRT_UE; \ 371 if (pp->p_toxic & PR_MCE) \ 372 whichtype |= PRT_MCE; \ 373 pr_types[whichtype]++; \ 374 } 375 376 int recl_calls; 377 int recl_mtbf = 3; 378 int reloc_calls; 379 int reloc_mtbf = 7; 380 int pr_calls; 381 int pr_mtbf = 15; 382 383 #define MTBF(v, f) (((++(v)) & (f)) != (f)) 384 385 #else /* DEBUG */ 386 387 #define PR_DEBUG(foo) /* nothing */ 388 #define PR_TYPES(foo) /* nothing */ 389 #define MTBF(v, f) (1) 390 391 #endif /* DEBUG */ 392 393 /* 394 * page_retire_done() - completion processing 395 * 396 * Used by the page_retire code for common completion processing. 397 * It keeps track of how many times a given result has happened, 398 * and writes out an occasional message. 399 * 400 * May be called with a NULL pp (PRD_INVALID_PA case). 401 */ 402 #define PRD_INVALID_KEY -1 403 #define PRD_SUCCESS 0 404 #define PRD_PENDING 1 405 #define PRD_FAILED 2 406 #define PRD_DUPLICATE 3 407 #define PRD_INVALID_PA 4 408 #define PRD_LIMIT 5 409 #define PRD_UE_SCRUBBED 6 410 #define PRD_UNR_SUCCESS 7 411 #define PRD_UNR_CANTLOCK 8 412 #define PRD_UNR_NOT 9 413 414 typedef struct page_retire_op { 415 int pr_key; /* one of the PRD_* defines from above */ 416 int pr_count; /* How many times this has happened */ 417 int pr_retval; /* return value */ 418 int pr_msglvl; /* message level - when to print */ 419 char *pr_message; /* Cryptic message for field service */ 420 } page_retire_op_t; 421 422 static page_retire_op_t page_retire_ops[] = { 423 /* key count retval msglvl message */ 424 {PRD_SUCCESS, 0, 0, 1, 425 "Page 0x%08x.%08x removed from service"}, 426 {PRD_PENDING, 0, EAGAIN, 2, 427 "Page 0x%08x.%08x will be retired on free"}, 428 {PRD_FAILED, 0, EAGAIN, 0, NULL}, 429 {PRD_DUPLICATE, 0, EIO, 2, 430 "Page 0x%08x.%08x already retired or pending"}, 431 {PRD_INVALID_PA, 0, EINVAL, 2, 432 "PA 0x%08x.%08x is not a relocatable page"}, 433 {PRD_LIMIT, 0, 0, 1, 434 "Page 0x%08x.%08x not retired due to limit exceeded"}, 435 {PRD_UE_SCRUBBED, 0, 0, 1, 436 "Previously reported error on page 0x%08x.%08x cleared"}, 437 {PRD_UNR_SUCCESS, 0, 0, 1, 438 "Page 0x%08x.%08x returned to service"}, 439 {PRD_UNR_CANTLOCK, 0, EAGAIN, 2, 440 "Page 0x%08x.%08x could not be unretired"}, 441 {PRD_UNR_NOT, 0, EIO, 2, 442 "Page 0x%08x.%08x is not retired"}, 443 {PRD_INVALID_KEY, 0, 0, 0, NULL} /* MUST BE LAST! */ 444 }; 445 446 /* 447 * print a message if page_retire_messages is true. 448 */ 449 #define PR_MESSAGE(debuglvl, msglvl, msg, pa) \ 450 { \ 451 uint64_t p = (uint64_t)pa; \ 452 if (page_retire_messages >= msglvl && msg != NULL) { \ 453 cmn_err(debuglvl, msg, \ 454 (uint32_t)(p >> 32), (uint32_t)p); \ 455 } \ 456 } 457 458 /* 459 * Note that multiple bits may be set in a single settoxic operation. 460 * May be called without the page locked. 461 */ 462 void 463 page_settoxic(page_t *pp, uchar_t bits) 464 { 465 atomic_or_8(&pp->p_toxic, bits); 466 } 467 468 /* 469 * Note that multiple bits may cleared in a single clrtoxic operation. 470 * Must be called with the page exclusively locked to prevent races which 471 * may attempt to retire a page without any toxic bits set. 472 * Note that the PR_CAPTURE bit can be cleared without the exclusive lock 473 * being held as there is a separate mutex which protects that bit. 474 */ 475 void 476 page_clrtoxic(page_t *pp, uchar_t bits) 477 { 478 ASSERT((bits & PR_CAPTURE) || PAGE_EXCL(pp)); 479 atomic_and_8(&pp->p_toxic, ~bits); 480 } 481 482 /* 483 * Prints any page retire messages to the user, and decides what 484 * error code is appropriate for the condition reported. 485 */ 486 static int 487 page_retire_done(page_t *pp, int code) 488 { 489 page_retire_op_t *prop; 490 uint64_t pa = 0; 491 int i; 492 493 if (pp != NULL) { 494 pa = mmu_ptob((uint64_t)pp->p_pagenum); 495 } 496 497 prop = NULL; 498 for (i = 0; page_retire_ops[i].pr_key != PRD_INVALID_KEY; i++) { 499 if (page_retire_ops[i].pr_key == code) { 500 prop = &page_retire_ops[i]; 501 break; 502 } 503 } 504 505 #ifdef DEBUG 506 if (page_retire_ops[i].pr_key == PRD_INVALID_KEY) { 507 cmn_err(CE_PANIC, "page_retire_done: Invalid opcode %d", code); 508 } 509 #endif 510 511 ASSERT(prop->pr_key == code); 512 513 prop->pr_count++; 514 515 PR_MESSAGE(CE_NOTE, prop->pr_msglvl, prop->pr_message, pa); 516 if (pp != NULL) { 517 page_settoxic(pp, PR_MSG); 518 } 519 520 return (prop->pr_retval); 521 } 522 523 /* 524 * Act like page_destroy(), but instead of freeing the page, hash it onto 525 * the retired_pages vnode, and mark it retired. 526 * 527 * For fun, we try to scrub the page until it's squeaky clean. 528 * availrmem is adjusted here. 529 */ 530 static void 531 page_retire_destroy(page_t *pp) 532 { 533 u_offset_t off = (u_offset_t)((uintptr_t)pp); 534 535 ASSERT(PAGE_EXCL(pp)); 536 ASSERT(!PP_ISFREE(pp)); 537 ASSERT(pp->p_szc == 0); 538 ASSERT(!hat_page_is_mapped(pp)); 539 ASSERT(!pp->p_vnode); 540 541 page_clr_all_props(pp); 542 pagescrub(pp, 0, MMU_PAGESIZE); 543 544 pp->p_next = NULL; 545 pp->p_prev = NULL; 546 if (page_hashin(pp, retired_pages, off, NULL) == 0) { 547 cmn_err(CE_PANIC, "retired page %p hashin failed", (void *)pp); 548 } 549 550 page_settoxic(pp, PR_RETIRED); 551 PR_INCR_KSTAT(pr_retired); 552 553 if (pp->p_toxic & PR_FMA) { 554 PR_INCR_KSTAT(pr_fma); 555 } else if (pp->p_toxic & PR_UE) { 556 PR_INCR_KSTAT(pr_ue); 557 } else { 558 PR_INCR_KSTAT(pr_mce); 559 } 560 561 mutex_enter(&freemem_lock); 562 availrmem--; 563 mutex_exit(&freemem_lock); 564 565 page_unlock(pp); 566 } 567 568 /* 569 * Check whether the number of pages which have been retired already exceeds 570 * the maximum allowable percentage of memory which may be retired. 571 * 572 * Returns 1 if the limit has been exceeded. 573 */ 574 static int 575 page_retire_limit(void) 576 { 577 if (PR_KSTAT_RETIRED_NOTUE >= (uint64_t)PAGE_RETIRE_LIMIT) { 578 PR_INCR_KSTAT(pr_limit_exceeded); 579 return (1); 580 } 581 582 return (0); 583 } 584 585 #define MSG_DM "Data Mismatch occurred at PA 0x%08x.%08x" \ 586 "[ 0x%x != 0x%x ] while attempting to clear previously " \ 587 "reported error; page removed from service" 588 589 #define MSG_UE "Uncorrectable Error occurred at PA 0x%08x.%08x while " \ 590 "attempting to clear previously reported error; page removed " \ 591 "from service" 592 593 /* 594 * Attempt to clear a UE from a page. 595 * Returns 1 if the error has been successfully cleared. 596 */ 597 static int 598 page_clear_transient_ue(page_t *pp) 599 { 600 caddr_t kaddr; 601 uint8_t rb, wb; 602 uint64_t pa; 603 uint32_t pa_hi, pa_lo; 604 on_trap_data_t otd; 605 int errors = 0; 606 int i; 607 608 ASSERT(PAGE_EXCL(pp)); 609 ASSERT(PP_PR_REQ(pp)); 610 ASSERT(pp->p_szc == 0); 611 ASSERT(!hat_page_is_mapped(pp)); 612 613 /* 614 * Clear the page and attempt to clear the UE. If we trap 615 * on the next access to the page, we know the UE has recurred. 616 */ 617 pagescrub(pp, 0, PAGESIZE); 618 619 /* 620 * Map the page and write a bunch of bit patterns to compare 621 * what we wrote with what we read back. This isn't a perfect 622 * test but it should be good enough to catch most of the 623 * recurring UEs. If this fails to catch a recurrent UE, we'll 624 * retire the page the next time we see a UE on the page. 625 */ 626 kaddr = ppmapin(pp, PROT_READ|PROT_WRITE, (caddr_t)-1); 627 628 pa = ptob((uint64_t)page_pptonum(pp)); 629 pa_hi = (uint32_t)(pa >> 32); 630 pa_lo = (uint32_t)pa; 631 632 /* 633 * Disable preemption to prevent the off chance that 634 * we migrate while in the middle of running through 635 * the bit pattern and run on a different processor 636 * than what we started on. 637 */ 638 kpreempt_disable(); 639 640 /* 641 * Fill the page with each (0x00 - 0xFF] bit pattern, flushing 642 * the cache in between reading and writing. We do this under 643 * on_trap() protection to avoid recursion. 644 */ 645 if (on_trap(&otd, OT_DATA_EC)) { 646 PR_MESSAGE(CE_WARN, 1, MSG_UE, pa); 647 errors = 1; 648 } else { 649 for (wb = 0xff; wb > 0; wb--) { 650 for (i = 0; i < PAGESIZE; i++) { 651 kaddr[i] = wb; 652 } 653 654 sync_data_memory(kaddr, PAGESIZE); 655 656 for (i = 0; i < PAGESIZE; i++) { 657 rb = kaddr[i]; 658 if (rb != wb) { 659 /* 660 * We had a mismatch without a trap. 661 * Uh-oh. Something is really wrong 662 * with this system. 663 */ 664 if (page_retire_messages) { 665 cmn_err(CE_WARN, MSG_DM, 666 pa_hi, pa_lo, rb, wb); 667 } 668 errors = 1; 669 goto out; /* double break */ 670 } 671 } 672 } 673 } 674 out: 675 no_trap(); 676 kpreempt_enable(); 677 ppmapout(kaddr); 678 679 return (errors ? 0 : 1); 680 } 681 682 /* 683 * Try to clear a page_t with a single UE. If the UE was transient, it is 684 * returned to service, and we return 1. Otherwise we return 0 meaning 685 * that further processing is required to retire the page. 686 */ 687 static int 688 page_retire_transient_ue(page_t *pp) 689 { 690 ASSERT(PAGE_EXCL(pp)); 691 ASSERT(!hat_page_is_mapped(pp)); 692 693 /* 694 * If this page is a repeat offender, retire it under the 695 * "two strikes and you're out" rule. The caller is responsible 696 * for scrubbing the page to try to clear the error. 697 */ 698 if (pp->p_toxic & PR_UE_SCRUBBED) { 699 PR_INCR_KSTAT(pr_ue_persistent); 700 return (0); 701 } 702 703 if (page_clear_transient_ue(pp)) { 704 /* 705 * We set the PR_SCRUBBED_UE bit; if we ever see this 706 * page again, we will retire it, no questions asked. 707 */ 708 page_settoxic(pp, PR_UE_SCRUBBED); 709 710 if (page_retire_first_ue) { 711 PR_INCR_KSTAT(pr_ue_cleared_retire); 712 return (0); 713 } else { 714 PR_INCR_KSTAT(pr_ue_cleared_free); 715 716 page_clrtoxic(pp, PR_UE | PR_MCE | PR_MSG); 717 718 /* LINTED: CONSTCOND */ 719 VN_DISPOSE(pp, B_FREE, 1, kcred); 720 return (1); 721 } 722 } 723 724 PR_INCR_KSTAT(pr_ue_persistent); 725 return (0); 726 } 727 728 /* 729 * Update the statistics dynamically when our kstat is read. 730 */ 731 static int 732 page_retire_kstat_update(kstat_t *ksp, int rw) 733 { 734 struct page_retire_kstat *pr; 735 736 if (ksp == NULL) 737 return (EINVAL); 738 739 switch (rw) { 740 741 case KSTAT_READ: 742 pr = (struct page_retire_kstat *)ksp->ks_data; 743 ASSERT(pr == &page_retire_kstat); 744 pr->pr_limit.value.ui64 = PAGE_RETIRE_LIMIT; 745 return (0); 746 747 case KSTAT_WRITE: 748 return (EACCES); 749 750 default: 751 return (EINVAL); 752 } 753 /*NOTREACHED*/ 754 } 755 756 static int 757 pr_list_kstat_update(kstat_t *ksp, int rw) 758 { 759 uint_t count; 760 page_t *pp; 761 kmutex_t *vphm; 762 763 if (rw == KSTAT_WRITE) 764 return (EACCES); 765 766 vphm = page_vnode_mutex(retired_pages); 767 mutex_enter(vphm); 768 /* Needs to be under a lock so that for loop will work right */ 769 if (retired_pages->v_pages == NULL) { 770 mutex_exit(vphm); 771 ksp->ks_ndata = 0; 772 ksp->ks_data_size = 0; 773 return (0); 774 } 775 776 count = 1; 777 for (pp = retired_pages->v_pages->p_vpnext; 778 pp != retired_pages->v_pages; pp = pp->p_vpnext) { 779 count++; 780 } 781 mutex_exit(vphm); 782 783 ksp->ks_ndata = count; 784 ksp->ks_data_size = count * 2 * sizeof (uint64_t); 785 786 return (0); 787 } 788 789 /* 790 * all spans will be pagesize and no coalescing will be done with the 791 * list produced. 792 */ 793 static int 794 pr_list_kstat_snapshot(kstat_t *ksp, void *buf, int rw) 795 { 796 kmutex_t *vphm; 797 page_t *pp; 798 struct memunit { 799 uint64_t address; 800 uint64_t size; 801 } *kspmem; 802 803 if (rw == KSTAT_WRITE) 804 return (EACCES); 805 806 ksp->ks_snaptime = gethrtime(); 807 808 kspmem = (struct memunit *)buf; 809 810 vphm = page_vnode_mutex(retired_pages); 811 mutex_enter(vphm); 812 pp = retired_pages->v_pages; 813 if (((caddr_t)kspmem >= (caddr_t)buf + ksp->ks_data_size) || 814 (pp == NULL)) { 815 mutex_exit(vphm); 816 return (0); 817 } 818 kspmem->address = ptob(pp->p_pagenum); 819 kspmem->size = PAGESIZE; 820 kspmem++; 821 for (pp = pp->p_vpnext; pp != retired_pages->v_pages; 822 pp = pp->p_vpnext, kspmem++) { 823 if ((caddr_t)kspmem >= (caddr_t)buf + ksp->ks_data_size) 824 break; 825 kspmem->address = ptob(pp->p_pagenum); 826 kspmem->size = PAGESIZE; 827 } 828 mutex_exit(vphm); 829 830 return (0); 831 } 832 833 /* 834 * page_retire_pend_count -- helper function for page_capture_thread, 835 * returns the number of pages pending retirement. 836 */ 837 uint64_t 838 page_retire_pend_count(void) 839 { 840 return (PR_KSTAT_PENDING); 841 } 842 843 uint64_t 844 page_retire_pend_kas_count(void) 845 { 846 return (PR_KSTAT_PENDING_KAS); 847 } 848 849 void 850 page_retire_incr_pend_count(void *datap) 851 { 852 PR_INCR_KSTAT(pr_pending); 853 854 if ((datap == &kvp) || (datap == &zvp)) { 855 PR_INCR_KSTAT(pr_pending_kas); 856 } 857 } 858 859 void 860 page_retire_decr_pend_count(void *datap) 861 { 862 PR_DECR_KSTAT(pr_pending); 863 864 if ((datap == &kvp) || (datap == &zvp)) { 865 PR_DECR_KSTAT(pr_pending_kas); 866 } 867 } 868 869 /* 870 * Initialize the page retire mechanism: 871 * 872 * - Establish the correctable error retire limit. 873 * - Initialize locks. 874 * - Build the retired_pages vnode. 875 * - Set up the kstats. 876 * - Fire off the background thread. 877 * - Tell page_retire() it's OK to start retiring pages. 878 */ 879 void 880 page_retire_init(void) 881 { 882 const fs_operation_def_t retired_vnodeops_template[] = { 883 { NULL, NULL } 884 }; 885 struct vnodeops *vops; 886 kstat_t *ksp; 887 888 const uint_t page_retire_ndata = 889 sizeof (page_retire_kstat) / sizeof (kstat_named_t); 890 891 ASSERT(page_retire_ksp == NULL); 892 893 if (max_pages_retired_bps <= 0) { 894 max_pages_retired_bps = MCE_BPT; 895 } 896 897 mutex_init(&pr_q_mutex, NULL, MUTEX_DEFAULT, NULL); 898 899 retired_pages = vn_alloc(KM_SLEEP); 900 if (vn_make_ops("retired_pages", retired_vnodeops_template, &vops)) { 901 cmn_err(CE_PANIC, 902 "page_retired_init: can't make retired vnodeops"); 903 } 904 vn_setops(retired_pages, vops); 905 906 if ((page_retire_ksp = kstat_create("unix", 0, "page_retire", 907 "misc", KSTAT_TYPE_NAMED, page_retire_ndata, 908 KSTAT_FLAG_VIRTUAL)) == NULL) { 909 cmn_err(CE_WARN, "kstat_create for page_retire failed"); 910 } else { 911 page_retire_ksp->ks_data = (void *)&page_retire_kstat; 912 page_retire_ksp->ks_update = page_retire_kstat_update; 913 kstat_install(page_retire_ksp); 914 } 915 916 mutex_init(&pr_list_kstat_mutex, NULL, MUTEX_DEFAULT, NULL); 917 ksp = kstat_create("unix", 0, "page_retire_list", "misc", 918 KSTAT_TYPE_RAW, 0, KSTAT_FLAG_VAR_SIZE | KSTAT_FLAG_VIRTUAL); 919 if (ksp != NULL) { 920 ksp->ks_update = pr_list_kstat_update; 921 ksp->ks_snapshot = pr_list_kstat_snapshot; 922 ksp->ks_lock = &pr_list_kstat_mutex; 923 kstat_install(ksp); 924 } 925 926 memscrub_notify_func = 927 (void(*)(uint64_t))kobj_getsymvalue("memscrub_notify", 0); 928 929 page_capture_register_callback(PC_RETIRE, -1, page_retire_pp_finish); 930 pr_enable = 1; 931 } 932 933 /* 934 * page_retire_hunt() callback for the retire thread. 935 */ 936 static void 937 page_retire_thread_cb(page_t *pp) 938 { 939 PR_DEBUG(prd_tctop); 940 if (!PP_ISKAS(pp) && page_trylock(pp, SE_EXCL)) { 941 PR_DEBUG(prd_tclocked); 942 page_unlock(pp); 943 } 944 } 945 946 /* 947 * Callback used by page_trycapture() to finish off retiring a page. 948 * The page has already been cleaned and we've been given sole access to 949 * it. 950 * Always returns 0 to indicate that callback succeded as the callback never 951 * fails to finish retiring the given page. 952 */ 953 /*ARGSUSED*/ 954 static int 955 page_retire_pp_finish(page_t *pp, void *notused, uint_t flags) 956 { 957 int toxic; 958 959 ASSERT(PAGE_EXCL(pp)); 960 ASSERT(pp->p_iolock_state == 0); 961 ASSERT(pp->p_szc == 0); 962 963 toxic = pp->p_toxic; 964 965 /* 966 * The problem page is locked, demoted, unmapped, not free, 967 * hashed out, and not COW or mlocked (whew!). 968 * 969 * Now we select our ammunition, take it around back, and shoot it. 970 */ 971 if (toxic & PR_UE) { 972 ue_error: 973 if (page_retire_transient_ue(pp)) { 974 PR_DEBUG(prd_uescrubbed); 975 (void) page_retire_done(pp, PRD_UE_SCRUBBED); 976 } else { 977 PR_DEBUG(prd_uenotscrubbed); 978 page_retire_destroy(pp); 979 (void) page_retire_done(pp, PRD_SUCCESS); 980 } 981 return (0); 982 } else if (toxic & PR_FMA) { 983 PR_DEBUG(prd_fma); 984 page_retire_destroy(pp); 985 (void) page_retire_done(pp, PRD_SUCCESS); 986 return (0); 987 } else if (toxic & PR_MCE) { 988 PR_DEBUG(prd_mce); 989 page_retire_destroy(pp); 990 (void) page_retire_done(pp, PRD_SUCCESS); 991 return (0); 992 } 993 994 /* 995 * When page_retire_first_ue is set to zero and a UE occurs which is 996 * transient, it's possible that we clear some flags set by a second 997 * UE error on the page which occurs while the first is currently being 998 * handled and thus we need to handle the case where none of the above 999 * are set. In this instance, PR_UE_SCRUBBED should be set and thus 1000 * we should execute the UE code above. 1001 */ 1002 if (toxic & PR_UE_SCRUBBED) { 1003 goto ue_error; 1004 } 1005 1006 /* 1007 * It's impossible to get here. 1008 */ 1009 panic("bad toxic flags 0x%x in page_retire_pp_finish\n", toxic); 1010 return (0); 1011 } 1012 1013 /* 1014 * page_retire() - the front door in to retire a page. 1015 * 1016 * Ideally, page_retire() would instantly retire the requested page. 1017 * Unfortunately, some pages are locked or otherwise tied up and cannot be 1018 * retired right away. We use the page capture logic to deal with this 1019 * situation as it will continuously try to retire the page in the background 1020 * if the first attempt fails. Success is determined by looking to see whether 1021 * the page has been retired after the page_trycapture() attempt. 1022 * 1023 * Returns: 1024 * 1025 * - 0 on success, 1026 * - EINVAL when the PA is whacko, 1027 * - EIO if the page is already retired or already pending retirement, or 1028 * - EAGAIN if the page could not be _immediately_ retired but is pending. 1029 */ 1030 int 1031 page_retire(uint64_t pa, uchar_t reason) 1032 { 1033 page_t *pp; 1034 1035 ASSERT(reason & PR_REASONS); /* there must be a reason */ 1036 ASSERT(!(reason & ~PR_REASONS)); /* but no other bits */ 1037 1038 pp = page_numtopp_nolock(mmu_btop(pa)); 1039 if (pp == NULL) { 1040 PR_MESSAGE(CE_WARN, 1, "Cannot schedule clearing of error on" 1041 " page 0x%08x.%08x; page is not relocatable memory", pa); 1042 return (page_retire_done(pp, PRD_INVALID_PA)); 1043 } 1044 if (PP_RETIRED(pp)) { 1045 PR_DEBUG(prd_dup1); 1046 return (page_retire_done(pp, PRD_DUPLICATE)); 1047 } 1048 1049 if (memscrub_notify_func != NULL) { 1050 (void) memscrub_notify_func(pa); 1051 } 1052 1053 if ((reason & PR_UE) && !PP_TOXIC(pp)) { 1054 PR_MESSAGE(CE_NOTE, 1, "Scheduling clearing of error on" 1055 " page 0x%08x.%08x", pa); 1056 } else if (PP_PR_REQ(pp)) { 1057 PR_DEBUG(prd_dup2); 1058 return (page_retire_done(pp, PRD_DUPLICATE)); 1059 } else { 1060 PR_MESSAGE(CE_NOTE, 1, "Scheduling removal of" 1061 " page 0x%08x.%08x", pa); 1062 } 1063 1064 /* Avoid setting toxic bits in the first place */ 1065 if ((reason & (PR_FMA | PR_MCE)) && !(reason & PR_UE) && 1066 page_retire_limit()) { 1067 return (page_retire_done(pp, PRD_LIMIT)); 1068 } 1069 1070 if (MTBF(pr_calls, pr_mtbf)) { 1071 page_settoxic(pp, reason); 1072 if (page_trycapture(pp, 0, CAPTURE_RETIRE, pp->p_vnode) == 0) { 1073 PR_DEBUG(prd_prlocked); 1074 } else { 1075 PR_DEBUG(prd_prnotlocked); 1076 } 1077 } else { 1078 PR_DEBUG(prd_prnotlocked); 1079 } 1080 1081 if (PP_RETIRED(pp)) { 1082 PR_DEBUG(prd_prretired); 1083 return (0); 1084 } else { 1085 cv_signal(&pc_cv); 1086 PR_INCR_KSTAT(pr_failed); 1087 1088 if (pp->p_toxic & PR_MSG) { 1089 return (page_retire_done(pp, PRD_FAILED)); 1090 } else { 1091 return (page_retire_done(pp, PRD_PENDING)); 1092 } 1093 } 1094 } 1095 1096 /* 1097 * Take a retired page off the retired-pages vnode and clear the toxic flags. 1098 * If "free" is nonzero, lock it and put it back on the freelist. If "free" 1099 * is zero, the caller already holds SE_EXCL lock so we simply unretire it 1100 * and don't do anything else with it. 1101 * 1102 * Any unretire messages are printed from this routine. 1103 * 1104 * Returns 0 if page pp was unretired; else an error code. 1105 * 1106 * If flags is: 1107 * PR_UNR_FREE - lock the page, clear the toxic flags and free it 1108 * to the freelist. 1109 * PR_UNR_TEMP - lock the page, unretire it, leave the toxic 1110 * bits set as is and return it to the caller. 1111 * PR_UNR_CLEAN - page is SE_EXCL locked, unretire it, clear the 1112 * toxic flags and return it to caller as is. 1113 */ 1114 int 1115 page_unretire_pp(page_t *pp, int flags) 1116 { 1117 /* 1118 * To be retired, a page has to be hashed onto the retired_pages vnode 1119 * and have PR_RETIRED set in p_toxic. 1120 */ 1121 if (flags == PR_UNR_CLEAN || 1122 page_try_reclaim_lock(pp, SE_EXCL, SE_RETIRED)) { 1123 ASSERT(PAGE_EXCL(pp)); 1124 PR_DEBUG(prd_ulocked); 1125 if (!PP_RETIRED(pp)) { 1126 PR_DEBUG(prd_unotretired); 1127 page_unlock(pp); 1128 return (page_retire_done(pp, PRD_UNR_NOT)); 1129 } 1130 1131 PR_MESSAGE(CE_NOTE, 1, "unretiring retired" 1132 " page 0x%08x.%08x", mmu_ptob((uint64_t)pp->p_pagenum)); 1133 if (pp->p_toxic & PR_FMA) { 1134 PR_DECR_KSTAT(pr_fma); 1135 } else if (pp->p_toxic & PR_UE) { 1136 PR_DECR_KSTAT(pr_ue); 1137 } else { 1138 PR_DECR_KSTAT(pr_mce); 1139 } 1140 1141 if (flags == PR_UNR_TEMP) 1142 page_clrtoxic(pp, PR_RETIRED); 1143 else 1144 page_clrtoxic(pp, PR_TOXICFLAGS); 1145 1146 if (flags == PR_UNR_FREE) { 1147 PR_DEBUG(prd_udestroy); 1148 page_destroy(pp, 0); 1149 } else { 1150 PR_DEBUG(prd_uhashout); 1151 page_hashout(pp, NULL); 1152 } 1153 1154 mutex_enter(&freemem_lock); 1155 availrmem++; 1156 mutex_exit(&freemem_lock); 1157 1158 PR_DEBUG(prd_uunretired); 1159 PR_DECR_KSTAT(pr_retired); 1160 PR_INCR_KSTAT(pr_unretired); 1161 return (page_retire_done(pp, PRD_UNR_SUCCESS)); 1162 } 1163 PR_DEBUG(prd_unotlocked); 1164 return (page_retire_done(pp, PRD_UNR_CANTLOCK)); 1165 } 1166 1167 /* 1168 * Return a page to service by moving it from the retired_pages vnode 1169 * onto the freelist. 1170 * 1171 * Called from mmioctl_page_retire() on behalf of the FMA DE. 1172 * 1173 * Returns: 1174 * 1175 * - 0 if the page is unretired, 1176 * - EAGAIN if the pp can not be locked, 1177 * - EINVAL if the PA is whacko, and 1178 * - EIO if the pp is not retired. 1179 */ 1180 int 1181 page_unretire(uint64_t pa) 1182 { 1183 page_t *pp; 1184 1185 pp = page_numtopp_nolock(mmu_btop(pa)); 1186 if (pp == NULL) { 1187 return (page_retire_done(pp, PRD_INVALID_PA)); 1188 } 1189 1190 return (page_unretire_pp(pp, PR_UNR_FREE)); 1191 } 1192 1193 /* 1194 * Test a page to see if it is retired. If errors is non-NULL, the toxic 1195 * bits of the page are returned. Returns 0 on success, error code on failure. 1196 */ 1197 int 1198 page_retire_check_pp(page_t *pp, uint64_t *errors) 1199 { 1200 int rc; 1201 1202 if (PP_RETIRED(pp)) { 1203 PR_DEBUG(prd_checkhit); 1204 rc = 0; 1205 } else if (PP_PR_REQ(pp)) { 1206 PR_DEBUG(prd_checkmiss_pend); 1207 rc = EAGAIN; 1208 } else { 1209 PR_DEBUG(prd_checkmiss_noerr); 1210 rc = EIO; 1211 } 1212 1213 /* 1214 * We have magically arranged the bit values returned to fmd(1M) 1215 * to line up with the FMA, MCE, and UE bits of the page_t. 1216 */ 1217 if (errors) { 1218 uint64_t toxic = (uint64_t)(pp->p_toxic & PR_ERRMASK); 1219 if (toxic & PR_UE_SCRUBBED) { 1220 toxic &= ~PR_UE_SCRUBBED; 1221 toxic |= PR_UE; 1222 } 1223 *errors = toxic; 1224 } 1225 1226 return (rc); 1227 } 1228 1229 /* 1230 * Test to see if the page_t for a given PA is retired, and return the 1231 * hardware errors we have seen on the page if requested. 1232 * 1233 * Called from mmioctl_page_retire on behalf of the FMA DE. 1234 * 1235 * Returns: 1236 * 1237 * - 0 if the page is retired, 1238 * - EIO if the page is not retired and has no errors, 1239 * - EAGAIN if the page is not retired but is pending; and 1240 * - EINVAL if the PA is whacko. 1241 */ 1242 int 1243 page_retire_check(uint64_t pa, uint64_t *errors) 1244 { 1245 page_t *pp; 1246 1247 if (errors) { 1248 *errors = 0; 1249 } 1250 1251 pp = page_numtopp_nolock(mmu_btop(pa)); 1252 if (pp == NULL) { 1253 return (page_retire_done(pp, PRD_INVALID_PA)); 1254 } 1255 1256 return (page_retire_check_pp(pp, errors)); 1257 } 1258 1259 /* 1260 * Page retire self-test. For now, it always returns 0. 1261 */ 1262 int 1263 page_retire_test(void) 1264 { 1265 page_t *first, *pp, *cpp, *cpp2, *lpp; 1266 1267 /* 1268 * Tests the corner case where a large page can't be retired 1269 * because one of the constituent pages is locked. We mark 1270 * one page to be retired and try to retire it, and mark the 1271 * other page to be retired but don't try to retire it, so 1272 * that page_unlock() in the failure path will recurse and try 1273 * to retire THAT page. This is the worst possible situation 1274 * we can get ourselves into. 1275 */ 1276 memsegs_lock(0); 1277 pp = first = page_first(); 1278 do { 1279 if (pp->p_szc && PP_PAGEROOT(pp) == pp) { 1280 cpp = pp + 1; 1281 lpp = PP_ISFREE(pp)? pp : pp + 2; 1282 cpp2 = pp + 3; 1283 if (!page_trylock(lpp, pp == lpp? SE_EXCL : SE_SHARED)) 1284 continue; 1285 if (!page_trylock(cpp, SE_EXCL)) { 1286 page_unlock(lpp); 1287 continue; 1288 } 1289 1290 /* fails */ 1291 (void) page_retire(ptob(cpp->p_pagenum), PR_FMA); 1292 1293 page_unlock(lpp); 1294 page_unlock(cpp); 1295 (void) page_retire(ptob(cpp->p_pagenum), PR_FMA); 1296 (void) page_retire(ptob(cpp2->p_pagenum), PR_FMA); 1297 } 1298 } while ((pp = page_next(pp)) != first); 1299 memsegs_unlock(0); 1300 1301 return (0); 1302 } 1303