17c478bd9Sstevel@tonic-gate /* 27c478bd9Sstevel@tonic-gate * CDDL HEADER START 37c478bd9Sstevel@tonic-gate * 47c478bd9Sstevel@tonic-gate * The contents of this file are subject to the terms of the 5c210ded4Sesaxe * Common Development and Distribution License (the "License"). 6c210ded4Sesaxe * You may not use this file except in compliance with the License. 77c478bd9Sstevel@tonic-gate * 87c478bd9Sstevel@tonic-gate * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 97c478bd9Sstevel@tonic-gate * or http://www.opensolaris.org/os/licensing. 107c478bd9Sstevel@tonic-gate * See the License for the specific language governing permissions 117c478bd9Sstevel@tonic-gate * and limitations under the License. 127c478bd9Sstevel@tonic-gate * 137c478bd9Sstevel@tonic-gate * When distributing Covered Code, include this CDDL HEADER in each 147c478bd9Sstevel@tonic-gate * file and include the License file at usr/src/OPENSOLARIS.LICENSE. 157c478bd9Sstevel@tonic-gate * If applicable, add the following below this CDDL HEADER, with the 167c478bd9Sstevel@tonic-gate * fields enclosed by brackets "[]" replaced with your own identifying 177c478bd9Sstevel@tonic-gate * information: Portions Copyright [yyyy] [name of copyright owner] 187c478bd9Sstevel@tonic-gate * 197c478bd9Sstevel@tonic-gate * CDDL HEADER END 207c478bd9Sstevel@tonic-gate */ 217c478bd9Sstevel@tonic-gate /* 22c210ded4Sesaxe * Copyright 2008 Sun Microsystems, Inc. All rights reserved. 237c478bd9Sstevel@tonic-gate * Use is subject to license terms. 247c478bd9Sstevel@tonic-gate */ 257c478bd9Sstevel@tonic-gate 267c478bd9Sstevel@tonic-gate /* 277c478bd9Sstevel@tonic-gate * The Cyclic Subsystem 287c478bd9Sstevel@tonic-gate * -------------------- 297c478bd9Sstevel@tonic-gate * 307c478bd9Sstevel@tonic-gate * Prehistory 317c478bd9Sstevel@tonic-gate * 327c478bd9Sstevel@tonic-gate * Historically, most computer architectures have specified interval-based 337c478bd9Sstevel@tonic-gate * timer parts (e.g. SPARCstation's counter/timer; Intel's i8254). While 347c478bd9Sstevel@tonic-gate * these parts deal in relative (i.e. not absolute) time values, they are 357c478bd9Sstevel@tonic-gate * typically used by the operating system to implement the abstraction of 367c478bd9Sstevel@tonic-gate * absolute time. As a result, these parts cannot typically be reprogrammed 377c478bd9Sstevel@tonic-gate * without introducing error in the system's notion of time. 387c478bd9Sstevel@tonic-gate * 397c478bd9Sstevel@tonic-gate * Starting in about 1994, chip architectures began specifying high resolution 407c478bd9Sstevel@tonic-gate * timestamp registers. As of this writing (1999), all major chip families 417c478bd9Sstevel@tonic-gate * (UltraSPARC, PentiumPro, MIPS, PowerPC, Alpha) have high resolution 427c478bd9Sstevel@tonic-gate * timestamp registers, and two (UltraSPARC and MIPS) have added the capacity 437c478bd9Sstevel@tonic-gate * to interrupt based on timestamp values. These timestamp-compare registers 447c478bd9Sstevel@tonic-gate * present a time-based interrupt source which can be reprogrammed arbitrarily 457c478bd9Sstevel@tonic-gate * often without introducing error. Given the low cost of implementing such a 467c478bd9Sstevel@tonic-gate * timestamp-compare register (and the tangible benefit of eliminating 477c478bd9Sstevel@tonic-gate * discrete timer parts), it is reasonable to expect that future chip 487c478bd9Sstevel@tonic-gate * architectures will adopt this feature. 497c478bd9Sstevel@tonic-gate * 507c478bd9Sstevel@tonic-gate * The cyclic subsystem has been designed to take advantage of chip 517c478bd9Sstevel@tonic-gate * architectures with the capacity to interrupt based on absolute, high 527c478bd9Sstevel@tonic-gate * resolution values of time. 537c478bd9Sstevel@tonic-gate * 547c478bd9Sstevel@tonic-gate * Subsystem Overview 557c478bd9Sstevel@tonic-gate * 567c478bd9Sstevel@tonic-gate * The cyclic subsystem is a low-level kernel subsystem designed to provide 577c478bd9Sstevel@tonic-gate * arbitrarily high resolution, per-CPU interval timers (to avoid colliding 587c478bd9Sstevel@tonic-gate * with existing terms, we dub such an interval timer a "cyclic"). Cyclics 597c478bd9Sstevel@tonic-gate * can be specified to fire at high, lock or low interrupt level, and may be 607c478bd9Sstevel@tonic-gate * optionally bound to a CPU or a CPU partition. A cyclic's CPU or CPU 617c478bd9Sstevel@tonic-gate * partition binding may be changed dynamically; the cyclic will be "juggled" 627c478bd9Sstevel@tonic-gate * to a CPU which satisfies the new binding. Alternatively, a cyclic may 637c478bd9Sstevel@tonic-gate * be specified to be "omnipresent", denoting firing on all online CPUs. 647c478bd9Sstevel@tonic-gate * 657c478bd9Sstevel@tonic-gate * Cyclic Subsystem Interface Overview 667c478bd9Sstevel@tonic-gate * ----------------------------------- 677c478bd9Sstevel@tonic-gate * 687c478bd9Sstevel@tonic-gate * The cyclic subsystem has interfaces with the kernel at-large, with other 697c478bd9Sstevel@tonic-gate * kernel subsystems (e.g. the processor management subsystem, the checkpoint 707c478bd9Sstevel@tonic-gate * resume subsystem) and with the platform (the cyclic backend). Each 717c478bd9Sstevel@tonic-gate * of these interfaces is given a brief synopsis here, and is described 727c478bd9Sstevel@tonic-gate * in full above the interface's implementation. 737c478bd9Sstevel@tonic-gate * 747c478bd9Sstevel@tonic-gate * The following diagram displays the cyclic subsystem's interfaces to 757c478bd9Sstevel@tonic-gate * other kernel components. The arrows denote a "calls" relationship, with 767c478bd9Sstevel@tonic-gate * the large arrow indicating the cyclic subsystem's consumer interface. 777c478bd9Sstevel@tonic-gate * Each arrow is labeled with the section in which the corresponding 787c478bd9Sstevel@tonic-gate * interface is described. 797c478bd9Sstevel@tonic-gate * 807c478bd9Sstevel@tonic-gate * Kernel at-large consumers 817c478bd9Sstevel@tonic-gate * -----------++------------ 827c478bd9Sstevel@tonic-gate * || 837c478bd9Sstevel@tonic-gate * || 847c478bd9Sstevel@tonic-gate * _||_ 857c478bd9Sstevel@tonic-gate * \ / 867c478bd9Sstevel@tonic-gate * \/ 877c478bd9Sstevel@tonic-gate * +---------------------+ 887c478bd9Sstevel@tonic-gate * | | 897c478bd9Sstevel@tonic-gate * | Cyclic subsystem |<----------- Other kernel subsystems 907c478bd9Sstevel@tonic-gate * | | 917c478bd9Sstevel@tonic-gate * +---------------------+ 927c478bd9Sstevel@tonic-gate * ^ | 937c478bd9Sstevel@tonic-gate * | | 947c478bd9Sstevel@tonic-gate * | | 957c478bd9Sstevel@tonic-gate * | v 967c478bd9Sstevel@tonic-gate * +---------------------+ 977c478bd9Sstevel@tonic-gate * | | 987c478bd9Sstevel@tonic-gate * | Cyclic backend | 997c478bd9Sstevel@tonic-gate * | (platform specific) | 1007c478bd9Sstevel@tonic-gate * | | 1017c478bd9Sstevel@tonic-gate * +---------------------+ 1027c478bd9Sstevel@tonic-gate * 1037c478bd9Sstevel@tonic-gate * 1047c478bd9Sstevel@tonic-gate * Kernel At-Large Interfaces 1057c478bd9Sstevel@tonic-gate * 1067c478bd9Sstevel@tonic-gate * cyclic_add() <-- Creates a cyclic 1077c478bd9Sstevel@tonic-gate * cyclic_add_omni() <-- Creates an omnipresent cyclic 1087c478bd9Sstevel@tonic-gate * cyclic_remove() <-- Removes a cyclic 1097c478bd9Sstevel@tonic-gate * cyclic_bind() <-- Change a cyclic's CPU or partition binding 110*87a18d3fSMadhavan Venkataraman * cyclic_reprogram() <-- Reprogram a cyclic's expiration 1117c478bd9Sstevel@tonic-gate * 1127c478bd9Sstevel@tonic-gate * Inter-subsystem Interfaces 1137c478bd9Sstevel@tonic-gate * 1147c478bd9Sstevel@tonic-gate * cyclic_juggle() <-- Juggles cyclics away from a CPU 1157c478bd9Sstevel@tonic-gate * cyclic_offline() <-- Offlines cyclic operation on a CPU 1167c478bd9Sstevel@tonic-gate * cyclic_online() <-- Reenables operation on an offlined CPU 1177c478bd9Sstevel@tonic-gate * cyclic_move_in() <-- Notifies subsystem of change in CPU partition 1187c478bd9Sstevel@tonic-gate * cyclic_move_out() <-- Notifies subsystem of change in CPU partition 1197c478bd9Sstevel@tonic-gate * cyclic_suspend() <-- Suspends the cyclic subsystem on all CPUs 1207c478bd9Sstevel@tonic-gate * cyclic_resume() <-- Resumes the cyclic subsystem on all CPUs 1217c478bd9Sstevel@tonic-gate * 1227c478bd9Sstevel@tonic-gate * Backend Interfaces 1237c478bd9Sstevel@tonic-gate * 1247c478bd9Sstevel@tonic-gate * cyclic_init() <-- Initializes the cyclic subsystem 1257c478bd9Sstevel@tonic-gate * cyclic_fire() <-- CY_HIGH_LEVEL interrupt entry point 1267c478bd9Sstevel@tonic-gate * cyclic_softint() <-- CY_LOCK/LOW_LEVEL soft interrupt entry point 1277c478bd9Sstevel@tonic-gate * 1287c478bd9Sstevel@tonic-gate * The backend-supplied interfaces (through the cyc_backend structure) are 1297c478bd9Sstevel@tonic-gate * documented in detail in <sys/cyclic_impl.h> 1307c478bd9Sstevel@tonic-gate * 1317c478bd9Sstevel@tonic-gate * 1327c478bd9Sstevel@tonic-gate * Cyclic Subsystem Implementation Overview 1337c478bd9Sstevel@tonic-gate * ---------------------------------------- 1347c478bd9Sstevel@tonic-gate * 1357c478bd9Sstevel@tonic-gate * The cyclic subsystem is designed to minimize interference between cyclics 1367c478bd9Sstevel@tonic-gate * on different CPUs. Thus, all of the cyclic subsystem's data structures 1377c478bd9Sstevel@tonic-gate * hang off of a per-CPU structure, cyc_cpu. 1387c478bd9Sstevel@tonic-gate * 1397c478bd9Sstevel@tonic-gate * Each cyc_cpu has a power-of-two sized array of cyclic structures (the 1407c478bd9Sstevel@tonic-gate * cyp_cyclics member of the cyc_cpu structure). If cyclic_add() is called 1417c478bd9Sstevel@tonic-gate * and there does not exist a free slot in the cyp_cyclics array, the size of 1427c478bd9Sstevel@tonic-gate * the array will be doubled. The array will never shrink. Cyclics are 1437c478bd9Sstevel@tonic-gate * referred to by their index in the cyp_cyclics array, which is of type 1447c478bd9Sstevel@tonic-gate * cyc_index_t. 1457c478bd9Sstevel@tonic-gate * 1467c478bd9Sstevel@tonic-gate * The cyclics are kept sorted by expiration time in the cyc_cpu's heap. The 1477c478bd9Sstevel@tonic-gate * heap is keyed by cyclic expiration time, with parents expiring earlier 1487c478bd9Sstevel@tonic-gate * than their children. 1497c478bd9Sstevel@tonic-gate * 1507c478bd9Sstevel@tonic-gate * Heap Management 1517c478bd9Sstevel@tonic-gate * 1527c478bd9Sstevel@tonic-gate * The heap is managed primarily by cyclic_fire(). Upon entry, cyclic_fire() 1537c478bd9Sstevel@tonic-gate * compares the root cyclic's expiration time to the current time. If the 1547c478bd9Sstevel@tonic-gate * expiration time is in the past, cyclic_expire() is called on the root 1557c478bd9Sstevel@tonic-gate * cyclic. Upon return from cyclic_expire(), the cyclic's new expiration time 1567c478bd9Sstevel@tonic-gate * is derived by adding its interval to its old expiration time, and a 1577c478bd9Sstevel@tonic-gate * downheap operation is performed. After the downheap, cyclic_fire() 1587c478bd9Sstevel@tonic-gate * examines the (potentially changed) root cyclic, repeating the 1597c478bd9Sstevel@tonic-gate * cyclic_expire()/add interval/cyclic_downheap() sequence until the root 1607c478bd9Sstevel@tonic-gate * cyclic has an expiration time in the future. This expiration time 1617c478bd9Sstevel@tonic-gate * (guaranteed to be the earliest in the heap) is then communicated to the 1627c478bd9Sstevel@tonic-gate * backend via cyb_reprogram. Optimal backends will next call cyclic_fire() 1637c478bd9Sstevel@tonic-gate * shortly after the root cyclic's expiration time. 1647c478bd9Sstevel@tonic-gate * 1657c478bd9Sstevel@tonic-gate * To allow efficient, deterministic downheap operations, we implement the 1667c478bd9Sstevel@tonic-gate * heap as an array (the cyp_heap member of the cyc_cpu structure), with each 1677c478bd9Sstevel@tonic-gate * element containing an index into the CPU's cyp_cyclics array. 1687c478bd9Sstevel@tonic-gate * 1697c478bd9Sstevel@tonic-gate * The heap is laid out in the array according to the following: 1707c478bd9Sstevel@tonic-gate * 1717c478bd9Sstevel@tonic-gate * 1. The root of the heap is always in the 0th element of the heap array 1727c478bd9Sstevel@tonic-gate * 2. The left and right children of the nth element are element 1737c478bd9Sstevel@tonic-gate * (((n + 1) << 1) - 1) and element ((n + 1) << 1), respectively. 1747c478bd9Sstevel@tonic-gate * 1757c478bd9Sstevel@tonic-gate * This layout is standard (see, e.g., Cormen's "Algorithms"); the proof 1767c478bd9Sstevel@tonic-gate * that these constraints correctly lay out a heap (or indeed, any binary 1777c478bd9Sstevel@tonic-gate * tree) is trivial and left to the reader. 1787c478bd9Sstevel@tonic-gate * 1797c478bd9Sstevel@tonic-gate * To see the heap by example, assume our cyclics array has the following 1807c478bd9Sstevel@tonic-gate * members (at time t): 1817c478bd9Sstevel@tonic-gate * 1827c478bd9Sstevel@tonic-gate * cy_handler cy_level cy_expire 1837c478bd9Sstevel@tonic-gate * --------------------------------------------- 1847c478bd9Sstevel@tonic-gate * [ 0] clock() LOCK t+10000000 1857c478bd9Sstevel@tonic-gate * [ 1] deadman() HIGH t+1000000000 1867c478bd9Sstevel@tonic-gate * [ 2] clock_highres_fire() LOW t+100 1877c478bd9Sstevel@tonic-gate * [ 3] clock_highres_fire() LOW t+1000 1887c478bd9Sstevel@tonic-gate * [ 4] clock_highres_fire() LOW t+500 1897c478bd9Sstevel@tonic-gate * [ 5] (free) -- -- 1907c478bd9Sstevel@tonic-gate * [ 6] (free) -- -- 1917c478bd9Sstevel@tonic-gate * [ 7] (free) -- -- 1927c478bd9Sstevel@tonic-gate * 1937c478bd9Sstevel@tonic-gate * The heap array could be: 1947c478bd9Sstevel@tonic-gate * 1957c478bd9Sstevel@tonic-gate * [0] [1] [2] [3] [4] [5] [6] [7] 1967c478bd9Sstevel@tonic-gate * +-----+-----+-----+-----+-----+-----+-----+-----+ 1977c478bd9Sstevel@tonic-gate * | | | | | | | | | 1987c478bd9Sstevel@tonic-gate * | 2 | 3 | 4 | 0 | 1 | x | x | x | 1997c478bd9Sstevel@tonic-gate * | | | | | | | | | 2007c478bd9Sstevel@tonic-gate * +-----+-----+-----+-----+-----+-----+-----+-----+ 2017c478bd9Sstevel@tonic-gate * 2027c478bd9Sstevel@tonic-gate * Graphically, this array corresponds to the following (excuse the ASCII art): 2037c478bd9Sstevel@tonic-gate * 2047c478bd9Sstevel@tonic-gate * 2 2057c478bd9Sstevel@tonic-gate * | 2067c478bd9Sstevel@tonic-gate * +------------------+------------------+ 2077c478bd9Sstevel@tonic-gate * 3 4 2087c478bd9Sstevel@tonic-gate * | 2097c478bd9Sstevel@tonic-gate * +---------+--------+ 2107c478bd9Sstevel@tonic-gate * 0 1 2117c478bd9Sstevel@tonic-gate * 2127c478bd9Sstevel@tonic-gate * Note that the heap is laid out by layer: all nodes at a given depth are 2137c478bd9Sstevel@tonic-gate * stored in consecutive elements of the array. Moreover, layers of 2147c478bd9Sstevel@tonic-gate * consecutive depths are in adjacent element ranges. This property 2157c478bd9Sstevel@tonic-gate * guarantees high locality of reference during downheap operations. 2167c478bd9Sstevel@tonic-gate * Specifically, we are guaranteed that we can downheap to a depth of 2177c478bd9Sstevel@tonic-gate * 2187c478bd9Sstevel@tonic-gate * lg (cache_line_size / sizeof (cyc_index_t)) 2197c478bd9Sstevel@tonic-gate * 2207c478bd9Sstevel@tonic-gate * nodes with at most one cache miss. On UltraSPARC (64 byte e-cache line 2217c478bd9Sstevel@tonic-gate * size), this corresponds to a depth of four nodes. Thus, if there are 2227c478bd9Sstevel@tonic-gate * fewer than sixteen cyclics in the heap, downheaps on UltraSPARC miss at 2237c478bd9Sstevel@tonic-gate * most once in the e-cache. 2247c478bd9Sstevel@tonic-gate * 2257c478bd9Sstevel@tonic-gate * Downheaps are required to compare siblings as they proceed down the 2267c478bd9Sstevel@tonic-gate * heap. For downheaps proceeding beyond the one-cache-miss depth, every 2277c478bd9Sstevel@tonic-gate * access to a left child could potentially miss in the cache. However, 2287c478bd9Sstevel@tonic-gate * if we assume 2297c478bd9Sstevel@tonic-gate * 2307c478bd9Sstevel@tonic-gate * (cache_line_size / sizeof (cyc_index_t)) > 2, 2317c478bd9Sstevel@tonic-gate * 2327c478bd9Sstevel@tonic-gate * then all siblings are guaranteed to be on the same cache line. Thus, the 2337c478bd9Sstevel@tonic-gate * miss on the left child will guarantee a hit on the right child; downheaps 2347c478bd9Sstevel@tonic-gate * will incur at most one cache miss per layer beyond the one-cache-miss 2357c478bd9Sstevel@tonic-gate * depth. The total number of cache misses for heap management during a 2367c478bd9Sstevel@tonic-gate * downheap operation is thus bounded by 2377c478bd9Sstevel@tonic-gate * 2387c478bd9Sstevel@tonic-gate * lg (n) - lg (cache_line_size / sizeof (cyc_index_t)) 2397c478bd9Sstevel@tonic-gate * 2407c478bd9Sstevel@tonic-gate * Traditional pointer-based heaps are implemented without regard to 2417c478bd9Sstevel@tonic-gate * locality. Downheaps can thus incur two cache misses per layer (one for 2427c478bd9Sstevel@tonic-gate * each child), but at most one cache miss at the root. This yields a bound 2437c478bd9Sstevel@tonic-gate * of 2447c478bd9Sstevel@tonic-gate * 2457c478bd9Sstevel@tonic-gate * 2 * lg (n) - 1 2467c478bd9Sstevel@tonic-gate * 2477c478bd9Sstevel@tonic-gate * on the total cache misses. 2487c478bd9Sstevel@tonic-gate * 2497c478bd9Sstevel@tonic-gate * This difference may seem theoretically trivial (the difference is, after 2507c478bd9Sstevel@tonic-gate * all, constant), but can become substantial in practice -- especially for 2517c478bd9Sstevel@tonic-gate * caches with very large cache lines and high miss penalties (e.g. TLBs). 2527c478bd9Sstevel@tonic-gate * 2537c478bd9Sstevel@tonic-gate * Heaps must always be full, balanced trees. Heap management must therefore 2547c478bd9Sstevel@tonic-gate * track the next point-of-insertion into the heap. In pointer-based heaps, 2557c478bd9Sstevel@tonic-gate * recomputing this point takes O(lg (n)). Given the layout of the 2567c478bd9Sstevel@tonic-gate * array-based implementation, however, the next point-of-insertion is 2577c478bd9Sstevel@tonic-gate * always: 2587c478bd9Sstevel@tonic-gate * 2597c478bd9Sstevel@tonic-gate * heap[number_of_elements] 2607c478bd9Sstevel@tonic-gate * 2617c478bd9Sstevel@tonic-gate * We exploit this property by implementing the free-list in the usused 2627c478bd9Sstevel@tonic-gate * heap elements. Heap insertion, therefore, consists only of filling in 2637c478bd9Sstevel@tonic-gate * the cyclic at cyp_cyclics[cyp_heap[number_of_elements]], incrementing 2647c478bd9Sstevel@tonic-gate * the number of elements, and performing an upheap. Heap deletion consists 2657c478bd9Sstevel@tonic-gate * of decrementing the number of elements, swapping the to-be-deleted element 2667c478bd9Sstevel@tonic-gate * with the element at cyp_heap[number_of_elements], and downheaping. 2677c478bd9Sstevel@tonic-gate * 2687c478bd9Sstevel@tonic-gate * Filling in more details in our earlier example: 2697c478bd9Sstevel@tonic-gate * 2707c478bd9Sstevel@tonic-gate * +--- free list head 2717c478bd9Sstevel@tonic-gate * | 2727c478bd9Sstevel@tonic-gate * V 2737c478bd9Sstevel@tonic-gate * 2747c478bd9Sstevel@tonic-gate * [0] [1] [2] [3] [4] [5] [6] [7] 2757c478bd9Sstevel@tonic-gate * +-----+-----+-----+-----+-----+-----+-----+-----+ 2767c478bd9Sstevel@tonic-gate * | | | | | | | | | 2777c478bd9Sstevel@tonic-gate * | 2 | 3 | 4 | 0 | 1 | 5 | 6 | 7 | 2787c478bd9Sstevel@tonic-gate * | | | | | | | | | 2797c478bd9Sstevel@tonic-gate * +-----+-----+-----+-----+-----+-----+-----+-----+ 2807c478bd9Sstevel@tonic-gate * 2817c478bd9Sstevel@tonic-gate * To insert into this heap, we would just need to fill in the cyclic at 2827c478bd9Sstevel@tonic-gate * cyp_cyclics[5], bump the number of elements (from 5 to 6) and perform 2837c478bd9Sstevel@tonic-gate * an upheap. 2847c478bd9Sstevel@tonic-gate * 2857c478bd9Sstevel@tonic-gate * If we wanted to remove, say, cyp_cyclics[3], we would first scan for it 2867c478bd9Sstevel@tonic-gate * in the cyp_heap, and discover it at cyp_heap[1]. We would then decrement 2877c478bd9Sstevel@tonic-gate * the number of elements (from 5 to 4), swap cyp_heap[1] with cyp_heap[4], 2887c478bd9Sstevel@tonic-gate * and perform a downheap from cyp_heap[1]. The linear scan is required 2897c478bd9Sstevel@tonic-gate * because the cyclic does not keep a backpointer into the heap. This makes 2907c478bd9Sstevel@tonic-gate * heap manipulation (e.g. downheaps) faster at the expense of removal 2917c478bd9Sstevel@tonic-gate * operations. 2927c478bd9Sstevel@tonic-gate * 2937c478bd9Sstevel@tonic-gate * Expiry processing 2947c478bd9Sstevel@tonic-gate * 2957c478bd9Sstevel@tonic-gate * As alluded to above, cyclic_expire() is called by cyclic_fire() at 2967c478bd9Sstevel@tonic-gate * CY_HIGH_LEVEL to expire a cyclic. Cyclic subsystem consumers are 2977c478bd9Sstevel@tonic-gate * guaranteed that for an arbitrary time t in the future, their cyclic 2987c478bd9Sstevel@tonic-gate * handler will have been called (t - cyt_when) / cyt_interval times. Thus, 2997c478bd9Sstevel@tonic-gate * there must be a one-to-one mapping between a cyclic's expiration at 3007c478bd9Sstevel@tonic-gate * CY_HIGH_LEVEL and its execution at the desired level (either CY_HIGH_LEVEL, 3017c478bd9Sstevel@tonic-gate * CY_LOCK_LEVEL or CY_LOW_LEVEL). 3027c478bd9Sstevel@tonic-gate * 3037c478bd9Sstevel@tonic-gate * For CY_HIGH_LEVEL cyclics, this is trivial; cyclic_expire() simply needs 3047c478bd9Sstevel@tonic-gate * to call the handler. 3057c478bd9Sstevel@tonic-gate * 3067c478bd9Sstevel@tonic-gate * For CY_LOCK_LEVEL and CY_LOW_LEVEL cyclics, however, there exists a 3077c478bd9Sstevel@tonic-gate * potential disconnect: if the CPU is at an interrupt level less than 3087c478bd9Sstevel@tonic-gate * CY_HIGH_LEVEL but greater than the level of a cyclic for a period of 3097c478bd9Sstevel@tonic-gate * time longer than twice the cyclic's interval, the cyclic will be expired 3107c478bd9Sstevel@tonic-gate * twice before it can be handled. 3117c478bd9Sstevel@tonic-gate * 3127c478bd9Sstevel@tonic-gate * To maintain the one-to-one mapping, we track the difference between the 3137c478bd9Sstevel@tonic-gate * number of times a cyclic has been expired and the number of times it's 3147c478bd9Sstevel@tonic-gate * been handled in a "pending count" (the cy_pend field of the cyclic 3157c478bd9Sstevel@tonic-gate * structure). cyclic_expire() thus increments the cy_pend count for the 3167c478bd9Sstevel@tonic-gate * expired cyclic and posts a soft interrupt at the desired level. In the 3177c478bd9Sstevel@tonic-gate * cyclic subsystem's soft interrupt handler, cyclic_softint(), we repeatedly 3187c478bd9Sstevel@tonic-gate * call the cyclic handler and decrement cy_pend until we have decremented 3197c478bd9Sstevel@tonic-gate * cy_pend to zero. 3207c478bd9Sstevel@tonic-gate * 3217c478bd9Sstevel@tonic-gate * The Producer/Consumer Buffer 3227c478bd9Sstevel@tonic-gate * 3237c478bd9Sstevel@tonic-gate * If we wish to avoid a linear scan of the cyclics array at soft interrupt 3247c478bd9Sstevel@tonic-gate * level, cyclic_softint() must be able to quickly determine which cyclics 3257c478bd9Sstevel@tonic-gate * have a non-zero cy_pend count. We thus introduce a per-soft interrupt 3267c478bd9Sstevel@tonic-gate * level producer/consumer buffer shared with CY_HIGH_LEVEL. These buffers 3277c478bd9Sstevel@tonic-gate * are encapsulated in the cyc_pcbuffer structure, and, like cyp_heap, are 3287c478bd9Sstevel@tonic-gate * implemented as cyc_index_t arrays (the cypc_buf member of the cyc_pcbuffer 3297c478bd9Sstevel@tonic-gate * structure). 3307c478bd9Sstevel@tonic-gate * 3317c478bd9Sstevel@tonic-gate * The producer (cyclic_expire() running at CY_HIGH_LEVEL) enqueues a cyclic 3327c478bd9Sstevel@tonic-gate * by storing the cyclic's index to cypc_buf[cypc_prodndx] and incrementing 3337c478bd9Sstevel@tonic-gate * cypc_prodndx. The consumer (cyclic_softint() running at either 3347c478bd9Sstevel@tonic-gate * CY_LOCK_LEVEL or CY_LOW_LEVEL) dequeues a cyclic by loading from 3357c478bd9Sstevel@tonic-gate * cypc_buf[cypc_consndx] and bumping cypc_consndx. The buffer is empty when 3367c478bd9Sstevel@tonic-gate * cypc_prodndx == cypc_consndx. 3377c478bd9Sstevel@tonic-gate * 3387c478bd9Sstevel@tonic-gate * To bound the size of the producer/consumer buffer, cyclic_expire() only 3397c478bd9Sstevel@tonic-gate * enqueues a cyclic if its cy_pend was zero (if the cyclic's cy_pend is 3407c478bd9Sstevel@tonic-gate * non-zero, cyclic_expire() only bumps cy_pend). Symmetrically, 3417c478bd9Sstevel@tonic-gate * cyclic_softint() only consumes a cyclic after it has decremented the 3427c478bd9Sstevel@tonic-gate * cy_pend count to zero. 3437c478bd9Sstevel@tonic-gate * 3447c478bd9Sstevel@tonic-gate * Returning to our example, here is what the CY_LOW_LEVEL producer/consumer 3457c478bd9Sstevel@tonic-gate * buffer might look like: 3467c478bd9Sstevel@tonic-gate * 3477c478bd9Sstevel@tonic-gate * cypc_consndx ---+ +--- cypc_prodndx 3487c478bd9Sstevel@tonic-gate * | | 3497c478bd9Sstevel@tonic-gate * V V 3507c478bd9Sstevel@tonic-gate * 3517c478bd9Sstevel@tonic-gate * [0] [1] [2] [3] [4] [5] [6] [7] 3527c478bd9Sstevel@tonic-gate * +-----+-----+-----+-----+-----+-----+-----+-----+ 3537c478bd9Sstevel@tonic-gate * | | | | | | | | | 3547c478bd9Sstevel@tonic-gate * | x | x | 3 | 2 | 4 | x | x | x | <== cypc_buf 3557c478bd9Sstevel@tonic-gate * | | | . | . | . | | | | 3567c478bd9Sstevel@tonic-gate * +-----+-----+- | -+- | -+- | -+-----+-----+-----+ 3577c478bd9Sstevel@tonic-gate * | | | 3587c478bd9Sstevel@tonic-gate * | | | cy_pend cy_handler 3597c478bd9Sstevel@tonic-gate * | | | ------------------------- 3607c478bd9Sstevel@tonic-gate * | | | [ 0] 1 clock() 3617c478bd9Sstevel@tonic-gate * | | | [ 1] 0 deadman() 3627c478bd9Sstevel@tonic-gate * | +---- | -------> [ 2] 3 clock_highres_fire() 3637c478bd9Sstevel@tonic-gate * +---------- | -------> [ 3] 1 clock_highres_fire() 3647c478bd9Sstevel@tonic-gate * +--------> [ 4] 1 clock_highres_fire() 3657c478bd9Sstevel@tonic-gate * [ 5] - (free) 3667c478bd9Sstevel@tonic-gate * [ 6] - (free) 3677c478bd9Sstevel@tonic-gate * [ 7] - (free) 3687c478bd9Sstevel@tonic-gate * 3697c478bd9Sstevel@tonic-gate * In particular, note that clock()'s cy_pend is 1 but that it is _not_ in 3707c478bd9Sstevel@tonic-gate * this producer/consumer buffer; it would be enqueued in the CY_LOCK_LEVEL 3717c478bd9Sstevel@tonic-gate * producer/consumer buffer. 3727c478bd9Sstevel@tonic-gate * 3737c478bd9Sstevel@tonic-gate * Locking 3747c478bd9Sstevel@tonic-gate * 3757c478bd9Sstevel@tonic-gate * Traditionally, access to per-CPU data structures shared between 3767c478bd9Sstevel@tonic-gate * interrupt levels is serialized by manipulating programmable interrupt 3777c478bd9Sstevel@tonic-gate * level: readers and writers are required to raise their interrupt level 3787c478bd9Sstevel@tonic-gate * to that of the highest level writer. 3797c478bd9Sstevel@tonic-gate * 3807c478bd9Sstevel@tonic-gate * For the producer/consumer buffers (shared between cyclic_fire()/ 3817c478bd9Sstevel@tonic-gate * cyclic_expire() executing at CY_HIGH_LEVEL and cyclic_softint() executing 3827c478bd9Sstevel@tonic-gate * at one of CY_LOCK_LEVEL or CY_LOW_LEVEL), forcing cyclic_softint() to raise 3837c478bd9Sstevel@tonic-gate * programmable interrupt level is undesirable: aside from the additional 3847c478bd9Sstevel@tonic-gate * latency incurred by manipulating interrupt level in the hot cy_pend 3857c478bd9Sstevel@tonic-gate * processing path, this would create the potential for soft level cy_pend 3867c478bd9Sstevel@tonic-gate * processing to delay CY_HIGH_LEVEL firing and expiry processing. 3877c478bd9Sstevel@tonic-gate * CY_LOCK/LOW_LEVEL cyclics could thereby induce jitter in CY_HIGH_LEVEL 3887c478bd9Sstevel@tonic-gate * cyclics. 3897c478bd9Sstevel@tonic-gate * 3907c478bd9Sstevel@tonic-gate * To minimize jitter, then, we would like the cyclic_fire()/cyclic_expire() 3917c478bd9Sstevel@tonic-gate * and cyclic_softint() code paths to be lock-free. 3927c478bd9Sstevel@tonic-gate * 3937c478bd9Sstevel@tonic-gate * For cyclic_fire()/cyclic_expire(), lock-free execution is straightforward: 3947c478bd9Sstevel@tonic-gate * because these routines execute at a higher interrupt level than 3957c478bd9Sstevel@tonic-gate * cyclic_softint(), their actions on the producer/consumer buffer appear 3967c478bd9Sstevel@tonic-gate * atomic. In particular, the increment of cy_pend appears to occur 3977c478bd9Sstevel@tonic-gate * atomically with the increment of cypc_prodndx. 3987c478bd9Sstevel@tonic-gate * 3997c478bd9Sstevel@tonic-gate * For cyclic_softint(), however, lock-free execution requires more delicacy. 4007c478bd9Sstevel@tonic-gate * When cyclic_softint() discovers a cyclic in the producer/consumer buffer, 4017c478bd9Sstevel@tonic-gate * it calls the cyclic's handler and attempts to atomically decrement the 4027c478bd9Sstevel@tonic-gate * cy_pend count with a compare&swap operation. 4037c478bd9Sstevel@tonic-gate * 4047c478bd9Sstevel@tonic-gate * If the compare&swap operation succeeds, cyclic_softint() behaves 4057c478bd9Sstevel@tonic-gate * conditionally based on the value it atomically wrote to cy_pend: 4067c478bd9Sstevel@tonic-gate * 4077c478bd9Sstevel@tonic-gate * - If the cy_pend was decremented to 0, the cyclic has been consumed; 4087c478bd9Sstevel@tonic-gate * cyclic_softint() increments the cypc_consndx and checks for more 4097c478bd9Sstevel@tonic-gate * enqueued work. 4107c478bd9Sstevel@tonic-gate * 4117c478bd9Sstevel@tonic-gate * - If the count was decremented to a non-zero value, there is more work 4127c478bd9Sstevel@tonic-gate * to be done on the cyclic; cyclic_softint() calls the cyclic handler 4137c478bd9Sstevel@tonic-gate * and repeats the atomic decrement process. 4147c478bd9Sstevel@tonic-gate * 4157c478bd9Sstevel@tonic-gate * If the compare&swap operation fails, cyclic_softint() knows that 4167c478bd9Sstevel@tonic-gate * cyclic_expire() has intervened and bumped the cy_pend count (resizes 4177c478bd9Sstevel@tonic-gate * and removals complicate this, however -- see the sections on their 4187c478bd9Sstevel@tonic-gate * operation, below). cyclic_softint() thus reloads cy_pend, and re-attempts 4197c478bd9Sstevel@tonic-gate * the atomic decrement. 4207c478bd9Sstevel@tonic-gate * 4217c478bd9Sstevel@tonic-gate * Recall that we bound the size of the producer/consumer buffer by 4227c478bd9Sstevel@tonic-gate * having cyclic_expire() only enqueue the specified cyclic if its 4237c478bd9Sstevel@tonic-gate * cy_pend count is zero; this assures that each cyclic is enqueued at 4247c478bd9Sstevel@tonic-gate * most once. This leads to a critical constraint on cyclic_softint(), 4257c478bd9Sstevel@tonic-gate * however: after the compare&swap operation which successfully decrements 4267c478bd9Sstevel@tonic-gate * cy_pend to zero, cyclic_softint() must _not_ re-examine the consumed 4277c478bd9Sstevel@tonic-gate * cyclic. In part to obey this constraint, cyclic_softint() calls the 4287c478bd9Sstevel@tonic-gate * cyclic handler before decrementing cy_pend. 4297c478bd9Sstevel@tonic-gate * 4307c478bd9Sstevel@tonic-gate * Resizing 4317c478bd9Sstevel@tonic-gate * 4327c478bd9Sstevel@tonic-gate * All of the discussion thus far has assumed a static number of cyclics. 4337c478bd9Sstevel@tonic-gate * Obviously, static limitations are not practical; we need the capacity 4347c478bd9Sstevel@tonic-gate * to resize our data structures dynamically. 4357c478bd9Sstevel@tonic-gate * 4367c478bd9Sstevel@tonic-gate * We resize our data structures lazily, and only on a per-CPU basis. 4377c478bd9Sstevel@tonic-gate * The size of the data structures always doubles and never shrinks. We 4387c478bd9Sstevel@tonic-gate * serialize adds (and thus resizes) on cpu_lock; we never need to deal 4397c478bd9Sstevel@tonic-gate * with concurrent resizes. Resizes should be rare; they may induce jitter 4407c478bd9Sstevel@tonic-gate * on the CPU being resized, but should not affect cyclic operation on other 4417c478bd9Sstevel@tonic-gate * CPUs. Pending cyclics may not be dropped during a resize operation. 4427c478bd9Sstevel@tonic-gate * 4437c478bd9Sstevel@tonic-gate * Three key cyc_cpu data structures need to be resized: the cyclics array, 4447c478bd9Sstevel@tonic-gate * the heap array and the producer/consumer buffers. Resizing the first two 4457c478bd9Sstevel@tonic-gate * is relatively straightforward: 4467c478bd9Sstevel@tonic-gate * 4477c478bd9Sstevel@tonic-gate * 1. The new, larger arrays are allocated in cyclic_expand() (called 4487c478bd9Sstevel@tonic-gate * from cyclic_add()). 4497c478bd9Sstevel@tonic-gate * 2. cyclic_expand() cross calls cyclic_expand_xcall() on the CPU 4507c478bd9Sstevel@tonic-gate * undergoing the resize. 4517c478bd9Sstevel@tonic-gate * 3. cyclic_expand_xcall() raises interrupt level to CY_HIGH_LEVEL 4527c478bd9Sstevel@tonic-gate * 4. The contents of the old arrays are copied into the new arrays. 4537c478bd9Sstevel@tonic-gate * 5. The old cyclics array is bzero()'d 4547c478bd9Sstevel@tonic-gate * 6. The pointers are updated. 4557c478bd9Sstevel@tonic-gate * 4567c478bd9Sstevel@tonic-gate * The producer/consumer buffer is dicier: cyclic_expand_xcall() may have 4577c478bd9Sstevel@tonic-gate * interrupted cyclic_softint() in the middle of consumption. To resize the 4587c478bd9Sstevel@tonic-gate * producer/consumer buffer, we implement up to two buffers per soft interrupt 4597c478bd9Sstevel@tonic-gate * level: a hard buffer (the buffer being produced into by cyclic_expire()) 4607c478bd9Sstevel@tonic-gate * and a soft buffer (the buffer from which cyclic_softint() is consuming). 4617c478bd9Sstevel@tonic-gate * During normal operation, the hard buffer and soft buffer point to the 4627c478bd9Sstevel@tonic-gate * same underlying producer/consumer buffer. 4637c478bd9Sstevel@tonic-gate * 4647c478bd9Sstevel@tonic-gate * During a resize, however, cyclic_expand_xcall() changes the hard buffer 4657c478bd9Sstevel@tonic-gate * to point to the new, larger producer/consumer buffer; all future 4667c478bd9Sstevel@tonic-gate * cyclic_expire()'s will produce into the new buffer. cyclic_expand_xcall() 4677c478bd9Sstevel@tonic-gate * then posts a CY_LOCK_LEVEL soft interrupt, landing in cyclic_softint(). 4687c478bd9Sstevel@tonic-gate * 4697c478bd9Sstevel@tonic-gate * As under normal operation, cyclic_softint() will consume cyclics from 4707c478bd9Sstevel@tonic-gate * its soft buffer. After the soft buffer is drained, however, 4717c478bd9Sstevel@tonic-gate * cyclic_softint() will see that the hard buffer has changed. At that time, 4727c478bd9Sstevel@tonic-gate * cyclic_softint() will change its soft buffer to point to the hard buffer, 4737c478bd9Sstevel@tonic-gate * and repeat the producer/consumer buffer draining procedure. 4747c478bd9Sstevel@tonic-gate * 4757c478bd9Sstevel@tonic-gate * After the new buffer is drained, cyclic_softint() will determine if both 4767c478bd9Sstevel@tonic-gate * soft levels have seen their new producer/consumer buffer. If both have, 4777c478bd9Sstevel@tonic-gate * cyclic_softint() will post on the semaphore cyp_modify_wait. If not, a 4787c478bd9Sstevel@tonic-gate * soft interrupt will be generated for the remaining level. 4797c478bd9Sstevel@tonic-gate * 4807c478bd9Sstevel@tonic-gate * cyclic_expand() blocks on the cyp_modify_wait semaphore (a semaphore is 4817c478bd9Sstevel@tonic-gate * used instead of a condition variable because of the race between the 4827c478bd9Sstevel@tonic-gate * sema_p() in cyclic_expand() and the sema_v() in cyclic_softint()). This 4837c478bd9Sstevel@tonic-gate * allows cyclic_expand() to know when the resize operation is complete; 4847c478bd9Sstevel@tonic-gate * all of the old buffers (the heap, the cyclics array and the producer/ 4857c478bd9Sstevel@tonic-gate * consumer buffers) can be freed. 4867c478bd9Sstevel@tonic-gate * 4877c478bd9Sstevel@tonic-gate * A final caveat on resizing: we described step (5) in the 4887c478bd9Sstevel@tonic-gate * cyclic_expand_xcall() procedure without providing any motivation. This 4897c478bd9Sstevel@tonic-gate * step addresses the problem of a cyclic_softint() attempting to decrement 4907c478bd9Sstevel@tonic-gate * a cy_pend count while interrupted by a cyclic_expand_xcall(). Because 4917c478bd9Sstevel@tonic-gate * cyclic_softint() has already called the handler by the time cy_pend is 4927c478bd9Sstevel@tonic-gate * decremented, we want to assure that it doesn't decrement a cy_pend 4937c478bd9Sstevel@tonic-gate * count in the old cyclics array. By zeroing the old cyclics array in 4947c478bd9Sstevel@tonic-gate * cyclic_expand_xcall(), we are zeroing out every cy_pend count; when 4957c478bd9Sstevel@tonic-gate * cyclic_softint() attempts to compare&swap on the cy_pend count, it will 4967c478bd9Sstevel@tonic-gate * fail and recognize that the count has been zeroed. cyclic_softint() will 4977c478bd9Sstevel@tonic-gate * update its stale copy of the cyp_cyclics pointer, re-read the cy_pend 4987c478bd9Sstevel@tonic-gate * count from the new cyclics array, and re-attempt the compare&swap. 4997c478bd9Sstevel@tonic-gate * 5007c478bd9Sstevel@tonic-gate * Removals 5017c478bd9Sstevel@tonic-gate * 5027c478bd9Sstevel@tonic-gate * Cyclic removals should be rare. To simplify the implementation (and to 5037c478bd9Sstevel@tonic-gate * allow optimization for the cyclic_fire()/cyclic_expire()/cyclic_softint() 5047c478bd9Sstevel@tonic-gate * path), we force removals and adds to serialize on cpu_lock. 5057c478bd9Sstevel@tonic-gate * 5067c478bd9Sstevel@tonic-gate * Cyclic removal is complicated by a guarantee made to the consumer of 5077c478bd9Sstevel@tonic-gate * the cyclic subsystem: after cyclic_remove() returns, the cyclic handler 5087c478bd9Sstevel@tonic-gate * has returned and will never again be called. 5097c478bd9Sstevel@tonic-gate * 5107c478bd9Sstevel@tonic-gate * Here is the procedure for cyclic removal: 5117c478bd9Sstevel@tonic-gate * 5127c478bd9Sstevel@tonic-gate * 1. cyclic_remove() calls cyclic_remove_xcall() on the CPU undergoing 5137c478bd9Sstevel@tonic-gate * the removal. 5147c478bd9Sstevel@tonic-gate * 2. cyclic_remove_xcall() raises interrupt level to CY_HIGH_LEVEL 5157c478bd9Sstevel@tonic-gate * 3. The current expiration time for the removed cyclic is recorded. 5167c478bd9Sstevel@tonic-gate * 4. If the cy_pend count on the removed cyclic is non-zero, it 5177c478bd9Sstevel@tonic-gate * is copied into cyp_rpend and subsequently zeroed. 5187c478bd9Sstevel@tonic-gate * 5. The cyclic is removed from the heap 5197c478bd9Sstevel@tonic-gate * 6. If the root of the heap has changed, the backend is reprogrammed. 5207c478bd9Sstevel@tonic-gate * 7. If the cy_pend count was non-zero cyclic_remove() blocks on the 5217c478bd9Sstevel@tonic-gate * cyp_modify_wait semaphore. 5227c478bd9Sstevel@tonic-gate * 5237c478bd9Sstevel@tonic-gate * The motivation for step (3) is explained in "Juggling", below. 5247c478bd9Sstevel@tonic-gate * 5257c478bd9Sstevel@tonic-gate * The cy_pend count is decremented in cyclic_softint() after the cyclic 5267c478bd9Sstevel@tonic-gate * handler returns. Thus, if we find a cy_pend count of zero in step 5277c478bd9Sstevel@tonic-gate * (4), we know that cyclic_remove() doesn't need to block. 5287c478bd9Sstevel@tonic-gate * 5297c478bd9Sstevel@tonic-gate * If the cy_pend count is non-zero, however, we must block in cyclic_remove() 5307c478bd9Sstevel@tonic-gate * until cyclic_softint() has finished calling the cyclic handler. To let 5317c478bd9Sstevel@tonic-gate * cyclic_softint() know that this cyclic has been removed, we zero the 5327c478bd9Sstevel@tonic-gate * cy_pend count. This will cause cyclic_softint()'s compare&swap to fail. 5337c478bd9Sstevel@tonic-gate * When cyclic_softint() sees the zero cy_pend count, it knows that it's been 5347c478bd9Sstevel@tonic-gate * caught during a resize (see "Resizing", above) or that the cyclic has been 5357c478bd9Sstevel@tonic-gate * removed. In the latter case, it calls cyclic_remove_pend() to call the 5367c478bd9Sstevel@tonic-gate * cyclic handler cyp_rpend - 1 times, and posts on cyp_modify_wait. 5377c478bd9Sstevel@tonic-gate * 5387c478bd9Sstevel@tonic-gate * Juggling 5397c478bd9Sstevel@tonic-gate * 5407c478bd9Sstevel@tonic-gate * At first glance, cyclic juggling seems to be a difficult problem. The 5417c478bd9Sstevel@tonic-gate * subsystem must guarantee that a cyclic doesn't execute simultaneously on 5427c478bd9Sstevel@tonic-gate * different CPUs, while also assuring that a cyclic fires exactly once 5437c478bd9Sstevel@tonic-gate * per interval. We solve this problem by leveraging a property of the 5447c478bd9Sstevel@tonic-gate * platform: gethrtime() is required to increase in lock-step across 5457c478bd9Sstevel@tonic-gate * multiple CPUs. Therefore, to juggle a cyclic, we remove it from its 5467c478bd9Sstevel@tonic-gate * CPU, recording its expiration time in the remove cross call (step (3) 5477c478bd9Sstevel@tonic-gate * in "Removing", above). We then add the cyclic to the new CPU, explicitly 5487c478bd9Sstevel@tonic-gate * setting its expiration time to the time recorded in the removal. This 5497c478bd9Sstevel@tonic-gate * leverages the existing cyclic expiry processing, which will compensate 5507c478bd9Sstevel@tonic-gate * for any time lost while juggling. 5517c478bd9Sstevel@tonic-gate * 552*87a18d3fSMadhavan Venkataraman * Reprogramming 553*87a18d3fSMadhavan Venkataraman * 554*87a18d3fSMadhavan Venkataraman * Normally, after a cyclic fires, its next expiration is computed from 555*87a18d3fSMadhavan Venkataraman * the current time and the cyclic interval. But there are situations when 556*87a18d3fSMadhavan Venkataraman * the next expiration needs to be reprogrammed by the kernel subsystem that 557*87a18d3fSMadhavan Venkataraman * is using the cyclic. cyclic_reprogram() allows this to be done. This, 558*87a18d3fSMadhavan Venkataraman * unlike the other kernel at-large cyclic API functions, is permitted to 559*87a18d3fSMadhavan Venkataraman * be called from the cyclic handler. This is because it does not use the 560*87a18d3fSMadhavan Venkataraman * cpu_lock to serialize access. 561*87a18d3fSMadhavan Venkataraman * 562*87a18d3fSMadhavan Venkataraman * When cyclic_reprogram() is called for an omni-cyclic, the operation is 563*87a18d3fSMadhavan Venkataraman * applied to the omni-cyclic's component on the current CPU. 564*87a18d3fSMadhavan Venkataraman * 565*87a18d3fSMadhavan Venkataraman * If a high-level cyclic handler reprograms its own cyclic, then 566*87a18d3fSMadhavan Venkataraman * cyclic_fire() detects that and does not recompute the cyclic's next 567*87a18d3fSMadhavan Venkataraman * expiration. However, for a lock-level or a low-level cyclic, the 568*87a18d3fSMadhavan Venkataraman * actual cyclic handler will execute at the lower PIL only after 569*87a18d3fSMadhavan Venkataraman * cyclic_fire() is done with all expired cyclics. To deal with this, such 570*87a18d3fSMadhavan Venkataraman * cyclics can be specified with a special interval of CY_INFINITY (INT64_MAX). 571*87a18d3fSMadhavan Venkataraman * cyclic_fire() recognizes this special value and recomputes the next 572*87a18d3fSMadhavan Venkataraman * expiration to CY_INFINITY. This effectively moves the cyclic to the 573*87a18d3fSMadhavan Venkataraman * bottom of the heap and prevents it from going off until its handler has 574*87a18d3fSMadhavan Venkataraman * had a chance to reprogram it. Infact, this is the way to create and reuse 575*87a18d3fSMadhavan Venkataraman * "one-shot" timers in the context of the cyclic subsystem without using 576*87a18d3fSMadhavan Venkataraman * cyclic_remove(). 577*87a18d3fSMadhavan Venkataraman * 578*87a18d3fSMadhavan Venkataraman * Here is the procedure for cyclic reprogramming: 579*87a18d3fSMadhavan Venkataraman * 580*87a18d3fSMadhavan Venkataraman * 1. cyclic_reprogram() calls cyclic_reprogram_xcall() on the CPU 581*87a18d3fSMadhavan Venkataraman * that houses the cyclic. 582*87a18d3fSMadhavan Venkataraman * 2. cyclic_reprogram_xcall() raises interrupt level to CY_HIGH_LEVEL 583*87a18d3fSMadhavan Venkataraman * 3. The cyclic is located in the cyclic heap. The search for this is 584*87a18d3fSMadhavan Venkataraman * done from the bottom of the heap to the top as reprogrammable cyclics 585*87a18d3fSMadhavan Venkataraman * would be located closer to the bottom than the top. 586*87a18d3fSMadhavan Venkataraman * 4. The cyclic expiration is set and the cyclic is moved to its 587*87a18d3fSMadhavan Venkataraman * correct position in the heap (up or down depending on whether the 588*87a18d3fSMadhavan Venkataraman * new expiration is less than or greater than the old one). 589*87a18d3fSMadhavan Venkataraman * 5. If the cyclic move modified the root of the heap, the backend is 590*87a18d3fSMadhavan Venkataraman * reprogrammed. 591*87a18d3fSMadhavan Venkataraman * 592*87a18d3fSMadhavan Venkataraman * Reprogramming can be a frequent event (see the callout subsystem). So, 593*87a18d3fSMadhavan Venkataraman * the serialization used has to be efficient. As with all other cyclic 594*87a18d3fSMadhavan Venkataraman * operations, the interrupt level is raised during reprogramming. Plus, 595*87a18d3fSMadhavan Venkataraman * during reprogramming, the cyclic must not be juggled (regular cyclic) 596*87a18d3fSMadhavan Venkataraman * or stopped (omni-cyclic). The implementation defines a per-cyclic 597*87a18d3fSMadhavan Venkataraman * reader-writer lock to accomplish this. This lock is acquired in the 598*87a18d3fSMadhavan Venkataraman * reader mode by cyclic_reprogram() and writer mode by cyclic_juggle() and 599*87a18d3fSMadhavan Venkataraman * cyclic_omni_stop(). The reader-writer lock makes it efficient if 600*87a18d3fSMadhavan Venkataraman * an omni-cyclic is reprogrammed on different CPUs frequently. 601*87a18d3fSMadhavan Venkataraman * 602*87a18d3fSMadhavan Venkataraman * Note that since the cpu_lock is not used during reprogramming, it is 603*87a18d3fSMadhavan Venkataraman * the responsibility of the user of the reprogrammable cyclic to make sure 604*87a18d3fSMadhavan Venkataraman * that the cyclic is not removed via cyclic_remove() during reprogramming. 605*87a18d3fSMadhavan Venkataraman * This is not an unreasonable requirement as the user will typically have 606*87a18d3fSMadhavan Venkataraman * some sort of synchronization for its cyclic-related activities. This 607*87a18d3fSMadhavan Venkataraman * little caveat exists because the cyclic ID is not really an ID. It is 608*87a18d3fSMadhavan Venkataraman * implemented as a pointer to a structure. 6097c478bd9Sstevel@tonic-gate */ 6107c478bd9Sstevel@tonic-gate #include <sys/cyclic_impl.h> 6117c478bd9Sstevel@tonic-gate #include <sys/sysmacros.h> 6127c478bd9Sstevel@tonic-gate #include <sys/systm.h> 6137c478bd9Sstevel@tonic-gate #include <sys/atomic.h> 6147c478bd9Sstevel@tonic-gate #include <sys/kmem.h> 6157c478bd9Sstevel@tonic-gate #include <sys/cmn_err.h> 6167c478bd9Sstevel@tonic-gate #include <sys/ddi.h> 617c210ded4Sesaxe #include <sys/sdt.h> 6187c478bd9Sstevel@tonic-gate 6197c478bd9Sstevel@tonic-gate #ifdef CYCLIC_TRACE 6207c478bd9Sstevel@tonic-gate 6217c478bd9Sstevel@tonic-gate /* 6227c478bd9Sstevel@tonic-gate * cyc_trace_enabled is for the benefit of kernel debuggers. 6237c478bd9Sstevel@tonic-gate */ 6247c478bd9Sstevel@tonic-gate int cyc_trace_enabled = 1; 6257c478bd9Sstevel@tonic-gate static cyc_tracebuf_t cyc_ptrace; 6267c478bd9Sstevel@tonic-gate static cyc_coverage_t cyc_coverage[CY_NCOVERAGE]; 6277c478bd9Sstevel@tonic-gate 6287c478bd9Sstevel@tonic-gate /* 6297c478bd9Sstevel@tonic-gate * Seen this anywhere? 6307c478bd9Sstevel@tonic-gate */ 6317c478bd9Sstevel@tonic-gate static uint_t 6327c478bd9Sstevel@tonic-gate cyclic_coverage_hash(char *p) 6337c478bd9Sstevel@tonic-gate { 6347c478bd9Sstevel@tonic-gate unsigned int g; 6357c478bd9Sstevel@tonic-gate uint_t hval; 6367c478bd9Sstevel@tonic-gate 6377c478bd9Sstevel@tonic-gate hval = 0; 6387c478bd9Sstevel@tonic-gate while (*p) { 6397c478bd9Sstevel@tonic-gate hval = (hval << 4) + *p++; 6407c478bd9Sstevel@tonic-gate if ((g = (hval & 0xf0000000)) != 0) 6417c478bd9Sstevel@tonic-gate hval ^= g >> 24; 6427c478bd9Sstevel@tonic-gate hval &= ~g; 6437c478bd9Sstevel@tonic-gate } 6447c478bd9Sstevel@tonic-gate return (hval); 6457c478bd9Sstevel@tonic-gate } 6467c478bd9Sstevel@tonic-gate 6477c478bd9Sstevel@tonic-gate static void 6487c478bd9Sstevel@tonic-gate cyclic_coverage(char *why, int level, uint64_t arg0, uint64_t arg1) 6497c478bd9Sstevel@tonic-gate { 6507c478bd9Sstevel@tonic-gate uint_t ndx, orig; 6517c478bd9Sstevel@tonic-gate 6527c478bd9Sstevel@tonic-gate for (ndx = orig = cyclic_coverage_hash(why) % CY_NCOVERAGE; ; ) { 6537c478bd9Sstevel@tonic-gate if (cyc_coverage[ndx].cyv_why == why) 6547c478bd9Sstevel@tonic-gate break; 6557c478bd9Sstevel@tonic-gate 6567c478bd9Sstevel@tonic-gate if (cyc_coverage[ndx].cyv_why != NULL || 6577c478bd9Sstevel@tonic-gate casptr(&cyc_coverage[ndx].cyv_why, NULL, why) != NULL) { 6587c478bd9Sstevel@tonic-gate 6597c478bd9Sstevel@tonic-gate if (++ndx == CY_NCOVERAGE) 6607c478bd9Sstevel@tonic-gate ndx = 0; 6617c478bd9Sstevel@tonic-gate 6627c478bd9Sstevel@tonic-gate if (ndx == orig) 6637c478bd9Sstevel@tonic-gate panic("too many cyclic coverage points"); 6647c478bd9Sstevel@tonic-gate continue; 6657c478bd9Sstevel@tonic-gate } 6667c478bd9Sstevel@tonic-gate 6677c478bd9Sstevel@tonic-gate /* 6687c478bd9Sstevel@tonic-gate * If we're here, we have successfully swung our guy into 6697c478bd9Sstevel@tonic-gate * the position at "ndx". 6707c478bd9Sstevel@tonic-gate */ 6717c478bd9Sstevel@tonic-gate break; 6727c478bd9Sstevel@tonic-gate } 6737c478bd9Sstevel@tonic-gate 6747c478bd9Sstevel@tonic-gate if (level == CY_PASSIVE_LEVEL) 6757c478bd9Sstevel@tonic-gate cyc_coverage[ndx].cyv_passive_count++; 6767c478bd9Sstevel@tonic-gate else 6777c478bd9Sstevel@tonic-gate cyc_coverage[ndx].cyv_count[level]++; 6787c478bd9Sstevel@tonic-gate 6797c478bd9Sstevel@tonic-gate cyc_coverage[ndx].cyv_arg0 = arg0; 6807c478bd9Sstevel@tonic-gate cyc_coverage[ndx].cyv_arg1 = arg1; 6817c478bd9Sstevel@tonic-gate } 6827c478bd9Sstevel@tonic-gate 6837c478bd9Sstevel@tonic-gate #define CYC_TRACE(cpu, level, why, arg0, arg1) \ 6847c478bd9Sstevel@tonic-gate CYC_TRACE_IMPL(&cpu->cyp_trace[level], level, why, arg0, arg1) 6857c478bd9Sstevel@tonic-gate 6867c478bd9Sstevel@tonic-gate #define CYC_PTRACE(why, arg0, arg1) \ 6877c478bd9Sstevel@tonic-gate CYC_TRACE_IMPL(&cyc_ptrace, CY_PASSIVE_LEVEL, why, arg0, arg1) 6887c478bd9Sstevel@tonic-gate 6897c478bd9Sstevel@tonic-gate #define CYC_TRACE_IMPL(buf, level, why, a0, a1) { \ 6907c478bd9Sstevel@tonic-gate if (panicstr == NULL) { \ 6917c478bd9Sstevel@tonic-gate int _ndx = (buf)->cyt_ndx; \ 6927c478bd9Sstevel@tonic-gate cyc_tracerec_t *_rec = &(buf)->cyt_buf[_ndx]; \ 6937c478bd9Sstevel@tonic-gate (buf)->cyt_ndx = (++_ndx == CY_NTRACEREC) ? 0 : _ndx; \ 6947c478bd9Sstevel@tonic-gate _rec->cyt_tstamp = gethrtime_unscaled(); \ 6957c478bd9Sstevel@tonic-gate _rec->cyt_why = (why); \ 6967c478bd9Sstevel@tonic-gate _rec->cyt_arg0 = (uint64_t)(uintptr_t)(a0); \ 6977c478bd9Sstevel@tonic-gate _rec->cyt_arg1 = (uint64_t)(uintptr_t)(a1); \ 6987c478bd9Sstevel@tonic-gate cyclic_coverage(why, level, \ 6997c478bd9Sstevel@tonic-gate (uint64_t)(uintptr_t)(a0), (uint64_t)(uintptr_t)(a1)); \ 7007c478bd9Sstevel@tonic-gate } \ 7017c478bd9Sstevel@tonic-gate } 7027c478bd9Sstevel@tonic-gate 7037c478bd9Sstevel@tonic-gate #else 7047c478bd9Sstevel@tonic-gate 7057c478bd9Sstevel@tonic-gate static int cyc_trace_enabled = 0; 7067c478bd9Sstevel@tonic-gate 7077c478bd9Sstevel@tonic-gate #define CYC_TRACE(cpu, level, why, arg0, arg1) 7087c478bd9Sstevel@tonic-gate #define CYC_PTRACE(why, arg0, arg1) 7097c478bd9Sstevel@tonic-gate 7107c478bd9Sstevel@tonic-gate #endif 7117c478bd9Sstevel@tonic-gate 7127c478bd9Sstevel@tonic-gate #define CYC_TRACE0(cpu, level, why) CYC_TRACE(cpu, level, why, 0, 0) 7137c478bd9Sstevel@tonic-gate #define CYC_TRACE1(cpu, level, why, arg0) CYC_TRACE(cpu, level, why, arg0, 0) 7147c478bd9Sstevel@tonic-gate 7157c478bd9Sstevel@tonic-gate #define CYC_PTRACE0(why) CYC_PTRACE(why, 0, 0) 7167c478bd9Sstevel@tonic-gate #define CYC_PTRACE1(why, arg0) CYC_PTRACE(why, arg0, 0) 7177c478bd9Sstevel@tonic-gate 7187c478bd9Sstevel@tonic-gate static kmem_cache_t *cyclic_id_cache; 7197c478bd9Sstevel@tonic-gate static cyc_id_t *cyclic_id_head; 7207c478bd9Sstevel@tonic-gate static hrtime_t cyclic_resolution; 7217c478bd9Sstevel@tonic-gate static cyc_backend_t cyclic_backend; 7227c478bd9Sstevel@tonic-gate 7237c478bd9Sstevel@tonic-gate /* 7247c478bd9Sstevel@tonic-gate * Returns 1 if the upheap propagated to the root, 0 if it did not. This 7257c478bd9Sstevel@tonic-gate * allows the caller to reprogram the backend only when the root has been 7267c478bd9Sstevel@tonic-gate * modified. 7277c478bd9Sstevel@tonic-gate */ 7287c478bd9Sstevel@tonic-gate static int 7297c478bd9Sstevel@tonic-gate cyclic_upheap(cyc_cpu_t *cpu, cyc_index_t ndx) 7307c478bd9Sstevel@tonic-gate { 7317c478bd9Sstevel@tonic-gate cyclic_t *cyclics; 7327c478bd9Sstevel@tonic-gate cyc_index_t *heap; 7337c478bd9Sstevel@tonic-gate cyc_index_t heap_parent, heap_current = ndx; 7347c478bd9Sstevel@tonic-gate cyc_index_t parent, current; 7357c478bd9Sstevel@tonic-gate 7367c478bd9Sstevel@tonic-gate if (heap_current == 0) 7377c478bd9Sstevel@tonic-gate return (1); 7387c478bd9Sstevel@tonic-gate 7397c478bd9Sstevel@tonic-gate heap = cpu->cyp_heap; 7407c478bd9Sstevel@tonic-gate cyclics = cpu->cyp_cyclics; 7417c478bd9Sstevel@tonic-gate heap_parent = CYC_HEAP_PARENT(heap_current); 7427c478bd9Sstevel@tonic-gate 7437c478bd9Sstevel@tonic-gate for (;;) { 7447c478bd9Sstevel@tonic-gate current = heap[heap_current]; 7457c478bd9Sstevel@tonic-gate parent = heap[heap_parent]; 7467c478bd9Sstevel@tonic-gate 7477c478bd9Sstevel@tonic-gate /* 7487c478bd9Sstevel@tonic-gate * We have an expiration time later than our parent; we're 7497c478bd9Sstevel@tonic-gate * done. 7507c478bd9Sstevel@tonic-gate */ 7517c478bd9Sstevel@tonic-gate if (cyclics[current].cy_expire >= cyclics[parent].cy_expire) 7527c478bd9Sstevel@tonic-gate return (0); 7537c478bd9Sstevel@tonic-gate 7547c478bd9Sstevel@tonic-gate /* 7557c478bd9Sstevel@tonic-gate * We need to swap with our parent, and continue up the heap. 7567c478bd9Sstevel@tonic-gate */ 7577c478bd9Sstevel@tonic-gate heap[heap_parent] = current; 7587c478bd9Sstevel@tonic-gate heap[heap_current] = parent; 7597c478bd9Sstevel@tonic-gate 7607c478bd9Sstevel@tonic-gate /* 7617c478bd9Sstevel@tonic-gate * If we just reached the root, we're done. 7627c478bd9Sstevel@tonic-gate */ 7637c478bd9Sstevel@tonic-gate if (heap_parent == 0) 7647c478bd9Sstevel@tonic-gate return (1); 7657c478bd9Sstevel@tonic-gate 7667c478bd9Sstevel@tonic-gate heap_current = heap_parent; 7677c478bd9Sstevel@tonic-gate heap_parent = CYC_HEAP_PARENT(heap_current); 7687c478bd9Sstevel@tonic-gate } 7697c478bd9Sstevel@tonic-gate } 7707c478bd9Sstevel@tonic-gate 7717c478bd9Sstevel@tonic-gate static void 7727c478bd9Sstevel@tonic-gate cyclic_downheap(cyc_cpu_t *cpu, cyc_index_t ndx) 7737c478bd9Sstevel@tonic-gate { 7747c478bd9Sstevel@tonic-gate cyclic_t *cyclics = cpu->cyp_cyclics; 7757c478bd9Sstevel@tonic-gate cyc_index_t *heap = cpu->cyp_heap; 7767c478bd9Sstevel@tonic-gate 7777c478bd9Sstevel@tonic-gate cyc_index_t heap_left, heap_right, heap_me = ndx; 7787c478bd9Sstevel@tonic-gate cyc_index_t left, right, me; 7797c478bd9Sstevel@tonic-gate cyc_index_t nelems = cpu->cyp_nelems; 7807c478bd9Sstevel@tonic-gate 7817c478bd9Sstevel@tonic-gate for (;;) { 7827c478bd9Sstevel@tonic-gate /* 7837c478bd9Sstevel@tonic-gate * If we don't have a left child (i.e., we're a leaf), we're 7847c478bd9Sstevel@tonic-gate * done. 7857c478bd9Sstevel@tonic-gate */ 7867c478bd9Sstevel@tonic-gate if ((heap_left = CYC_HEAP_LEFT(heap_me)) >= nelems) 7877c478bd9Sstevel@tonic-gate return; 7887c478bd9Sstevel@tonic-gate 7897c478bd9Sstevel@tonic-gate left = heap[heap_left]; 7907c478bd9Sstevel@tonic-gate me = heap[heap_me]; 7917c478bd9Sstevel@tonic-gate 7927c478bd9Sstevel@tonic-gate heap_right = CYC_HEAP_RIGHT(heap_me); 7937c478bd9Sstevel@tonic-gate 7947c478bd9Sstevel@tonic-gate /* 7957c478bd9Sstevel@tonic-gate * Even if we don't have a right child, we still need to compare 7967c478bd9Sstevel@tonic-gate * our expiration time against that of our left child. 7977c478bd9Sstevel@tonic-gate */ 7987c478bd9Sstevel@tonic-gate if (heap_right >= nelems) 7997c478bd9Sstevel@tonic-gate goto comp_left; 8007c478bd9Sstevel@tonic-gate 8017c478bd9Sstevel@tonic-gate right = heap[heap_right]; 8027c478bd9Sstevel@tonic-gate 8037c478bd9Sstevel@tonic-gate /* 8047c478bd9Sstevel@tonic-gate * We have both a left and a right child. We need to compare 8057c478bd9Sstevel@tonic-gate * the expiration times of the children to determine which 8067c478bd9Sstevel@tonic-gate * expires earlier. 8077c478bd9Sstevel@tonic-gate */ 8087c478bd9Sstevel@tonic-gate if (cyclics[right].cy_expire < cyclics[left].cy_expire) { 8097c478bd9Sstevel@tonic-gate /* 8107c478bd9Sstevel@tonic-gate * Our right child is the earlier of our children. 8117c478bd9Sstevel@tonic-gate * We'll now compare our expiration time to its; if 8127c478bd9Sstevel@tonic-gate * ours is the earlier, we're done. 8137c478bd9Sstevel@tonic-gate */ 8147c478bd9Sstevel@tonic-gate if (cyclics[me].cy_expire <= cyclics[right].cy_expire) 8157c478bd9Sstevel@tonic-gate return; 8167c478bd9Sstevel@tonic-gate 8177c478bd9Sstevel@tonic-gate /* 8187c478bd9Sstevel@tonic-gate * Our right child expires earlier than we do; swap 8197c478bd9Sstevel@tonic-gate * with our right child, and descend right. 8207c478bd9Sstevel@tonic-gate */ 8217c478bd9Sstevel@tonic-gate heap[heap_right] = me; 8227c478bd9Sstevel@tonic-gate heap[heap_me] = right; 8237c478bd9Sstevel@tonic-gate heap_me = heap_right; 8247c478bd9Sstevel@tonic-gate continue; 8257c478bd9Sstevel@tonic-gate } 8267c478bd9Sstevel@tonic-gate 8277c478bd9Sstevel@tonic-gate comp_left: 8287c478bd9Sstevel@tonic-gate /* 8297c478bd9Sstevel@tonic-gate * Our left child is the earlier of our children (or we have 8307c478bd9Sstevel@tonic-gate * no right child). We'll now compare our expiration time 8317c478bd9Sstevel@tonic-gate * to its; if ours is the earlier, we're done. 8327c478bd9Sstevel@tonic-gate */ 8337c478bd9Sstevel@tonic-gate if (cyclics[me].cy_expire <= cyclics[left].cy_expire) 8347c478bd9Sstevel@tonic-gate return; 8357c478bd9Sstevel@tonic-gate 8367c478bd9Sstevel@tonic-gate /* 8377c478bd9Sstevel@tonic-gate * Our left child expires earlier than we do; swap with our 8387c478bd9Sstevel@tonic-gate * left child, and descend left. 8397c478bd9Sstevel@tonic-gate */ 8407c478bd9Sstevel@tonic-gate heap[heap_left] = me; 8417c478bd9Sstevel@tonic-gate heap[heap_me] = left; 8427c478bd9Sstevel@tonic-gate heap_me = heap_left; 8437c478bd9Sstevel@tonic-gate } 8447c478bd9Sstevel@tonic-gate } 8457c478bd9Sstevel@tonic-gate 8467c478bd9Sstevel@tonic-gate static void 8477c478bd9Sstevel@tonic-gate cyclic_expire(cyc_cpu_t *cpu, cyc_index_t ndx, cyclic_t *cyclic) 8487c478bd9Sstevel@tonic-gate { 8497c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 8507c478bd9Sstevel@tonic-gate cyc_level_t level = cyclic->cy_level; 8517c478bd9Sstevel@tonic-gate 8527c478bd9Sstevel@tonic-gate /* 8537c478bd9Sstevel@tonic-gate * If this is a CY_HIGH_LEVEL cyclic, just call the handler; we don't 8547c478bd9Sstevel@tonic-gate * need to worry about the pend count for CY_HIGH_LEVEL cyclics. 8557c478bd9Sstevel@tonic-gate */ 8567c478bd9Sstevel@tonic-gate if (level == CY_HIGH_LEVEL) { 8577c478bd9Sstevel@tonic-gate cyc_func_t handler = cyclic->cy_handler; 8587c478bd9Sstevel@tonic-gate void *arg = cyclic->cy_arg; 8597c478bd9Sstevel@tonic-gate 8607c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "handler-in", handler, arg); 861c210ded4Sesaxe DTRACE_PROBE1(cyclic__start, cyclic_t *, cyclic); 862c210ded4Sesaxe 8637c478bd9Sstevel@tonic-gate (*handler)(arg); 864c210ded4Sesaxe 865c210ded4Sesaxe DTRACE_PROBE1(cyclic__end, cyclic_t *, cyclic); 8667c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "handler-out", handler, arg); 8677c478bd9Sstevel@tonic-gate 8687c478bd9Sstevel@tonic-gate return; 8697c478bd9Sstevel@tonic-gate } 8707c478bd9Sstevel@tonic-gate 8717c478bd9Sstevel@tonic-gate /* 8727c478bd9Sstevel@tonic-gate * We're at CY_HIGH_LEVEL; this modification to cy_pend need not 8737c478bd9Sstevel@tonic-gate * be atomic (the high interrupt level assures that it will appear 8747c478bd9Sstevel@tonic-gate * atomic to any softint currently running). 8757c478bd9Sstevel@tonic-gate */ 8767c478bd9Sstevel@tonic-gate if (cyclic->cy_pend++ == 0) { 8777c478bd9Sstevel@tonic-gate cyc_softbuf_t *softbuf = &cpu->cyp_softbuf[level]; 8787c478bd9Sstevel@tonic-gate cyc_pcbuffer_t *pc = &softbuf->cys_buf[softbuf->cys_hard]; 8797c478bd9Sstevel@tonic-gate 8807c478bd9Sstevel@tonic-gate /* 8817c478bd9Sstevel@tonic-gate * We need to enqueue this cyclic in the soft buffer. 8827c478bd9Sstevel@tonic-gate */ 8837c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "expire-enq", cyclic, 8847c478bd9Sstevel@tonic-gate pc->cypc_prodndx); 8857c478bd9Sstevel@tonic-gate pc->cypc_buf[pc->cypc_prodndx++ & pc->cypc_sizemask] = ndx; 8867c478bd9Sstevel@tonic-gate 8877c478bd9Sstevel@tonic-gate ASSERT(pc->cypc_prodndx != pc->cypc_consndx); 8887c478bd9Sstevel@tonic-gate } else { 8897c478bd9Sstevel@tonic-gate /* 8907c478bd9Sstevel@tonic-gate * If the pend count is zero after we incremented it, then 8917c478bd9Sstevel@tonic-gate * we've wrapped (i.e. we had a cy_pend count of over four 8927c478bd9Sstevel@tonic-gate * billion. In this case, we clamp the pend count at 8937c478bd9Sstevel@tonic-gate * UINT32_MAX. Yes, cyclics can be lost in this case. 8947c478bd9Sstevel@tonic-gate */ 8957c478bd9Sstevel@tonic-gate if (cyclic->cy_pend == 0) { 8967c478bd9Sstevel@tonic-gate CYC_TRACE1(cpu, CY_HIGH_LEVEL, "expire-wrap", cyclic); 8977c478bd9Sstevel@tonic-gate cyclic->cy_pend = UINT32_MAX; 8987c478bd9Sstevel@tonic-gate } 8997c478bd9Sstevel@tonic-gate 9007c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "expire-bump", cyclic, 0); 9017c478bd9Sstevel@tonic-gate } 9027c478bd9Sstevel@tonic-gate 9037c478bd9Sstevel@tonic-gate be->cyb_softint(be->cyb_arg, cyclic->cy_level); 9047c478bd9Sstevel@tonic-gate } 9057c478bd9Sstevel@tonic-gate 9067c478bd9Sstevel@tonic-gate /* 9077c478bd9Sstevel@tonic-gate * cyclic_fire(cpu_t *) 9087c478bd9Sstevel@tonic-gate * 9097c478bd9Sstevel@tonic-gate * Overview 9107c478bd9Sstevel@tonic-gate * 9117c478bd9Sstevel@tonic-gate * cyclic_fire() is the cyclic subsystem's CY_HIGH_LEVEL interrupt handler. 9127c478bd9Sstevel@tonic-gate * Called by the cyclic backend. 9137c478bd9Sstevel@tonic-gate * 9147c478bd9Sstevel@tonic-gate * Arguments and notes 9157c478bd9Sstevel@tonic-gate * 9167c478bd9Sstevel@tonic-gate * The only argument is the CPU on which the interrupt is executing; 9177c478bd9Sstevel@tonic-gate * backends must call into cyclic_fire() on the specified CPU. 9187c478bd9Sstevel@tonic-gate * 9197c478bd9Sstevel@tonic-gate * cyclic_fire() may be called spuriously without ill effect. Optimal 9207c478bd9Sstevel@tonic-gate * backends will call into cyclic_fire() at or shortly after the time 9217c478bd9Sstevel@tonic-gate * requested via cyb_reprogram(). However, calling cyclic_fire() 9227c478bd9Sstevel@tonic-gate * arbitrarily late will only manifest latency bubbles; the correctness 9237c478bd9Sstevel@tonic-gate * of the cyclic subsystem does not rely on the timeliness of the backend. 9247c478bd9Sstevel@tonic-gate * 9257c478bd9Sstevel@tonic-gate * cyclic_fire() is wait-free; it will not block or spin. 9267c478bd9Sstevel@tonic-gate * 9277c478bd9Sstevel@tonic-gate * Return values 9287c478bd9Sstevel@tonic-gate * 9297c478bd9Sstevel@tonic-gate * None. 9307c478bd9Sstevel@tonic-gate * 9317c478bd9Sstevel@tonic-gate * Caller's context 9327c478bd9Sstevel@tonic-gate * 9337c478bd9Sstevel@tonic-gate * cyclic_fire() must be called from CY_HIGH_LEVEL interrupt context. 9347c478bd9Sstevel@tonic-gate */ 9357c478bd9Sstevel@tonic-gate void 9367c478bd9Sstevel@tonic-gate cyclic_fire(cpu_t *c) 9377c478bd9Sstevel@tonic-gate { 9387c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = c->cpu_cyclic; 9397c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 9407c478bd9Sstevel@tonic-gate cyc_index_t *heap = cpu->cyp_heap; 9417c478bd9Sstevel@tonic-gate cyclic_t *cyclic, *cyclics = cpu->cyp_cyclics; 9427c478bd9Sstevel@tonic-gate void *arg = be->cyb_arg; 9437c478bd9Sstevel@tonic-gate hrtime_t now = gethrtime(); 9447c478bd9Sstevel@tonic-gate hrtime_t exp; 9457c478bd9Sstevel@tonic-gate 9467c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "fire", now, 0); 9477c478bd9Sstevel@tonic-gate 9487c478bd9Sstevel@tonic-gate if (cpu->cyp_nelems == 0) { 9497c478bd9Sstevel@tonic-gate /* 9507c478bd9Sstevel@tonic-gate * This is a spurious fire. Count it as such, and blow 9517c478bd9Sstevel@tonic-gate * out of here. 9527c478bd9Sstevel@tonic-gate */ 9537c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, CY_HIGH_LEVEL, "fire-spurious"); 9547c478bd9Sstevel@tonic-gate return; 9557c478bd9Sstevel@tonic-gate } 9567c478bd9Sstevel@tonic-gate 9577c478bd9Sstevel@tonic-gate for (;;) { 9587c478bd9Sstevel@tonic-gate cyc_index_t ndx = heap[0]; 9597c478bd9Sstevel@tonic-gate 9607c478bd9Sstevel@tonic-gate cyclic = &cyclics[ndx]; 9617c478bd9Sstevel@tonic-gate 9627c478bd9Sstevel@tonic-gate ASSERT(!(cyclic->cy_flags & CYF_FREE)); 9637c478bd9Sstevel@tonic-gate 9647c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "fire-check", cyclic, 9657c478bd9Sstevel@tonic-gate cyclic->cy_expire); 9667c478bd9Sstevel@tonic-gate 9677c478bd9Sstevel@tonic-gate if ((exp = cyclic->cy_expire) > now) 9687c478bd9Sstevel@tonic-gate break; 9697c478bd9Sstevel@tonic-gate 9707c478bd9Sstevel@tonic-gate cyclic_expire(cpu, ndx, cyclic); 9717c478bd9Sstevel@tonic-gate 9727c478bd9Sstevel@tonic-gate /* 973*87a18d3fSMadhavan Venkataraman * If the handler reprogrammed the cyclic, then don't 974*87a18d3fSMadhavan Venkataraman * recompute the expiration. Then, if the interval is 975*87a18d3fSMadhavan Venkataraman * infinity, set the expiration to infinity. This can 976*87a18d3fSMadhavan Venkataraman * be used to create one-shot timers. 977*87a18d3fSMadhavan Venkataraman */ 978*87a18d3fSMadhavan Venkataraman if (exp != cyclic->cy_expire) { 979*87a18d3fSMadhavan Venkataraman /* 980*87a18d3fSMadhavan Venkataraman * If a hi level cyclic reprograms itself, 981*87a18d3fSMadhavan Venkataraman * the heap adjustment and reprogramming of the 982*87a18d3fSMadhavan Venkataraman * clock source have already been done at this 983*87a18d3fSMadhavan Venkataraman * point. So, we can continue. 984*87a18d3fSMadhavan Venkataraman */ 985*87a18d3fSMadhavan Venkataraman continue; 986*87a18d3fSMadhavan Venkataraman } 987*87a18d3fSMadhavan Venkataraman 988*87a18d3fSMadhavan Venkataraman if (cyclic->cy_interval == CY_INFINITY) 989*87a18d3fSMadhavan Venkataraman exp = CY_INFINITY; 990*87a18d3fSMadhavan Venkataraman else 991*87a18d3fSMadhavan Venkataraman exp += cyclic->cy_interval; 992*87a18d3fSMadhavan Venkataraman 993*87a18d3fSMadhavan Venkataraman /* 9947c478bd9Sstevel@tonic-gate * If this cyclic will be set to next expire in the distant 9957c478bd9Sstevel@tonic-gate * past, we have one of two situations: 9967c478bd9Sstevel@tonic-gate * 9977c478bd9Sstevel@tonic-gate * a) This is the first firing of a cyclic which had 9987c478bd9Sstevel@tonic-gate * cy_expire set to 0. 9997c478bd9Sstevel@tonic-gate * 10007c478bd9Sstevel@tonic-gate * b) We are tragically late for a cyclic -- most likely 10017c478bd9Sstevel@tonic-gate * due to being in the debugger. 10027c478bd9Sstevel@tonic-gate * 10037c478bd9Sstevel@tonic-gate * In either case, we set the new expiration time to be the 10047c478bd9Sstevel@tonic-gate * the next interval boundary. This assures that the 10057c478bd9Sstevel@tonic-gate * expiration time modulo the interval is invariant. 10067c478bd9Sstevel@tonic-gate * 10077c478bd9Sstevel@tonic-gate * We arbitrarily define "distant" to be one second (one second 10087c478bd9Sstevel@tonic-gate * is chosen because it's shorter than any foray to the 10097c478bd9Sstevel@tonic-gate * debugger while still being longer than any legitimate 10107c478bd9Sstevel@tonic-gate * stretch at CY_HIGH_LEVEL). 10117c478bd9Sstevel@tonic-gate */ 10127c478bd9Sstevel@tonic-gate 10137c478bd9Sstevel@tonic-gate if (now - exp > NANOSEC) { 10147c478bd9Sstevel@tonic-gate hrtime_t interval = cyclic->cy_interval; 10157c478bd9Sstevel@tonic-gate 10167c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, exp == interval ? 10177c478bd9Sstevel@tonic-gate "fire-first" : "fire-swing", now, exp); 10187c478bd9Sstevel@tonic-gate 10197c478bd9Sstevel@tonic-gate exp += ((now - exp) / interval + 1) * interval; 10207c478bd9Sstevel@tonic-gate } 10217c478bd9Sstevel@tonic-gate 10227c478bd9Sstevel@tonic-gate cyclic->cy_expire = exp; 10237c478bd9Sstevel@tonic-gate cyclic_downheap(cpu, 0); 10247c478bd9Sstevel@tonic-gate } 10257c478bd9Sstevel@tonic-gate 10267c478bd9Sstevel@tonic-gate /* 10277c478bd9Sstevel@tonic-gate * Now we have a cyclic in the root slot which isn't in the past; 10287c478bd9Sstevel@tonic-gate * reprogram the interrupt source. 10297c478bd9Sstevel@tonic-gate */ 10307c478bd9Sstevel@tonic-gate be->cyb_reprogram(arg, exp); 10317c478bd9Sstevel@tonic-gate } 10327c478bd9Sstevel@tonic-gate 10337c478bd9Sstevel@tonic-gate static void 10347c478bd9Sstevel@tonic-gate cyclic_remove_pend(cyc_cpu_t *cpu, cyc_level_t level, cyclic_t *cyclic) 10357c478bd9Sstevel@tonic-gate { 10367c478bd9Sstevel@tonic-gate cyc_func_t handler = cyclic->cy_handler; 10377c478bd9Sstevel@tonic-gate void *arg = cyclic->cy_arg; 10387c478bd9Sstevel@tonic-gate uint32_t i, rpend = cpu->cyp_rpend - 1; 10397c478bd9Sstevel@tonic-gate 10407c478bd9Sstevel@tonic-gate ASSERT(cyclic->cy_flags & CYF_FREE); 10417c478bd9Sstevel@tonic-gate ASSERT(cyclic->cy_pend == 0); 10427c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_REMOVING); 10437c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_rpend > 0); 10447c478bd9Sstevel@tonic-gate 10457c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "remove-rpend", cyclic, cpu->cyp_rpend); 10467c478bd9Sstevel@tonic-gate 10477c478bd9Sstevel@tonic-gate /* 10487c478bd9Sstevel@tonic-gate * Note that we only call the handler cyp_rpend - 1 times; this is 10497c478bd9Sstevel@tonic-gate * to account for the handler call in cyclic_softint(). 10507c478bd9Sstevel@tonic-gate */ 10517c478bd9Sstevel@tonic-gate for (i = 0; i < rpend; i++) { 10527c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "rpend-in", handler, arg); 1053c210ded4Sesaxe DTRACE_PROBE1(cyclic__start, cyclic_t *, cyclic); 1054c210ded4Sesaxe 10557c478bd9Sstevel@tonic-gate (*handler)(arg); 1056c210ded4Sesaxe 1057c210ded4Sesaxe DTRACE_PROBE1(cyclic__end, cyclic_t *, cyclic); 10587c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "rpend-out", handler, arg); 10597c478bd9Sstevel@tonic-gate } 10607c478bd9Sstevel@tonic-gate 10617c478bd9Sstevel@tonic-gate /* 10627c478bd9Sstevel@tonic-gate * We can now let the remove operation complete. 10637c478bd9Sstevel@tonic-gate */ 10647c478bd9Sstevel@tonic-gate sema_v(&cpu->cyp_modify_wait); 10657c478bd9Sstevel@tonic-gate } 10667c478bd9Sstevel@tonic-gate 10677c478bd9Sstevel@tonic-gate /* 10687c478bd9Sstevel@tonic-gate * cyclic_softint(cpu_t *cpu, cyc_level_t level) 10697c478bd9Sstevel@tonic-gate * 10707c478bd9Sstevel@tonic-gate * Overview 10717c478bd9Sstevel@tonic-gate * 10727c478bd9Sstevel@tonic-gate * cyclic_softint() is the cyclic subsystem's CY_LOCK_LEVEL and CY_LOW_LEVEL 10737c478bd9Sstevel@tonic-gate * soft interrupt handler. Called by the cyclic backend. 10747c478bd9Sstevel@tonic-gate * 10757c478bd9Sstevel@tonic-gate * Arguments and notes 10767c478bd9Sstevel@tonic-gate * 10777c478bd9Sstevel@tonic-gate * The first argument to cyclic_softint() is the CPU on which the interrupt 10787c478bd9Sstevel@tonic-gate * is executing; backends must call into cyclic_softint() on the specified 10797c478bd9Sstevel@tonic-gate * CPU. The second argument is the level of the soft interrupt; it must 10807c478bd9Sstevel@tonic-gate * be one of CY_LOCK_LEVEL or CY_LOW_LEVEL. 10817c478bd9Sstevel@tonic-gate * 10827c478bd9Sstevel@tonic-gate * cyclic_softint() will call the handlers for cyclics pending at the 10837c478bd9Sstevel@tonic-gate * specified level. cyclic_softint() will not return until all pending 10847c478bd9Sstevel@tonic-gate * cyclics at the specified level have been dealt with; intervening 10857c478bd9Sstevel@tonic-gate * CY_HIGH_LEVEL interrupts which enqueue cyclics at the specified level 10867c478bd9Sstevel@tonic-gate * may therefore prolong cyclic_softint(). 10877c478bd9Sstevel@tonic-gate * 10887c478bd9Sstevel@tonic-gate * cyclic_softint() never disables interrupts, and, if neither a 10897c478bd9Sstevel@tonic-gate * cyclic_add() nor a cyclic_remove() is pending on the specified CPU, is 10907c478bd9Sstevel@tonic-gate * lock-free. This assures that in the common case, cyclic_softint() 10917c478bd9Sstevel@tonic-gate * completes without blocking, and never starves cyclic_fire(). If either 10927c478bd9Sstevel@tonic-gate * cyclic_add() or cyclic_remove() is pending, cyclic_softint() may grab 10937c478bd9Sstevel@tonic-gate * a dispatcher lock. 10947c478bd9Sstevel@tonic-gate * 10957c478bd9Sstevel@tonic-gate * While cyclic_softint() is designed for bounded latency, it is obviously 10967c478bd9Sstevel@tonic-gate * at the mercy of its cyclic handlers. Because cyclic handlers may block 10977c478bd9Sstevel@tonic-gate * arbitrarily, callers of cyclic_softint() should not rely upon 10987c478bd9Sstevel@tonic-gate * deterministic completion. 10997c478bd9Sstevel@tonic-gate * 11007c478bd9Sstevel@tonic-gate * cyclic_softint() may be called spuriously without ill effect. 11017c478bd9Sstevel@tonic-gate * 11027c478bd9Sstevel@tonic-gate * Return value 11037c478bd9Sstevel@tonic-gate * 11047c478bd9Sstevel@tonic-gate * None. 11057c478bd9Sstevel@tonic-gate * 11067c478bd9Sstevel@tonic-gate * Caller's context 11077c478bd9Sstevel@tonic-gate * 11087c478bd9Sstevel@tonic-gate * The caller must be executing in soft interrupt context at either 11097c478bd9Sstevel@tonic-gate * CY_LOCK_LEVEL or CY_LOW_LEVEL. The level passed to cyclic_softint() 11107c478bd9Sstevel@tonic-gate * must match the level at which it is executing. On optimal backends, 11117c478bd9Sstevel@tonic-gate * the caller will hold no locks. In any case, the caller may not hold 11127c478bd9Sstevel@tonic-gate * cpu_lock or any lock acquired by any cyclic handler or held across 11137c478bd9Sstevel@tonic-gate * any of cyclic_add(), cyclic_remove(), cyclic_bind() or cyclic_juggle(). 11147c478bd9Sstevel@tonic-gate */ 11157c478bd9Sstevel@tonic-gate void 11167c478bd9Sstevel@tonic-gate cyclic_softint(cpu_t *c, cyc_level_t level) 11177c478bd9Sstevel@tonic-gate { 11187c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = c->cpu_cyclic; 11197c478bd9Sstevel@tonic-gate cyc_softbuf_t *softbuf; 11207c478bd9Sstevel@tonic-gate int soft, *buf, consndx, resized = 0, intr_resized = 0; 11217c478bd9Sstevel@tonic-gate cyc_pcbuffer_t *pc; 11227c478bd9Sstevel@tonic-gate cyclic_t *cyclics = cpu->cyp_cyclics; 11237c478bd9Sstevel@tonic-gate int sizemask; 11247c478bd9Sstevel@tonic-gate 11257c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "softint", cyclics, 0); 11267c478bd9Sstevel@tonic-gate 11277c478bd9Sstevel@tonic-gate ASSERT(level < CY_LOW_LEVEL + CY_SOFT_LEVELS); 11287c478bd9Sstevel@tonic-gate 11297c478bd9Sstevel@tonic-gate softbuf = &cpu->cyp_softbuf[level]; 11307c478bd9Sstevel@tonic-gate top: 11317c478bd9Sstevel@tonic-gate soft = softbuf->cys_soft; 11327c478bd9Sstevel@tonic-gate ASSERT(soft == 0 || soft == 1); 11337c478bd9Sstevel@tonic-gate 11347c478bd9Sstevel@tonic-gate pc = &softbuf->cys_buf[soft]; 11357c478bd9Sstevel@tonic-gate buf = pc->cypc_buf; 11367c478bd9Sstevel@tonic-gate consndx = pc->cypc_consndx; 11377c478bd9Sstevel@tonic-gate sizemask = pc->cypc_sizemask; 11387c478bd9Sstevel@tonic-gate 11397c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "softint-top", cyclics, pc); 11407c478bd9Sstevel@tonic-gate 11417c478bd9Sstevel@tonic-gate while (consndx != pc->cypc_prodndx) { 11427c478bd9Sstevel@tonic-gate int pend, npend, opend; 11437c478bd9Sstevel@tonic-gate int consmasked = consndx & sizemask; 11447c478bd9Sstevel@tonic-gate cyclic_t *cyclic = &cyclics[buf[consmasked]]; 11457c478bd9Sstevel@tonic-gate cyc_func_t handler = cyclic->cy_handler; 11467c478bd9Sstevel@tonic-gate void *arg = cyclic->cy_arg; 11477c478bd9Sstevel@tonic-gate 11487c478bd9Sstevel@tonic-gate ASSERT(buf[consmasked] < cpu->cyp_size); 11497c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "consuming", consndx, cyclic); 11507c478bd9Sstevel@tonic-gate 11517c478bd9Sstevel@tonic-gate /* 11527c478bd9Sstevel@tonic-gate * We have found this cyclic in the pcbuffer. We know that 11537c478bd9Sstevel@tonic-gate * one of the following is true: 11547c478bd9Sstevel@tonic-gate * 11557c478bd9Sstevel@tonic-gate * (a) The pend is non-zero. We need to execute the handler 11567c478bd9Sstevel@tonic-gate * at least once. 11577c478bd9Sstevel@tonic-gate * 11587c478bd9Sstevel@tonic-gate * (b) The pend _was_ non-zero, but it's now zero due to a 11597c478bd9Sstevel@tonic-gate * resize. We will call the handler once, see that we 11607c478bd9Sstevel@tonic-gate * are in this case, and read the new cyclics buffer 11617c478bd9Sstevel@tonic-gate * (and hence the old non-zero pend). 11627c478bd9Sstevel@tonic-gate * 11637c478bd9Sstevel@tonic-gate * (c) The pend _was_ non-zero, but it's now zero due to a 11647c478bd9Sstevel@tonic-gate * removal. We will call the handler once, see that we 11657c478bd9Sstevel@tonic-gate * are in this case, and call into cyclic_remove_pend() 11667c478bd9Sstevel@tonic-gate * to call the cyclic rpend times. We will take into 11677c478bd9Sstevel@tonic-gate * account that we have already called the handler once. 11687c478bd9Sstevel@tonic-gate * 11697c478bd9Sstevel@tonic-gate * Point is: it's safe to call the handler without first 11707c478bd9Sstevel@tonic-gate * checking the pend. 11717c478bd9Sstevel@tonic-gate */ 11727c478bd9Sstevel@tonic-gate do { 11737c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "handler-in", handler, arg); 1174c210ded4Sesaxe DTRACE_PROBE1(cyclic__start, cyclic_t *, cyclic); 1175c210ded4Sesaxe 11767c478bd9Sstevel@tonic-gate (*handler)(arg); 1177c210ded4Sesaxe 1178c210ded4Sesaxe DTRACE_PROBE1(cyclic__end, cyclic_t *, cyclic); 11797c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "handler-out", handler, arg); 11807c478bd9Sstevel@tonic-gate reread: 11817c478bd9Sstevel@tonic-gate pend = cyclic->cy_pend; 11827c478bd9Sstevel@tonic-gate npend = pend - 1; 11837c478bd9Sstevel@tonic-gate 11847c478bd9Sstevel@tonic-gate if (pend == 0) { 11857c478bd9Sstevel@tonic-gate if (cpu->cyp_state == CYS_REMOVING) { 11867c478bd9Sstevel@tonic-gate /* 11877c478bd9Sstevel@tonic-gate * This cyclic has been removed while 11887c478bd9Sstevel@tonic-gate * it had a non-zero pend count (we 11897c478bd9Sstevel@tonic-gate * know it was non-zero because we 11907c478bd9Sstevel@tonic-gate * found this cyclic in the pcbuffer). 11917c478bd9Sstevel@tonic-gate * There must be a non-zero rpend for 11927c478bd9Sstevel@tonic-gate * this CPU, and there must be a remove 11937c478bd9Sstevel@tonic-gate * operation blocking; we'll call into 11947c478bd9Sstevel@tonic-gate * cyclic_remove_pend() to clean this 11957c478bd9Sstevel@tonic-gate * up, and break out of the pend loop. 11967c478bd9Sstevel@tonic-gate */ 11977c478bd9Sstevel@tonic-gate cyclic_remove_pend(cpu, level, cyclic); 11987c478bd9Sstevel@tonic-gate break; 11997c478bd9Sstevel@tonic-gate } 12007c478bd9Sstevel@tonic-gate 12017c478bd9Sstevel@tonic-gate /* 12027c478bd9Sstevel@tonic-gate * We must have had a resize interrupt us. 12037c478bd9Sstevel@tonic-gate */ 12047c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "resize-int", cyclics, 0); 12057c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_EXPANDING); 12067c478bd9Sstevel@tonic-gate ASSERT(cyclics != cpu->cyp_cyclics); 12077c478bd9Sstevel@tonic-gate ASSERT(resized == 0); 12087c478bd9Sstevel@tonic-gate ASSERT(intr_resized == 0); 12097c478bd9Sstevel@tonic-gate intr_resized = 1; 12107c478bd9Sstevel@tonic-gate cyclics = cpu->cyp_cyclics; 12117c478bd9Sstevel@tonic-gate cyclic = &cyclics[buf[consmasked]]; 12127c478bd9Sstevel@tonic-gate ASSERT(cyclic->cy_handler == handler); 12137c478bd9Sstevel@tonic-gate ASSERT(cyclic->cy_arg == arg); 12147c478bd9Sstevel@tonic-gate goto reread; 12157c478bd9Sstevel@tonic-gate } 12167c478bd9Sstevel@tonic-gate 12177c478bd9Sstevel@tonic-gate if ((opend = 12187c478bd9Sstevel@tonic-gate cas32(&cyclic->cy_pend, pend, npend)) != pend) { 12197c478bd9Sstevel@tonic-gate /* 12207c478bd9Sstevel@tonic-gate * Our cas32 can fail for one of several 12217c478bd9Sstevel@tonic-gate * reasons: 12227c478bd9Sstevel@tonic-gate * 12237c478bd9Sstevel@tonic-gate * (a) An intervening high level bumped up the 12247c478bd9Sstevel@tonic-gate * pend count on this cyclic. In this 12257c478bd9Sstevel@tonic-gate * case, we will see a higher pend. 12267c478bd9Sstevel@tonic-gate * 12277c478bd9Sstevel@tonic-gate * (b) The cyclics array has been yanked out 12287c478bd9Sstevel@tonic-gate * from underneath us by a resize 12297c478bd9Sstevel@tonic-gate * operation. In this case, pend is 0 and 12307c478bd9Sstevel@tonic-gate * cyp_state is CYS_EXPANDING. 12317c478bd9Sstevel@tonic-gate * 12327c478bd9Sstevel@tonic-gate * (c) The cyclic has been removed by an 12337c478bd9Sstevel@tonic-gate * intervening remove-xcall. In this case, 12347c478bd9Sstevel@tonic-gate * pend will be 0, the cyp_state will be 12357c478bd9Sstevel@tonic-gate * CYS_REMOVING, and the cyclic will be 12367c478bd9Sstevel@tonic-gate * marked CYF_FREE. 12377c478bd9Sstevel@tonic-gate * 12387c478bd9Sstevel@tonic-gate * The assertion below checks that we are 12397c478bd9Sstevel@tonic-gate * in one of the above situations. The 12407c478bd9Sstevel@tonic-gate * action under all three is to return to 12417c478bd9Sstevel@tonic-gate * the top of the loop. 12427c478bd9Sstevel@tonic-gate */ 12437c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "cas-fail", opend, pend); 12447c478bd9Sstevel@tonic-gate ASSERT(opend > pend || (opend == 0 && 12457c478bd9Sstevel@tonic-gate ((cyclics != cpu->cyp_cyclics && 12467c478bd9Sstevel@tonic-gate cpu->cyp_state == CYS_EXPANDING) || 12477c478bd9Sstevel@tonic-gate (cpu->cyp_state == CYS_REMOVING && 12487c478bd9Sstevel@tonic-gate (cyclic->cy_flags & CYF_FREE))))); 12497c478bd9Sstevel@tonic-gate goto reread; 12507c478bd9Sstevel@tonic-gate } 12517c478bd9Sstevel@tonic-gate 12527c478bd9Sstevel@tonic-gate /* 12537c478bd9Sstevel@tonic-gate * Okay, so we've managed to successfully decrement 12547c478bd9Sstevel@tonic-gate * pend. If we just decremented the pend to 0, we're 12557c478bd9Sstevel@tonic-gate * done. 12567c478bd9Sstevel@tonic-gate */ 12577c478bd9Sstevel@tonic-gate } while (npend > 0); 12587c478bd9Sstevel@tonic-gate 12597c478bd9Sstevel@tonic-gate pc->cypc_consndx = ++consndx; 12607c478bd9Sstevel@tonic-gate } 12617c478bd9Sstevel@tonic-gate 12627c478bd9Sstevel@tonic-gate /* 12637c478bd9Sstevel@tonic-gate * If the high level handler is no longer writing to the same 12647c478bd9Sstevel@tonic-gate * buffer, then we've had a resize. We need to switch our soft 12657c478bd9Sstevel@tonic-gate * index, and goto top. 12667c478bd9Sstevel@tonic-gate */ 12677c478bd9Sstevel@tonic-gate if (soft != softbuf->cys_hard) { 12687c478bd9Sstevel@tonic-gate /* 12697c478bd9Sstevel@tonic-gate * We can assert that the other buffer has grown by exactly 12707c478bd9Sstevel@tonic-gate * one factor of two. 12717c478bd9Sstevel@tonic-gate */ 12727c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, level, "buffer-grow", 0, 0); 12737c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_EXPANDING); 12747c478bd9Sstevel@tonic-gate ASSERT(softbuf->cys_buf[softbuf->cys_hard].cypc_sizemask == 12757c478bd9Sstevel@tonic-gate (softbuf->cys_buf[soft].cypc_sizemask << 1) + 1 || 12767c478bd9Sstevel@tonic-gate softbuf->cys_buf[soft].cypc_sizemask == 0); 12777c478bd9Sstevel@tonic-gate ASSERT(softbuf->cys_hard == (softbuf->cys_soft ^ 1)); 12787c478bd9Sstevel@tonic-gate 12797c478bd9Sstevel@tonic-gate /* 12807c478bd9Sstevel@tonic-gate * If our cached cyclics pointer doesn't match cyp_cyclics, 12817c478bd9Sstevel@tonic-gate * then we took a resize between our last iteration of the 12827c478bd9Sstevel@tonic-gate * pend loop and the check against softbuf->cys_hard. 12837c478bd9Sstevel@tonic-gate */ 12847c478bd9Sstevel@tonic-gate if (cpu->cyp_cyclics != cyclics) { 12857c478bd9Sstevel@tonic-gate CYC_TRACE1(cpu, level, "resize-int-int", consndx); 12867c478bd9Sstevel@tonic-gate cyclics = cpu->cyp_cyclics; 12877c478bd9Sstevel@tonic-gate } 12887c478bd9Sstevel@tonic-gate 12897c478bd9Sstevel@tonic-gate softbuf->cys_soft = softbuf->cys_hard; 12907c478bd9Sstevel@tonic-gate 12917c478bd9Sstevel@tonic-gate ASSERT(resized == 0); 12927c478bd9Sstevel@tonic-gate resized = 1; 12937c478bd9Sstevel@tonic-gate goto top; 12947c478bd9Sstevel@tonic-gate } 12957c478bd9Sstevel@tonic-gate 12967c478bd9Sstevel@tonic-gate /* 12977c478bd9Sstevel@tonic-gate * If we were interrupted by a resize operation, then we must have 12987c478bd9Sstevel@tonic-gate * seen the hard index change. 12997c478bd9Sstevel@tonic-gate */ 13007c478bd9Sstevel@tonic-gate ASSERT(!(intr_resized == 1 && resized == 0)); 13017c478bd9Sstevel@tonic-gate 13027c478bd9Sstevel@tonic-gate if (resized) { 13037c478bd9Sstevel@tonic-gate uint32_t lev, nlev; 13047c478bd9Sstevel@tonic-gate 13057c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_EXPANDING); 13067c478bd9Sstevel@tonic-gate 13077c478bd9Sstevel@tonic-gate do { 13087c478bd9Sstevel@tonic-gate lev = cpu->cyp_modify_levels; 13097c478bd9Sstevel@tonic-gate nlev = lev + 1; 13107c478bd9Sstevel@tonic-gate } while (cas32(&cpu->cyp_modify_levels, lev, nlev) != lev); 13117c478bd9Sstevel@tonic-gate 13127c478bd9Sstevel@tonic-gate /* 13137c478bd9Sstevel@tonic-gate * If we are the last soft level to see the modification, 13147c478bd9Sstevel@tonic-gate * post on cyp_modify_wait. Otherwise, (if we're not 13157c478bd9Sstevel@tonic-gate * already at low level), post down to the next soft level. 13167c478bd9Sstevel@tonic-gate */ 13177c478bd9Sstevel@tonic-gate if (nlev == CY_SOFT_LEVELS) { 13187c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, level, "resize-kick"); 13197c478bd9Sstevel@tonic-gate sema_v(&cpu->cyp_modify_wait); 13207c478bd9Sstevel@tonic-gate } else { 13217c478bd9Sstevel@tonic-gate ASSERT(nlev < CY_SOFT_LEVELS); 13227c478bd9Sstevel@tonic-gate if (level != CY_LOW_LEVEL) { 13237c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 13247c478bd9Sstevel@tonic-gate 13257c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, level, "resize-post"); 13267c478bd9Sstevel@tonic-gate be->cyb_softint(be->cyb_arg, level - 1); 13277c478bd9Sstevel@tonic-gate } 13287c478bd9Sstevel@tonic-gate } 13297c478bd9Sstevel@tonic-gate } 13307c478bd9Sstevel@tonic-gate } 13317c478bd9Sstevel@tonic-gate 13327c478bd9Sstevel@tonic-gate static void 13337c478bd9Sstevel@tonic-gate cyclic_expand_xcall(cyc_xcallarg_t *arg) 13347c478bd9Sstevel@tonic-gate { 13357c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = arg->cyx_cpu; 13367c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 13377c478bd9Sstevel@tonic-gate cyb_arg_t bar = be->cyb_arg; 13387c478bd9Sstevel@tonic-gate cyc_cookie_t cookie; 13397c478bd9Sstevel@tonic-gate cyc_index_t new_size = arg->cyx_size, size = cpu->cyp_size, i; 13407c478bd9Sstevel@tonic-gate cyc_index_t *new_heap = arg->cyx_heap; 13417c478bd9Sstevel@tonic-gate cyclic_t *cyclics = cpu->cyp_cyclics, *new_cyclics = arg->cyx_cyclics; 13427c478bd9Sstevel@tonic-gate 13437c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_EXPANDING); 13447c478bd9Sstevel@tonic-gate 13457c478bd9Sstevel@tonic-gate /* 13467c478bd9Sstevel@tonic-gate * This is a little dicey. First, we'll raise our interrupt level 13477c478bd9Sstevel@tonic-gate * to CY_HIGH_LEVEL. This CPU already has a new heap, cyclic array, 13487c478bd9Sstevel@tonic-gate * etc.; we just need to bcopy them across. As for the softint 13497c478bd9Sstevel@tonic-gate * buffers, we'll switch the active buffers. The actual softints will 13507c478bd9Sstevel@tonic-gate * take care of consuming any pending cyclics in the old buffer. 13517c478bd9Sstevel@tonic-gate */ 13527c478bd9Sstevel@tonic-gate cookie = be->cyb_set_level(bar, CY_HIGH_LEVEL); 13537c478bd9Sstevel@tonic-gate 13547c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "expand", new_size, 0); 13557c478bd9Sstevel@tonic-gate 13567c478bd9Sstevel@tonic-gate /* 13577c478bd9Sstevel@tonic-gate * Assert that the new size is a power of 2. 13587c478bd9Sstevel@tonic-gate */ 13597c478bd9Sstevel@tonic-gate ASSERT((new_size & new_size - 1) == 0); 13607c478bd9Sstevel@tonic-gate ASSERT(new_size == (size << 1)); 13617c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_heap != NULL && cpu->cyp_cyclics != NULL); 13627c478bd9Sstevel@tonic-gate 13637c478bd9Sstevel@tonic-gate bcopy(cpu->cyp_heap, new_heap, sizeof (cyc_index_t) * size); 13647c478bd9Sstevel@tonic-gate bcopy(cyclics, new_cyclics, sizeof (cyclic_t) * size); 13657c478bd9Sstevel@tonic-gate 13667c478bd9Sstevel@tonic-gate /* 13677c478bd9Sstevel@tonic-gate * Now run through the old cyclics array, setting pend to 0. To 13687c478bd9Sstevel@tonic-gate * softints (which are executing at a lower priority level), the 13697c478bd9Sstevel@tonic-gate * pends dropping to 0 will appear atomic with the cyp_cyclics 13707c478bd9Sstevel@tonic-gate * pointer changing. 13717c478bd9Sstevel@tonic-gate */ 13727c478bd9Sstevel@tonic-gate for (i = 0; i < size; i++) 13737c478bd9Sstevel@tonic-gate cyclics[i].cy_pend = 0; 13747c478bd9Sstevel@tonic-gate 13757c478bd9Sstevel@tonic-gate /* 13767c478bd9Sstevel@tonic-gate * Set up the free list, and set all of the new cyclics to be CYF_FREE. 13777c478bd9Sstevel@tonic-gate */ 13787c478bd9Sstevel@tonic-gate for (i = size; i < new_size; i++) { 13797c478bd9Sstevel@tonic-gate new_heap[i] = i; 13807c478bd9Sstevel@tonic-gate new_cyclics[i].cy_flags = CYF_FREE; 13817c478bd9Sstevel@tonic-gate } 13827c478bd9Sstevel@tonic-gate 13837c478bd9Sstevel@tonic-gate /* 13847c478bd9Sstevel@tonic-gate * We can go ahead and plow the value of cyp_heap and cyp_cyclics; 13857c478bd9Sstevel@tonic-gate * cyclic_expand() has kept a copy. 13867c478bd9Sstevel@tonic-gate */ 13877c478bd9Sstevel@tonic-gate cpu->cyp_heap = new_heap; 13887c478bd9Sstevel@tonic-gate cpu->cyp_cyclics = new_cyclics; 13897c478bd9Sstevel@tonic-gate cpu->cyp_size = new_size; 13907c478bd9Sstevel@tonic-gate 13917c478bd9Sstevel@tonic-gate /* 13927c478bd9Sstevel@tonic-gate * We've switched over the heap and the cyclics array. Now we need 13937c478bd9Sstevel@tonic-gate * to switch over our active softint buffer pointers. 13947c478bd9Sstevel@tonic-gate */ 13957c478bd9Sstevel@tonic-gate for (i = CY_LOW_LEVEL; i < CY_LOW_LEVEL + CY_SOFT_LEVELS; i++) { 13967c478bd9Sstevel@tonic-gate cyc_softbuf_t *softbuf = &cpu->cyp_softbuf[i]; 13977c478bd9Sstevel@tonic-gate uchar_t hard = softbuf->cys_hard; 13987c478bd9Sstevel@tonic-gate 13997c478bd9Sstevel@tonic-gate /* 14007c478bd9Sstevel@tonic-gate * Assert that we're not in the middle of a resize operation. 14017c478bd9Sstevel@tonic-gate */ 14027c478bd9Sstevel@tonic-gate ASSERT(hard == softbuf->cys_soft); 14037c478bd9Sstevel@tonic-gate ASSERT(hard == 0 || hard == 1); 14047c478bd9Sstevel@tonic-gate ASSERT(softbuf->cys_buf[hard].cypc_buf != NULL); 14057c478bd9Sstevel@tonic-gate 14067c478bd9Sstevel@tonic-gate softbuf->cys_hard = hard ^ 1; 14077c478bd9Sstevel@tonic-gate 14087c478bd9Sstevel@tonic-gate /* 14097c478bd9Sstevel@tonic-gate * The caller (cyclic_expand()) is responsible for setting 14107c478bd9Sstevel@tonic-gate * up the new producer-consumer buffer; assert that it's 14117c478bd9Sstevel@tonic-gate * been done correctly. 14127c478bd9Sstevel@tonic-gate */ 14137c478bd9Sstevel@tonic-gate ASSERT(softbuf->cys_buf[hard ^ 1].cypc_buf != NULL); 14147c478bd9Sstevel@tonic-gate ASSERT(softbuf->cys_buf[hard ^ 1].cypc_prodndx == 0); 14157c478bd9Sstevel@tonic-gate ASSERT(softbuf->cys_buf[hard ^ 1].cypc_consndx == 0); 14167c478bd9Sstevel@tonic-gate } 14177c478bd9Sstevel@tonic-gate 14187c478bd9Sstevel@tonic-gate /* 14197c478bd9Sstevel@tonic-gate * That's all there is to it; now we just need to postdown to 14207c478bd9Sstevel@tonic-gate * get the softint chain going. 14217c478bd9Sstevel@tonic-gate */ 14227c478bd9Sstevel@tonic-gate be->cyb_softint(bar, CY_HIGH_LEVEL - 1); 14237c478bd9Sstevel@tonic-gate be->cyb_restore_level(bar, cookie); 14247c478bd9Sstevel@tonic-gate } 14257c478bd9Sstevel@tonic-gate 14267c478bd9Sstevel@tonic-gate /* 14277c478bd9Sstevel@tonic-gate * cyclic_expand() will cross call onto the CPU to perform the actual 14287c478bd9Sstevel@tonic-gate * expand operation. 14297c478bd9Sstevel@tonic-gate */ 14307c478bd9Sstevel@tonic-gate static void 14317c478bd9Sstevel@tonic-gate cyclic_expand(cyc_cpu_t *cpu) 14327c478bd9Sstevel@tonic-gate { 14337c478bd9Sstevel@tonic-gate cyc_index_t new_size, old_size; 14347c478bd9Sstevel@tonic-gate cyc_index_t *new_heap, *old_heap; 14357c478bd9Sstevel@tonic-gate cyclic_t *new_cyclics, *old_cyclics; 14367c478bd9Sstevel@tonic-gate cyc_xcallarg_t arg; 14377c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 14387c478bd9Sstevel@tonic-gate char old_hard; 14397c478bd9Sstevel@tonic-gate int i; 14407c478bd9Sstevel@tonic-gate 14417c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 14427c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 14437c478bd9Sstevel@tonic-gate 14447c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_EXPANDING; 14457c478bd9Sstevel@tonic-gate 14467c478bd9Sstevel@tonic-gate old_heap = cpu->cyp_heap; 14477c478bd9Sstevel@tonic-gate old_cyclics = cpu->cyp_cyclics; 14487c478bd9Sstevel@tonic-gate 14497c478bd9Sstevel@tonic-gate if ((new_size = ((old_size = cpu->cyp_size) << 1)) == 0) { 14507c478bd9Sstevel@tonic-gate new_size = CY_DEFAULT_PERCPU; 14517c478bd9Sstevel@tonic-gate ASSERT(old_heap == NULL && old_cyclics == NULL); 14527c478bd9Sstevel@tonic-gate } 14537c478bd9Sstevel@tonic-gate 14547c478bd9Sstevel@tonic-gate /* 14557c478bd9Sstevel@tonic-gate * Check that the new_size is a power of 2. 14567c478bd9Sstevel@tonic-gate */ 14577c478bd9Sstevel@tonic-gate ASSERT((new_size - 1 & new_size) == 0); 14587c478bd9Sstevel@tonic-gate 14597c478bd9Sstevel@tonic-gate new_heap = kmem_alloc(sizeof (cyc_index_t) * new_size, KM_SLEEP); 14607c478bd9Sstevel@tonic-gate new_cyclics = kmem_zalloc(sizeof (cyclic_t) * new_size, KM_SLEEP); 14617c478bd9Sstevel@tonic-gate 14627c478bd9Sstevel@tonic-gate /* 14637c478bd9Sstevel@tonic-gate * We know that no other expansions are in progress (they serialize 14647c478bd9Sstevel@tonic-gate * on cpu_lock), so we can safely read the softbuf metadata. 14657c478bd9Sstevel@tonic-gate */ 14667c478bd9Sstevel@tonic-gate old_hard = cpu->cyp_softbuf[0].cys_hard; 14677c478bd9Sstevel@tonic-gate 14687c478bd9Sstevel@tonic-gate for (i = CY_LOW_LEVEL; i < CY_LOW_LEVEL + CY_SOFT_LEVELS; i++) { 14697c478bd9Sstevel@tonic-gate cyc_softbuf_t *softbuf = &cpu->cyp_softbuf[i]; 14707c478bd9Sstevel@tonic-gate char hard = softbuf->cys_hard; 14717c478bd9Sstevel@tonic-gate cyc_pcbuffer_t *pc = &softbuf->cys_buf[hard ^ 1]; 14727c478bd9Sstevel@tonic-gate 14737c478bd9Sstevel@tonic-gate ASSERT(hard == old_hard); 14747c478bd9Sstevel@tonic-gate ASSERT(hard == softbuf->cys_soft); 14757c478bd9Sstevel@tonic-gate ASSERT(pc->cypc_buf == NULL); 14767c478bd9Sstevel@tonic-gate 14777c478bd9Sstevel@tonic-gate pc->cypc_buf = 14787c478bd9Sstevel@tonic-gate kmem_alloc(sizeof (cyc_index_t) * new_size, KM_SLEEP); 14797c478bd9Sstevel@tonic-gate pc->cypc_prodndx = pc->cypc_consndx = 0; 14807c478bd9Sstevel@tonic-gate pc->cypc_sizemask = new_size - 1; 14817c478bd9Sstevel@tonic-gate } 14827c478bd9Sstevel@tonic-gate 14837c478bd9Sstevel@tonic-gate arg.cyx_cpu = cpu; 14847c478bd9Sstevel@tonic-gate arg.cyx_heap = new_heap; 14857c478bd9Sstevel@tonic-gate arg.cyx_cyclics = new_cyclics; 14867c478bd9Sstevel@tonic-gate arg.cyx_size = new_size; 14877c478bd9Sstevel@tonic-gate 14887c478bd9Sstevel@tonic-gate cpu->cyp_modify_levels = 0; 14897c478bd9Sstevel@tonic-gate 14907c478bd9Sstevel@tonic-gate be->cyb_xcall(be->cyb_arg, cpu->cyp_cpu, 14917c478bd9Sstevel@tonic-gate (cyc_func_t)cyclic_expand_xcall, &arg); 14927c478bd9Sstevel@tonic-gate 14937c478bd9Sstevel@tonic-gate /* 14947c478bd9Sstevel@tonic-gate * Now block, waiting for the resize operation to complete. 14957c478bd9Sstevel@tonic-gate */ 14967c478bd9Sstevel@tonic-gate sema_p(&cpu->cyp_modify_wait); 14977c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_modify_levels == CY_SOFT_LEVELS); 14987c478bd9Sstevel@tonic-gate 14997c478bd9Sstevel@tonic-gate /* 15007c478bd9Sstevel@tonic-gate * The operation is complete; we can now free the old buffers. 15017c478bd9Sstevel@tonic-gate */ 15027c478bd9Sstevel@tonic-gate for (i = CY_LOW_LEVEL; i < CY_LOW_LEVEL + CY_SOFT_LEVELS; i++) { 15037c478bd9Sstevel@tonic-gate cyc_softbuf_t *softbuf = &cpu->cyp_softbuf[i]; 15047c478bd9Sstevel@tonic-gate char hard = softbuf->cys_hard; 15057c478bd9Sstevel@tonic-gate cyc_pcbuffer_t *pc = &softbuf->cys_buf[hard ^ 1]; 15067c478bd9Sstevel@tonic-gate 15077c478bd9Sstevel@tonic-gate ASSERT(hard == (old_hard ^ 1)); 15087c478bd9Sstevel@tonic-gate ASSERT(hard == softbuf->cys_soft); 15097c478bd9Sstevel@tonic-gate 15107c478bd9Sstevel@tonic-gate if (pc->cypc_buf == NULL) 15117c478bd9Sstevel@tonic-gate continue; 15127c478bd9Sstevel@tonic-gate 15137c478bd9Sstevel@tonic-gate ASSERT(pc->cypc_sizemask == ((new_size - 1) >> 1)); 15147c478bd9Sstevel@tonic-gate 15157c478bd9Sstevel@tonic-gate kmem_free(pc->cypc_buf, 15167c478bd9Sstevel@tonic-gate sizeof (cyc_index_t) * (pc->cypc_sizemask + 1)); 15177c478bd9Sstevel@tonic-gate pc->cypc_buf = NULL; 15187c478bd9Sstevel@tonic-gate } 15197c478bd9Sstevel@tonic-gate 15207c478bd9Sstevel@tonic-gate if (old_cyclics != NULL) { 15217c478bd9Sstevel@tonic-gate ASSERT(old_heap != NULL); 15227c478bd9Sstevel@tonic-gate ASSERT(old_size != 0); 15237c478bd9Sstevel@tonic-gate kmem_free(old_cyclics, sizeof (cyclic_t) * old_size); 15247c478bd9Sstevel@tonic-gate kmem_free(old_heap, sizeof (cyc_index_t) * old_size); 15257c478bd9Sstevel@tonic-gate } 15267c478bd9Sstevel@tonic-gate 15277c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_EXPANDING); 15287c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_ONLINE; 15297c478bd9Sstevel@tonic-gate } 15307c478bd9Sstevel@tonic-gate 15317c478bd9Sstevel@tonic-gate /* 15327c478bd9Sstevel@tonic-gate * cyclic_pick_cpu will attempt to pick a CPU according to the constraints 15337c478bd9Sstevel@tonic-gate * specified by the partition, bound CPU, and flags. Additionally, 15347c478bd9Sstevel@tonic-gate * cyclic_pick_cpu() will not pick the avoid CPU; it will return NULL if 15357c478bd9Sstevel@tonic-gate * the avoid CPU is the only CPU which satisfies the constraints. 15367c478bd9Sstevel@tonic-gate * 15377c478bd9Sstevel@tonic-gate * If CYF_CPU_BOUND is set in flags, the specified CPU must be non-NULL. 15387c478bd9Sstevel@tonic-gate * If CYF_PART_BOUND is set in flags, the specified partition must be non-NULL. 15397c478bd9Sstevel@tonic-gate * If both CYF_CPU_BOUND and CYF_PART_BOUND are set, the specified CPU must 15407c478bd9Sstevel@tonic-gate * be in the specified partition. 15417c478bd9Sstevel@tonic-gate */ 15427c478bd9Sstevel@tonic-gate static cyc_cpu_t * 15437c478bd9Sstevel@tonic-gate cyclic_pick_cpu(cpupart_t *part, cpu_t *bound, cpu_t *avoid, uint16_t flags) 15447c478bd9Sstevel@tonic-gate { 15457c478bd9Sstevel@tonic-gate cpu_t *c, *start = (part != NULL) ? part->cp_cpulist : CPU; 15467c478bd9Sstevel@tonic-gate cpu_t *online = NULL; 15477c478bd9Sstevel@tonic-gate uintptr_t offset; 15487c478bd9Sstevel@tonic-gate 15497c478bd9Sstevel@tonic-gate CYC_PTRACE("pick-cpu", part, bound); 15507c478bd9Sstevel@tonic-gate 15517c478bd9Sstevel@tonic-gate ASSERT(!(flags & CYF_CPU_BOUND) || bound != NULL); 15527c478bd9Sstevel@tonic-gate ASSERT(!(flags & CYF_PART_BOUND) || part != NULL); 15537c478bd9Sstevel@tonic-gate 15547c478bd9Sstevel@tonic-gate /* 15557c478bd9Sstevel@tonic-gate * If we're bound to our CPU, there isn't much choice involved. We 15567c478bd9Sstevel@tonic-gate * need to check that the CPU passed as bound is in the cpupart, and 15577c478bd9Sstevel@tonic-gate * that the CPU that we're binding to has been configured. 15587c478bd9Sstevel@tonic-gate */ 15597c478bd9Sstevel@tonic-gate if (flags & CYF_CPU_BOUND) { 15607c478bd9Sstevel@tonic-gate CYC_PTRACE("pick-cpu-bound", bound, avoid); 15617c478bd9Sstevel@tonic-gate 15627c478bd9Sstevel@tonic-gate if ((flags & CYF_PART_BOUND) && bound->cpu_part != part) 15637c478bd9Sstevel@tonic-gate panic("cyclic_pick_cpu: " 15647c478bd9Sstevel@tonic-gate "CPU binding contradicts partition binding"); 15657c478bd9Sstevel@tonic-gate 15667c478bd9Sstevel@tonic-gate if (bound == avoid) 15677c478bd9Sstevel@tonic-gate return (NULL); 15687c478bd9Sstevel@tonic-gate 15697c478bd9Sstevel@tonic-gate if (bound->cpu_cyclic == NULL) 15707c478bd9Sstevel@tonic-gate panic("cyclic_pick_cpu: " 15717c478bd9Sstevel@tonic-gate "attempt to bind to non-configured CPU"); 15727c478bd9Sstevel@tonic-gate 15737c478bd9Sstevel@tonic-gate return (bound->cpu_cyclic); 15747c478bd9Sstevel@tonic-gate } 15757c478bd9Sstevel@tonic-gate 15767c478bd9Sstevel@tonic-gate if (flags & CYF_PART_BOUND) { 15777c478bd9Sstevel@tonic-gate CYC_PTRACE("pick-part-bound", bound, avoid); 15787c478bd9Sstevel@tonic-gate offset = offsetof(cpu_t, cpu_next_part); 15797c478bd9Sstevel@tonic-gate } else { 15807c478bd9Sstevel@tonic-gate offset = offsetof(cpu_t, cpu_next_onln); 15817c478bd9Sstevel@tonic-gate } 15827c478bd9Sstevel@tonic-gate 15837c478bd9Sstevel@tonic-gate c = start; 15847c478bd9Sstevel@tonic-gate do { 15857c478bd9Sstevel@tonic-gate if (c->cpu_cyclic == NULL) 15867c478bd9Sstevel@tonic-gate continue; 15877c478bd9Sstevel@tonic-gate 15887c478bd9Sstevel@tonic-gate if (c->cpu_cyclic->cyp_state == CYS_OFFLINE) 15897c478bd9Sstevel@tonic-gate continue; 15907c478bd9Sstevel@tonic-gate 15917c478bd9Sstevel@tonic-gate if (c == avoid) 15927c478bd9Sstevel@tonic-gate continue; 15937c478bd9Sstevel@tonic-gate 15947c478bd9Sstevel@tonic-gate if (c->cpu_flags & CPU_ENABLE) 15957c478bd9Sstevel@tonic-gate goto found; 15967c478bd9Sstevel@tonic-gate 15977c478bd9Sstevel@tonic-gate if (online == NULL) 15987c478bd9Sstevel@tonic-gate online = c; 15997c478bd9Sstevel@tonic-gate } while ((c = *(cpu_t **)((uintptr_t)c + offset)) != start); 16007c478bd9Sstevel@tonic-gate 16017c478bd9Sstevel@tonic-gate /* 16027c478bd9Sstevel@tonic-gate * If we're here, we're in one of two situations: 16037c478bd9Sstevel@tonic-gate * 16047c478bd9Sstevel@tonic-gate * (a) We have a partition-bound cyclic, and there is no CPU in 16057c478bd9Sstevel@tonic-gate * our partition which is CPU_ENABLE'd. If we saw another 16067c478bd9Sstevel@tonic-gate * non-CYS_OFFLINE CPU in our partition, we'll go with it. 16077c478bd9Sstevel@tonic-gate * If not, the avoid CPU must be the only non-CYS_OFFLINE 16087c478bd9Sstevel@tonic-gate * CPU in the partition; we're forced to return NULL. 16097c478bd9Sstevel@tonic-gate * 16107c478bd9Sstevel@tonic-gate * (b) We have a partition-unbound cyclic, in which case there 16117c478bd9Sstevel@tonic-gate * must only be one CPU CPU_ENABLE'd, and it must be the one 16127c478bd9Sstevel@tonic-gate * we're trying to avoid. If cyclic_juggle()/cyclic_offline() 16137c478bd9Sstevel@tonic-gate * are called appropriately, this generally shouldn't happen 16147c478bd9Sstevel@tonic-gate * (the offline should fail before getting to this code). 16157c478bd9Sstevel@tonic-gate * At any rate: we can't avoid the avoid CPU, so we return 16167c478bd9Sstevel@tonic-gate * NULL. 16177c478bd9Sstevel@tonic-gate */ 16187c478bd9Sstevel@tonic-gate if (!(flags & CYF_PART_BOUND)) { 16197c478bd9Sstevel@tonic-gate ASSERT(avoid->cpu_flags & CPU_ENABLE); 16207c478bd9Sstevel@tonic-gate return (NULL); 16217c478bd9Sstevel@tonic-gate } 16227c478bd9Sstevel@tonic-gate 16237c478bd9Sstevel@tonic-gate CYC_PTRACE("pick-no-intr", part, avoid); 16247c478bd9Sstevel@tonic-gate 16257c478bd9Sstevel@tonic-gate if ((c = online) != NULL) 16267c478bd9Sstevel@tonic-gate goto found; 16277c478bd9Sstevel@tonic-gate 16287c478bd9Sstevel@tonic-gate CYC_PTRACE("pick-fail", part, avoid); 16297c478bd9Sstevel@tonic-gate ASSERT(avoid->cpu_part == start->cpu_part); 16307c478bd9Sstevel@tonic-gate return (NULL); 16317c478bd9Sstevel@tonic-gate 16327c478bd9Sstevel@tonic-gate found: 16337c478bd9Sstevel@tonic-gate CYC_PTRACE("pick-cpu-found", c, avoid); 16347c478bd9Sstevel@tonic-gate ASSERT(c != avoid); 16357c478bd9Sstevel@tonic-gate ASSERT(c->cpu_cyclic != NULL); 16367c478bd9Sstevel@tonic-gate 16377c478bd9Sstevel@tonic-gate return (c->cpu_cyclic); 16387c478bd9Sstevel@tonic-gate } 16397c478bd9Sstevel@tonic-gate 16407c478bd9Sstevel@tonic-gate static void 16417c478bd9Sstevel@tonic-gate cyclic_add_xcall(cyc_xcallarg_t *arg) 16427c478bd9Sstevel@tonic-gate { 16437c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = arg->cyx_cpu; 16447c478bd9Sstevel@tonic-gate cyc_handler_t *hdlr = arg->cyx_hdlr; 16457c478bd9Sstevel@tonic-gate cyc_time_t *when = arg->cyx_when; 16467c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 16477c478bd9Sstevel@tonic-gate cyc_index_t ndx, nelems; 16487c478bd9Sstevel@tonic-gate cyc_cookie_t cookie; 16497c478bd9Sstevel@tonic-gate cyb_arg_t bar = be->cyb_arg; 16507c478bd9Sstevel@tonic-gate cyclic_t *cyclic; 16517c478bd9Sstevel@tonic-gate 16527c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_nelems < cpu->cyp_size); 16537c478bd9Sstevel@tonic-gate 16547c478bd9Sstevel@tonic-gate cookie = be->cyb_set_level(bar, CY_HIGH_LEVEL); 16557c478bd9Sstevel@tonic-gate 16567c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, 16577c478bd9Sstevel@tonic-gate "add-xcall", when->cyt_when, when->cyt_interval); 16587c478bd9Sstevel@tonic-gate 16597c478bd9Sstevel@tonic-gate nelems = cpu->cyp_nelems++; 16607c478bd9Sstevel@tonic-gate 16617c478bd9Sstevel@tonic-gate if (nelems == 0) { 16627c478bd9Sstevel@tonic-gate /* 16637c478bd9Sstevel@tonic-gate * If this is the first element, we need to enable the 16647c478bd9Sstevel@tonic-gate * backend on this CPU. 16657c478bd9Sstevel@tonic-gate */ 16667c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, CY_HIGH_LEVEL, "enabled"); 16677c478bd9Sstevel@tonic-gate be->cyb_enable(bar); 16687c478bd9Sstevel@tonic-gate } 16697c478bd9Sstevel@tonic-gate 16707c478bd9Sstevel@tonic-gate ndx = cpu->cyp_heap[nelems]; 16717c478bd9Sstevel@tonic-gate cyclic = &cpu->cyp_cyclics[ndx]; 16727c478bd9Sstevel@tonic-gate 16737c478bd9Sstevel@tonic-gate ASSERT(cyclic->cy_flags == CYF_FREE); 16747c478bd9Sstevel@tonic-gate cyclic->cy_interval = when->cyt_interval; 16757c478bd9Sstevel@tonic-gate 16767c478bd9Sstevel@tonic-gate if (when->cyt_when == 0) { 16777c478bd9Sstevel@tonic-gate /* 16787c478bd9Sstevel@tonic-gate * If a start time hasn't been explicitly specified, we'll 16797c478bd9Sstevel@tonic-gate * start on the next interval boundary. 16807c478bd9Sstevel@tonic-gate */ 16817c478bd9Sstevel@tonic-gate cyclic->cy_expire = (gethrtime() / cyclic->cy_interval + 1) * 16827c478bd9Sstevel@tonic-gate cyclic->cy_interval; 16837c478bd9Sstevel@tonic-gate } else { 16847c478bd9Sstevel@tonic-gate cyclic->cy_expire = when->cyt_when; 16857c478bd9Sstevel@tonic-gate } 16867c478bd9Sstevel@tonic-gate 16877c478bd9Sstevel@tonic-gate cyclic->cy_handler = hdlr->cyh_func; 16887c478bd9Sstevel@tonic-gate cyclic->cy_arg = hdlr->cyh_arg; 16897c478bd9Sstevel@tonic-gate cyclic->cy_level = hdlr->cyh_level; 16907c478bd9Sstevel@tonic-gate cyclic->cy_flags = arg->cyx_flags; 16917c478bd9Sstevel@tonic-gate 16927c478bd9Sstevel@tonic-gate if (cyclic_upheap(cpu, nelems)) { 16937c478bd9Sstevel@tonic-gate hrtime_t exp = cyclic->cy_expire; 16947c478bd9Sstevel@tonic-gate 16957c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "add-reprog", cyclic, exp); 16967c478bd9Sstevel@tonic-gate 16977c478bd9Sstevel@tonic-gate /* 16987c478bd9Sstevel@tonic-gate * If our upheap propagated to the root, we need to 16997c478bd9Sstevel@tonic-gate * reprogram the interrupt source. 17007c478bd9Sstevel@tonic-gate */ 17017c478bd9Sstevel@tonic-gate be->cyb_reprogram(bar, exp); 17027c478bd9Sstevel@tonic-gate } 17037c478bd9Sstevel@tonic-gate be->cyb_restore_level(bar, cookie); 17047c478bd9Sstevel@tonic-gate 17057c478bd9Sstevel@tonic-gate arg->cyx_ndx = ndx; 17067c478bd9Sstevel@tonic-gate } 17077c478bd9Sstevel@tonic-gate 17087c478bd9Sstevel@tonic-gate static cyc_index_t 17097c478bd9Sstevel@tonic-gate cyclic_add_here(cyc_cpu_t *cpu, cyc_handler_t *hdlr, 17107c478bd9Sstevel@tonic-gate cyc_time_t *when, uint16_t flags) 17117c478bd9Sstevel@tonic-gate { 17127c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 17137c478bd9Sstevel@tonic-gate cyb_arg_t bar = be->cyb_arg; 17147c478bd9Sstevel@tonic-gate cyc_xcallarg_t arg; 17157c478bd9Sstevel@tonic-gate 17167c478bd9Sstevel@tonic-gate CYC_PTRACE("add-cpu", cpu, hdlr->cyh_func); 17177c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 17187c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 17197c478bd9Sstevel@tonic-gate ASSERT(!(cpu->cyp_cpu->cpu_flags & CPU_OFFLINE)); 17207c478bd9Sstevel@tonic-gate ASSERT(when->cyt_when >= 0 && when->cyt_interval > 0); 17217c478bd9Sstevel@tonic-gate 17227c478bd9Sstevel@tonic-gate if (cpu->cyp_nelems == cpu->cyp_size) { 17237c478bd9Sstevel@tonic-gate /* 17247c478bd9Sstevel@tonic-gate * This is expensive; it will cross call onto the other 17257c478bd9Sstevel@tonic-gate * CPU to perform the expansion. 17267c478bd9Sstevel@tonic-gate */ 17277c478bd9Sstevel@tonic-gate cyclic_expand(cpu); 17287c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_nelems < cpu->cyp_size); 17297c478bd9Sstevel@tonic-gate } 17307c478bd9Sstevel@tonic-gate 17317c478bd9Sstevel@tonic-gate /* 17327c478bd9Sstevel@tonic-gate * By now, we know that we're going to be able to successfully 17337c478bd9Sstevel@tonic-gate * perform the add. Now cross call over to the CPU of interest to 17347c478bd9Sstevel@tonic-gate * actually add our cyclic. 17357c478bd9Sstevel@tonic-gate */ 17367c478bd9Sstevel@tonic-gate arg.cyx_cpu = cpu; 17377c478bd9Sstevel@tonic-gate arg.cyx_hdlr = hdlr; 17387c478bd9Sstevel@tonic-gate arg.cyx_when = when; 17397c478bd9Sstevel@tonic-gate arg.cyx_flags = flags; 17407c478bd9Sstevel@tonic-gate 17417c478bd9Sstevel@tonic-gate be->cyb_xcall(bar, cpu->cyp_cpu, (cyc_func_t)cyclic_add_xcall, &arg); 17427c478bd9Sstevel@tonic-gate 17437c478bd9Sstevel@tonic-gate CYC_PTRACE("add-cpu-done", cpu, arg.cyx_ndx); 17447c478bd9Sstevel@tonic-gate 17457c478bd9Sstevel@tonic-gate return (arg.cyx_ndx); 17467c478bd9Sstevel@tonic-gate } 17477c478bd9Sstevel@tonic-gate 17487c478bd9Sstevel@tonic-gate static void 17497c478bd9Sstevel@tonic-gate cyclic_remove_xcall(cyc_xcallarg_t *arg) 17507c478bd9Sstevel@tonic-gate { 17517c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = arg->cyx_cpu; 17527c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 17537c478bd9Sstevel@tonic-gate cyb_arg_t bar = be->cyb_arg; 17547c478bd9Sstevel@tonic-gate cyc_cookie_t cookie; 1755*87a18d3fSMadhavan Venkataraman cyc_index_t ndx = arg->cyx_ndx, nelems, i; 1756*87a18d3fSMadhavan Venkataraman cyc_index_t *heap, last; 17577c478bd9Sstevel@tonic-gate cyclic_t *cyclic; 17587c478bd9Sstevel@tonic-gate #ifdef DEBUG 17597c478bd9Sstevel@tonic-gate cyc_index_t root; 17607c478bd9Sstevel@tonic-gate #endif 17617c478bd9Sstevel@tonic-gate 17627c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_REMOVING); 17637c478bd9Sstevel@tonic-gate 17647c478bd9Sstevel@tonic-gate cookie = be->cyb_set_level(bar, CY_HIGH_LEVEL); 17657c478bd9Sstevel@tonic-gate 17667c478bd9Sstevel@tonic-gate CYC_TRACE1(cpu, CY_HIGH_LEVEL, "remove-xcall", ndx); 17677c478bd9Sstevel@tonic-gate 1768*87a18d3fSMadhavan Venkataraman heap = cpu->cyp_heap; 1769*87a18d3fSMadhavan Venkataraman nelems = cpu->cyp_nelems; 1770*87a18d3fSMadhavan Venkataraman ASSERT(nelems > 0); 17717c478bd9Sstevel@tonic-gate cyclic = &cpu->cyp_cyclics[ndx]; 17727c478bd9Sstevel@tonic-gate 17737c478bd9Sstevel@tonic-gate /* 17747c478bd9Sstevel@tonic-gate * Grab the current expiration time. If this cyclic is being 17757c478bd9Sstevel@tonic-gate * removed as part of a juggling operation, the expiration time 17767c478bd9Sstevel@tonic-gate * will be used when the cyclic is added to the new CPU. 17777c478bd9Sstevel@tonic-gate */ 17787c478bd9Sstevel@tonic-gate if (arg->cyx_when != NULL) { 17797c478bd9Sstevel@tonic-gate arg->cyx_when->cyt_when = cyclic->cy_expire; 17807c478bd9Sstevel@tonic-gate arg->cyx_when->cyt_interval = cyclic->cy_interval; 17817c478bd9Sstevel@tonic-gate } 17827c478bd9Sstevel@tonic-gate 17837c478bd9Sstevel@tonic-gate if (cyclic->cy_pend != 0) { 17847c478bd9Sstevel@tonic-gate /* 17857c478bd9Sstevel@tonic-gate * The pend is non-zero; this cyclic is currently being 17867c478bd9Sstevel@tonic-gate * executed (or will be executed shortly). If the caller 17877c478bd9Sstevel@tonic-gate * refuses to wait, we must return (doing nothing). Otherwise, 17887c478bd9Sstevel@tonic-gate * we will stash the pend value * in this CPU's rpend, and 17897c478bd9Sstevel@tonic-gate * then zero it out. The softint in the pend loop will see 17907c478bd9Sstevel@tonic-gate * that we have zeroed out pend, and will call the cyclic 17917c478bd9Sstevel@tonic-gate * handler rpend times. The caller will wait until the 17927c478bd9Sstevel@tonic-gate * softint has completed calling the cyclic handler. 17937c478bd9Sstevel@tonic-gate */ 17947c478bd9Sstevel@tonic-gate if (arg->cyx_wait == CY_NOWAIT) { 17957c478bd9Sstevel@tonic-gate arg->cyx_wait = CY_WAIT; 17967c478bd9Sstevel@tonic-gate goto out; 17977c478bd9Sstevel@tonic-gate } 17987c478bd9Sstevel@tonic-gate 17997c478bd9Sstevel@tonic-gate ASSERT(cyclic->cy_level != CY_HIGH_LEVEL); 18007c478bd9Sstevel@tonic-gate CYC_TRACE1(cpu, CY_HIGH_LEVEL, "remove-pend", cyclic->cy_pend); 18017c478bd9Sstevel@tonic-gate cpu->cyp_rpend = cyclic->cy_pend; 18027c478bd9Sstevel@tonic-gate cyclic->cy_pend = 0; 18037c478bd9Sstevel@tonic-gate } 18047c478bd9Sstevel@tonic-gate 18057c478bd9Sstevel@tonic-gate /* 18067c478bd9Sstevel@tonic-gate * Now set the flags to CYF_FREE. We don't need a membar_enter() 18077c478bd9Sstevel@tonic-gate * between zeroing pend and setting the flags because we're at 18087c478bd9Sstevel@tonic-gate * CY_HIGH_LEVEL (that is, the zeroing of pend and the setting 18097c478bd9Sstevel@tonic-gate * of cy_flags appear atomic to softints). 18107c478bd9Sstevel@tonic-gate */ 18117c478bd9Sstevel@tonic-gate cyclic->cy_flags = CYF_FREE; 18127c478bd9Sstevel@tonic-gate 18137c478bd9Sstevel@tonic-gate for (i = 0; i < nelems; i++) { 18147c478bd9Sstevel@tonic-gate if (heap[i] == ndx) 18157c478bd9Sstevel@tonic-gate break; 18167c478bd9Sstevel@tonic-gate } 18177c478bd9Sstevel@tonic-gate 18187c478bd9Sstevel@tonic-gate if (i == nelems) 18197c478bd9Sstevel@tonic-gate panic("attempt to remove non-existent cyclic"); 18207c478bd9Sstevel@tonic-gate 18217c478bd9Sstevel@tonic-gate cpu->cyp_nelems = --nelems; 18227c478bd9Sstevel@tonic-gate 18237c478bd9Sstevel@tonic-gate if (nelems == 0) { 18247c478bd9Sstevel@tonic-gate /* 18257c478bd9Sstevel@tonic-gate * If we just removed the last element, then we need to 18267c478bd9Sstevel@tonic-gate * disable the backend on this CPU. 18277c478bd9Sstevel@tonic-gate */ 18287c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, CY_HIGH_LEVEL, "disabled"); 18297c478bd9Sstevel@tonic-gate be->cyb_disable(bar); 18307c478bd9Sstevel@tonic-gate } 18317c478bd9Sstevel@tonic-gate 18327c478bd9Sstevel@tonic-gate if (i == nelems) { 18337c478bd9Sstevel@tonic-gate /* 18347c478bd9Sstevel@tonic-gate * If we just removed the last element of the heap, then 18357c478bd9Sstevel@tonic-gate * we don't have to downheap. 18367c478bd9Sstevel@tonic-gate */ 18377c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, CY_HIGH_LEVEL, "remove-bottom"); 18387c478bd9Sstevel@tonic-gate goto out; 18397c478bd9Sstevel@tonic-gate } 18407c478bd9Sstevel@tonic-gate 18417c478bd9Sstevel@tonic-gate #ifdef DEBUG 18427c478bd9Sstevel@tonic-gate root = heap[0]; 18437c478bd9Sstevel@tonic-gate #endif 18447c478bd9Sstevel@tonic-gate 18457c478bd9Sstevel@tonic-gate /* 18467c478bd9Sstevel@tonic-gate * Swap the last element of the heap with the one we want to 18477c478bd9Sstevel@tonic-gate * remove, and downheap (this has the implicit effect of putting 18487c478bd9Sstevel@tonic-gate * the newly freed element on the free list). 18497c478bd9Sstevel@tonic-gate */ 18507c478bd9Sstevel@tonic-gate heap[i] = (last = heap[nelems]); 18517c478bd9Sstevel@tonic-gate heap[nelems] = ndx; 18527c478bd9Sstevel@tonic-gate 18537c478bd9Sstevel@tonic-gate if (i == 0) { 18547c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, CY_HIGH_LEVEL, "remove-root"); 18557c478bd9Sstevel@tonic-gate cyclic_downheap(cpu, 0); 18567c478bd9Sstevel@tonic-gate } else { 18577c478bd9Sstevel@tonic-gate if (cyclic_upheap(cpu, i) == 0) { 18587c478bd9Sstevel@tonic-gate /* 18597c478bd9Sstevel@tonic-gate * The upheap didn't propagate to the root; if it 18607c478bd9Sstevel@tonic-gate * didn't propagate at all, we need to downheap. 18617c478bd9Sstevel@tonic-gate */ 18627c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, CY_HIGH_LEVEL, "remove-no-root"); 18637c478bd9Sstevel@tonic-gate if (heap[i] == last) { 18647c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, CY_HIGH_LEVEL, "remove-no-up"); 18657c478bd9Sstevel@tonic-gate cyclic_downheap(cpu, i); 18667c478bd9Sstevel@tonic-gate } 18677c478bd9Sstevel@tonic-gate ASSERT(heap[0] == root); 18687c478bd9Sstevel@tonic-gate goto out; 18697c478bd9Sstevel@tonic-gate } 18707c478bd9Sstevel@tonic-gate } 18717c478bd9Sstevel@tonic-gate 18727c478bd9Sstevel@tonic-gate /* 18737c478bd9Sstevel@tonic-gate * We're here because we changed the root; we need to reprogram 18747c478bd9Sstevel@tonic-gate * the clock source. 18757c478bd9Sstevel@tonic-gate */ 18767c478bd9Sstevel@tonic-gate cyclic = &cpu->cyp_cyclics[heap[0]]; 18777c478bd9Sstevel@tonic-gate 18787c478bd9Sstevel@tonic-gate CYC_TRACE0(cpu, CY_HIGH_LEVEL, "remove-reprog"); 18797c478bd9Sstevel@tonic-gate 18807c478bd9Sstevel@tonic-gate ASSERT(nelems != 0); 18817c478bd9Sstevel@tonic-gate be->cyb_reprogram(bar, cyclic->cy_expire); 18827c478bd9Sstevel@tonic-gate out: 18837c478bd9Sstevel@tonic-gate be->cyb_restore_level(bar, cookie); 18847c478bd9Sstevel@tonic-gate } 18857c478bd9Sstevel@tonic-gate 18867c478bd9Sstevel@tonic-gate static int 18877c478bd9Sstevel@tonic-gate cyclic_remove_here(cyc_cpu_t *cpu, cyc_index_t ndx, cyc_time_t *when, int wait) 18887c478bd9Sstevel@tonic-gate { 18897c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 18907c478bd9Sstevel@tonic-gate cyc_xcallarg_t arg; 18917c478bd9Sstevel@tonic-gate cyclic_t *cyclic = &cpu->cyp_cyclics[ndx]; 18927c478bd9Sstevel@tonic-gate cyc_level_t level = cyclic->cy_level; 18937c478bd9Sstevel@tonic-gate 18947c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 18957c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_rpend == 0); 18967c478bd9Sstevel@tonic-gate ASSERT(wait == CY_WAIT || wait == CY_NOWAIT); 18977c478bd9Sstevel@tonic-gate 18987c478bd9Sstevel@tonic-gate arg.cyx_ndx = ndx; 18997c478bd9Sstevel@tonic-gate arg.cyx_cpu = cpu; 19007c478bd9Sstevel@tonic-gate arg.cyx_when = when; 19017c478bd9Sstevel@tonic-gate arg.cyx_wait = wait; 19027c478bd9Sstevel@tonic-gate 19037c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 19047c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_REMOVING; 19057c478bd9Sstevel@tonic-gate 19067c478bd9Sstevel@tonic-gate be->cyb_xcall(be->cyb_arg, cpu->cyp_cpu, 19077c478bd9Sstevel@tonic-gate (cyc_func_t)cyclic_remove_xcall, &arg); 19087c478bd9Sstevel@tonic-gate 19097c478bd9Sstevel@tonic-gate /* 19107c478bd9Sstevel@tonic-gate * If the cyclic we removed wasn't at CY_HIGH_LEVEL, then we need to 19117c478bd9Sstevel@tonic-gate * check the cyp_rpend. If it's non-zero, then we need to wait here 19127c478bd9Sstevel@tonic-gate * for all pending cyclic handlers to run. 19137c478bd9Sstevel@tonic-gate */ 19147c478bd9Sstevel@tonic-gate ASSERT(!(level == CY_HIGH_LEVEL && cpu->cyp_rpend != 0)); 19157c478bd9Sstevel@tonic-gate ASSERT(!(wait == CY_NOWAIT && cpu->cyp_rpend != 0)); 19167c478bd9Sstevel@tonic-gate ASSERT(!(arg.cyx_wait == CY_NOWAIT && cpu->cyp_rpend != 0)); 19177c478bd9Sstevel@tonic-gate 19187c478bd9Sstevel@tonic-gate if (wait != arg.cyx_wait) { 19197c478bd9Sstevel@tonic-gate /* 19207c478bd9Sstevel@tonic-gate * We are being told that we must wait if we want to 19217c478bd9Sstevel@tonic-gate * remove this cyclic; put the CPU back in the CYS_ONLINE 19227c478bd9Sstevel@tonic-gate * state and return failure. 19237c478bd9Sstevel@tonic-gate */ 19247c478bd9Sstevel@tonic-gate ASSERT(wait == CY_NOWAIT && arg.cyx_wait == CY_WAIT); 19257c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_REMOVING); 19267c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_ONLINE; 19277c478bd9Sstevel@tonic-gate 19287c478bd9Sstevel@tonic-gate return (0); 19297c478bd9Sstevel@tonic-gate } 19307c478bd9Sstevel@tonic-gate 19317c478bd9Sstevel@tonic-gate if (cpu->cyp_rpend != 0) 19327c478bd9Sstevel@tonic-gate sema_p(&cpu->cyp_modify_wait); 19337c478bd9Sstevel@tonic-gate 19347c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_REMOVING); 19357c478bd9Sstevel@tonic-gate 19367c478bd9Sstevel@tonic-gate cpu->cyp_rpend = 0; 19377c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_ONLINE; 19387c478bd9Sstevel@tonic-gate 19397c478bd9Sstevel@tonic-gate return (1); 19407c478bd9Sstevel@tonic-gate } 19417c478bd9Sstevel@tonic-gate 19427c478bd9Sstevel@tonic-gate /* 1943*87a18d3fSMadhavan Venkataraman * If cyclic_reprogram() is called on the same CPU as the cyclic's CPU, then 1944*87a18d3fSMadhavan Venkataraman * it calls this function directly. Else, it invokes this function through 1945*87a18d3fSMadhavan Venkataraman * an X-call to the cyclic's CPU. 1946*87a18d3fSMadhavan Venkataraman */ 1947*87a18d3fSMadhavan Venkataraman static void 1948*87a18d3fSMadhavan Venkataraman cyclic_reprogram_cyclic(cyc_cpu_t *cpu, cyc_index_t ndx, hrtime_t expire) 1949*87a18d3fSMadhavan Venkataraman { 1950*87a18d3fSMadhavan Venkataraman cyc_backend_t *be = cpu->cyp_backend; 1951*87a18d3fSMadhavan Venkataraman cyb_arg_t bar = be->cyb_arg; 1952*87a18d3fSMadhavan Venkataraman cyc_cookie_t cookie; 1953*87a18d3fSMadhavan Venkataraman cyc_index_t nelems, i; 1954*87a18d3fSMadhavan Venkataraman cyc_index_t *heap; 1955*87a18d3fSMadhavan Venkataraman cyclic_t *cyclic; 1956*87a18d3fSMadhavan Venkataraman hrtime_t oexpire; 1957*87a18d3fSMadhavan Venkataraman int reprog; 1958*87a18d3fSMadhavan Venkataraman 1959*87a18d3fSMadhavan Venkataraman cookie = be->cyb_set_level(bar, CY_HIGH_LEVEL); 1960*87a18d3fSMadhavan Venkataraman 1961*87a18d3fSMadhavan Venkataraman CYC_TRACE1(cpu, CY_HIGH_LEVEL, "reprog-xcall", ndx); 1962*87a18d3fSMadhavan Venkataraman 1963*87a18d3fSMadhavan Venkataraman nelems = cpu->cyp_nelems; 1964*87a18d3fSMadhavan Venkataraman ASSERT(nelems > 0); 1965*87a18d3fSMadhavan Venkataraman heap = cpu->cyp_heap; 1966*87a18d3fSMadhavan Venkataraman 1967*87a18d3fSMadhavan Venkataraman /* 1968*87a18d3fSMadhavan Venkataraman * Reprogrammed cyclics are typically one-shot ones that get 1969*87a18d3fSMadhavan Venkataraman * set to infinity on every expiration. We shorten the search by 1970*87a18d3fSMadhavan Venkataraman * searching from the bottom of the heap to the top instead of the 1971*87a18d3fSMadhavan Venkataraman * other way around. 1972*87a18d3fSMadhavan Venkataraman */ 1973*87a18d3fSMadhavan Venkataraman for (i = nelems - 1; i >= 0; i--) { 1974*87a18d3fSMadhavan Venkataraman if (heap[i] == ndx) 1975*87a18d3fSMadhavan Venkataraman break; 1976*87a18d3fSMadhavan Venkataraman } 1977*87a18d3fSMadhavan Venkataraman if (i < 0) 1978*87a18d3fSMadhavan Venkataraman panic("attempt to reprogram non-existent cyclic"); 1979*87a18d3fSMadhavan Venkataraman 1980*87a18d3fSMadhavan Venkataraman cyclic = &cpu->cyp_cyclics[ndx]; 1981*87a18d3fSMadhavan Venkataraman oexpire = cyclic->cy_expire; 1982*87a18d3fSMadhavan Venkataraman cyclic->cy_expire = expire; 1983*87a18d3fSMadhavan Venkataraman 1984*87a18d3fSMadhavan Venkataraman reprog = (i == 0); 1985*87a18d3fSMadhavan Venkataraman if (expire > oexpire) { 1986*87a18d3fSMadhavan Venkataraman CYC_TRACE1(cpu, CY_HIGH_LEVEL, "reprog-down", i); 1987*87a18d3fSMadhavan Venkataraman cyclic_downheap(cpu, i); 1988*87a18d3fSMadhavan Venkataraman } else if (i > 0) { 1989*87a18d3fSMadhavan Venkataraman CYC_TRACE1(cpu, CY_HIGH_LEVEL, "reprog-up", i); 1990*87a18d3fSMadhavan Venkataraman reprog = cyclic_upheap(cpu, i); 1991*87a18d3fSMadhavan Venkataraman } 1992*87a18d3fSMadhavan Venkataraman 1993*87a18d3fSMadhavan Venkataraman if (reprog && (cpu->cyp_state != CYS_SUSPENDED)) { 1994*87a18d3fSMadhavan Venkataraman /* 1995*87a18d3fSMadhavan Venkataraman * The root changed. Reprogram the clock source. 1996*87a18d3fSMadhavan Venkataraman */ 1997*87a18d3fSMadhavan Venkataraman CYC_TRACE0(cpu, CY_HIGH_LEVEL, "reprog-root"); 1998*87a18d3fSMadhavan Venkataraman cyclic = &cpu->cyp_cyclics[heap[0]]; 1999*87a18d3fSMadhavan Venkataraman be->cyb_reprogram(bar, cyclic->cy_expire); 2000*87a18d3fSMadhavan Venkataraman } 2001*87a18d3fSMadhavan Venkataraman 2002*87a18d3fSMadhavan Venkataraman be->cyb_restore_level(bar, cookie); 2003*87a18d3fSMadhavan Venkataraman } 2004*87a18d3fSMadhavan Venkataraman 2005*87a18d3fSMadhavan Venkataraman static void 2006*87a18d3fSMadhavan Venkataraman cyclic_reprogram_xcall(cyc_xcallarg_t *arg) 2007*87a18d3fSMadhavan Venkataraman { 2008*87a18d3fSMadhavan Venkataraman cyclic_reprogram_cyclic(arg->cyx_cpu, arg->cyx_ndx, 2009*87a18d3fSMadhavan Venkataraman arg->cyx_when->cyt_when); 2010*87a18d3fSMadhavan Venkataraman } 2011*87a18d3fSMadhavan Venkataraman 2012*87a18d3fSMadhavan Venkataraman static void 2013*87a18d3fSMadhavan Venkataraman cyclic_reprogram_here(cyc_cpu_t *cpu, cyc_index_t ndx, hrtime_t expiration) 2014*87a18d3fSMadhavan Venkataraman { 2015*87a18d3fSMadhavan Venkataraman cyc_backend_t *be = cpu->cyp_backend; 2016*87a18d3fSMadhavan Venkataraman cyc_xcallarg_t arg; 2017*87a18d3fSMadhavan Venkataraman cyc_time_t when; 2018*87a18d3fSMadhavan Venkataraman 2019*87a18d3fSMadhavan Venkataraman ASSERT(expiration > 0); 2020*87a18d3fSMadhavan Venkataraman 2021*87a18d3fSMadhavan Venkataraman arg.cyx_ndx = ndx; 2022*87a18d3fSMadhavan Venkataraman arg.cyx_cpu = cpu; 2023*87a18d3fSMadhavan Venkataraman arg.cyx_when = &when; 2024*87a18d3fSMadhavan Venkataraman when.cyt_when = expiration; 2025*87a18d3fSMadhavan Venkataraman 2026*87a18d3fSMadhavan Venkataraman be->cyb_xcall(be->cyb_arg, cpu->cyp_cpu, 2027*87a18d3fSMadhavan Venkataraman (cyc_func_t)cyclic_reprogram_xcall, &arg); 2028*87a18d3fSMadhavan Venkataraman } 2029*87a18d3fSMadhavan Venkataraman 2030*87a18d3fSMadhavan Venkataraman /* 20317c478bd9Sstevel@tonic-gate * cyclic_juggle_one_to() should only be called when the source cyclic 20327c478bd9Sstevel@tonic-gate * can be juggled and the destination CPU is known to be able to accept 20337c478bd9Sstevel@tonic-gate * it. 20347c478bd9Sstevel@tonic-gate */ 20357c478bd9Sstevel@tonic-gate static void 20367c478bd9Sstevel@tonic-gate cyclic_juggle_one_to(cyc_id_t *idp, cyc_cpu_t *dest) 20377c478bd9Sstevel@tonic-gate { 20387c478bd9Sstevel@tonic-gate cyc_cpu_t *src = idp->cyi_cpu; 20397c478bd9Sstevel@tonic-gate cyc_index_t ndx = idp->cyi_ndx; 20407c478bd9Sstevel@tonic-gate cyc_time_t when; 20417c478bd9Sstevel@tonic-gate cyc_handler_t hdlr; 20427c478bd9Sstevel@tonic-gate cyclic_t *cyclic; 20437c478bd9Sstevel@tonic-gate uint16_t flags; 20447c478bd9Sstevel@tonic-gate hrtime_t delay; 20457c478bd9Sstevel@tonic-gate 20467c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 20477c478bd9Sstevel@tonic-gate ASSERT(src != NULL && idp->cyi_omni_list == NULL); 20487c478bd9Sstevel@tonic-gate ASSERT(!(dest->cyp_cpu->cpu_flags & (CPU_QUIESCED | CPU_OFFLINE))); 20497c478bd9Sstevel@tonic-gate CYC_PTRACE("juggle-one-to", idp, dest); 20507c478bd9Sstevel@tonic-gate 20517c478bd9Sstevel@tonic-gate cyclic = &src->cyp_cyclics[ndx]; 20527c478bd9Sstevel@tonic-gate 20537c478bd9Sstevel@tonic-gate flags = cyclic->cy_flags; 20547c478bd9Sstevel@tonic-gate ASSERT(!(flags & CYF_CPU_BOUND) && !(flags & CYF_FREE)); 20557c478bd9Sstevel@tonic-gate 20567c478bd9Sstevel@tonic-gate hdlr.cyh_func = cyclic->cy_handler; 20577c478bd9Sstevel@tonic-gate hdlr.cyh_level = cyclic->cy_level; 20587c478bd9Sstevel@tonic-gate hdlr.cyh_arg = cyclic->cy_arg; 20597c478bd9Sstevel@tonic-gate 20607c478bd9Sstevel@tonic-gate /* 20617c478bd9Sstevel@tonic-gate * Before we begin the juggling process, see if the destination 20627c478bd9Sstevel@tonic-gate * CPU requires an expansion. If it does, we'll perform the 20637c478bd9Sstevel@tonic-gate * expansion before removing the cyclic. This is to prevent us 20647c478bd9Sstevel@tonic-gate * from blocking while a system-critical cyclic (notably, the clock 20657c478bd9Sstevel@tonic-gate * cyclic) isn't on a CPU. 20667c478bd9Sstevel@tonic-gate */ 20677c478bd9Sstevel@tonic-gate if (dest->cyp_nelems == dest->cyp_size) { 20687c478bd9Sstevel@tonic-gate CYC_PTRACE("remove-expand", idp, dest); 20697c478bd9Sstevel@tonic-gate cyclic_expand(dest); 20707c478bd9Sstevel@tonic-gate ASSERT(dest->cyp_nelems < dest->cyp_size); 20717c478bd9Sstevel@tonic-gate } 20727c478bd9Sstevel@tonic-gate 20737c478bd9Sstevel@tonic-gate /* 2074*87a18d3fSMadhavan Venkataraman * Prevent a reprogram of this cyclic while we are relocating it. 2075*87a18d3fSMadhavan Venkataraman * Otherwise, cyclic_reprogram_here() will end up sending an X-call 2076*87a18d3fSMadhavan Venkataraman * to the wrong CPU. 2077*87a18d3fSMadhavan Venkataraman */ 2078*87a18d3fSMadhavan Venkataraman rw_enter(&idp->cyi_lock, RW_WRITER); 2079*87a18d3fSMadhavan Venkataraman 2080*87a18d3fSMadhavan Venkataraman /* 20817c478bd9Sstevel@tonic-gate * Remove the cyclic from the source. As mentioned above, we cannot 20827c478bd9Sstevel@tonic-gate * block during this operation; if we cannot remove the cyclic 20837c478bd9Sstevel@tonic-gate * without waiting, we spin for a time shorter than the interval, and 20847c478bd9Sstevel@tonic-gate * reattempt the (non-blocking) removal. If we continue to fail, 20857c478bd9Sstevel@tonic-gate * we will exponentially back off (up to half of the interval). 20867c478bd9Sstevel@tonic-gate * Note that the removal will ultimately succeed -- even if the 20877c478bd9Sstevel@tonic-gate * cyclic handler is blocked on a resource held by a thread which we 20887c478bd9Sstevel@tonic-gate * have preempted, priority inheritance assures that the preempted 20897c478bd9Sstevel@tonic-gate * thread will preempt us and continue to progress. 20907c478bd9Sstevel@tonic-gate */ 20917c478bd9Sstevel@tonic-gate for (delay = NANOSEC / MICROSEC; ; delay <<= 1) { 20927c478bd9Sstevel@tonic-gate /* 20937c478bd9Sstevel@tonic-gate * Before we begin this operation, disable kernel preemption. 20947c478bd9Sstevel@tonic-gate */ 20957c478bd9Sstevel@tonic-gate kpreempt_disable(); 20967c478bd9Sstevel@tonic-gate if (cyclic_remove_here(src, ndx, &when, CY_NOWAIT)) 20977c478bd9Sstevel@tonic-gate break; 20987c478bd9Sstevel@tonic-gate 20997c478bd9Sstevel@tonic-gate /* 21007c478bd9Sstevel@tonic-gate * The operation failed; enable kernel preemption while 21017c478bd9Sstevel@tonic-gate * spinning. 21027c478bd9Sstevel@tonic-gate */ 21037c478bd9Sstevel@tonic-gate kpreempt_enable(); 21047c478bd9Sstevel@tonic-gate 21057c478bd9Sstevel@tonic-gate CYC_PTRACE("remove-retry", idp, src); 21067c478bd9Sstevel@tonic-gate 21077c478bd9Sstevel@tonic-gate if (delay > (cyclic->cy_interval >> 1)) 21087c478bd9Sstevel@tonic-gate delay = cyclic->cy_interval >> 1; 21097c478bd9Sstevel@tonic-gate 2110*87a18d3fSMadhavan Venkataraman /* 2111*87a18d3fSMadhavan Venkataraman * Drop the RW lock to avoid a deadlock with the cyclic 2112*87a18d3fSMadhavan Venkataraman * handler (because it can potentially call cyclic_reprogram(). 2113*87a18d3fSMadhavan Venkataraman */ 2114*87a18d3fSMadhavan Venkataraman rw_exit(&idp->cyi_lock); 21157c478bd9Sstevel@tonic-gate drv_usecwait((clock_t)(delay / (NANOSEC / MICROSEC))); 2116*87a18d3fSMadhavan Venkataraman rw_enter(&idp->cyi_lock, RW_WRITER); 21177c478bd9Sstevel@tonic-gate } 21187c478bd9Sstevel@tonic-gate 21197c478bd9Sstevel@tonic-gate /* 21207c478bd9Sstevel@tonic-gate * Now add the cyclic to the destination. This won't block; we 21217c478bd9Sstevel@tonic-gate * performed any necessary (blocking) expansion of the destination 21227c478bd9Sstevel@tonic-gate * CPU before removing the cyclic from the source CPU. 21237c478bd9Sstevel@tonic-gate */ 21247c478bd9Sstevel@tonic-gate idp->cyi_ndx = cyclic_add_here(dest, &hdlr, &when, flags); 21257c478bd9Sstevel@tonic-gate idp->cyi_cpu = dest; 21267c478bd9Sstevel@tonic-gate kpreempt_enable(); 2127*87a18d3fSMadhavan Venkataraman 2128*87a18d3fSMadhavan Venkataraman /* 2129*87a18d3fSMadhavan Venkataraman * Now that we have successfully relocated the cyclic, allow 2130*87a18d3fSMadhavan Venkataraman * it to be reprogrammed. 2131*87a18d3fSMadhavan Venkataraman */ 2132*87a18d3fSMadhavan Venkataraman rw_exit(&idp->cyi_lock); 21337c478bd9Sstevel@tonic-gate } 21347c478bd9Sstevel@tonic-gate 21357c478bd9Sstevel@tonic-gate static int 21367c478bd9Sstevel@tonic-gate cyclic_juggle_one(cyc_id_t *idp) 21377c478bd9Sstevel@tonic-gate { 21387c478bd9Sstevel@tonic-gate cyc_index_t ndx = idp->cyi_ndx; 21397c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = idp->cyi_cpu, *dest; 21407c478bd9Sstevel@tonic-gate cyclic_t *cyclic = &cpu->cyp_cyclics[ndx]; 21417c478bd9Sstevel@tonic-gate cpu_t *c = cpu->cyp_cpu; 21427c478bd9Sstevel@tonic-gate cpupart_t *part = c->cpu_part; 21437c478bd9Sstevel@tonic-gate 21447c478bd9Sstevel@tonic-gate CYC_PTRACE("juggle-one", idp, cpu); 21457c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 21467c478bd9Sstevel@tonic-gate ASSERT(!(c->cpu_flags & CPU_OFFLINE)); 21477c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 21487c478bd9Sstevel@tonic-gate ASSERT(!(cyclic->cy_flags & CYF_FREE)); 21497c478bd9Sstevel@tonic-gate 21507c478bd9Sstevel@tonic-gate if ((dest = cyclic_pick_cpu(part, c, c, cyclic->cy_flags)) == NULL) { 21517c478bd9Sstevel@tonic-gate /* 21527c478bd9Sstevel@tonic-gate * Bad news: this cyclic can't be juggled. 21537c478bd9Sstevel@tonic-gate */ 21547c478bd9Sstevel@tonic-gate CYC_PTRACE("juggle-fail", idp, cpu) 21557c478bd9Sstevel@tonic-gate return (0); 21567c478bd9Sstevel@tonic-gate } 21577c478bd9Sstevel@tonic-gate 21587c478bd9Sstevel@tonic-gate cyclic_juggle_one_to(idp, dest); 21597c478bd9Sstevel@tonic-gate 21607c478bd9Sstevel@tonic-gate return (1); 21617c478bd9Sstevel@tonic-gate } 21627c478bd9Sstevel@tonic-gate 21637c478bd9Sstevel@tonic-gate static void 21647c478bd9Sstevel@tonic-gate cyclic_unbind_cpu(cyclic_id_t id) 21657c478bd9Sstevel@tonic-gate { 21667c478bd9Sstevel@tonic-gate cyc_id_t *idp = (cyc_id_t *)id; 21677c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = idp->cyi_cpu; 21687c478bd9Sstevel@tonic-gate cpu_t *c = cpu->cyp_cpu; 21697c478bd9Sstevel@tonic-gate cyclic_t *cyclic = &cpu->cyp_cyclics[idp->cyi_ndx]; 21707c478bd9Sstevel@tonic-gate 21717c478bd9Sstevel@tonic-gate CYC_PTRACE("unbind-cpu", id, cpu); 21727c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 21737c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 21747c478bd9Sstevel@tonic-gate ASSERT(!(cyclic->cy_flags & CYF_FREE)); 21757c478bd9Sstevel@tonic-gate ASSERT(cyclic->cy_flags & CYF_CPU_BOUND); 21767c478bd9Sstevel@tonic-gate 21777c478bd9Sstevel@tonic-gate cyclic->cy_flags &= ~CYF_CPU_BOUND; 21787c478bd9Sstevel@tonic-gate 21797c478bd9Sstevel@tonic-gate /* 21807c478bd9Sstevel@tonic-gate * If we were bound to CPU which has interrupts disabled, we need 21817c478bd9Sstevel@tonic-gate * to juggle away. This can only fail if we are bound to a 21827c478bd9Sstevel@tonic-gate * processor set, and if every CPU in the processor set has 21837c478bd9Sstevel@tonic-gate * interrupts disabled. 21847c478bd9Sstevel@tonic-gate */ 21857c478bd9Sstevel@tonic-gate if (!(c->cpu_flags & CPU_ENABLE)) { 21867c478bd9Sstevel@tonic-gate int res = cyclic_juggle_one(idp); 21877c478bd9Sstevel@tonic-gate 21887c478bd9Sstevel@tonic-gate ASSERT((res && idp->cyi_cpu != cpu) || 21897c478bd9Sstevel@tonic-gate (!res && (cyclic->cy_flags & CYF_PART_BOUND))); 21907c478bd9Sstevel@tonic-gate } 21917c478bd9Sstevel@tonic-gate } 21927c478bd9Sstevel@tonic-gate 21937c478bd9Sstevel@tonic-gate static void 21947c478bd9Sstevel@tonic-gate cyclic_bind_cpu(cyclic_id_t id, cpu_t *d) 21957c478bd9Sstevel@tonic-gate { 21967c478bd9Sstevel@tonic-gate cyc_id_t *idp = (cyc_id_t *)id; 21977c478bd9Sstevel@tonic-gate cyc_cpu_t *dest = d->cpu_cyclic, *cpu = idp->cyi_cpu; 21987c478bd9Sstevel@tonic-gate cpu_t *c = cpu->cyp_cpu; 21997c478bd9Sstevel@tonic-gate cyclic_t *cyclic = &cpu->cyp_cyclics[idp->cyi_ndx]; 22007c478bd9Sstevel@tonic-gate cpupart_t *part = c->cpu_part; 22017c478bd9Sstevel@tonic-gate 22027c478bd9Sstevel@tonic-gate CYC_PTRACE("bind-cpu", id, dest); 22037c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 22047c478bd9Sstevel@tonic-gate ASSERT(!(d->cpu_flags & CPU_OFFLINE)); 22057c478bd9Sstevel@tonic-gate ASSERT(!(c->cpu_flags & CPU_OFFLINE)); 22067c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 22077c478bd9Sstevel@tonic-gate ASSERT(dest != NULL); 22087c478bd9Sstevel@tonic-gate ASSERT(dest->cyp_state == CYS_ONLINE); 22097c478bd9Sstevel@tonic-gate ASSERT(!(cyclic->cy_flags & CYF_FREE)); 22107c478bd9Sstevel@tonic-gate ASSERT(!(cyclic->cy_flags & CYF_CPU_BOUND)); 22117c478bd9Sstevel@tonic-gate 22127c478bd9Sstevel@tonic-gate dest = cyclic_pick_cpu(part, d, NULL, cyclic->cy_flags | CYF_CPU_BOUND); 22137c478bd9Sstevel@tonic-gate 22147c478bd9Sstevel@tonic-gate if (dest != cpu) { 22157c478bd9Sstevel@tonic-gate cyclic_juggle_one_to(idp, dest); 22167c478bd9Sstevel@tonic-gate cyclic = &dest->cyp_cyclics[idp->cyi_ndx]; 22177c478bd9Sstevel@tonic-gate } 22187c478bd9Sstevel@tonic-gate 22197c478bd9Sstevel@tonic-gate cyclic->cy_flags |= CYF_CPU_BOUND; 22207c478bd9Sstevel@tonic-gate } 22217c478bd9Sstevel@tonic-gate 22227c478bd9Sstevel@tonic-gate static void 22237c478bd9Sstevel@tonic-gate cyclic_unbind_cpupart(cyclic_id_t id) 22247c478bd9Sstevel@tonic-gate { 22257c478bd9Sstevel@tonic-gate cyc_id_t *idp = (cyc_id_t *)id; 22267c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = idp->cyi_cpu; 22277c478bd9Sstevel@tonic-gate cpu_t *c = cpu->cyp_cpu; 22287c478bd9Sstevel@tonic-gate cyclic_t *cyc = &cpu->cyp_cyclics[idp->cyi_ndx]; 22297c478bd9Sstevel@tonic-gate 22307c478bd9Sstevel@tonic-gate CYC_PTRACE("unbind-part", idp, c->cpu_part); 22317c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 22327c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 22337c478bd9Sstevel@tonic-gate ASSERT(!(cyc->cy_flags & CYF_FREE)); 22347c478bd9Sstevel@tonic-gate ASSERT(cyc->cy_flags & CYF_PART_BOUND); 22357c478bd9Sstevel@tonic-gate 22367c478bd9Sstevel@tonic-gate cyc->cy_flags &= ~CYF_PART_BOUND; 22377c478bd9Sstevel@tonic-gate 22387c478bd9Sstevel@tonic-gate /* 22397c478bd9Sstevel@tonic-gate * If we're on a CPU which has interrupts disabled (and if this cyclic 22407c478bd9Sstevel@tonic-gate * isn't bound to the CPU), we need to juggle away. 22417c478bd9Sstevel@tonic-gate */ 22427c478bd9Sstevel@tonic-gate if (!(c->cpu_flags & CPU_ENABLE) && !(cyc->cy_flags & CYF_CPU_BOUND)) { 22437c478bd9Sstevel@tonic-gate int res = cyclic_juggle_one(idp); 22447c478bd9Sstevel@tonic-gate 22457c478bd9Sstevel@tonic-gate ASSERT(res && idp->cyi_cpu != cpu); 22467c478bd9Sstevel@tonic-gate } 22477c478bd9Sstevel@tonic-gate } 22487c478bd9Sstevel@tonic-gate 22497c478bd9Sstevel@tonic-gate static void 22507c478bd9Sstevel@tonic-gate cyclic_bind_cpupart(cyclic_id_t id, cpupart_t *part) 22517c478bd9Sstevel@tonic-gate { 22527c478bd9Sstevel@tonic-gate cyc_id_t *idp = (cyc_id_t *)id; 22537c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = idp->cyi_cpu, *dest; 22547c478bd9Sstevel@tonic-gate cpu_t *c = cpu->cyp_cpu; 22557c478bd9Sstevel@tonic-gate cyclic_t *cyc = &cpu->cyp_cyclics[idp->cyi_ndx]; 22567c478bd9Sstevel@tonic-gate 22577c478bd9Sstevel@tonic-gate CYC_PTRACE("bind-part", idp, part); 22587c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 22597c478bd9Sstevel@tonic-gate ASSERT(!(c->cpu_flags & CPU_OFFLINE)); 22607c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 22617c478bd9Sstevel@tonic-gate ASSERT(!(cyc->cy_flags & CYF_FREE)); 22627c478bd9Sstevel@tonic-gate ASSERT(!(cyc->cy_flags & CYF_PART_BOUND)); 22637c478bd9Sstevel@tonic-gate ASSERT(part->cp_ncpus > 0); 22647c478bd9Sstevel@tonic-gate 22657c478bd9Sstevel@tonic-gate dest = cyclic_pick_cpu(part, c, NULL, cyc->cy_flags | CYF_PART_BOUND); 22667c478bd9Sstevel@tonic-gate 22677c478bd9Sstevel@tonic-gate if (dest != cpu) { 22687c478bd9Sstevel@tonic-gate cyclic_juggle_one_to(idp, dest); 22697c478bd9Sstevel@tonic-gate cyc = &dest->cyp_cyclics[idp->cyi_ndx]; 22707c478bd9Sstevel@tonic-gate } 22717c478bd9Sstevel@tonic-gate 22727c478bd9Sstevel@tonic-gate cyc->cy_flags |= CYF_PART_BOUND; 22737c478bd9Sstevel@tonic-gate } 22747c478bd9Sstevel@tonic-gate 22757c478bd9Sstevel@tonic-gate static void 22767c478bd9Sstevel@tonic-gate cyclic_configure(cpu_t *c) 22777c478bd9Sstevel@tonic-gate { 22787c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = kmem_zalloc(sizeof (cyc_cpu_t), KM_SLEEP); 22797c478bd9Sstevel@tonic-gate cyc_backend_t *nbe = kmem_zalloc(sizeof (cyc_backend_t), KM_SLEEP); 22807c478bd9Sstevel@tonic-gate int i; 22817c478bd9Sstevel@tonic-gate 22827c478bd9Sstevel@tonic-gate CYC_PTRACE1("configure", cpu); 22837c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 22847c478bd9Sstevel@tonic-gate 22857c478bd9Sstevel@tonic-gate if (cyclic_id_cache == NULL) 22867c478bd9Sstevel@tonic-gate cyclic_id_cache = kmem_cache_create("cyclic_id_cache", 22877c478bd9Sstevel@tonic-gate sizeof (cyc_id_t), 0, NULL, NULL, NULL, NULL, NULL, 0); 22887c478bd9Sstevel@tonic-gate 22897c478bd9Sstevel@tonic-gate cpu->cyp_cpu = c; 22907c478bd9Sstevel@tonic-gate 22917c478bd9Sstevel@tonic-gate sema_init(&cpu->cyp_modify_wait, 0, NULL, SEMA_DEFAULT, NULL); 22927c478bd9Sstevel@tonic-gate 22937c478bd9Sstevel@tonic-gate cpu->cyp_size = 1; 22947c478bd9Sstevel@tonic-gate cpu->cyp_heap = kmem_zalloc(sizeof (cyc_index_t), KM_SLEEP); 22957c478bd9Sstevel@tonic-gate cpu->cyp_cyclics = kmem_zalloc(sizeof (cyclic_t), KM_SLEEP); 22967c478bd9Sstevel@tonic-gate cpu->cyp_cyclics->cy_flags = CYF_FREE; 22977c478bd9Sstevel@tonic-gate 22987c478bd9Sstevel@tonic-gate for (i = CY_LOW_LEVEL; i < CY_LOW_LEVEL + CY_SOFT_LEVELS; i++) { 22997c478bd9Sstevel@tonic-gate /* 23007c478bd9Sstevel@tonic-gate * We don't need to set the sizemask; it's already zero 23017c478bd9Sstevel@tonic-gate * (which is the appropriate sizemask for a size of 1). 23027c478bd9Sstevel@tonic-gate */ 23037c478bd9Sstevel@tonic-gate cpu->cyp_softbuf[i].cys_buf[0].cypc_buf = 23047c478bd9Sstevel@tonic-gate kmem_alloc(sizeof (cyc_index_t), KM_SLEEP); 23057c478bd9Sstevel@tonic-gate } 23067c478bd9Sstevel@tonic-gate 23077c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_OFFLINE; 23087c478bd9Sstevel@tonic-gate 23097c478bd9Sstevel@tonic-gate /* 23107c478bd9Sstevel@tonic-gate * Setup the backend for this CPU. 23117c478bd9Sstevel@tonic-gate */ 23127c478bd9Sstevel@tonic-gate bcopy(&cyclic_backend, nbe, sizeof (cyc_backend_t)); 23137c478bd9Sstevel@tonic-gate nbe->cyb_arg = nbe->cyb_configure(c); 23147c478bd9Sstevel@tonic-gate cpu->cyp_backend = nbe; 23157c478bd9Sstevel@tonic-gate 23167c478bd9Sstevel@tonic-gate /* 23177c478bd9Sstevel@tonic-gate * On platforms where stray interrupts may be taken during startup, 23187c478bd9Sstevel@tonic-gate * the CPU's cpu_cyclic pointer serves as an indicator that the 23197c478bd9Sstevel@tonic-gate * cyclic subsystem for this CPU is prepared to field interrupts. 23207c478bd9Sstevel@tonic-gate */ 23217c478bd9Sstevel@tonic-gate membar_producer(); 23227c478bd9Sstevel@tonic-gate 23237c478bd9Sstevel@tonic-gate c->cpu_cyclic = cpu; 23247c478bd9Sstevel@tonic-gate } 23257c478bd9Sstevel@tonic-gate 23267c478bd9Sstevel@tonic-gate static void 23277c478bd9Sstevel@tonic-gate cyclic_unconfigure(cpu_t *c) 23287c478bd9Sstevel@tonic-gate { 23297c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = c->cpu_cyclic; 23307c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 23317c478bd9Sstevel@tonic-gate cyb_arg_t bar = be->cyb_arg; 23327c478bd9Sstevel@tonic-gate int i; 23337c478bd9Sstevel@tonic-gate 23347c478bd9Sstevel@tonic-gate CYC_PTRACE1("unconfigure", cpu); 23357c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 23367c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_OFFLINE); 23377c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_nelems == 0); 23387c478bd9Sstevel@tonic-gate 23397c478bd9Sstevel@tonic-gate /* 23407c478bd9Sstevel@tonic-gate * Let the backend know that the CPU is being yanked, and free up 23417c478bd9Sstevel@tonic-gate * the backend structure. 23427c478bd9Sstevel@tonic-gate */ 23437c478bd9Sstevel@tonic-gate be->cyb_unconfigure(bar); 23447c478bd9Sstevel@tonic-gate kmem_free(be, sizeof (cyc_backend_t)); 23457c478bd9Sstevel@tonic-gate cpu->cyp_backend = NULL; 23467c478bd9Sstevel@tonic-gate 23477c478bd9Sstevel@tonic-gate /* 23487c478bd9Sstevel@tonic-gate * Free up the producer/consumer buffers at each of the soft levels. 23497c478bd9Sstevel@tonic-gate */ 23507c478bd9Sstevel@tonic-gate for (i = CY_LOW_LEVEL; i < CY_LOW_LEVEL + CY_SOFT_LEVELS; i++) { 23517c478bd9Sstevel@tonic-gate cyc_softbuf_t *softbuf = &cpu->cyp_softbuf[i]; 23527c478bd9Sstevel@tonic-gate uchar_t hard = softbuf->cys_hard; 23537c478bd9Sstevel@tonic-gate cyc_pcbuffer_t *pc = &softbuf->cys_buf[hard]; 23547c478bd9Sstevel@tonic-gate size_t bufsize = sizeof (cyc_index_t) * (pc->cypc_sizemask + 1); 23557c478bd9Sstevel@tonic-gate 23567c478bd9Sstevel@tonic-gate /* 23577c478bd9Sstevel@tonic-gate * Assert that we're not in the middle of a resize operation. 23587c478bd9Sstevel@tonic-gate */ 23597c478bd9Sstevel@tonic-gate ASSERT(hard == softbuf->cys_soft); 23607c478bd9Sstevel@tonic-gate ASSERT(hard == 0 || hard == 1); 23617c478bd9Sstevel@tonic-gate ASSERT(pc->cypc_buf != NULL); 23627c478bd9Sstevel@tonic-gate ASSERT(softbuf->cys_buf[hard ^ 1].cypc_buf == NULL); 23637c478bd9Sstevel@tonic-gate 23647c478bd9Sstevel@tonic-gate kmem_free(pc->cypc_buf, bufsize); 23657c478bd9Sstevel@tonic-gate pc->cypc_buf = NULL; 23667c478bd9Sstevel@tonic-gate } 23677c478bd9Sstevel@tonic-gate 23687c478bd9Sstevel@tonic-gate /* 23697c478bd9Sstevel@tonic-gate * Finally, clean up our remaining dynamic structures and NULL out 23707c478bd9Sstevel@tonic-gate * the cpu_cyclic pointer. 23717c478bd9Sstevel@tonic-gate */ 23727c478bd9Sstevel@tonic-gate kmem_free(cpu->cyp_cyclics, cpu->cyp_size * sizeof (cyclic_t)); 23737c478bd9Sstevel@tonic-gate kmem_free(cpu->cyp_heap, cpu->cyp_size * sizeof (cyc_index_t)); 23747c478bd9Sstevel@tonic-gate kmem_free(cpu, sizeof (cyc_cpu_t)); 23757c478bd9Sstevel@tonic-gate 23767c478bd9Sstevel@tonic-gate c->cpu_cyclic = NULL; 23777c478bd9Sstevel@tonic-gate } 23787c478bd9Sstevel@tonic-gate 23797c478bd9Sstevel@tonic-gate static int 23807c478bd9Sstevel@tonic-gate cyclic_cpu_setup(cpu_setup_t what, int id) 23817c478bd9Sstevel@tonic-gate { 23827c478bd9Sstevel@tonic-gate /* 23837c478bd9Sstevel@tonic-gate * We are guaranteed that there is still/already an entry in the 23847c478bd9Sstevel@tonic-gate * cpu array for this CPU. 23857c478bd9Sstevel@tonic-gate */ 23867c478bd9Sstevel@tonic-gate cpu_t *c = cpu[id]; 23877c478bd9Sstevel@tonic-gate cyc_cpu_t *cyp = c->cpu_cyclic; 23887c478bd9Sstevel@tonic-gate 23897c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 23907c478bd9Sstevel@tonic-gate 23917c478bd9Sstevel@tonic-gate switch (what) { 23927c478bd9Sstevel@tonic-gate case CPU_CONFIG: 23937c478bd9Sstevel@tonic-gate ASSERT(cyp == NULL); 23947c478bd9Sstevel@tonic-gate cyclic_configure(c); 23957c478bd9Sstevel@tonic-gate break; 23967c478bd9Sstevel@tonic-gate 23977c478bd9Sstevel@tonic-gate case CPU_UNCONFIG: 23987c478bd9Sstevel@tonic-gate ASSERT(cyp != NULL && cyp->cyp_state == CYS_OFFLINE); 23997c478bd9Sstevel@tonic-gate cyclic_unconfigure(c); 24007c478bd9Sstevel@tonic-gate break; 24017c478bd9Sstevel@tonic-gate 24027c478bd9Sstevel@tonic-gate default: 24037c478bd9Sstevel@tonic-gate break; 24047c478bd9Sstevel@tonic-gate } 24057c478bd9Sstevel@tonic-gate 24067c478bd9Sstevel@tonic-gate return (0); 24077c478bd9Sstevel@tonic-gate } 24087c478bd9Sstevel@tonic-gate 24097c478bd9Sstevel@tonic-gate static void 24107c478bd9Sstevel@tonic-gate cyclic_suspend_xcall(cyc_xcallarg_t *arg) 24117c478bd9Sstevel@tonic-gate { 24127c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = arg->cyx_cpu; 24137c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 24147c478bd9Sstevel@tonic-gate cyc_cookie_t cookie; 24157c478bd9Sstevel@tonic-gate cyb_arg_t bar = be->cyb_arg; 24167c478bd9Sstevel@tonic-gate 24177c478bd9Sstevel@tonic-gate cookie = be->cyb_set_level(bar, CY_HIGH_LEVEL); 24187c478bd9Sstevel@tonic-gate 24197c478bd9Sstevel@tonic-gate CYC_TRACE1(cpu, CY_HIGH_LEVEL, "suspend-xcall", cpu->cyp_nelems); 24207c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE || cpu->cyp_state == CYS_OFFLINE); 24217c478bd9Sstevel@tonic-gate 24227c478bd9Sstevel@tonic-gate /* 24237c478bd9Sstevel@tonic-gate * We won't disable this CPU unless it has a non-zero number of 24247c478bd9Sstevel@tonic-gate * elements (cpu_lock assures that no one else may be attempting 24257c478bd9Sstevel@tonic-gate * to disable this CPU). 24267c478bd9Sstevel@tonic-gate */ 24277c478bd9Sstevel@tonic-gate if (cpu->cyp_nelems > 0) { 24287c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 24297c478bd9Sstevel@tonic-gate be->cyb_disable(bar); 24307c478bd9Sstevel@tonic-gate } 24317c478bd9Sstevel@tonic-gate 24327c478bd9Sstevel@tonic-gate if (cpu->cyp_state == CYS_ONLINE) 24337c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_SUSPENDED; 24347c478bd9Sstevel@tonic-gate 24357c478bd9Sstevel@tonic-gate be->cyb_suspend(bar); 24367c478bd9Sstevel@tonic-gate be->cyb_restore_level(bar, cookie); 24377c478bd9Sstevel@tonic-gate } 24387c478bd9Sstevel@tonic-gate 24397c478bd9Sstevel@tonic-gate static void 24407c478bd9Sstevel@tonic-gate cyclic_resume_xcall(cyc_xcallarg_t *arg) 24417c478bd9Sstevel@tonic-gate { 24427c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = arg->cyx_cpu; 24437c478bd9Sstevel@tonic-gate cyc_backend_t *be = cpu->cyp_backend; 24447c478bd9Sstevel@tonic-gate cyc_cookie_t cookie; 24457c478bd9Sstevel@tonic-gate cyb_arg_t bar = be->cyb_arg; 24467c478bd9Sstevel@tonic-gate cyc_state_t state = cpu->cyp_state; 24477c478bd9Sstevel@tonic-gate 24487c478bd9Sstevel@tonic-gate cookie = be->cyb_set_level(bar, CY_HIGH_LEVEL); 24497c478bd9Sstevel@tonic-gate 24507c478bd9Sstevel@tonic-gate CYC_TRACE1(cpu, CY_HIGH_LEVEL, "resume-xcall", cpu->cyp_nelems); 24517c478bd9Sstevel@tonic-gate ASSERT(state == CYS_SUSPENDED || state == CYS_OFFLINE); 24527c478bd9Sstevel@tonic-gate 24537c478bd9Sstevel@tonic-gate be->cyb_resume(bar); 24547c478bd9Sstevel@tonic-gate 24557c478bd9Sstevel@tonic-gate /* 24567c478bd9Sstevel@tonic-gate * We won't enable this CPU unless it has a non-zero number of 24577c478bd9Sstevel@tonic-gate * elements. 24587c478bd9Sstevel@tonic-gate */ 24597c478bd9Sstevel@tonic-gate if (cpu->cyp_nelems > 0) { 24607c478bd9Sstevel@tonic-gate cyclic_t *cyclic = &cpu->cyp_cyclics[cpu->cyp_heap[0]]; 24617c478bd9Sstevel@tonic-gate hrtime_t exp = cyclic->cy_expire; 24627c478bd9Sstevel@tonic-gate 24637c478bd9Sstevel@tonic-gate CYC_TRACE(cpu, CY_HIGH_LEVEL, "resume-reprog", cyclic, exp); 24647c478bd9Sstevel@tonic-gate ASSERT(state == CYS_SUSPENDED); 24657c478bd9Sstevel@tonic-gate be->cyb_enable(bar); 24667c478bd9Sstevel@tonic-gate be->cyb_reprogram(bar, exp); 24677c478bd9Sstevel@tonic-gate } 24687c478bd9Sstevel@tonic-gate 24697c478bd9Sstevel@tonic-gate if (state == CYS_SUSPENDED) 24707c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_ONLINE; 24717c478bd9Sstevel@tonic-gate 24727c478bd9Sstevel@tonic-gate CYC_TRACE1(cpu, CY_HIGH_LEVEL, "resume-done", cpu->cyp_nelems); 24737c478bd9Sstevel@tonic-gate be->cyb_restore_level(bar, cookie); 24747c478bd9Sstevel@tonic-gate } 24757c478bd9Sstevel@tonic-gate 24767c478bd9Sstevel@tonic-gate static void 24777c478bd9Sstevel@tonic-gate cyclic_omni_start(cyc_id_t *idp, cyc_cpu_t *cpu) 24787c478bd9Sstevel@tonic-gate { 24797c478bd9Sstevel@tonic-gate cyc_omni_handler_t *omni = &idp->cyi_omni_hdlr; 24807c478bd9Sstevel@tonic-gate cyc_omni_cpu_t *ocpu = kmem_alloc(sizeof (cyc_omni_cpu_t), KM_SLEEP); 24817c478bd9Sstevel@tonic-gate cyc_handler_t hdlr; 24827c478bd9Sstevel@tonic-gate cyc_time_t when; 24837c478bd9Sstevel@tonic-gate 24847c478bd9Sstevel@tonic-gate CYC_PTRACE("omni-start", cpu, idp); 24857c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 24867c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 24877c478bd9Sstevel@tonic-gate ASSERT(idp->cyi_cpu == NULL); 24887c478bd9Sstevel@tonic-gate 24897c478bd9Sstevel@tonic-gate hdlr.cyh_func = NULL; 24907c478bd9Sstevel@tonic-gate hdlr.cyh_arg = NULL; 24917c478bd9Sstevel@tonic-gate hdlr.cyh_level = CY_LEVELS; 24927c478bd9Sstevel@tonic-gate 24937c478bd9Sstevel@tonic-gate when.cyt_when = 0; 24947c478bd9Sstevel@tonic-gate when.cyt_interval = 0; 24957c478bd9Sstevel@tonic-gate 24967c478bd9Sstevel@tonic-gate omni->cyo_online(omni->cyo_arg, cpu->cyp_cpu, &hdlr, &when); 24977c478bd9Sstevel@tonic-gate 24987c478bd9Sstevel@tonic-gate ASSERT(hdlr.cyh_func != NULL); 24997c478bd9Sstevel@tonic-gate ASSERT(hdlr.cyh_level < CY_LEVELS); 25007c478bd9Sstevel@tonic-gate ASSERT(when.cyt_when >= 0 && when.cyt_interval > 0); 25017c478bd9Sstevel@tonic-gate 25027c478bd9Sstevel@tonic-gate ocpu->cyo_cpu = cpu; 25037c478bd9Sstevel@tonic-gate ocpu->cyo_arg = hdlr.cyh_arg; 25047c478bd9Sstevel@tonic-gate ocpu->cyo_ndx = cyclic_add_here(cpu, &hdlr, &when, 0); 25057c478bd9Sstevel@tonic-gate ocpu->cyo_next = idp->cyi_omni_list; 25067c478bd9Sstevel@tonic-gate idp->cyi_omni_list = ocpu; 25077c478bd9Sstevel@tonic-gate } 25087c478bd9Sstevel@tonic-gate 25097c478bd9Sstevel@tonic-gate static void 25107c478bd9Sstevel@tonic-gate cyclic_omni_stop(cyc_id_t *idp, cyc_cpu_t *cpu) 25117c478bd9Sstevel@tonic-gate { 25127c478bd9Sstevel@tonic-gate cyc_omni_handler_t *omni = &idp->cyi_omni_hdlr; 25137c478bd9Sstevel@tonic-gate cyc_omni_cpu_t *ocpu = idp->cyi_omni_list, *prev = NULL; 2514*87a18d3fSMadhavan Venkataraman clock_t delay; 2515*87a18d3fSMadhavan Venkataraman int ret; 25167c478bd9Sstevel@tonic-gate 25177c478bd9Sstevel@tonic-gate CYC_PTRACE("omni-stop", cpu, idp); 25187c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 25197c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 25207c478bd9Sstevel@tonic-gate ASSERT(idp->cyi_cpu == NULL); 25217c478bd9Sstevel@tonic-gate ASSERT(ocpu != NULL); 25227c478bd9Sstevel@tonic-gate 2523*87a18d3fSMadhavan Venkataraman /* 2524*87a18d3fSMadhavan Venkataraman * Prevent a reprogram of this cyclic while we are removing it. 2525*87a18d3fSMadhavan Venkataraman * Otherwise, cyclic_reprogram_here() will end up sending an X-call 2526*87a18d3fSMadhavan Venkataraman * to the offlined CPU. 2527*87a18d3fSMadhavan Venkataraman */ 2528*87a18d3fSMadhavan Venkataraman rw_enter(&idp->cyi_lock, RW_WRITER); 2529*87a18d3fSMadhavan Venkataraman 25307c478bd9Sstevel@tonic-gate while (ocpu != NULL && ocpu->cyo_cpu != cpu) { 25317c478bd9Sstevel@tonic-gate prev = ocpu; 25327c478bd9Sstevel@tonic-gate ocpu = ocpu->cyo_next; 25337c478bd9Sstevel@tonic-gate } 25347c478bd9Sstevel@tonic-gate 25357c478bd9Sstevel@tonic-gate /* 25367c478bd9Sstevel@tonic-gate * We _must_ have found an cyc_omni_cpu which corresponds to this 25377c478bd9Sstevel@tonic-gate * CPU -- the definition of an omnipresent cyclic is that it runs 25387c478bd9Sstevel@tonic-gate * on all online CPUs. 25397c478bd9Sstevel@tonic-gate */ 25407c478bd9Sstevel@tonic-gate ASSERT(ocpu != NULL); 25417c478bd9Sstevel@tonic-gate 25427c478bd9Sstevel@tonic-gate if (prev == NULL) { 25437c478bd9Sstevel@tonic-gate idp->cyi_omni_list = ocpu->cyo_next; 25447c478bd9Sstevel@tonic-gate } else { 25457c478bd9Sstevel@tonic-gate prev->cyo_next = ocpu->cyo_next; 25467c478bd9Sstevel@tonic-gate } 25477c478bd9Sstevel@tonic-gate 2548*87a18d3fSMadhavan Venkataraman /* 2549*87a18d3fSMadhavan Venkataraman * Remove the cyclic from the source. We cannot block during this 2550*87a18d3fSMadhavan Venkataraman * operation because we are holding the cyi_lock which can be held 2551*87a18d3fSMadhavan Venkataraman * by the cyclic handler via cyclic_reprogram(). 2552*87a18d3fSMadhavan Venkataraman * 2553*87a18d3fSMadhavan Venkataraman * If we cannot remove the cyclic without waiting, we spin for a time, 2554*87a18d3fSMadhavan Venkataraman * and reattempt the (non-blocking) removal. If the handler is blocked 2555*87a18d3fSMadhavan Venkataraman * on the cyi_lock, then we let go of it in the spin loop to give 2556*87a18d3fSMadhavan Venkataraman * the handler a chance to run. Note that the removal will ultimately 2557*87a18d3fSMadhavan Venkataraman * succeed -- even if the cyclic handler is blocked on a resource 2558*87a18d3fSMadhavan Venkataraman * held by a thread which we have preempted, priority inheritance 2559*87a18d3fSMadhavan Venkataraman * assures that the preempted thread will preempt us and continue 2560*87a18d3fSMadhavan Venkataraman * to progress. 2561*87a18d3fSMadhavan Venkataraman */ 2562*87a18d3fSMadhavan Venkataraman for (delay = 1; ; delay <<= 1) { 2563*87a18d3fSMadhavan Venkataraman /* 2564*87a18d3fSMadhavan Venkataraman * Before we begin this operation, disable kernel preemption. 2565*87a18d3fSMadhavan Venkataraman */ 2566*87a18d3fSMadhavan Venkataraman kpreempt_disable(); 2567*87a18d3fSMadhavan Venkataraman ret = cyclic_remove_here(ocpu->cyo_cpu, ocpu->cyo_ndx, NULL, 2568*87a18d3fSMadhavan Venkataraman CY_NOWAIT); 2569*87a18d3fSMadhavan Venkataraman /* 2570*87a18d3fSMadhavan Venkataraman * Enable kernel preemption while spinning. 2571*87a18d3fSMadhavan Venkataraman */ 2572*87a18d3fSMadhavan Venkataraman kpreempt_enable(); 2573*87a18d3fSMadhavan Venkataraman 2574*87a18d3fSMadhavan Venkataraman if (ret) 2575*87a18d3fSMadhavan Venkataraman break; 2576*87a18d3fSMadhavan Venkataraman 2577*87a18d3fSMadhavan Venkataraman CYC_PTRACE("remove-omni-retry", idp, ocpu->cyo_cpu); 2578*87a18d3fSMadhavan Venkataraman 2579*87a18d3fSMadhavan Venkataraman /* 2580*87a18d3fSMadhavan Venkataraman * Drop the RW lock to avoid a deadlock with the cyclic 2581*87a18d3fSMadhavan Venkataraman * handler (because it can potentially call cyclic_reprogram(). 2582*87a18d3fSMadhavan Venkataraman */ 2583*87a18d3fSMadhavan Venkataraman rw_exit(&idp->cyi_lock); 2584*87a18d3fSMadhavan Venkataraman drv_usecwait(delay); 2585*87a18d3fSMadhavan Venkataraman rw_enter(&idp->cyi_lock, RW_WRITER); 2586*87a18d3fSMadhavan Venkataraman } 2587*87a18d3fSMadhavan Venkataraman 2588*87a18d3fSMadhavan Venkataraman /* 2589*87a18d3fSMadhavan Venkataraman * Now that we have successfully removed the cyclic, allow the omni 2590*87a18d3fSMadhavan Venkataraman * cyclic to be reprogrammed on other CPUs. 2591*87a18d3fSMadhavan Venkataraman */ 2592*87a18d3fSMadhavan Venkataraman rw_exit(&idp->cyi_lock); 25937c478bd9Sstevel@tonic-gate 25947c478bd9Sstevel@tonic-gate /* 25957c478bd9Sstevel@tonic-gate * The cyclic has been removed from this CPU; time to call the 25967c478bd9Sstevel@tonic-gate * omnipresent offline handler. 25977c478bd9Sstevel@tonic-gate */ 25987c478bd9Sstevel@tonic-gate if (omni->cyo_offline != NULL) 25997c478bd9Sstevel@tonic-gate omni->cyo_offline(omni->cyo_arg, cpu->cyp_cpu, ocpu->cyo_arg); 26007c478bd9Sstevel@tonic-gate 26017c478bd9Sstevel@tonic-gate kmem_free(ocpu, sizeof (cyc_omni_cpu_t)); 26027c478bd9Sstevel@tonic-gate } 26037c478bd9Sstevel@tonic-gate 26047c478bd9Sstevel@tonic-gate static cyc_id_t * 26057c478bd9Sstevel@tonic-gate cyclic_new_id() 26067c478bd9Sstevel@tonic-gate { 26077c478bd9Sstevel@tonic-gate cyc_id_t *idp; 26087c478bd9Sstevel@tonic-gate 26097c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 26107c478bd9Sstevel@tonic-gate 26117c478bd9Sstevel@tonic-gate idp = kmem_cache_alloc(cyclic_id_cache, KM_SLEEP); 26127c478bd9Sstevel@tonic-gate 26137c478bd9Sstevel@tonic-gate /* 26147c478bd9Sstevel@tonic-gate * The cyi_cpu field of the cyc_id_t structure tracks the CPU 26157c478bd9Sstevel@tonic-gate * associated with the cyclic. If and only if this field is NULL, the 26167c478bd9Sstevel@tonic-gate * cyc_id_t is an omnipresent cyclic. Note that cyi_omni_list may be 26177c478bd9Sstevel@tonic-gate * NULL for an omnipresent cyclic while the cyclic is being created 26187c478bd9Sstevel@tonic-gate * or destroyed. 26197c478bd9Sstevel@tonic-gate */ 26207c478bd9Sstevel@tonic-gate idp->cyi_cpu = NULL; 26217c478bd9Sstevel@tonic-gate idp->cyi_ndx = 0; 2622*87a18d3fSMadhavan Venkataraman rw_init(&idp->cyi_lock, NULL, RW_DEFAULT, NULL); 26237c478bd9Sstevel@tonic-gate 26247c478bd9Sstevel@tonic-gate idp->cyi_next = cyclic_id_head; 26257c478bd9Sstevel@tonic-gate idp->cyi_prev = NULL; 26267c478bd9Sstevel@tonic-gate idp->cyi_omni_list = NULL; 26277c478bd9Sstevel@tonic-gate 26287c478bd9Sstevel@tonic-gate if (cyclic_id_head != NULL) { 26297c478bd9Sstevel@tonic-gate ASSERT(cyclic_id_head->cyi_prev == NULL); 26307c478bd9Sstevel@tonic-gate cyclic_id_head->cyi_prev = idp; 26317c478bd9Sstevel@tonic-gate } 26327c478bd9Sstevel@tonic-gate 26337c478bd9Sstevel@tonic-gate cyclic_id_head = idp; 26347c478bd9Sstevel@tonic-gate 26357c478bd9Sstevel@tonic-gate return (idp); 26367c478bd9Sstevel@tonic-gate } 26377c478bd9Sstevel@tonic-gate 26387c478bd9Sstevel@tonic-gate /* 26397c478bd9Sstevel@tonic-gate * cyclic_id_t cyclic_add(cyc_handler_t *, cyc_time_t *) 26407c478bd9Sstevel@tonic-gate * 26417c478bd9Sstevel@tonic-gate * Overview 26427c478bd9Sstevel@tonic-gate * 26437c478bd9Sstevel@tonic-gate * cyclic_add() will create an unbound cyclic with the specified handler and 26447c478bd9Sstevel@tonic-gate * interval. The cyclic will run on a CPU which both has interrupts enabled 26457c478bd9Sstevel@tonic-gate * and is in the system CPU partition. 26467c478bd9Sstevel@tonic-gate * 26477c478bd9Sstevel@tonic-gate * Arguments and notes 26487c478bd9Sstevel@tonic-gate * 26497c478bd9Sstevel@tonic-gate * As its first argument, cyclic_add() takes a cyc_handler, which has the 26507c478bd9Sstevel@tonic-gate * following members: 26517c478bd9Sstevel@tonic-gate * 26527c478bd9Sstevel@tonic-gate * cyc_func_t cyh_func <-- Cyclic handler 26537c478bd9Sstevel@tonic-gate * void *cyh_arg <-- Argument to cyclic handler 26547c478bd9Sstevel@tonic-gate * cyc_level_t cyh_level <-- Level at which to fire; must be one of 26557c478bd9Sstevel@tonic-gate * CY_LOW_LEVEL, CY_LOCK_LEVEL or CY_HIGH_LEVEL 26567c478bd9Sstevel@tonic-gate * 26577c478bd9Sstevel@tonic-gate * Note that cyh_level is _not_ an ipl or spl; it must be one the 26587c478bd9Sstevel@tonic-gate * CY_*_LEVELs. This layer of abstraction allows the platform to define 26597c478bd9Sstevel@tonic-gate * the precise interrupt priority levels, within the following constraints: 26607c478bd9Sstevel@tonic-gate * 26617c478bd9Sstevel@tonic-gate * CY_LOCK_LEVEL must map to LOCK_LEVEL 26627c478bd9Sstevel@tonic-gate * CY_HIGH_LEVEL must map to an ipl greater than LOCK_LEVEL 26637c478bd9Sstevel@tonic-gate * CY_LOW_LEVEL must map to an ipl below LOCK_LEVEL 26647c478bd9Sstevel@tonic-gate * 26657c478bd9Sstevel@tonic-gate * In addition to a cyc_handler, cyclic_add() takes a cyc_time, which 26667c478bd9Sstevel@tonic-gate * has the following members: 26677c478bd9Sstevel@tonic-gate * 26687c478bd9Sstevel@tonic-gate * hrtime_t cyt_when <-- Absolute time, in nanoseconds since boot, at 26697c478bd9Sstevel@tonic-gate * which to start firing 26707c478bd9Sstevel@tonic-gate * hrtime_t cyt_interval <-- Length of interval, in nanoseconds 26717c478bd9Sstevel@tonic-gate * 26727c478bd9Sstevel@tonic-gate * gethrtime() is the time source for nanoseconds since boot. If cyt_when 26737c478bd9Sstevel@tonic-gate * is set to 0, the cyclic will start to fire when cyt_interval next 26747c478bd9Sstevel@tonic-gate * divides the number of nanoseconds since boot. 26757c478bd9Sstevel@tonic-gate * 26767c478bd9Sstevel@tonic-gate * The cyt_interval field _must_ be filled in by the caller; one-shots are 26777c478bd9Sstevel@tonic-gate * _not_ explicitly supported by the cyclic subsystem (cyclic_add() will 26787c478bd9Sstevel@tonic-gate * assert that cyt_interval is non-zero). The maximum value for either 26797c478bd9Sstevel@tonic-gate * field is INT64_MAX; the caller is responsible for assuring that 26807c478bd9Sstevel@tonic-gate * cyt_when + cyt_interval <= INT64_MAX. Neither field may be negative. 26817c478bd9Sstevel@tonic-gate * 26827c478bd9Sstevel@tonic-gate * For an arbitrary time t in the future, the cyclic handler is guaranteed 26837c478bd9Sstevel@tonic-gate * to have been called (t - cyt_when) / cyt_interval times. This will 26847c478bd9Sstevel@tonic-gate * be true even if interrupts have been disabled for periods greater than 26857c478bd9Sstevel@tonic-gate * cyt_interval nanoseconds. In order to compensate for such periods, 26867c478bd9Sstevel@tonic-gate * the cyclic handler may be called a finite number of times with an 26877c478bd9Sstevel@tonic-gate * arbitrarily small interval. 26887c478bd9Sstevel@tonic-gate * 26897c478bd9Sstevel@tonic-gate * The cyclic subsystem will not enforce any lower bound on the interval; 26907c478bd9Sstevel@tonic-gate * if the interval is less than the time required to process an interrupt, 26917c478bd9Sstevel@tonic-gate * the CPU will wedge. It's the responsibility of the caller to assure that 26927c478bd9Sstevel@tonic-gate * either the value of the interval is sane, or that its caller has 26937c478bd9Sstevel@tonic-gate * sufficient privilege to deny service (i.e. its caller is root). 26947c478bd9Sstevel@tonic-gate * 26957c478bd9Sstevel@tonic-gate * The cyclic handler is guaranteed to be single threaded, even while the 26967c478bd9Sstevel@tonic-gate * cyclic is being juggled between CPUs (see cyclic_juggle(), below). 26977c478bd9Sstevel@tonic-gate * That is, a given cyclic handler will never be executed simultaneously 26987c478bd9Sstevel@tonic-gate * on different CPUs. 26997c478bd9Sstevel@tonic-gate * 27007c478bd9Sstevel@tonic-gate * Return value 27017c478bd9Sstevel@tonic-gate * 27027c478bd9Sstevel@tonic-gate * cyclic_add() returns a cyclic_id_t, which is guaranteed to be a value 27037c478bd9Sstevel@tonic-gate * other than CYCLIC_NONE. cyclic_add() cannot fail. 27047c478bd9Sstevel@tonic-gate * 27057c478bd9Sstevel@tonic-gate * Caller's context 27067c478bd9Sstevel@tonic-gate * 27077c478bd9Sstevel@tonic-gate * cpu_lock must be held by the caller, and the caller must not be in 27087c478bd9Sstevel@tonic-gate * interrupt context. cyclic_add() will perform a KM_SLEEP kernel 27097c478bd9Sstevel@tonic-gate * memory allocation, so the usual rules (e.g. p_lock cannot be held) 27107c478bd9Sstevel@tonic-gate * apply. A cyclic may be added even in the presence of CPUs that have 27117c478bd9Sstevel@tonic-gate * not been configured with respect to the cyclic subsystem, but only 27127c478bd9Sstevel@tonic-gate * configured CPUs will be eligible to run the new cyclic. 27137c478bd9Sstevel@tonic-gate * 27147c478bd9Sstevel@tonic-gate * Cyclic handler's context 27157c478bd9Sstevel@tonic-gate * 27167c478bd9Sstevel@tonic-gate * Cyclic handlers will be executed in the interrupt context corresponding 27177c478bd9Sstevel@tonic-gate * to the specified level (i.e. either high, lock or low level). The 27187c478bd9Sstevel@tonic-gate * usual context rules apply. 27197c478bd9Sstevel@tonic-gate * 27207c478bd9Sstevel@tonic-gate * A cyclic handler may not grab ANY locks held by the caller of any of 27217c478bd9Sstevel@tonic-gate * cyclic_add(), cyclic_remove() or cyclic_bind(); the implementation of 27227c478bd9Sstevel@tonic-gate * these functions may require blocking on cyclic handler completion. 27237c478bd9Sstevel@tonic-gate * Moreover, cyclic handlers may not make any call back into the cyclic 27247c478bd9Sstevel@tonic-gate * subsystem. 27257c478bd9Sstevel@tonic-gate */ 27267c478bd9Sstevel@tonic-gate cyclic_id_t 27277c478bd9Sstevel@tonic-gate cyclic_add(cyc_handler_t *hdlr, cyc_time_t *when) 27287c478bd9Sstevel@tonic-gate { 27297c478bd9Sstevel@tonic-gate cyc_id_t *idp = cyclic_new_id(); 27307c478bd9Sstevel@tonic-gate 27317c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 27327c478bd9Sstevel@tonic-gate ASSERT(when->cyt_when >= 0 && when->cyt_interval > 0); 27337c478bd9Sstevel@tonic-gate 27347c478bd9Sstevel@tonic-gate idp->cyi_cpu = cyclic_pick_cpu(NULL, NULL, NULL, 0); 27357c478bd9Sstevel@tonic-gate idp->cyi_ndx = cyclic_add_here(idp->cyi_cpu, hdlr, when, 0); 27367c478bd9Sstevel@tonic-gate 27377c478bd9Sstevel@tonic-gate return ((uintptr_t)idp); 27387c478bd9Sstevel@tonic-gate } 27397c478bd9Sstevel@tonic-gate 27407c478bd9Sstevel@tonic-gate /* 27417c478bd9Sstevel@tonic-gate * cyclic_id_t cyclic_add_omni(cyc_omni_handler_t *) 27427c478bd9Sstevel@tonic-gate * 27437c478bd9Sstevel@tonic-gate * Overview 27447c478bd9Sstevel@tonic-gate * 27457c478bd9Sstevel@tonic-gate * cyclic_add_omni() will create an omnipresent cyclic with the specified 27467c478bd9Sstevel@tonic-gate * online and offline handlers. Omnipresent cyclics run on all online 27477c478bd9Sstevel@tonic-gate * CPUs, including CPUs which have unbound interrupts disabled. 27487c478bd9Sstevel@tonic-gate * 27497c478bd9Sstevel@tonic-gate * Arguments 27507c478bd9Sstevel@tonic-gate * 27517c478bd9Sstevel@tonic-gate * As its only argument, cyclic_add_omni() takes a cyc_omni_handler, which 27527c478bd9Sstevel@tonic-gate * has the following members: 27537c478bd9Sstevel@tonic-gate * 27547c478bd9Sstevel@tonic-gate * void (*cyo_online)() <-- Online handler 27557c478bd9Sstevel@tonic-gate * void (*cyo_offline)() <-- Offline handler 27567c478bd9Sstevel@tonic-gate * void *cyo_arg <-- Argument to be passed to on/offline handlers 27577c478bd9Sstevel@tonic-gate * 27587c478bd9Sstevel@tonic-gate * Online handler 27597c478bd9Sstevel@tonic-gate * 27607c478bd9Sstevel@tonic-gate * The cyo_online member is a pointer to a function which has the following 27617c478bd9Sstevel@tonic-gate * four arguments: 27627c478bd9Sstevel@tonic-gate * 27637c478bd9Sstevel@tonic-gate * void * <-- Argument (cyo_arg) 27647c478bd9Sstevel@tonic-gate * cpu_t * <-- Pointer to CPU about to be onlined 27657c478bd9Sstevel@tonic-gate * cyc_handler_t * <-- Pointer to cyc_handler_t; must be filled in 27667c478bd9Sstevel@tonic-gate * by omni online handler 27677c478bd9Sstevel@tonic-gate * cyc_time_t * <-- Pointer to cyc_time_t; must be filled in by 27687c478bd9Sstevel@tonic-gate * omni online handler 27697c478bd9Sstevel@tonic-gate * 27707c478bd9Sstevel@tonic-gate * The omni cyclic online handler is always called _before_ the omni 27717c478bd9Sstevel@tonic-gate * cyclic begins to fire on the specified CPU. As the above argument 27727c478bd9Sstevel@tonic-gate * description implies, the online handler must fill in the two structures 27737c478bd9Sstevel@tonic-gate * passed to it: the cyc_handler_t and the cyc_time_t. These are the 27747c478bd9Sstevel@tonic-gate * same two structures passed to cyclic_add(), outlined above. This 27757c478bd9Sstevel@tonic-gate * allows the omni cyclic to have maximum flexibility; different CPUs may 27767c478bd9Sstevel@tonic-gate * optionally 27777c478bd9Sstevel@tonic-gate * 27787c478bd9Sstevel@tonic-gate * (a) have different intervals 27797c478bd9Sstevel@tonic-gate * (b) be explicitly in or out of phase with one another 27807c478bd9Sstevel@tonic-gate * (c) have different handlers 27817c478bd9Sstevel@tonic-gate * (d) have different handler arguments 27827c478bd9Sstevel@tonic-gate * (e) fire at different levels 27837c478bd9Sstevel@tonic-gate * 27847c478bd9Sstevel@tonic-gate * Of these, (e) seems somewhat dubious, but is nonetheless allowed. 27857c478bd9Sstevel@tonic-gate * 27867c478bd9Sstevel@tonic-gate * The omni online handler is called in the same context as cyclic_add(), 27877c478bd9Sstevel@tonic-gate * and has the same liberties: omni online handlers may perform KM_SLEEP 27887c478bd9Sstevel@tonic-gate * kernel memory allocations, and may grab locks which are also acquired 27897c478bd9Sstevel@tonic-gate * by cyclic handlers. However, omni cyclic online handlers may _not_ 27907c478bd9Sstevel@tonic-gate * call back into the cyclic subsystem, and should be generally careful 27917c478bd9Sstevel@tonic-gate * about calling into arbitrary kernel subsystems. 27927c478bd9Sstevel@tonic-gate * 27937c478bd9Sstevel@tonic-gate * Offline handler 27947c478bd9Sstevel@tonic-gate * 27957c478bd9Sstevel@tonic-gate * The cyo_offline member is a pointer to a function which has the following 27967c478bd9Sstevel@tonic-gate * three arguments: 27977c478bd9Sstevel@tonic-gate * 27987c478bd9Sstevel@tonic-gate * void * <-- Argument (cyo_arg) 27997c478bd9Sstevel@tonic-gate * cpu_t * <-- Pointer to CPU about to be offlined 28007c478bd9Sstevel@tonic-gate * void * <-- CPU's cyclic argument (that is, value 28017c478bd9Sstevel@tonic-gate * to which cyh_arg member of the cyc_handler_t 28027c478bd9Sstevel@tonic-gate * was set in the omni online handler) 28037c478bd9Sstevel@tonic-gate * 28047c478bd9Sstevel@tonic-gate * The omni cyclic offline handler is always called _after_ the omni 28057c478bd9Sstevel@tonic-gate * cyclic has ceased firing on the specified CPU. Its purpose is to 28067c478bd9Sstevel@tonic-gate * allow cleanup of any resources dynamically allocated in the omni cyclic 28077c478bd9Sstevel@tonic-gate * online handler. The context of the offline handler is identical to 28087c478bd9Sstevel@tonic-gate * that of the online handler; the same constraints and liberties apply. 28097c478bd9Sstevel@tonic-gate * 28107c478bd9Sstevel@tonic-gate * The offline handler is optional; it may be NULL. 28117c478bd9Sstevel@tonic-gate * 28127c478bd9Sstevel@tonic-gate * Return value 28137c478bd9Sstevel@tonic-gate * 28147c478bd9Sstevel@tonic-gate * cyclic_add_omni() returns a cyclic_id_t, which is guaranteed to be a 28157c478bd9Sstevel@tonic-gate * value other than CYCLIC_NONE. cyclic_add_omni() cannot fail. 28167c478bd9Sstevel@tonic-gate * 28177c478bd9Sstevel@tonic-gate * Caller's context 28187c478bd9Sstevel@tonic-gate * 28197c478bd9Sstevel@tonic-gate * The caller's context is identical to that of cyclic_add(), specified 28207c478bd9Sstevel@tonic-gate * above. 28217c478bd9Sstevel@tonic-gate */ 28227c478bd9Sstevel@tonic-gate cyclic_id_t 28237c478bd9Sstevel@tonic-gate cyclic_add_omni(cyc_omni_handler_t *omni) 28247c478bd9Sstevel@tonic-gate { 28257c478bd9Sstevel@tonic-gate cyc_id_t *idp = cyclic_new_id(); 28267c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu; 28277c478bd9Sstevel@tonic-gate cpu_t *c; 28287c478bd9Sstevel@tonic-gate 28297c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 28307c478bd9Sstevel@tonic-gate ASSERT(omni != NULL && omni->cyo_online != NULL); 28317c478bd9Sstevel@tonic-gate 28327c478bd9Sstevel@tonic-gate idp->cyi_omni_hdlr = *omni; 28337c478bd9Sstevel@tonic-gate 28347c478bd9Sstevel@tonic-gate c = cpu_list; 28357c478bd9Sstevel@tonic-gate do { 28367c478bd9Sstevel@tonic-gate if ((cpu = c->cpu_cyclic) == NULL) 28377c478bd9Sstevel@tonic-gate continue; 28387c478bd9Sstevel@tonic-gate 28397c478bd9Sstevel@tonic-gate if (cpu->cyp_state != CYS_ONLINE) { 28407c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_OFFLINE); 28417c478bd9Sstevel@tonic-gate continue; 28427c478bd9Sstevel@tonic-gate } 28437c478bd9Sstevel@tonic-gate 28447c478bd9Sstevel@tonic-gate cyclic_omni_start(idp, cpu); 28457c478bd9Sstevel@tonic-gate } while ((c = c->cpu_next) != cpu_list); 28467c478bd9Sstevel@tonic-gate 28477c478bd9Sstevel@tonic-gate /* 28487c478bd9Sstevel@tonic-gate * We must have found at least one online CPU on which to run 28497c478bd9Sstevel@tonic-gate * this cyclic. 28507c478bd9Sstevel@tonic-gate */ 28517c478bd9Sstevel@tonic-gate ASSERT(idp->cyi_omni_list != NULL); 28527c478bd9Sstevel@tonic-gate ASSERT(idp->cyi_cpu == NULL); 28537c478bd9Sstevel@tonic-gate 28547c478bd9Sstevel@tonic-gate return ((uintptr_t)idp); 28557c478bd9Sstevel@tonic-gate } 28567c478bd9Sstevel@tonic-gate 28577c478bd9Sstevel@tonic-gate /* 28587c478bd9Sstevel@tonic-gate * void cyclic_remove(cyclic_id_t) 28597c478bd9Sstevel@tonic-gate * 28607c478bd9Sstevel@tonic-gate * Overview 28617c478bd9Sstevel@tonic-gate * 28627c478bd9Sstevel@tonic-gate * cyclic_remove() will remove the specified cyclic from the system. 28637c478bd9Sstevel@tonic-gate * 28647c478bd9Sstevel@tonic-gate * Arguments and notes 28657c478bd9Sstevel@tonic-gate * 28667c478bd9Sstevel@tonic-gate * The only argument is a cyclic_id returned from either cyclic_add() or 28677c478bd9Sstevel@tonic-gate * cyclic_add_omni(). 28687c478bd9Sstevel@tonic-gate * 28697c478bd9Sstevel@tonic-gate * By the time cyclic_remove() returns, the caller is guaranteed that the 28707c478bd9Sstevel@tonic-gate * removed cyclic handler has completed execution (this is the same 28717c478bd9Sstevel@tonic-gate * semantic that untimeout() provides). As a result, cyclic_remove() may 28727c478bd9Sstevel@tonic-gate * need to block, waiting for the removed cyclic to complete execution. 28737c478bd9Sstevel@tonic-gate * This leads to an important constraint on the caller: no lock may be 28747c478bd9Sstevel@tonic-gate * held across cyclic_remove() that also may be acquired by a cyclic 28757c478bd9Sstevel@tonic-gate * handler. 28767c478bd9Sstevel@tonic-gate * 28777c478bd9Sstevel@tonic-gate * Return value 28787c478bd9Sstevel@tonic-gate * 28797c478bd9Sstevel@tonic-gate * None; cyclic_remove() always succeeds. 28807c478bd9Sstevel@tonic-gate * 28817c478bd9Sstevel@tonic-gate * Caller's context 28827c478bd9Sstevel@tonic-gate * 28837c478bd9Sstevel@tonic-gate * cpu_lock must be held by the caller, and the caller must not be in 28847c478bd9Sstevel@tonic-gate * interrupt context. The caller may not hold any locks which are also 28857c478bd9Sstevel@tonic-gate * grabbed by any cyclic handler. See "Arguments and notes", above. 28867c478bd9Sstevel@tonic-gate */ 28877c478bd9Sstevel@tonic-gate void 28887c478bd9Sstevel@tonic-gate cyclic_remove(cyclic_id_t id) 28897c478bd9Sstevel@tonic-gate { 28907c478bd9Sstevel@tonic-gate cyc_id_t *idp = (cyc_id_t *)id; 28917c478bd9Sstevel@tonic-gate cyc_id_t *prev = idp->cyi_prev, *next = idp->cyi_next; 28927c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = idp->cyi_cpu; 28937c478bd9Sstevel@tonic-gate 28947c478bd9Sstevel@tonic-gate CYC_PTRACE("remove", idp, idp->cyi_cpu); 28957c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 28967c478bd9Sstevel@tonic-gate 28977c478bd9Sstevel@tonic-gate if (cpu != NULL) { 28987c478bd9Sstevel@tonic-gate (void) cyclic_remove_here(cpu, idp->cyi_ndx, NULL, CY_WAIT); 28997c478bd9Sstevel@tonic-gate } else { 29007c478bd9Sstevel@tonic-gate ASSERT(idp->cyi_omni_list != NULL); 29017c478bd9Sstevel@tonic-gate while (idp->cyi_omni_list != NULL) 29027c478bd9Sstevel@tonic-gate cyclic_omni_stop(idp, idp->cyi_omni_list->cyo_cpu); 29037c478bd9Sstevel@tonic-gate } 29047c478bd9Sstevel@tonic-gate 29057c478bd9Sstevel@tonic-gate if (prev != NULL) { 29067c478bd9Sstevel@tonic-gate ASSERT(cyclic_id_head != idp); 29077c478bd9Sstevel@tonic-gate prev->cyi_next = next; 29087c478bd9Sstevel@tonic-gate } else { 29097c478bd9Sstevel@tonic-gate ASSERT(cyclic_id_head == idp); 29107c478bd9Sstevel@tonic-gate cyclic_id_head = next; 29117c478bd9Sstevel@tonic-gate } 29127c478bd9Sstevel@tonic-gate 29137c478bd9Sstevel@tonic-gate if (next != NULL) 29147c478bd9Sstevel@tonic-gate next->cyi_prev = prev; 29157c478bd9Sstevel@tonic-gate 29167c478bd9Sstevel@tonic-gate kmem_cache_free(cyclic_id_cache, idp); 29177c478bd9Sstevel@tonic-gate } 29187c478bd9Sstevel@tonic-gate 29197c478bd9Sstevel@tonic-gate /* 29207c478bd9Sstevel@tonic-gate * void cyclic_bind(cyclic_id_t, cpu_t *, cpupart_t *) 29217c478bd9Sstevel@tonic-gate * 29227c478bd9Sstevel@tonic-gate * Overview 29237c478bd9Sstevel@tonic-gate * 29247c478bd9Sstevel@tonic-gate * cyclic_bind() atomically changes the CPU and CPU partition bindings 29257c478bd9Sstevel@tonic-gate * of a cyclic. 29267c478bd9Sstevel@tonic-gate * 29277c478bd9Sstevel@tonic-gate * Arguments and notes 29287c478bd9Sstevel@tonic-gate * 29297c478bd9Sstevel@tonic-gate * The first argument is a cyclic_id retuned from cyclic_add(). 29307c478bd9Sstevel@tonic-gate * cyclic_bind() may _not_ be called on a cyclic_id returned from 29317c478bd9Sstevel@tonic-gate * cyclic_add_omni(). 29327c478bd9Sstevel@tonic-gate * 29337c478bd9Sstevel@tonic-gate * The second argument specifies the CPU to which to bind the specified 29347c478bd9Sstevel@tonic-gate * cyclic. If the specified cyclic is bound to a CPU other than the one 29357c478bd9Sstevel@tonic-gate * specified, it will be unbound from its bound CPU. Unbinding the cyclic 29367c478bd9Sstevel@tonic-gate * from its CPU may cause it to be juggled to another CPU. If the specified 29377c478bd9Sstevel@tonic-gate * CPU is non-NULL, the cyclic will be subsequently rebound to the specified 29387c478bd9Sstevel@tonic-gate * CPU. 29397c478bd9Sstevel@tonic-gate * 29407c478bd9Sstevel@tonic-gate * If a CPU with bound cyclics is transitioned into the P_NOINTR state, 29417c478bd9Sstevel@tonic-gate * only cyclics not bound to the CPU can be juggled away; CPU-bound cyclics 29427c478bd9Sstevel@tonic-gate * will continue to fire on the P_NOINTR CPU. A CPU with bound cyclics 29437c478bd9Sstevel@tonic-gate * cannot be offlined (attempts to offline the CPU will return EBUSY). 29447c478bd9Sstevel@tonic-gate * Likewise, cyclics may not be bound to an offline CPU; if the caller 29457c478bd9Sstevel@tonic-gate * attempts to bind a cyclic to an offline CPU, the cyclic subsystem will 29467c478bd9Sstevel@tonic-gate * panic. 29477c478bd9Sstevel@tonic-gate * 29487c478bd9Sstevel@tonic-gate * The third argument specifies the CPU partition to which to bind the 29497c478bd9Sstevel@tonic-gate * specified cyclic. If the specified cyclic is bound to a CPU partition 29507c478bd9Sstevel@tonic-gate * other than the one specified, it will be unbound from its bound 29517c478bd9Sstevel@tonic-gate * partition. Unbinding the cyclic from its CPU partition may cause it 29527c478bd9Sstevel@tonic-gate * to be juggled to another CPU. If the specified CPU partition is 29537c478bd9Sstevel@tonic-gate * non-NULL, the cyclic will be subsequently rebound to the specified CPU 29547c478bd9Sstevel@tonic-gate * partition. 29557c478bd9Sstevel@tonic-gate * 29567c478bd9Sstevel@tonic-gate * It is the caller's responsibility to assure that the specified CPU 29577c478bd9Sstevel@tonic-gate * partition contains a CPU. If it does not, the cyclic subsystem will 29587c478bd9Sstevel@tonic-gate * panic. A CPU partition with bound cyclics cannot be destroyed (attempts 29597c478bd9Sstevel@tonic-gate * to destroy the partition will return EBUSY). If a CPU with 29607c478bd9Sstevel@tonic-gate * partition-bound cyclics is transitioned into the P_NOINTR state, cyclics 29617c478bd9Sstevel@tonic-gate * bound to the CPU's partition (but not bound to the CPU) will be juggled 29627c478bd9Sstevel@tonic-gate * away only if there exists another CPU in the partition in the P_ONLINE 29637c478bd9Sstevel@tonic-gate * state. 29647c478bd9Sstevel@tonic-gate * 29657c478bd9Sstevel@tonic-gate * It is the caller's responsibility to assure that the specified CPU and 29667c478bd9Sstevel@tonic-gate * CPU partition are self-consistent. If both parameters are non-NULL, 29677c478bd9Sstevel@tonic-gate * and the specified CPU partition does not contain the specified CPU, the 29687c478bd9Sstevel@tonic-gate * cyclic subsystem will panic. 29697c478bd9Sstevel@tonic-gate * 29707c478bd9Sstevel@tonic-gate * It is the caller's responsibility to assure that the specified CPU has 29717c478bd9Sstevel@tonic-gate * been configured with respect to the cyclic subsystem. Generally, this 29727c478bd9Sstevel@tonic-gate * is always true for valid, on-line CPUs. The only periods of time during 29737c478bd9Sstevel@tonic-gate * which this may not be true are during MP boot (i.e. after cyclic_init() 29747c478bd9Sstevel@tonic-gate * is called but before cyclic_mp_init() is called) or during dynamic 29757c478bd9Sstevel@tonic-gate * reconfiguration; cyclic_bind() should only be called with great care 29767c478bd9Sstevel@tonic-gate * from these contexts. 29777c478bd9Sstevel@tonic-gate * 29787c478bd9Sstevel@tonic-gate * Return value 29797c478bd9Sstevel@tonic-gate * 29807c478bd9Sstevel@tonic-gate * None; cyclic_bind() always succeeds. 29817c478bd9Sstevel@tonic-gate * 29827c478bd9Sstevel@tonic-gate * Caller's context 29837c478bd9Sstevel@tonic-gate * 29847c478bd9Sstevel@tonic-gate * cpu_lock must be held by the caller, and the caller must not be in 29857c478bd9Sstevel@tonic-gate * interrupt context. The caller may not hold any locks which are also 29867c478bd9Sstevel@tonic-gate * grabbed by any cyclic handler. 29877c478bd9Sstevel@tonic-gate */ 29887c478bd9Sstevel@tonic-gate void 29897c478bd9Sstevel@tonic-gate cyclic_bind(cyclic_id_t id, cpu_t *d, cpupart_t *part) 29907c478bd9Sstevel@tonic-gate { 29917c478bd9Sstevel@tonic-gate cyc_id_t *idp = (cyc_id_t *)id; 29927c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = idp->cyi_cpu; 29937c478bd9Sstevel@tonic-gate cpu_t *c; 29947c478bd9Sstevel@tonic-gate uint16_t flags; 29957c478bd9Sstevel@tonic-gate 29967c478bd9Sstevel@tonic-gate CYC_PTRACE("bind", d, part); 29977c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 29987c478bd9Sstevel@tonic-gate ASSERT(part == NULL || d == NULL || d->cpu_part == part); 29997c478bd9Sstevel@tonic-gate 30007c478bd9Sstevel@tonic-gate if (cpu == NULL) { 30017c478bd9Sstevel@tonic-gate ASSERT(idp->cyi_omni_list != NULL); 30027c478bd9Sstevel@tonic-gate panic("attempt to change binding of omnipresent cyclic"); 30037c478bd9Sstevel@tonic-gate } 30047c478bd9Sstevel@tonic-gate 30057c478bd9Sstevel@tonic-gate c = cpu->cyp_cpu; 30067c478bd9Sstevel@tonic-gate flags = cpu->cyp_cyclics[idp->cyi_ndx].cy_flags; 30077c478bd9Sstevel@tonic-gate 30087c478bd9Sstevel@tonic-gate if (c != d && (flags & CYF_CPU_BOUND)) 30097c478bd9Sstevel@tonic-gate cyclic_unbind_cpu(id); 30107c478bd9Sstevel@tonic-gate 30117c478bd9Sstevel@tonic-gate /* 30127c478bd9Sstevel@tonic-gate * Reload our cpu (we may have migrated). We don't have to reload 30137c478bd9Sstevel@tonic-gate * the flags field here; if we were CYF_PART_BOUND on entry, we are 30147c478bd9Sstevel@tonic-gate * CYF_PART_BOUND now. 30157c478bd9Sstevel@tonic-gate */ 30167c478bd9Sstevel@tonic-gate cpu = idp->cyi_cpu; 30177c478bd9Sstevel@tonic-gate c = cpu->cyp_cpu; 30187c478bd9Sstevel@tonic-gate 30197c478bd9Sstevel@tonic-gate if (part != c->cpu_part && (flags & CYF_PART_BOUND)) 30207c478bd9Sstevel@tonic-gate cyclic_unbind_cpupart(id); 30217c478bd9Sstevel@tonic-gate 30227c478bd9Sstevel@tonic-gate /* 30237c478bd9Sstevel@tonic-gate * Now reload the flags field, asserting that if we are CPU bound, 30247c478bd9Sstevel@tonic-gate * the CPU was specified (and likewise, if we are partition bound, 30257c478bd9Sstevel@tonic-gate * the partition was specified). 30267c478bd9Sstevel@tonic-gate */ 30277c478bd9Sstevel@tonic-gate cpu = idp->cyi_cpu; 30287c478bd9Sstevel@tonic-gate c = cpu->cyp_cpu; 30297c478bd9Sstevel@tonic-gate flags = cpu->cyp_cyclics[idp->cyi_ndx].cy_flags; 30307c478bd9Sstevel@tonic-gate ASSERT(!(flags & CYF_CPU_BOUND) || c == d); 30317c478bd9Sstevel@tonic-gate ASSERT(!(flags & CYF_PART_BOUND) || c->cpu_part == part); 30327c478bd9Sstevel@tonic-gate 30337c478bd9Sstevel@tonic-gate if (!(flags & CYF_CPU_BOUND) && d != NULL) 30347c478bd9Sstevel@tonic-gate cyclic_bind_cpu(id, d); 30357c478bd9Sstevel@tonic-gate 30367c478bd9Sstevel@tonic-gate if (!(flags & CYF_PART_BOUND) && part != NULL) 30377c478bd9Sstevel@tonic-gate cyclic_bind_cpupart(id, part); 30387c478bd9Sstevel@tonic-gate } 30397c478bd9Sstevel@tonic-gate 3040*87a18d3fSMadhavan Venkataraman int 3041*87a18d3fSMadhavan Venkataraman cyclic_reprogram(cyclic_id_t id, hrtime_t expiration) 3042*87a18d3fSMadhavan Venkataraman { 3043*87a18d3fSMadhavan Venkataraman cyc_id_t *idp = (cyc_id_t *)id; 3044*87a18d3fSMadhavan Venkataraman cyc_cpu_t *cpu; 3045*87a18d3fSMadhavan Venkataraman cyc_omni_cpu_t *ocpu; 3046*87a18d3fSMadhavan Venkataraman cyc_index_t ndx; 3047*87a18d3fSMadhavan Venkataraman 3048*87a18d3fSMadhavan Venkataraman ASSERT(expiration > 0); 3049*87a18d3fSMadhavan Venkataraman 3050*87a18d3fSMadhavan Venkataraman CYC_PTRACE("reprog", idp, idp->cyi_cpu); 3051*87a18d3fSMadhavan Venkataraman 3052*87a18d3fSMadhavan Venkataraman kpreempt_disable(); 3053*87a18d3fSMadhavan Venkataraman 3054*87a18d3fSMadhavan Venkataraman /* 3055*87a18d3fSMadhavan Venkataraman * Prevent the cyclic from moving or disappearing while we reprogram. 3056*87a18d3fSMadhavan Venkataraman */ 3057*87a18d3fSMadhavan Venkataraman rw_enter(&idp->cyi_lock, RW_READER); 3058*87a18d3fSMadhavan Venkataraman 3059*87a18d3fSMadhavan Venkataraman if (idp->cyi_cpu == NULL) { 3060*87a18d3fSMadhavan Venkataraman ASSERT(curthread->t_preempt > 0); 3061*87a18d3fSMadhavan Venkataraman cpu = CPU->cpu_cyclic; 3062*87a18d3fSMadhavan Venkataraman 3063*87a18d3fSMadhavan Venkataraman /* 3064*87a18d3fSMadhavan Venkataraman * For an omni cyclic, we reprogram the cyclic corresponding 3065*87a18d3fSMadhavan Venkataraman * to the current CPU. Look for it in the list. 3066*87a18d3fSMadhavan Venkataraman */ 3067*87a18d3fSMadhavan Venkataraman ocpu = idp->cyi_omni_list; 3068*87a18d3fSMadhavan Venkataraman while (ocpu != NULL) { 3069*87a18d3fSMadhavan Venkataraman if (ocpu->cyo_cpu == cpu) 3070*87a18d3fSMadhavan Venkataraman break; 3071*87a18d3fSMadhavan Venkataraman ocpu = ocpu->cyo_next; 3072*87a18d3fSMadhavan Venkataraman } 3073*87a18d3fSMadhavan Venkataraman 3074*87a18d3fSMadhavan Venkataraman if (ocpu == NULL) { 3075*87a18d3fSMadhavan Venkataraman /* 3076*87a18d3fSMadhavan Venkataraman * Didn't find it. This means that CPU offline 3077*87a18d3fSMadhavan Venkataraman * must have removed it racing with us. So, 3078*87a18d3fSMadhavan Venkataraman * nothing to do. 3079*87a18d3fSMadhavan Venkataraman */ 3080*87a18d3fSMadhavan Venkataraman rw_exit(&idp->cyi_lock); 3081*87a18d3fSMadhavan Venkataraman 3082*87a18d3fSMadhavan Venkataraman kpreempt_enable(); 3083*87a18d3fSMadhavan Venkataraman 3084*87a18d3fSMadhavan Venkataraman return (0); 3085*87a18d3fSMadhavan Venkataraman } 3086*87a18d3fSMadhavan Venkataraman ndx = ocpu->cyo_ndx; 3087*87a18d3fSMadhavan Venkataraman } else { 3088*87a18d3fSMadhavan Venkataraman cpu = idp->cyi_cpu; 3089*87a18d3fSMadhavan Venkataraman ndx = idp->cyi_ndx; 3090*87a18d3fSMadhavan Venkataraman } 3091*87a18d3fSMadhavan Venkataraman 3092*87a18d3fSMadhavan Venkataraman if (cpu->cyp_cpu == CPU) 3093*87a18d3fSMadhavan Venkataraman cyclic_reprogram_cyclic(cpu, ndx, expiration); 3094*87a18d3fSMadhavan Venkataraman else 3095*87a18d3fSMadhavan Venkataraman cyclic_reprogram_here(cpu, ndx, expiration); 3096*87a18d3fSMadhavan Venkataraman 3097*87a18d3fSMadhavan Venkataraman /* 3098*87a18d3fSMadhavan Venkataraman * Allow the cyclic to be moved or removed. 3099*87a18d3fSMadhavan Venkataraman */ 3100*87a18d3fSMadhavan Venkataraman rw_exit(&idp->cyi_lock); 3101*87a18d3fSMadhavan Venkataraman 3102*87a18d3fSMadhavan Venkataraman kpreempt_enable(); 3103*87a18d3fSMadhavan Venkataraman 3104*87a18d3fSMadhavan Venkataraman return (1); 3105*87a18d3fSMadhavan Venkataraman } 3106*87a18d3fSMadhavan Venkataraman 31077c478bd9Sstevel@tonic-gate hrtime_t 31087c478bd9Sstevel@tonic-gate cyclic_getres() 31097c478bd9Sstevel@tonic-gate { 31107c478bd9Sstevel@tonic-gate return (cyclic_resolution); 31117c478bd9Sstevel@tonic-gate } 31127c478bd9Sstevel@tonic-gate 31137c478bd9Sstevel@tonic-gate void 31147c478bd9Sstevel@tonic-gate cyclic_init(cyc_backend_t *be, hrtime_t resolution) 31157c478bd9Sstevel@tonic-gate { 31167c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 31177c478bd9Sstevel@tonic-gate 31187c478bd9Sstevel@tonic-gate CYC_PTRACE("init", be, resolution); 31197c478bd9Sstevel@tonic-gate cyclic_resolution = resolution; 31207c478bd9Sstevel@tonic-gate 31217c478bd9Sstevel@tonic-gate /* 31227c478bd9Sstevel@tonic-gate * Copy the passed cyc_backend into the backend template. This must 31237c478bd9Sstevel@tonic-gate * be done before the CPU can be configured. 31247c478bd9Sstevel@tonic-gate */ 31257c478bd9Sstevel@tonic-gate bcopy(be, &cyclic_backend, sizeof (cyc_backend_t)); 31267c478bd9Sstevel@tonic-gate 31277c478bd9Sstevel@tonic-gate /* 31287c478bd9Sstevel@tonic-gate * It's safe to look at the "CPU" pointer without disabling kernel 31297c478bd9Sstevel@tonic-gate * preemption; cyclic_init() is called only during startup by the 31307c478bd9Sstevel@tonic-gate * cyclic backend. 31317c478bd9Sstevel@tonic-gate */ 31327c478bd9Sstevel@tonic-gate cyclic_configure(CPU); 31337c478bd9Sstevel@tonic-gate cyclic_online(CPU); 31347c478bd9Sstevel@tonic-gate } 31357c478bd9Sstevel@tonic-gate 31367c478bd9Sstevel@tonic-gate /* 31377c478bd9Sstevel@tonic-gate * It is assumed that cyclic_mp_init() is called some time after cyclic 31387c478bd9Sstevel@tonic-gate * init (and therefore, after cpu0 has been initialized). We grab cpu_lock, 31397c478bd9Sstevel@tonic-gate * find the already initialized CPU, and initialize every other CPU with the 31407c478bd9Sstevel@tonic-gate * same backend. Finally, we register a cpu_setup function. 31417c478bd9Sstevel@tonic-gate */ 31427c478bd9Sstevel@tonic-gate void 31437c478bd9Sstevel@tonic-gate cyclic_mp_init() 31447c478bd9Sstevel@tonic-gate { 31457c478bd9Sstevel@tonic-gate cpu_t *c; 31467c478bd9Sstevel@tonic-gate 31477c478bd9Sstevel@tonic-gate mutex_enter(&cpu_lock); 31487c478bd9Sstevel@tonic-gate 31497c478bd9Sstevel@tonic-gate c = cpu_list; 31507c478bd9Sstevel@tonic-gate do { 31517c478bd9Sstevel@tonic-gate if (c->cpu_cyclic == NULL) { 31527c478bd9Sstevel@tonic-gate cyclic_configure(c); 31537c478bd9Sstevel@tonic-gate cyclic_online(c); 31547c478bd9Sstevel@tonic-gate } 31557c478bd9Sstevel@tonic-gate } while ((c = c->cpu_next) != cpu_list); 31567c478bd9Sstevel@tonic-gate 31577c478bd9Sstevel@tonic-gate register_cpu_setup_func((cpu_setup_func_t *)cyclic_cpu_setup, NULL); 31587c478bd9Sstevel@tonic-gate mutex_exit(&cpu_lock); 31597c478bd9Sstevel@tonic-gate } 31607c478bd9Sstevel@tonic-gate 31617c478bd9Sstevel@tonic-gate /* 31627c478bd9Sstevel@tonic-gate * int cyclic_juggle(cpu_t *) 31637c478bd9Sstevel@tonic-gate * 31647c478bd9Sstevel@tonic-gate * Overview 31657c478bd9Sstevel@tonic-gate * 31667c478bd9Sstevel@tonic-gate * cyclic_juggle() juggles as many cyclics as possible away from the 31677c478bd9Sstevel@tonic-gate * specified CPU; all remaining cyclics on the CPU will either be CPU- 31687c478bd9Sstevel@tonic-gate * or partition-bound. 31697c478bd9Sstevel@tonic-gate * 31707c478bd9Sstevel@tonic-gate * Arguments and notes 31717c478bd9Sstevel@tonic-gate * 31727c478bd9Sstevel@tonic-gate * The only argument to cyclic_juggle() is the CPU from which cyclics 31737c478bd9Sstevel@tonic-gate * should be juggled. CPU-bound cyclics are never juggled; partition-bound 31747c478bd9Sstevel@tonic-gate * cyclics are only juggled if the specified CPU is in the P_NOINTR state 31757c478bd9Sstevel@tonic-gate * and there exists a P_ONLINE CPU in the partition. The cyclic subsystem 31767c478bd9Sstevel@tonic-gate * assures that a cyclic will never fire late or spuriously, even while 31777c478bd9Sstevel@tonic-gate * being juggled. 31787c478bd9Sstevel@tonic-gate * 31797c478bd9Sstevel@tonic-gate * Return value 31807c478bd9Sstevel@tonic-gate * 31817c478bd9Sstevel@tonic-gate * cyclic_juggle() returns a non-zero value if all cyclics were able to 31827c478bd9Sstevel@tonic-gate * be juggled away from the CPU, and zero if one or more cyclics could 31837c478bd9Sstevel@tonic-gate * not be juggled away. 31847c478bd9Sstevel@tonic-gate * 31857c478bd9Sstevel@tonic-gate * Caller's context 31867c478bd9Sstevel@tonic-gate * 31877c478bd9Sstevel@tonic-gate * cpu_lock must be held by the caller, and the caller must not be in 31887c478bd9Sstevel@tonic-gate * interrupt context. The caller may not hold any locks which are also 31897c478bd9Sstevel@tonic-gate * grabbed by any cyclic handler. While cyclic_juggle() _may_ be called 31907c478bd9Sstevel@tonic-gate * in any context satisfying these constraints, it _must_ be called 31917c478bd9Sstevel@tonic-gate * immediately after clearing CPU_ENABLE (i.e. before dropping cpu_lock). 31927c478bd9Sstevel@tonic-gate * Failure to do so could result in an assertion failure in the cyclic 31937c478bd9Sstevel@tonic-gate * subsystem. 31947c478bd9Sstevel@tonic-gate */ 31957c478bd9Sstevel@tonic-gate int 31967c478bd9Sstevel@tonic-gate cyclic_juggle(cpu_t *c) 31977c478bd9Sstevel@tonic-gate { 31987c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = c->cpu_cyclic; 31997c478bd9Sstevel@tonic-gate cyc_id_t *idp; 32007c478bd9Sstevel@tonic-gate int all_juggled = 1; 32017c478bd9Sstevel@tonic-gate 32027c478bd9Sstevel@tonic-gate CYC_PTRACE1("juggle", c); 32037c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 32047c478bd9Sstevel@tonic-gate 32057c478bd9Sstevel@tonic-gate /* 32067c478bd9Sstevel@tonic-gate * We'll go through each cyclic on the CPU, attempting to juggle 32077c478bd9Sstevel@tonic-gate * each one elsewhere. 32087c478bd9Sstevel@tonic-gate */ 32097c478bd9Sstevel@tonic-gate for (idp = cyclic_id_head; idp != NULL; idp = idp->cyi_next) { 32107c478bd9Sstevel@tonic-gate if (idp->cyi_cpu != cpu) 32117c478bd9Sstevel@tonic-gate continue; 32127c478bd9Sstevel@tonic-gate 32137c478bd9Sstevel@tonic-gate if (cyclic_juggle_one(idp) == 0) { 32147c478bd9Sstevel@tonic-gate all_juggled = 0; 32157c478bd9Sstevel@tonic-gate continue; 32167c478bd9Sstevel@tonic-gate } 32177c478bd9Sstevel@tonic-gate 32187c478bd9Sstevel@tonic-gate ASSERT(idp->cyi_cpu != cpu); 32197c478bd9Sstevel@tonic-gate } 32207c478bd9Sstevel@tonic-gate 32217c478bd9Sstevel@tonic-gate return (all_juggled); 32227c478bd9Sstevel@tonic-gate } 32237c478bd9Sstevel@tonic-gate 32247c478bd9Sstevel@tonic-gate /* 32257c478bd9Sstevel@tonic-gate * int cyclic_offline(cpu_t *) 32267c478bd9Sstevel@tonic-gate * 32277c478bd9Sstevel@tonic-gate * Overview 32287c478bd9Sstevel@tonic-gate * 32297c478bd9Sstevel@tonic-gate * cyclic_offline() offlines the cyclic subsystem on the specified CPU. 32307c478bd9Sstevel@tonic-gate * 32317c478bd9Sstevel@tonic-gate * Arguments and notes 32327c478bd9Sstevel@tonic-gate * 32337c478bd9Sstevel@tonic-gate * The only argument to cyclic_offline() is a CPU to offline. 32347c478bd9Sstevel@tonic-gate * cyclic_offline() will attempt to juggle cyclics away from the specified 32357c478bd9Sstevel@tonic-gate * CPU. 32367c478bd9Sstevel@tonic-gate * 32377c478bd9Sstevel@tonic-gate * Return value 32387c478bd9Sstevel@tonic-gate * 32397c478bd9Sstevel@tonic-gate * cyclic_offline() returns 1 if all cyclics on the CPU were juggled away 32407c478bd9Sstevel@tonic-gate * and the cyclic subsystem on the CPU was successfully offlines. 32417c478bd9Sstevel@tonic-gate * cyclic_offline returns 0 if some cyclics remain, blocking the cyclic 32427c478bd9Sstevel@tonic-gate * offline operation. All remaining cyclics on the CPU will either be 32437c478bd9Sstevel@tonic-gate * CPU- or partition-bound. 32447c478bd9Sstevel@tonic-gate * 32457c478bd9Sstevel@tonic-gate * See the "Arguments and notes" of cyclic_juggle(), below, for more detail 32467c478bd9Sstevel@tonic-gate * on cyclic juggling. 32477c478bd9Sstevel@tonic-gate * 32487c478bd9Sstevel@tonic-gate * Caller's context 32497c478bd9Sstevel@tonic-gate * 32507c478bd9Sstevel@tonic-gate * The only caller of cyclic_offline() should be the processor management 32517c478bd9Sstevel@tonic-gate * subsystem. It is expected that the caller of cyclic_offline() will 32527c478bd9Sstevel@tonic-gate * offline the CPU immediately after cyclic_offline() returns success (i.e. 32537c478bd9Sstevel@tonic-gate * before dropping cpu_lock). Moreover, it is expected that the caller will 32547c478bd9Sstevel@tonic-gate * fail the CPU offline operation if cyclic_offline() returns failure. 32557c478bd9Sstevel@tonic-gate */ 32567c478bd9Sstevel@tonic-gate int 32577c478bd9Sstevel@tonic-gate cyclic_offline(cpu_t *c) 32587c478bd9Sstevel@tonic-gate { 32597c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = c->cpu_cyclic; 32607c478bd9Sstevel@tonic-gate cyc_id_t *idp; 32617c478bd9Sstevel@tonic-gate 32627c478bd9Sstevel@tonic-gate CYC_PTRACE1("offline", cpu); 32637c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 32647c478bd9Sstevel@tonic-gate 32657c478bd9Sstevel@tonic-gate if (!cyclic_juggle(c)) 32667c478bd9Sstevel@tonic-gate return (0); 32677c478bd9Sstevel@tonic-gate 32687c478bd9Sstevel@tonic-gate /* 32697c478bd9Sstevel@tonic-gate * This CPU is headed offline; we need to now stop omnipresent 32707c478bd9Sstevel@tonic-gate * cyclic firing on this CPU. 32717c478bd9Sstevel@tonic-gate */ 32727c478bd9Sstevel@tonic-gate for (idp = cyclic_id_head; idp != NULL; idp = idp->cyi_next) { 32737c478bd9Sstevel@tonic-gate if (idp->cyi_cpu != NULL) 32747c478bd9Sstevel@tonic-gate continue; 32757c478bd9Sstevel@tonic-gate 32767c478bd9Sstevel@tonic-gate /* 32777c478bd9Sstevel@tonic-gate * We cannot possibly be offlining the last CPU; cyi_omni_list 32787c478bd9Sstevel@tonic-gate * must be non-NULL. 32797c478bd9Sstevel@tonic-gate */ 32807c478bd9Sstevel@tonic-gate ASSERT(idp->cyi_omni_list != NULL); 32817c478bd9Sstevel@tonic-gate cyclic_omni_stop(idp, cpu); 32827c478bd9Sstevel@tonic-gate } 32837c478bd9Sstevel@tonic-gate 32847c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_ONLINE); 32857c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_OFFLINE; 32867c478bd9Sstevel@tonic-gate 32877c478bd9Sstevel@tonic-gate return (1); 32887c478bd9Sstevel@tonic-gate } 32897c478bd9Sstevel@tonic-gate 32907c478bd9Sstevel@tonic-gate /* 32917c478bd9Sstevel@tonic-gate * void cyclic_online(cpu_t *) 32927c478bd9Sstevel@tonic-gate * 32937c478bd9Sstevel@tonic-gate * Overview 32947c478bd9Sstevel@tonic-gate * 32957c478bd9Sstevel@tonic-gate * cyclic_online() onlines a CPU previously offlined with cyclic_offline(). 32967c478bd9Sstevel@tonic-gate * 32977c478bd9Sstevel@tonic-gate * Arguments and notes 32987c478bd9Sstevel@tonic-gate * 32997c478bd9Sstevel@tonic-gate * cyclic_online()'s only argument is a CPU to online. The specified 33007c478bd9Sstevel@tonic-gate * CPU must have been previously offlined with cyclic_offline(). After 33017c478bd9Sstevel@tonic-gate * cyclic_online() returns, the specified CPU will be eligible to execute 33027c478bd9Sstevel@tonic-gate * cyclics. 33037c478bd9Sstevel@tonic-gate * 33047c478bd9Sstevel@tonic-gate * Return value 33057c478bd9Sstevel@tonic-gate * 33067c478bd9Sstevel@tonic-gate * None; cyclic_online() always succeeds. 33077c478bd9Sstevel@tonic-gate * 33087c478bd9Sstevel@tonic-gate * Caller's context 33097c478bd9Sstevel@tonic-gate * 33107c478bd9Sstevel@tonic-gate * cyclic_online() should only be called by the processor management 33117c478bd9Sstevel@tonic-gate * subsystem; cpu_lock must be held. 33127c478bd9Sstevel@tonic-gate */ 33137c478bd9Sstevel@tonic-gate void 33147c478bd9Sstevel@tonic-gate cyclic_online(cpu_t *c) 33157c478bd9Sstevel@tonic-gate { 33167c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = c->cpu_cyclic; 33177c478bd9Sstevel@tonic-gate cyc_id_t *idp; 33187c478bd9Sstevel@tonic-gate 33197c478bd9Sstevel@tonic-gate CYC_PTRACE1("online", cpu); 33207c478bd9Sstevel@tonic-gate ASSERT(c->cpu_flags & CPU_ENABLE); 33217c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 33227c478bd9Sstevel@tonic-gate ASSERT(cpu->cyp_state == CYS_OFFLINE); 33237c478bd9Sstevel@tonic-gate 33247c478bd9Sstevel@tonic-gate cpu->cyp_state = CYS_ONLINE; 33257c478bd9Sstevel@tonic-gate 33267c478bd9Sstevel@tonic-gate /* 33277c478bd9Sstevel@tonic-gate * Now that this CPU is open for business, we need to start firing 33287c478bd9Sstevel@tonic-gate * all omnipresent cyclics on it. 33297c478bd9Sstevel@tonic-gate */ 33307c478bd9Sstevel@tonic-gate for (idp = cyclic_id_head; idp != NULL; idp = idp->cyi_next) { 33317c478bd9Sstevel@tonic-gate if (idp->cyi_cpu != NULL) 33327c478bd9Sstevel@tonic-gate continue; 33337c478bd9Sstevel@tonic-gate 33347c478bd9Sstevel@tonic-gate cyclic_omni_start(idp, cpu); 33357c478bd9Sstevel@tonic-gate } 33367c478bd9Sstevel@tonic-gate } 33377c478bd9Sstevel@tonic-gate 33387c478bd9Sstevel@tonic-gate /* 33397c478bd9Sstevel@tonic-gate * void cyclic_move_in(cpu_t *) 33407c478bd9Sstevel@tonic-gate * 33417c478bd9Sstevel@tonic-gate * Overview 33427c478bd9Sstevel@tonic-gate * 33437c478bd9Sstevel@tonic-gate * cyclic_move_in() is called by the CPU partition code immediately after 33447c478bd9Sstevel@tonic-gate * the specified CPU has moved into a new partition. 33457c478bd9Sstevel@tonic-gate * 33467c478bd9Sstevel@tonic-gate * Arguments and notes 33477c478bd9Sstevel@tonic-gate * 33487c478bd9Sstevel@tonic-gate * The only argument to cyclic_move_in() is a CPU which has moved into a 33497c478bd9Sstevel@tonic-gate * new partition. If the specified CPU is P_ONLINE, and every other 33507c478bd9Sstevel@tonic-gate * CPU in the specified CPU's new partition is P_NOINTR, cyclic_move_in() 33517c478bd9Sstevel@tonic-gate * will juggle all partition-bound, CPU-unbound cyclics to the specified 33527c478bd9Sstevel@tonic-gate * CPU. 33537c478bd9Sstevel@tonic-gate * 33547c478bd9Sstevel@tonic-gate * Return value 33557c478bd9Sstevel@tonic-gate * 33567c478bd9Sstevel@tonic-gate * None; cyclic_move_in() always succeeds. 33577c478bd9Sstevel@tonic-gate * 33587c478bd9Sstevel@tonic-gate * Caller's context 33597c478bd9Sstevel@tonic-gate * 33607c478bd9Sstevel@tonic-gate * cyclic_move_in() should _only_ be called immediately after a CPU has 33617c478bd9Sstevel@tonic-gate * moved into a new partition, with cpu_lock held. As with other calls 33627c478bd9Sstevel@tonic-gate * into the cyclic subsystem, no lock may be held which is also grabbed 33637c478bd9Sstevel@tonic-gate * by any cyclic handler. 33647c478bd9Sstevel@tonic-gate */ 33657c478bd9Sstevel@tonic-gate void 33667c478bd9Sstevel@tonic-gate cyclic_move_in(cpu_t *d) 33677c478bd9Sstevel@tonic-gate { 33687c478bd9Sstevel@tonic-gate cyc_id_t *idp; 33697c478bd9Sstevel@tonic-gate cyc_cpu_t *dest = d->cpu_cyclic; 33707c478bd9Sstevel@tonic-gate cyclic_t *cyclic; 33717c478bd9Sstevel@tonic-gate cpupart_t *part = d->cpu_part; 33727c478bd9Sstevel@tonic-gate 33737c478bd9Sstevel@tonic-gate CYC_PTRACE("move-in", dest, part); 33747c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 33757c478bd9Sstevel@tonic-gate 33767c478bd9Sstevel@tonic-gate /* 33777c478bd9Sstevel@tonic-gate * Look for CYF_PART_BOUND cyclics in the new partition. If 33787c478bd9Sstevel@tonic-gate * we find one, check to see if it is currently on a CPU which has 33797c478bd9Sstevel@tonic-gate * interrupts disabled. If it is (and if this CPU currently has 33807c478bd9Sstevel@tonic-gate * interrupts enabled), we'll juggle those cyclics over here. 33817c478bd9Sstevel@tonic-gate */ 33827c478bd9Sstevel@tonic-gate if (!(d->cpu_flags & CPU_ENABLE)) { 33837c478bd9Sstevel@tonic-gate CYC_PTRACE1("move-in-none", dest); 33847c478bd9Sstevel@tonic-gate return; 33857c478bd9Sstevel@tonic-gate } 33867c478bd9Sstevel@tonic-gate 33877c478bd9Sstevel@tonic-gate for (idp = cyclic_id_head; idp != NULL; idp = idp->cyi_next) { 33887c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = idp->cyi_cpu; 33897c478bd9Sstevel@tonic-gate cpu_t *c; 33907c478bd9Sstevel@tonic-gate 33917c478bd9Sstevel@tonic-gate /* 33927c478bd9Sstevel@tonic-gate * Omnipresent cyclics are exempt from juggling. 33937c478bd9Sstevel@tonic-gate */ 33947c478bd9Sstevel@tonic-gate if (cpu == NULL) 33957c478bd9Sstevel@tonic-gate continue; 33967c478bd9Sstevel@tonic-gate 33977c478bd9Sstevel@tonic-gate c = cpu->cyp_cpu; 33987c478bd9Sstevel@tonic-gate 33997c478bd9Sstevel@tonic-gate if (c->cpu_part != part || (c->cpu_flags & CPU_ENABLE)) 34007c478bd9Sstevel@tonic-gate continue; 34017c478bd9Sstevel@tonic-gate 34027c478bd9Sstevel@tonic-gate cyclic = &cpu->cyp_cyclics[idp->cyi_ndx]; 34037c478bd9Sstevel@tonic-gate 34047c478bd9Sstevel@tonic-gate if (cyclic->cy_flags & CYF_CPU_BOUND) 34057c478bd9Sstevel@tonic-gate continue; 34067c478bd9Sstevel@tonic-gate 34077c478bd9Sstevel@tonic-gate /* 34087c478bd9Sstevel@tonic-gate * We know that this cyclic is bound to its processor set 34097c478bd9Sstevel@tonic-gate * (otherwise, it would not be on a CPU with interrupts 34107c478bd9Sstevel@tonic-gate * disabled); juggle it to our CPU. 34117c478bd9Sstevel@tonic-gate */ 34127c478bd9Sstevel@tonic-gate ASSERT(cyclic->cy_flags & CYF_PART_BOUND); 34137c478bd9Sstevel@tonic-gate cyclic_juggle_one_to(idp, dest); 34147c478bd9Sstevel@tonic-gate } 34157c478bd9Sstevel@tonic-gate 34167c478bd9Sstevel@tonic-gate CYC_PTRACE1("move-in-done", dest); 34177c478bd9Sstevel@tonic-gate } 34187c478bd9Sstevel@tonic-gate 34197c478bd9Sstevel@tonic-gate /* 34207c478bd9Sstevel@tonic-gate * int cyclic_move_out(cpu_t *) 34217c478bd9Sstevel@tonic-gate * 34227c478bd9Sstevel@tonic-gate * Overview 34237c478bd9Sstevel@tonic-gate * 34247c478bd9Sstevel@tonic-gate * cyclic_move_out() is called by the CPU partition code immediately before 34257c478bd9Sstevel@tonic-gate * the specified CPU is to move out of its partition. 34267c478bd9Sstevel@tonic-gate * 34277c478bd9Sstevel@tonic-gate * Arguments and notes 34287c478bd9Sstevel@tonic-gate * 34297c478bd9Sstevel@tonic-gate * The only argument to cyclic_move_out() is a CPU which is to move out of 34307c478bd9Sstevel@tonic-gate * its partition. 34317c478bd9Sstevel@tonic-gate * 34327c478bd9Sstevel@tonic-gate * cyclic_move_out() will attempt to juggle away all partition-bound 34337c478bd9Sstevel@tonic-gate * cyclics. If the specified CPU is the last CPU in a partition with 34347c478bd9Sstevel@tonic-gate * partition-bound cyclics, cyclic_move_out() will fail. If there exists 34357c478bd9Sstevel@tonic-gate * a partition-bound cyclic which is CPU-bound to the specified CPU, 34367c478bd9Sstevel@tonic-gate * cyclic_move_out() will fail. 34377c478bd9Sstevel@tonic-gate * 34387c478bd9Sstevel@tonic-gate * Note that cyclic_move_out() will _only_ attempt to juggle away 34397c478bd9Sstevel@tonic-gate * partition-bound cyclics; CPU-bound cyclics which are not partition-bound 34407c478bd9Sstevel@tonic-gate * and unbound cyclics are not affected by changing the partition 34417c478bd9Sstevel@tonic-gate * affiliation of the CPU. 34427c478bd9Sstevel@tonic-gate * 34437c478bd9Sstevel@tonic-gate * Return value 34447c478bd9Sstevel@tonic-gate * 34457c478bd9Sstevel@tonic-gate * cyclic_move_out() returns 1 if all partition-bound cyclics on the CPU 34467c478bd9Sstevel@tonic-gate * were juggled away; 0 if some cyclics remain. 34477c478bd9Sstevel@tonic-gate * 34487c478bd9Sstevel@tonic-gate * Caller's context 34497c478bd9Sstevel@tonic-gate * 34507c478bd9Sstevel@tonic-gate * cyclic_move_out() should _only_ be called immediately before a CPU has 34517c478bd9Sstevel@tonic-gate * moved out of its partition, with cpu_lock held. It is expected that 34527c478bd9Sstevel@tonic-gate * the caller of cyclic_move_out() will change the processor set affiliation 34537c478bd9Sstevel@tonic-gate * of the specified CPU immediately after cyclic_move_out() returns 34547c478bd9Sstevel@tonic-gate * success (i.e. before dropping cpu_lock). Moreover, it is expected that 34557c478bd9Sstevel@tonic-gate * the caller will fail the CPU repartitioning operation if cyclic_move_out() 34567c478bd9Sstevel@tonic-gate * returns failure. As with other calls into the cyclic subsystem, no lock 34577c478bd9Sstevel@tonic-gate * may be held which is also grabbed by any cyclic handler. 34587c478bd9Sstevel@tonic-gate */ 34597c478bd9Sstevel@tonic-gate int 34607c478bd9Sstevel@tonic-gate cyclic_move_out(cpu_t *c) 34617c478bd9Sstevel@tonic-gate { 34627c478bd9Sstevel@tonic-gate cyc_id_t *idp; 34637c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu = c->cpu_cyclic, *dest; 34647c478bd9Sstevel@tonic-gate cyclic_t *cyclic, *cyclics = cpu->cyp_cyclics; 34657c478bd9Sstevel@tonic-gate cpupart_t *part = c->cpu_part; 34667c478bd9Sstevel@tonic-gate 34677c478bd9Sstevel@tonic-gate CYC_PTRACE1("move-out", cpu); 34687c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 34697c478bd9Sstevel@tonic-gate 34707c478bd9Sstevel@tonic-gate /* 34717c478bd9Sstevel@tonic-gate * If there are any CYF_PART_BOUND cyclics on this CPU, we need 34727c478bd9Sstevel@tonic-gate * to try to juggle them away. 34737c478bd9Sstevel@tonic-gate */ 34747c478bd9Sstevel@tonic-gate for (idp = cyclic_id_head; idp != NULL; idp = idp->cyi_next) { 34757c478bd9Sstevel@tonic-gate 34767c478bd9Sstevel@tonic-gate if (idp->cyi_cpu != cpu) 34777c478bd9Sstevel@tonic-gate continue; 34787c478bd9Sstevel@tonic-gate 34797c478bd9Sstevel@tonic-gate cyclic = &cyclics[idp->cyi_ndx]; 34807c478bd9Sstevel@tonic-gate 34817c478bd9Sstevel@tonic-gate if (!(cyclic->cy_flags & CYF_PART_BOUND)) 34827c478bd9Sstevel@tonic-gate continue; 34837c478bd9Sstevel@tonic-gate 34847c478bd9Sstevel@tonic-gate dest = cyclic_pick_cpu(part, c, c, cyclic->cy_flags); 34857c478bd9Sstevel@tonic-gate 34867c478bd9Sstevel@tonic-gate if (dest == NULL) { 34877c478bd9Sstevel@tonic-gate /* 34887c478bd9Sstevel@tonic-gate * We can't juggle this cyclic; we need to return 34897c478bd9Sstevel@tonic-gate * failure (we won't bother trying to juggle away 34907c478bd9Sstevel@tonic-gate * other cyclics). 34917c478bd9Sstevel@tonic-gate */ 34927c478bd9Sstevel@tonic-gate CYC_PTRACE("move-out-fail", cpu, idp); 34937c478bd9Sstevel@tonic-gate return (0); 34947c478bd9Sstevel@tonic-gate } 34957c478bd9Sstevel@tonic-gate cyclic_juggle_one_to(idp, dest); 34967c478bd9Sstevel@tonic-gate } 34977c478bd9Sstevel@tonic-gate 34987c478bd9Sstevel@tonic-gate CYC_PTRACE1("move-out-done", cpu); 34997c478bd9Sstevel@tonic-gate return (1); 35007c478bd9Sstevel@tonic-gate } 35017c478bd9Sstevel@tonic-gate 35027c478bd9Sstevel@tonic-gate /* 35037c478bd9Sstevel@tonic-gate * void cyclic_suspend() 35047c478bd9Sstevel@tonic-gate * 35057c478bd9Sstevel@tonic-gate * Overview 35067c478bd9Sstevel@tonic-gate * 35077c478bd9Sstevel@tonic-gate * cyclic_suspend() suspends all cyclic activity throughout the cyclic 35087c478bd9Sstevel@tonic-gate * subsystem. It should be called only by subsystems which are attempting 35097c478bd9Sstevel@tonic-gate * to suspend the entire system (e.g. checkpoint/resume, dynamic 35107c478bd9Sstevel@tonic-gate * reconfiguration). 35117c478bd9Sstevel@tonic-gate * 35127c478bd9Sstevel@tonic-gate * Arguments and notes 35137c478bd9Sstevel@tonic-gate * 35147c478bd9Sstevel@tonic-gate * cyclic_suspend() takes no arguments. Each CPU with an active cyclic 35157c478bd9Sstevel@tonic-gate * disables its backend (offline CPUs disable their backends as part of 35167c478bd9Sstevel@tonic-gate * the cyclic_offline() operation), thereby disabling future CY_HIGH_LEVEL 35177c478bd9Sstevel@tonic-gate * interrupts. 35187c478bd9Sstevel@tonic-gate * 35197c478bd9Sstevel@tonic-gate * Note that disabling CY_HIGH_LEVEL interrupts does not completely preclude 35207c478bd9Sstevel@tonic-gate * cyclic handlers from being called after cyclic_suspend() returns: if a 35217c478bd9Sstevel@tonic-gate * CY_LOCK_LEVEL or CY_LOW_LEVEL interrupt thread was blocked at the time 35227c478bd9Sstevel@tonic-gate * of cyclic_suspend(), cyclic handlers at its level may continue to be 35237c478bd9Sstevel@tonic-gate * called after the interrupt thread becomes unblocked. The 35247c478bd9Sstevel@tonic-gate * post-cyclic_suspend() activity is bounded by the pend count on all 35257c478bd9Sstevel@tonic-gate * cyclics at the time of cyclic_suspend(). Callers concerned with more 35267c478bd9Sstevel@tonic-gate * than simply disabling future CY_HIGH_LEVEL interrupts must check for 35277c478bd9Sstevel@tonic-gate * this condition. 35287c478bd9Sstevel@tonic-gate * 35297c478bd9Sstevel@tonic-gate * On most platforms, timestamps from gethrtime() and gethrestime() are not 35307c478bd9Sstevel@tonic-gate * guaranteed to monotonically increase between cyclic_suspend() and 35317c478bd9Sstevel@tonic-gate * cyclic_resume(). However, timestamps are guaranteed to monotonically 35327c478bd9Sstevel@tonic-gate * increase across the entire cyclic_suspend()/cyclic_resume() operation. 35337c478bd9Sstevel@tonic-gate * That is, every timestamp obtained before cyclic_suspend() will be less 35347c478bd9Sstevel@tonic-gate * than every timestamp obtained after cyclic_resume(). 35357c478bd9Sstevel@tonic-gate * 35367c478bd9Sstevel@tonic-gate * Return value 35377c478bd9Sstevel@tonic-gate * 35387c478bd9Sstevel@tonic-gate * None; cyclic_suspend() always succeeds. 35397c478bd9Sstevel@tonic-gate * 35407c478bd9Sstevel@tonic-gate * Caller's context 35417c478bd9Sstevel@tonic-gate * 35427c478bd9Sstevel@tonic-gate * The cyclic subsystem must be configured on every valid CPU; 35437c478bd9Sstevel@tonic-gate * cyclic_suspend() may not be called during boot or during dynamic 35447c478bd9Sstevel@tonic-gate * reconfiguration. Additionally, cpu_lock must be held, and the caller 35457c478bd9Sstevel@tonic-gate * cannot be in high-level interrupt context. However, unlike most other 35467c478bd9Sstevel@tonic-gate * cyclic entry points, cyclic_suspend() may be called with locks held 35477c478bd9Sstevel@tonic-gate * which are also acquired by CY_LOCK_LEVEL or CY_LOW_LEVEL cyclic 35487c478bd9Sstevel@tonic-gate * handlers. 35497c478bd9Sstevel@tonic-gate */ 35507c478bd9Sstevel@tonic-gate void 35517c478bd9Sstevel@tonic-gate cyclic_suspend() 35527c478bd9Sstevel@tonic-gate { 35537c478bd9Sstevel@tonic-gate cpu_t *c; 35547c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu; 35557c478bd9Sstevel@tonic-gate cyc_xcallarg_t arg; 35567c478bd9Sstevel@tonic-gate cyc_backend_t *be; 35577c478bd9Sstevel@tonic-gate 35587c478bd9Sstevel@tonic-gate CYC_PTRACE0("suspend"); 35597c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 35607c478bd9Sstevel@tonic-gate c = cpu_list; 35617c478bd9Sstevel@tonic-gate 35627c478bd9Sstevel@tonic-gate do { 35637c478bd9Sstevel@tonic-gate cpu = c->cpu_cyclic; 35647c478bd9Sstevel@tonic-gate be = cpu->cyp_backend; 35657c478bd9Sstevel@tonic-gate arg.cyx_cpu = cpu; 35667c478bd9Sstevel@tonic-gate 35677c478bd9Sstevel@tonic-gate be->cyb_xcall(be->cyb_arg, c, 35687c478bd9Sstevel@tonic-gate (cyc_func_t)cyclic_suspend_xcall, &arg); 35697c478bd9Sstevel@tonic-gate } while ((c = c->cpu_next) != cpu_list); 35707c478bd9Sstevel@tonic-gate } 35717c478bd9Sstevel@tonic-gate 35727c478bd9Sstevel@tonic-gate /* 35737c478bd9Sstevel@tonic-gate * void cyclic_resume() 35747c478bd9Sstevel@tonic-gate * 35757c478bd9Sstevel@tonic-gate * cyclic_resume() resumes all cyclic activity throughout the cyclic 35767c478bd9Sstevel@tonic-gate * subsystem. It should be called only by system-suspending subsystems. 35777c478bd9Sstevel@tonic-gate * 35787c478bd9Sstevel@tonic-gate * Arguments and notes 35797c478bd9Sstevel@tonic-gate * 35807c478bd9Sstevel@tonic-gate * cyclic_resume() takes no arguments. Each CPU with an active cyclic 35817c478bd9Sstevel@tonic-gate * reenables and reprograms its backend (offline CPUs are not reenabled). 35827c478bd9Sstevel@tonic-gate * On most platforms, timestamps from gethrtime() and gethrestime() are not 35837c478bd9Sstevel@tonic-gate * guaranteed to monotonically increase between cyclic_suspend() and 35847c478bd9Sstevel@tonic-gate * cyclic_resume(). However, timestamps are guaranteed to monotonically 35857c478bd9Sstevel@tonic-gate * increase across the entire cyclic_suspend()/cyclic_resume() operation. 35867c478bd9Sstevel@tonic-gate * That is, every timestamp obtained before cyclic_suspend() will be less 35877c478bd9Sstevel@tonic-gate * than every timestamp obtained after cyclic_resume(). 35887c478bd9Sstevel@tonic-gate * 35897c478bd9Sstevel@tonic-gate * Return value 35907c478bd9Sstevel@tonic-gate * 35917c478bd9Sstevel@tonic-gate * None; cyclic_resume() always succeeds. 35927c478bd9Sstevel@tonic-gate * 35937c478bd9Sstevel@tonic-gate * Caller's context 35947c478bd9Sstevel@tonic-gate * 35957c478bd9Sstevel@tonic-gate * The cyclic subsystem must be configured on every valid CPU; 35967c478bd9Sstevel@tonic-gate * cyclic_resume() may not be called during boot or during dynamic 35977c478bd9Sstevel@tonic-gate * reconfiguration. Additionally, cpu_lock must be held, and the caller 35987c478bd9Sstevel@tonic-gate * cannot be in high-level interrupt context. However, unlike most other 35997c478bd9Sstevel@tonic-gate * cyclic entry points, cyclic_resume() may be called with locks held which 36007c478bd9Sstevel@tonic-gate * are also acquired by CY_LOCK_LEVEL or CY_LOW_LEVEL cyclic handlers. 36017c478bd9Sstevel@tonic-gate */ 36027c478bd9Sstevel@tonic-gate void 36037c478bd9Sstevel@tonic-gate cyclic_resume() 36047c478bd9Sstevel@tonic-gate { 36057c478bd9Sstevel@tonic-gate cpu_t *c; 36067c478bd9Sstevel@tonic-gate cyc_cpu_t *cpu; 36077c478bd9Sstevel@tonic-gate cyc_xcallarg_t arg; 36087c478bd9Sstevel@tonic-gate cyc_backend_t *be; 36097c478bd9Sstevel@tonic-gate 36107c478bd9Sstevel@tonic-gate CYC_PTRACE0("resume"); 36117c478bd9Sstevel@tonic-gate ASSERT(MUTEX_HELD(&cpu_lock)); 36127c478bd9Sstevel@tonic-gate 36137c478bd9Sstevel@tonic-gate c = cpu_list; 36147c478bd9Sstevel@tonic-gate 36157c478bd9Sstevel@tonic-gate do { 36167c478bd9Sstevel@tonic-gate cpu = c->cpu_cyclic; 36177c478bd9Sstevel@tonic-gate be = cpu->cyp_backend; 36187c478bd9Sstevel@tonic-gate arg.cyx_cpu = cpu; 36197c478bd9Sstevel@tonic-gate 36207c478bd9Sstevel@tonic-gate be->cyb_xcall(be->cyb_arg, c, 36217c478bd9Sstevel@tonic-gate (cyc_func_t)cyclic_resume_xcall, &arg); 36227c478bd9Sstevel@tonic-gate } while ((c = c->cpu_next) != cpu_list); 36237c478bd9Sstevel@tonic-gate } 3624