Lines Matching full:period
179 /* Delay in jiffies for grace-period initialization delays, debug only. */
216 * Return true if an RCU grace period is in progress. The READ_ONCE()s
354 * We inform the RCU core by emulating a zero-duration dyntick-idle period.
446 * How long the grace period must be before we start recruiting
455 * Make sure that we give the grace-period kthread time to detect any
549 * Send along grace-period-related data for rcutorture diagnostics.
558 /* Gather grace-period sequence numbers for rcutorture diagnostics. */
567 /* Format grace-period sequence numbers for rcutorture diagnostics. */
649 * in a timely manner, the RCU grace-period kthread sets that CPU's
658 * interrupt or exception. In that case, the RCU grace-period kthread
781 * @lag_gps: Set overflow lag to this many grace period worth of counters
830 * Ordering between remote CPU's pre idle accesses and post grace period in rcu_watching_snap_save()
868 * of the current RCU grace period. in rcu_watching_snap_recheck()
888 * the relevant portions the CPU-hotplug code, the grace-period in rcu_watching_snap_recheck()
953 * but only once per grace period, just to keep the IPIs down to in rcu_watching_snap_recheck()
999 * rcu_start_this_gp - Request the start of a particular grace period
1002 * @gp_seq_req: The gp_seq of the grace period to start.
1004 * Start the specified grace period, as needed to handle newly arrived
1007 * is reason to awaken the grace-period kthread.
1010 * is why the caller is responsible for waking the grace-period kthread.
1022 * structure's lock or bail out if the need for this grace period in rcu_start_this_gp()
1023 * has already been recorded -- or if that grace period has in in rcu_start_this_gp()
1024 * fact already started. If there is already a grace period in in rcu_start_this_gp()
1026 * end of the grace period will scan the leaf rcu_node structures. in rcu_start_this_gp()
1045 * grace period is in progress, which means that in rcu_start_this_gp()
1085 * Clean up any old requests for the just-ended grace period. Also return
1101 * Awaken the grace-period kthread. Don't do a self-awaken (unless in an
1103 * sleep upon return, resulting in a grace-period hang), and don't bother
1104 * awakening when there is nothing for the grace-period kthread to do
1110 * in the grace-period kthread's context? Because the kthread might have
1135 * the RCU grace-period kthread.
1154 * Callbacks are often registered with incomplete grace-period in rcu_accelerate_cbs()
1157 * makes a conservative estimate of the grace period number at which in rcu_accelerate_cbs()
1160 * accelerating callback invocation to an earlier grace-period in rcu_accelerate_cbs()
1182 * that a new grace-period request be made, invokes rcu_accelerate_cbs()
1206 * Move any callbacks whose grace period has completed to the
1211 * Returns true if the RCU grace-period kthread needs to be awakened.
1236 * that the RCU grace-period kthread be awakened.
1244 // The grace period cannot end while we hold the rcu_node lock. in rcu_advance_cbs_nowake()
1253 * a new grace period.
1267 * Returns true if the grace-period kthread needs to be awakened.
1298 * If the current grace period is waiting for this CPU, in __note_gp_changes()
1308 rdp->gp_seq = rnp->gp_seq; /* Remember new grace-period state. */ in __note_gp_changes()
1341 /* Register a counter to suppress debugging grace-period delays. */
1375 /* Allow rcutorture to stall the grace-period kthread. */
1407 // Make the polled API aware of the beginning of a grace period.
1423 // Make the polled API aware of the end of a grace period.
1443 // Make the polled API aware of the beginning of a grace period, but
1460 // Make the polled API aware of the end of a grace period, but where
1485 * period has elapsed. These nodes processing
1489 * At every grace period init, a new wait node is added
1491 * for this new grace period. Given that there are a fixed
1494 * is delayed) and additional grace period is requested.
1499 * grace period, therefore users which should wait due
1500 * to a slow process are handled by _this_ grace period
1649 "A full grace period is not passed yet!\n"); in rcu_sr_normal_complete()
1671 * grace period kthread in rcu_sr_normal_gp_cleanup(). in rcu_sr_normal_gp_cleanup_work()
1829 * Initialize a new grace period. Return false if no grace period required.
1852 * Grace period already in progress, don't start another. in rcu_gp_init()
1859 /* Advance to a new grace period and initialize state. */ in rcu_gp_init()
1871 * be missed by the current grace period, potentially leading to in rcu_gp_init()
1905 * the rcu_node tree. Note that this new grace period need not in rcu_gp_init()
1948 * If all waited-on tasks from prior grace period are in rcu_gp_init()
1974 * seeing that no grace period is in progress, at least until the in rcu_gp_init()
1977 * The grace period cannot complete until the initialization in rcu_gp_init()
2010 // If strict, make all CPUs aware of new grace period. in rcu_gp_init()
2045 // The current grace period has completed. in rcu_gp_fqs_check_wake()
2089 * Loop doing repeated quiescent-state forcing until the grace period ends.
2128 * Exit the loop if the root rcu_node structure indicates that the grace period in rcu_gp_fqs_loop()
2131 * the current grace period are queued only on leaf rcu_node structures. in rcu_gp_fqs_loop()
2175 * Clean up after the old grace period.
2196 * We know the grace period is complete, but to everyone else in rcu_gp_cleanup()
2199 * they can do to advance the grace period. It is therefore in rcu_gp_cleanup()
2201 * period as completed in all of the rcu_node structures. in rcu_gp_cleanup()
2209 * period to process their callbacks. This also avoids some nasty in rcu_gp_cleanup()
2210 * RCU grace-period initialization races by forcing the end of in rcu_gp_cleanup()
2211 * the current grace period to be completely recorded in all of in rcu_gp_cleanup()
2213 * period is recorded in any of the rcu_node structures. in rcu_gp_cleanup()
2228 * enqueued before re-checking the grace-period state. in rcu_gp_cleanup()
2253 /* Declare grace period done, trace first to use old GP number. */ in rcu_gp_cleanup()
2268 // We get here if a grace period was needed (“needgp”) in rcu_gp_cleanup()
2271 // the need for another grace period). The purpose in rcu_gp_cleanup()
2284 // additional grace period or if rcu_accelerate_cbs() has in rcu_gp_cleanup()
2293 // Make synchronize_rcu() users aware of the end of old grace period. in rcu_gp_cleanup()
2296 // If strict, make all CPUs aware of the end of the old grace period. in rcu_gp_cleanup()
2309 /* Handle grace-period start. */ in rcu_gp_kthread()
2332 /* Handle grace-period end. */ in rcu_gp_kthread()
2341 * Invoke rcu_gp_kthread_wake() to awaken the grace-period kthread if
2342 * another grace period is required. Whether we wake the grace-period
2345 * period. Note that the caller must hold rnp->lock, which is released
2364 * is the grace-period snapshot, which means that the quiescent states
2370 * during grace-period initialization.
2387 * relevant grace period is already over, so done. in rcu_report_qs_rnp()
2423 * state for this grace period. Invoke rcu_report_qs_rsp() in rcu_report_qs_rnp()
2424 * to clean up and start the next grace period if one is needed. in rcu_report_qs_rnp()
2432 * RCU grace period. The caller must hold the corresponding rnp->lock with
2489 * The grace period in which this quiescent state was in rcu_report_qs_rdp()
2492 * within the current grace period. in rcu_report_qs_rdp()
2525 * Check to see if there is a new grace period of which this CPU
2528 * quiescent state for this grace period, and record that fact if so.
2533 /* Check for grace-period ends and beginnings. */ in rcu_check_quiescent_state()
2537 * Does this CPU still need to do its part for current grace period? in rcu_check_quiescent_state()
2545 * period? If no, then exit and wait for the next call. in rcu_check_quiescent_state()
2571 * period. Throttle as specified by rdp->blimit.
2603 * Callbacks execution is fully ordered against preceding grace period in rcu_do_batch()
2725 * core processing. If the current grace period has gone on too long,
2761 * blocking the current grace period, initiate RCU priority boosting.
2906 /* No grace period and unregistered callbacks? */ in rcu_core()
3075 * Force the grace period if too many callbacks or too long waiting. in call_rcu_core()
3079 * is the only one waiting for a grace period to complete. in call_rcu_core()
3084 /* Are we ignoring a completed grace period? */ in call_rcu_core()
3087 /* Start a new grace period if one not already started. */ in call_rcu_core()
3091 /* Give the grace period a kick. */ in call_rcu_core()
3214 * call_rcu_hurry() - Queue RCU callback for invocation after grace period, and
3219 * @func: actual callback function to be invoked after the grace period
3222 * period elapses, in other words after all pre-existing RCU read-side
3243 * call_rcu() - Queue an RCU callback for invocation after a grace period.
3249 * @func: actual callback function to be invoked after the grace period
3252 * period elapses, in other words after all pre-existing RCU read-side
3259 * The specified function will be invoked after another full grace period
3270 * Note that all CPUs must agree that the grace period extended beyond
3294 * seconds before starting the grace period needed by the corresponding
3306 * During early boot, any blocking grace-period wait automatically
3307 * implies a grace period.
3313 * grace-period optimization is ignored once the scheduler is running.
3363 * synchronize_rcu - wait until a grace period has elapsed.
3366 * period has elapsed, in other words after all currently executing RCU
3426 // Update the normal grace-period counters to record in synchronize_rcu()
3427 // this grace period, but only those used by the boot CPU. in synchronize_rcu()
3444 * like poll_state_synchronize_rcu_full() as a cookie whose grace period
3459 * grace period has elapsed in the meantime.
3474 * @gsp: location to place combined normal/expedited grace-period state
3476 * Places the normal and expedited grace-period states in @gsp. This
3479 * grace period (whether normal or expedited) has elapsed in the meantime.
3485 * This does not guarantee that the needed grace period will actually
3520 // Note it is possible for a grace period to have elapsed between in start_poll_synchronize_rcu_common()
3523 // get a grace period that no one needed. These accesses are ordered in start_poll_synchronize_rcu_common()
3525 // from which they are updated at grace-period start, as required. in start_poll_synchronize_rcu_common()
3533 * start_poll_synchronize_rcu - Snapshot and start RCU grace period
3537 * grace period has elapsed in the meantime. If the needed grace period
3539 * grace period.
3551 * start_poll_synchronize_rcu_full - Take a full snapshot and start RCU grace period
3554 * Places the normal and expedited grace-period states in *@gs. This
3557 * grace period (whether normal or expedited) has elapsed in the meantime.
3558 * If the needed grace period is not already slated to start, notifies
3559 * RCU core of the need for that grace period.
3570 * poll_state_synchronize_rcu - Has the specified RCU grace period completed?
3573 * If a full RCU grace period has elapsed since the earlier call from
3577 * can explicitly wait for a grace period, for example, by passing @oldstate
3587 * either refresh them or set a flag indicating that the grace period has
3589 * to get a guaranteed-completed grace-period state.
3591 * In addition, because oldstate compresses the grace-period state for
3593 * it can miss a grace period when synchronize_rcu() runs concurrently
3615 * period recorded in *gsp has completed. Callers that need the full
3632 * poll_state_synchronize_rcu_full - Has the specified RCU grace period completed?
3635 * If a full RCU grace period has elapsed since the earlier call from
3639 * can explicitly wait for a grace period, for example, by passing @gsp
3648 * that the grace period has completed. Alternatively, they can use
3650 * grace-period state.
3657 * The problem is that the just-ending grace-period's callbacks can be
3666 smp_mb(); // Order against root rcu_node structure grace-period cleanup. in poll_state_synchronize_rcu_full()
3676 * cond_synchronize_rcu - Conditionally wait for an RCU grace period
3679 * If a full RCU grace period has elapsed since the earlier call to
3681 * Otherwise, invoke synchronize_rcu() to wait for a full grace period.
3701 * cond_synchronize_rcu_full - Conditionally wait for an RCU grace period
3704 * If a full RCU grace period has elapsed since the call to
3708 * for a full grace period.
3779 /* Has RCU gone idle with this CPU needing another grace period? */ in rcu_pending()
3785 /* Have RCU grace period completed or started? */ in rcu_pending()
3883 * Note that this primitive does not necessarily wait for an RCU grace period
3886 * immediately, without waiting for anything, much less an RCU grace period.
4341 * of the next grace period. in rcutree_prepare_cpu()
4441 /* An incoming CPU should never be blocking a grace period. */ in rcutree_report_cpu_starting()
4664 …if (WARN_ONCE(IS_ERR(t), "%s: Could not start grace-period kthread, OOM is now expected behavior\n… in rcu_spawn_gp_kthread()
4696 * contain synchronous grace-period primitives (during which time, this idle
4698 * function is called, any synchronous grace-period primitives are run as
4699 * expedited, with the requesting task driving the grace period forward.