Lines Matching +full:no +full:- +full:idle +full:- +full:on +full:- +full:init
1 /* SPDX-License-Identifier: GPL-2.0 */
3 * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst
32 SCX_EXIT_UNREG = 64, /* user-space initiated unregistration */
33 SCX_EXIT_UNREG_BPF, /* BPF-initiated unregistration */
34 SCX_EXIT_UNREG_KERN, /* kernel-initiated unregistration */
50 * SYS ACT: System-defined exit actions
51 * SYS RSN: System-defined exit reasons
52 * USR : User-defined exit codes and reasons
55 * actions and/or system reasons with a user-defined exit code.
67 * ops.exit() may be called even if the loading failed before ops.init()
69 * info communication. The following flag indicates whether ops.init()
80 /* %SCX_EXIT_* - broad category of the exit reason */
106 * Keep built-in idle tracking even if ops.update_idle() is implemented.
111 * By default, if there are no other task to run on the CPU, ext core
121 * scheduler depends on pid lookup for dispatching, the task will be
125 * dispatched to the local DSQ on enqueue. If the BPF scheduler doesn't
126 * depend on pid lookups and wants to handle these tasks directly, the
138 * A migration disabled task can only execute on its current CPU. By
139 * default, such tasks are automatically put on the CPU's local DSQ with
140 * the default slice on enqueue. If this ops flag is set, they also go
144 * only select the current CPU. Also, p->cpus_ptr will only contain its
145 * current CPU while p->nr_cpus_allowed keeps tracking p->user_cpus_ptr
146 * and thus may disagree with cpumask_weight(p->cpus_ptr).
152 * ops.enqueue() on the ops.select_cpu() selected or the wakee's
153 * previous CPU via IPI (inter-processor interrupt) to reduce cacheline
156 * As the BPF scheduler may depend on ops.select_cpu() being invoked
160 * the BPF scheduler must be able to handle ops.enqueue() invoked on the
162 * may be executed on multiple CPUs.
167 * If set, enable per-node idle cpumasks. If clear, use a single global
168 * flat idle cpumask.
175 SCX_OPS_HAS_CGROUP_WEIGHT = 1LLU << 16, /* DEPRECATED, will be removed on 6.18 */
195 * Set if ops.init_task() is being invoked on the fork path, as opposed
207 /* Whether the task exited before running on sched_ext. */
211 /* argument container for ops->cgroup_init() */
234 * Argument container for ops->cpu_acquire(). Currently empty, but may be
239 /* argument container for ops->cpu_release() */
244 /* the task that's going to be scheduled on the CPU */
260 * struct sched_ext_ops - Operation table for BPF scheduler implementation
271 * @prev_cpu: the cpu @p was on before sleeping
275 * is getting dispatched for execution later. However, as @p is not on
279 * If an idle CPU is returned, the CPU is kicked and will try to
281 * select_cpu() serves as the default way to wake up idle CPUs.
289 * on a single CPU or tasks with migration disabled, as they don't have
296 * @enqueue: Enqueue a task on the BPF scheduler
301 * scx_bpf_dsq_insert() or enqueue on the BPF scheduler. If not directly
321 * on the scheduling logic, this can lead to confusing behaviors - e.g.
338 * ops.dispatch_max_batch. See the comments on top of the two functions
343 * @prev->scx.flags, it is not enqueued yet and will be enqueued after
353 * This operation is called every 1/HZ seconds on CPUs which are
354 * executing an SCX task. Setting @p->scx.slice to 0 will trigger an
355 * immediate dispatch cycle on the CPU.
360 * @runnable: A task is becoming runnable on its associated CPU
365 * execution state transitions. A task becomes ->runnable() on a CPU,
366 * and then goes through one or more ->running() and ->stopping() pairs
367 * as it runs on the CPU, and eventually becomes ->quiescent() when it's
368 * done running on the CPU.
370 * @p is becoming runnable on the CPU because it's
372 * - waking up (%SCX_ENQ_WAKEUP)
373 * - being moved from another CPU
374 * - being restored after temporarily taken off the queue for an
377 * This and ->enqueue() are related but not coupled. This operation
378 * notifies @p's state transition and may not be followed by ->enqueue()
380 * being enqueued on a CPU experiencing a hotplug event. Likewise, a
381 * task may be ->enqueue()'d without being preceded by this operation
387 * @running: A task is starting to run on its associated CPU
391 * one the task is going to run on. This can happen when a task
393 * which triggers this callback, may run on a CPU different from
399 * See ->runnable() for explanation on the task state notifiers.
409 * one the task was running on. This can happen when a task
411 * which triggers this callback, may run on a CPU different from
415 * the task was running on.
417 * See ->runnable() for explanation on the task state notifiers. If
418 * !@runnable, ->quiescent() will be invoked after this operation
424 * @quiescent: A task is becoming not runnable on its associated CPU
428 * See ->runnable() for explanation on the task state notifiers.
430 * @p is becoming quiescent on the CPU because it's
432 * - sleeping (%SCX_DEQ_SLEEP)
433 * - being moved to another CPU
434 * - being temporarily taken off the queue for an attribute change
437 * This and ->dequeue() are related but not coupled. This operation
438 * notifies @p's state transition and may not be preceded by ->dequeue()
453 * If @to is not-NULL, @from wants to yield the CPU to @to. If the bpf
459 * @core_sched_before: Task ordering for core-sched
463 * Used by core-sched to determine the ordering between two tasks. See
464 * Documentation/admin-guide/hw-vuln/core-scheduling.rst for details on
465 * core-sched.
467 * Both @a and @b are runnable and may or may not currently be queued on
469 * %false if there's no required ordering or @b should run before @a.
488 * @cpumask: cpumask of cpus that @p can run on
496 * @update_idle: Update the idle state of a CPU
497 * @cpu: CPU to update the idle state for
498 * @idle: whether entering or exiting the idle state
500 * This operation is called when @rq's CPU goes or leaves the idle
501 * state. By default, implementing this operation disables the built-in
502 * idle CPU tracking and the following helpers become unavailable:
504 * - scx_bpf_select_cpu_dfl()
505 * - scx_bpf_select_cpu_and()
506 * - scx_bpf_test_and_clear_cpu_idle()
507 * - scx_bpf_pick_idle_cpu()
510 * implementation relies on scx_bpf_select_cpu_dfl().
512 * Specify the %SCX_OPS_KEEP_BUILTIN_IDLE flag to keep the built-in idle
515 void (*update_idle)(s32 cpu, bool idle);
532 * The specified CPU is no longer under the control of the BPF
535 * caller should consult @args->reason to determine the cause.
542 * @args: init arguments, see the struct definition
548 * Return 0 for success, -errno for failure. An error return while
555 * @exit_task: Exit a previously-running task from the system
568 * Enable @p for BPF scheduling. enable() is called on @p any time it
584 * @dump: Dump BPF scheduler state on error
592 * @dump_cpu: Dump BPF scheduler state for a CPU on error
595 * @idle: @cpu is currently idle without any runnable tasks
598 * @cpu. If @idle is %true and this operation doesn't produce any
601 void (*dump_cpu)(struct scx_dump_ctx *ctx, s32 cpu, bool idle);
604 * @dump_task: Dump BPF scheduler state for a runnable task on error
617 * @args: init arguments, see the struct definition
622 * Return 0 for success, -errno for failure. An error return while
647 * Return 0 for success, -errno for failure. An error return aborts the
731 * All CPU hotplug ops must come before ops.init().
735 * @init: Initialize the BPF scheduler
737 s32 (*init)(void); member
743 * ops.exit() is also called on ops.init() failure, which is a bit
744 * unusual. This is to allow rich reporting through @info on how
745 * ops.init() failed.
777 * If 0, no detection occurs. Otherwise, the scheduler will fail to
778 * load if the sequence number does not match @scx_hotplug_seq on the
786 * Must be a non-zero valid BPF object name including only isalnum(),
801 SCX_OPI_CPU_HOTPLUG_END = SCX_OP_IDX(init),
802 SCX_OPI_END = SCX_OP_IDX(init),
823 * continued to run because there were no other tasks on the CPU.
864 * The event counters are in a per-CPU variable to minimize the
865 * accounting overhead. A system-wide view on the event counter is
878 * The global DSQ (%SCX_DSQ_GLOBAL) is split per-node for scalability.
879 * This is to avoid live-locking in bypass mode where all tasks are
881 * per-node split isn't sufficient, it can be further split.
925 * The task being enqueued was previously enqueued on the current CPU's
928 * invoked in a ->cpu_release() callback, and the task is again
929 * dispatched back to %SCX_LOCAL_DSQ by this current ->enqueue(), the
930 * task will not be scheduled on the CPU until at least the next invocation
931 * of the ->cpu_acquire() callback.
941 * The BPF scheduler is responsible for triggering a follow-up
960 * The generic core-sched layer decided to execute the task even though
967 SCX_PICK_IDLE_CORE = 1LLU << 0, /* pick a CPU whose SMT siblings are also idle */
973 * Kick the target CPU if idle. Guarantees that the target CPU goes
974 * through at least one full scheduling cycle before going idle. If the
975 * target CPU can be determined to be currently not idle and going to go
976 * through a scheduling cycle before going idle, noop.
982 * current task of the target CPU is an SCX task, its ->scx.slice is
1015 * sched_ext_entity->ops_state
1020 * NONE -> QUEUEING -> QUEUED -> DISPATCHING
1023 * \-------------------------------/
1026 * sites for explanations on the conditions being waited upon and why they are
1032 * relaxes the requirements on the BPF scheduler. This allows the BPF scheduler
1045 * on the task being dispatched.
1048 * p->scx.ops_state is atomic_long_t which leaves 30 bits for QSEQ on
1056 #define SCX_OPSS_STATE_MASK ((1LU << SCX_OPSS_QSEQ_SHIFT) - 1)
1062 * Return the rq currently locked from an scx callback, or NULL if no rq is
1072 return !current->scx.kf_mask; in scx_kf_allowed_if_unlocked()
1077 return unlikely(rq->scx.flags & SCX_RQ_BYPASSING); in scx_rq_bypassing()