Merge tag 'sched_ext-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext

Pull sched_ext updates from Tejun Heo:
"Most of this cycle completes the enqueue-path support for hierarc

Merge tag 'sched_ext-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext

Pull sched_ext updates from Tejun Heo:
"Most of this cycle completes the enqueue-path support for hierarchical
sub-scheduling, which makes sub-scheduler support feature complete: a
root BPF scheduler can now hand a cgroup subtree over to a nested
sub-scheduler together with revocable CPU grants, and the
sub-scheduler owns all scheduling decisions for its tasks on those
CPUs.

Development volume was high and a number of changes plugging holes in
the new support landed late in the cycle. Also included are core
scheduling fixes that were completed too late for the v7.2 release and
are routed through this pull request.

Sub-scheduler CPU delegation:

- Parent schedulers now grant and revoke per-CPU capabilities
(enqueueing, preemption, CPU frequency control) on their children,
enforced on every path a scheduler can reach a CPU through.
Previously only dispatching could be delegated; this lets
sub-schedulers fully schedule their CPUs.

- Rescue execution: a task whose scheduler doesn't have access to the
CPUs the task needs to run on starved until the watchdog ejected
the whole scheduler. The kernel now runs such tasks directly on a
small bandwidth budget, turning a scheduler-killing failure into
bounded degradation.

- Cgroup integration: tasks migrating across a sub-scheduler boundary
weren't re-homed to the new owner, causing wrong-scheduler
scheduling and a use-after-free. Sub-schedulers now take over their
cgroup subtree and receive its cgroup callbacks.

- Arena objects now cross the kernel/BPF boundary as typed pointer
arguments, translated transparently by the BPF tree's new arena
argument support, replacing untyped arguments with manual
translation.

- scx_qmap now demonstrates full hierarchical sub-scheduling.

Other fixes and updates:

- Robustness improvements: the abort path is now NMI-safe, fixing
deadlocks when errors are raised from NMI context and making
hardlockup recovery direct. Reenqueue loops that could monopolize a
CPU ahead of the watchdog now eject the offending scheduler, and
stalls are blamed on the scheduler actually responsible.

- Hardening: BPF-writable arena memory is validated before kernel
use, and task slice and vtime writes got explicit synchronization
rules, closing corruption vectors open to buggy or malicious
schedulers.

- Core scheduling: sched_ext dispatching can drop the rq lock inside
the core-wide pick, which let interleaving selections corrupt each
other's state and hard-hang the machine. The selection now restarts
when the lock was released. The task ordering callback was also
invoked with its arguments swapped, and the default ordering is
updated to work across sub-scheduler boundaries. The fixes are
marked for stable.

- Other fixes headed for stable: a task init leak on fork failure
during enable, tooling compat macros that silently failed to detect
newer kernels, and a crash on reenqueueing against a destroyed
dispatch queue.

- Tooling: scx_pair moves off deprecated callbacks, and the
deprecated scx_bpf_cpu_rq() kfunc is removed"

* tag 'sched_ext-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (144 commits)
sched_ext: Drop the dead SCX_DEQ_CORE_SCHED_EXEC test in dequeue_task_scx()
sched_ext: Make core-sched task ordering hierarchy-aware
sched_ext: Use runnable_at for the default core-sched task ordering
sched_ext: Fix inverted ops.core_sched_before() invocation
sched_ext: Move the config-off sub-cap kfunc stubs into sub.c
sched_ext: Rename balance-era identifiers to dispatch terms
sched_ext: Drop the stale keep_prev fixup in dispatch_pick()
sched_ext: Keep kick_sync waiting on the rq's own CPU
sched_ext: Make SCHED_CLASS_EXT select GENERIC_ALLOCATOR
sched_ext/scx_flatcg: Fix cvtime true-up on slice expiry
sched_ext: Don't BUG_ON a destroyed DSQ in process_deferred_reenq_users
sched_ext: Fix scx_bpf_dsq_move_to_local___v2 compat detection
sched_ext: Make scx_bpf_events() read the calling scheduler's counters
sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable()
selftests/sched_ext: Fix flaky ddsp failure tests on busy systems
selftests/sched_ext: Make numa idle validation race-free
sched_ext: Fix scx_bpf_dsq_reenq___compat kfunc extern prototype
sched_ext/scx_flatcg: expire cached hweights on weight changes
sched_ext: Fix exit_task leak on fork failure during enable
sched_ext: fix stale references in doc comments
...

show more ...

sched_ext: Merge branch 'for-7.3-arena-args' into for-7.3

Pull to receive the __arena argument conversion:

67f1f4a48c24 ("sched_ext: Pass kernel arena pointers to ops_cid callbacks")
a8dc810968af

sched_ext: Merge branch 'for-7.3-arena-args' into for-7.3

Pull to receive the __arena argument conversion:

67f1f4a48c24 ("sched_ext: Pass kernel arena pointers to ops_cid callbacks")
a8dc810968af ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments")
a05c5b5cb5cf ("sched_ext: Convert scx_bpf_cid_override() to __arena array arguments")

along with the bpf-next branch carrying the __arena argument support they
depend on.

Conflict in kernel/sched/ext/ext.c between:

c384ab8a0b13 ("sched_ext: Move the config-off sub-cap kfunc stubs into sub.c")

and:

a8dc810968af ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments")

which updated the stubs in their old ext.c location. Resolved by keeping
ext.c without the stubs and applying the prototype conversion to the
relocated stubs in sub.c.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Move the config-off sub-cap kfunc stubs into sub.c

The EOPNOTSUPP stubs for the sub-cap kfuncs live in ext.c under #ifndef
CONFIG_EXT_SUB_SCHED while the real definitions live in sub.c. M

sched_ext: Move the config-off sub-cap kfunc stubs into sub.c

The EOPNOTSUPP stubs for the sub-cap kfuncs live in ext.c under #ifndef
CONFIG_EXT_SUB_SCHED while the real definitions live in sub.c. Move the
stubs into sub.c so all sub kfunc definitions live in one file. Pure code
move, no functional change.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Rename balance-era identifiers to dispatch terms

sched_class->balance() is gone from sched_ext and what balance_one() does is
run dispatch to produce something pickable. Update the balanc

sched_ext: Rename balance-era identifiers to dispatch terms

sched_class->balance() is gone from sched_ext and what balance_one() does is
run dispatch to produce something pickable. Update the balance-era names to
dispatch terms:

- balance_one() -> dispatch_one()
- SCX_RQ_IN_BALANCE -> SCX_RQ_IN_DISPATCH

No BPF scheduler reads the flag. The enum autogen headers gain the new name
with the old entry retained like other removed enumerators, zero-filling at
load time. No functional changes.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Set up ops.sub_ecaps_updated() dispatch context on the executing CPU

scx_process_sync_ecaps() sets up the dispatch context for
ops.sub_ecaps_updated() in the target cpu's pcpu context rec

sched_ext: Set up ops.sub_ecaps_updated() dispatch context on the executing CPU

scx_process_sync_ecaps() sets up the dispatch context for
ops.sub_ecaps_updated() in the target cpu's pcpu context recovered from the
llist node. However, the context is per executing cpu: the dispatch kfuncs
resolve it with this_cpu_ptr() and the dispatch buffer lives in it. What the
dispatches target is determined by the rq recorded in the context, not by
which cpu's context it is. Under core scheduling the pick runs balance_one()
for sibling rqs, so a sync processed for a sibling invokes the op with the
executing cpu's context not set up and its dispatch kfuncs misoperate on a
NULL or stale rq.

Set up the executing cpu's dsp_ctx instead, matching scx_dispatch_sched().
The recorded rq keeps the dispatches targeting the synced cpu.

Fixes: b81a6c018cde ("sched_ext: Add sub_ecaps_updated() effective-cap change notifier")
Reported-by: David Carlier <devnexen@gmail.com>
Link: https://lore.kernel.org/all/20260813045931.8691-1-devnexen@gmail.com/
Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Convert sub-cap kfuncs to __arena cmask arguments

The sub-cap kfuncs take their cmask arguments as __ign pointers. The values
cross the kfunc boundary as unchecked scalars and scx_cmask_r

sched_ext: Convert sub-cap kfuncs to __arena cmask arguments

The sub-cap kfuncs take their cmask arguments as __ign pointers. The values
cross the kfunc boundary as unchecked scalars and scx_cmask_ref_init()
rebases them into the arena by hand.

BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the cmask arguments __arena so the kfuncs receive kernel
addresses and scx_cmask_ref_init() loses the hand-rolled conversion. The
optional denied_out keeps its NULL not-provided signal via
__arena__nullable. The mandatory masks use plain __arena.

scx_qmap's call sites drop the (void *)(long) casts since the BPF-side
declarations type the cmask arguments __arena and take arena pointers
directly.

The arena argument address translation is currently implemented only on
x86-64. Schedulers calling these kfuncs load only there for now.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Pass kernel arena pointers to ops_cid callbacks

The cid-form set_cmask() and sub_caps_updated() callbacks receive cmasks
that the kernel builds in the arena, and the kernel converts the k

sched_ext: Pass kernel arena pointers to ops_cid callbacks

The cid-form set_cmask() and sub_caps_updated() callbacks receive cmasks
that the kernel builds in the arena, and the kernel converts the kernel
addresses to the BPF arena pointer form by hand before each call.

BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the arguments __arena in the cfi stubs and the ops_cid member
declarations and pass the kernel arena addresses directly, dropping the
manual scx_kaddr_to_arena() conversions and the now-unused helper. The
delivered value is unchanged and existing BPF-side code works as before.

The arena argument address translation is currently implemented only on
x86-64. cid-form schedulers implementing these callbacks load only there for
now.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

Merge branch 'for-7.3' into for-7.3-arena-args

sched_ext: Merge branch 'for-7.2-fixes' into for-7.3

Pull to receive:

c10b216a072f ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4af ("sched/core: Make core-sched flips wait f

sched_ext: Merge branch 'for-7.2-fixes' into for-7.3

Pull to receive:

c10b216a072f ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4af ("sched/core: Make core-sched flips wait for in-flight selections")
ffaab58d2175 ("sched_ext: Replace SCX_RQ_BAL_KEEP with a dispatch verdict return")
3dd52416e44a ("sched_ext: Fix this_rq() assumptions in dispatch kfuncs")
f2da9587118d ("sched_ext: Count rq lock releases in rq->scx.lock_drop_seq")
d954004205c1 ("sched_ext: Fix rq->core_pick corruption under core scheduling")

for the pending core scheduling follow-ups and to resolve the conflicts
with the code reorganization and cap gate work on for-7.3.

ffaab58d2175 converts scx_dispatch_sched() to a dispatch verdict return
which for-7.3 moved from ext.c into inlines.h. Resolved by applying the
conversion to the relocated copy and combining balance_one()'s verdict
returns with the scx_task_can_stay_on_cpu() gate from the cap work.

ffaab58d2175 and 3dd52416e44a update scx_bpf_sub_dispatch() which
for-7.3 moved into sub.c. Resolved by applying the scx_locked_rq()
switch and the verdict test to the sub.c copy.

f2da9587118d instruments the open-coded lock releases in
consume_remote_task() which for-7.3 folded into switch_rq_lock().
Resolved by keeping the accounting in switch_rq_lock() which covers all
its callers.

d954004205c1 widens the put_prev_task_scx() WARN suppression to all
core-sched rqs on the same condition that for-7.3 gated with
scx_task_can_stay_on_cpu(). Resolved by combining both.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Eject the top rescue consumer on overload

When rescue demand on a cpu persistently exceeds the configured bandwidth,
tasks age on that cpu's rescue DSQ until the stall watchdog fires. The

sched_ext: Eject the top rescue consumer on overload

When rescue demand on a cpu persistently exceeds the configured bandwidth,
tasks age on that cpu's rescue DSQ until the stall watchdog fires. The
watchdog blames the waiting task's owner, but the misbehaving party is
whoever floods the queue, not whoever happens to time out.

Track each sched's recent rescue consumption per cpu as a decaying average.
Once the oldest waiter on a cpu's rescue DSQ has been queued past a
threshold derived from the rescue knobs (4s at the defaults), the rescue
timer ejects the sub with the highest recent consumption on that cpu with
SCX_EXIT_ERROR_RESCUE. With no recent consumer there is no victim and
nothing is ejected - the generic stall watchdog eventually blames the
waiter's owner instead. Ejections on a cpu are spaced one threshold apart so
the freed bandwidth can drain the backlog before another sub is judged.

The overload check only wins the race against the stall watchdog when the
watchdog timeout clears the threshold, and a single in-budget wait must not
cross the trigger on its own. Warn on a scheduler whose timeout doesn't fit
and on knobs whose funding period exceeds half the threshold.

v2: - Track kill_at in jiffies_64 - on 32-bit, the time_before() grace check
wraps 2^31 ticks after the last ejection and suppresses ejections.
(sashiko AI)

- Track rescue_avg_at in jiffies_64 likewise - the unsigned long decay
delta truncates mod 2^32 on 32-bit and can revive a weeks-old usage
average in the victim pick.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Add bandwidth-limited rescue execution for stranded tasks

A local DSQ insert lacking the needed caps is diverted to the reject DSQ and
bounced back through ops.enqueue() so the scheduler

sched_ext: Add bandwidth-limited rescue execution for stranded tasks

A local DSQ insert lacking the needed caps is diverted to the reject DSQ and
bounced back through ops.enqueue() so the scheduler can re-decide. That
recovery assumes the scheduler has somewhere legal to send the task. When it
doesn't, e.g. when the task's affinity is restricted to cids delegated away,
the task starves until the stall watchdog ejects the scheduler. An exiting
task is worse - it skips ops.enqueue() and the rejection becomes a
self-requeuing cycle that burns the CPU until the watchdog fires.

Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the
insert would be rejected for missing caps, the kernel takes over and runs
the task on the target CPU without consulting the owning scheduler. The
kernel sets the flag itself when enqueueing an exiting task.

Rescue is a last-resort forward-progress backstop with a persistent
disadvantage, not a way around cap enforcement. A per-CPU token bucket
accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at
a time in arrival order. Each is granted a slice of the rescue_quantum_us
(default 5ms) quantum divided across the waiters, waits at the tail of the
local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival
once the slice is served.

The schedulers keep their normal control over an admitted rescuee and may
preempt or reslice it. Service is measured on CPU time actually received, so
neither shortens the rescue. Prolonged denial escalates - the remaining
slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee
preempts the current task. Escalation is paced by the same bucket, and
delivered service converges on the configured bandwidth no matter how
aggressively the schedulers dispatch.

Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making
SCX_ENQ_RESCUE inserts reject as usual.

v2: - Add SCX_OPS_OPEN() fix-ups for the new ops fields so cpu-form
schedulers setting them still load on older kernels. (Andrea)

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Make SCX_ENQ_IGNORE_CAPS waive the preemption cap too

SCX_ENQ_IGNORE_CAPS is kernel-internal and marks a placement the kernel
forces. scx_caps_for_enq() waives the enqueue cap for it, but

sched_ext: Make SCX_ENQ_IGNORE_CAPS waive the preemption cap too

SCX_ENQ_IGNORE_CAPS is kernel-internal and marks a placement the kernel
forces. scx_caps_for_enq() waives the enqueue cap for it, but a PREEMPT
insert still picks up the preemption cap requirement from
scx_caps_for_preempt(). Update scx_caps_for_preempt() to take enq_flags and
require nothing when SCX_ENQ_IGNORE_CAPS is set.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Rename scx_local_or_reject_dsq() to scx_resolve_local_dsq()

The following rescue execution addition gives the function a third possible
destination, making a name that enumerates the outc

sched_ext: Rename scx_local_or_reject_dsq() to scx_resolve_local_dsq()

The following rescue execution addition gives the function a third possible
destination, making a name that enumerates the outcomes a poor fit. Rename
to the destination-neutral scx_resolve_local_dsq(). No functional changes.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Format bstr exit messages after claiming the exit

The bstr exit kfuncs format the message into a shared static buffer under a
raw spinlock before initiating the exit. The lock can't be ta

sched_ext: Format bstr exit messages after claiming the exit

The bstr exit kfuncs format the message into a shared static buffer under a
raw spinlock before initiating the exit. The lock can't be taken from NMI
and needlessly serializes all bstr exits system-wide.

Now that exit claiming is lock-free, reverse the order: claim the exit first
and format directly into the exit_info message buffer which the claim winner
owns exclusively. The new scx_exit_bstr() implements the sequence, replacing
scx_bstr_format(), and the shared buffer and lock are deleted; the formatter
itself is what bpf_trace_printk() already runs from NMI. scx_prog_sched()
callers were relying on the lock for RCU protection, which is now provided
explicitly.

A malformed format no longer changes or fails the requested operation:
scx_bpf_exit_bstr() keeps its graceful exit kind and scx_bpf_sub_kill_bstr()
still kills the child, with a fallback message carrying the formatting
errno, while the sched that supplied the bad format is aborted for its bug.

Before this and the previous patch, an "any" category kfunc called from NMI
context could trigger scx_error() and deadlock - e.g. a tracing prog
attached to a function running in NMI calling scx_bpf_dsq_peek() on a
non-existent DSQ would try to grab scx_sched_lock, which may be held by the
interrupted CPU. This and the previous patch fix the deadlock: scx_error()
and the bstr exit kfuncs, and thus scx_bpf_error() and scx_bpf_exit(), are
now safe to call from any context including NMI.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Make exit claiming lock-free

scx_claim_exit() claims descendants' exits by walking the subtree under
scx_sched_lock, making exit claiming, and thus scx_error(), unusable from
NMI and from

sched_ext: Make exit claiming lock-free

scx_claim_exit() claims descendants' exits by walking the subtree under
scx_sched_lock, making exit claiming, and thus scx_error(), unusable from
NMI and from under scx_sched_lock. However, kfuncs raising errors can run
from NMI-attached BPF progs, the hardlockup handler runs in NMI, and
scx_link_sched() wants to report failures under the lock.

The walk does two things with different urgencies: ->aborting must be
asserted synchronously to break IRQs-off dispatch-path live-locks, while the
descendants' exit_kind claims can happen later. Split them: sweep ->aborting
locklessly under RCU to unwedge the system and defer the locked
SCX_EXIT_PARENT walk to a new irq_work, both of which are NMI-safe.

The sweep stores each node's ->aborting and then reads its children list
while scx_link_sched() inserts and then checks the parent's ->aborting, the
two sides paired by full barriers - one side always sees the other. A link
that sees ->aborting undoes its insert and fails. As the undo's
list_del_rcu() leaves ->sibling non-empty, list_empty() can no longer
identify a never-linked sched during teardown - add sch->linked instead.

trace_sched_ext_exit can now fire from NMI and is called after the
->aborting stores so that its callbacks don't hold up live-lock recovery.
The exit backtrace is skipped for NMI exits as stack_trace_save()'s
NMI-safety is arch-dependent and undocumented.

v2: Move trace_sched_ext_exit() after the ->aborting stores (Andrea).

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Bound per-task reenqueues and eject the owning scheduler

Unlike local reenqueues, cap rejections have no repeat limit. A
malfunctioning scheduler can keep re-inserting a task to a cid it

sched_ext: Bound per-task reenqueues and eject the owning scheduler

Unlike local reenqueues, cap rejections have no repeat limit. A
malfunctioning scheduler can keep re-inserting a task to a cid it lacks caps
on, cycling the task through reject and reenqueue. This was assumed safe
because a task that never runs trips the stall watchdog. However, the
reenqueue irq_work re-arms itself and outranks the timer vector, blocking
everything else on the CPU including stall detection and recovery, until the
NMI hardlockup detector fires.

Local reenqueues already have a repeat cap, SCX_REENQ_LOCAL_MAX_REPEAT,
which needs generalizing to cover all reenqueues. It also has an attribution
problem. Counted per-cpu on root, it tears down the whole hierarchy even
when a sub-scheduler caused the repeated reenqueues.

Generalize by bounding every reenqueue with one per-task counter. reenq_cnt
is bumped in scx_do_enqueue_task() on each SCX_ENQ_REENQ, the single path
every reenqueue producer passes through, and cleared in clr_task_runnable()
when the task is picked to run and in scx_disable_task() when it leaves the
scheduler's control. Past SCX_REENQ_MAX_REPEAT the task's owning scheduler
is ejected with a new SCX_EXIT_ERROR_REENQ and the task is left stranded to
be picked up during sched exit.

The SCX_EV_REENQ_LOCAL_REPEAT event becomes SCX_EV_REENQ_REPEAT, counting
repeat reenqueues from all sources.

v2: Count SCX_EV_REENQ_REPEAT only when a reenqueue leads to another
reenqueue, not on every reenqueue.

v3: - Also clear reenq_cnt in scx_disable_task() so that the count doesn't
carry over to the next owner across sched class switches, scheduler
replacement or sub-scheduler rehoming (Andrea Righi).

- Update the stale SCX_EV_REENQ_LOCAL_REPEAT references in sched-ext.rst
(Andrea Righi).

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Resolve most remaining scx_root accesses

scx_root is __rcu and naked accesses were left as transitional markers for
the multi-scheduler transition, to be converted to accesses through the

sched_ext: Resolve most remaining scx_root accesses

scx_root is __rcu and naked accesses were left as transitional markers for
the multi-scheduler transition, to be converted to accesses through the
associated scheduler instances. Most accesses have since been converted to
resolve the sched from the program or task at hand. The remaining naked
sites divide into ones that semantically always want the root sched, which
this patch resolves, and one that is left to a later patch.

The resolved sites:

- The SCX_OPS_TID_TO_TASK validation and the ecaps sync kick already hold a
sched whose ancestors[] pins the root as entry 0 with plain pointers
stable for the sched's lifetime. Reach the root through the sched at hand.

- The dispatch entry, class switch, idle notification and fork init paths
only execute while the scheduler is live and scx_root never changes inside
the live window, so no update can race them. Add scx_root_protected_live()
which documents that invariant and resolves with a plain load.

- The hotplug path, including the ecaps reseeds, runs with the hotplug lock
held, which excludes the scx_root writers. Add scx_root_protected(), which
accepts either the hotplug lock or scx_enable_mutex.

- Is-root tests use a zero level instead of comparing against the global.

touch_core_sched_dispatch() stays naked, to be resolved by a later patch.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Add scx_cgroup_sched() for cgrp->scx_sched reads

cgrp->scx_sched is __rcu and published with rcu_assign_pointer() but every
reader loads it with a plain access, so sparse flags all of the

sched_ext: Add scx_cgroup_sched() for cgrp->scx_sched reads

cgrp->scx_sched is __rcu and published with rcu_assign_pointer() but every
reader loads it with a plain access, so sparse flags all of them. The reads
are lock-protected: enable/disable paths rewrite the field under all of
scx_enable_mutex, scx_fork_rwsem and cgroup_mutex, and cgroup creation
inherits the parent's sched under cgroup_mutex before the new cgroup is
reachable, so holding any one of the three locks makes the read stable.

Add scx_cgroup_sched() which states the protection with
rcu_dereference_check() and convert the readers. No functional changes.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Fix stale errno in scx_sub_enable_workfn()

The nesting depth check and the cgroup online check in
scx_sub_enable_workfn() reach err_disable without setting ret, so
the fallback error adde

sched_ext: Fix stale errno in scx_sub_enable_workfn()

The nesting depth check and the cgroup online check in
scx_sub_enable_workfn() reach err_disable without setting ret, so
the fallback error added by commit db4e9defd2e8 ("sched_ext: Record
an error on errno-only sub-enable failure") reports
"scx_sub_enable() failed (0)".

This is currently harmless because both paths record their own
scx_error() first and the first error wins, but it leaves the
fallback broken for these paths. Set -EINVAL and -ENODEV there
so the fallback always reports a real errno.

v2: The validate_ops() path from v1 is already fixed in for-7.3
(sub.c already has ret = scx_validate_ops()), so only the two
remaining paths are addressed.

Signed-off-by: Cui Jian <cjian720@163.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Build the cid tables privately and publish them with RCU

The cid tables are visible to the cid kfuncs while being modified: the
first enable publishes the global pointers before filling t

sched_ext: Build the cid tables privately and publish them with RCU

The cid tables are visible to the cid kfuncs while being modified: the
first enable publishes the global pointers before filling them,
ops.init_cids() overrides rewrite them in place, and re-enables rebuild
them in place. A racing TRACING or SYSCALL program can read unfilled
entries, including uninitialized memory in the kmalloc'd tables, or torn
topo updates.

Tie the tables' lifetimes to the root sched instead: each root enable
builds a fresh set privately and publishes the per-table __rcu globals once
the layout is final, and root disable unpublishes and RCU-frees the set. A
non-NULL global is now always a fully built table which stays valid for the
reader's RCU read section, and lookups stay two loads. Kfuncs treat NULL as
no-mapping, also after the scheduler exits instead of reporting the stale
last mapping.

The cid kfuncs are available whether the root scheduler is cid-form or
cpu-form, the latter to allow gradual migration to cids. Every root
therefore builds and publishes a default mapping.

Every reader must either be gated on scheduler liveness or NULL-check
inside an RCU read section. Fix the two kfuncs that were neither:
scx_bpf_this_cid() read the table with no RCU or preemption protection and
scx_bpf_task_cid() relied on KF_RCU, which doesn't put a sleepable program
in an RCU read section. The hotplug callbacks are instead serialized by
retiring the tables inside the cpus_read_lock() section that clears
scx_root.

v2: Document why every root builds the tables (desc + cid.c comment).

Reported-by: Andrea Righi <arighi@nvidia.com>
Closes: https://lore.kernel.org/r/al3tLtPZZkFjMveK@gpd4
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Hand over cgroups at sub-scheduler enable/disable

Sub-schedulers don't get cgroups yet: every task_group is inited on the root
sched and the routing added by the previous patches always r

sched_ext: Hand over cgroups at sub-scheduler enable/disable

Sub-schedulers don't get cgroups yet: every task_group is inited on the root
sched and the routing added by the previous patches always resolves to it.
Add the handover: an enabling sub-scheduler takes over the cgroups in its
subtree and a disabling one returns them to its parent.

scx_cgroup_claim_subtree() runs while the sub enables, after the subtree's
cgrp->scx_sched's are set and before any task is claimed. It inits each
subtree task_group on the sub, exits it from the parent and updates
tg->scx.sched. A failed ops.cgroup_init() unwinds the sub-side inits and
aborts the enable with the parent untouched.

Disabling reverses it with scx_cgroup_return_subtree(): exit each cgroup
from the sub, then re-init it on the parent with the current tg->scx.*
values, resyncing weight and bandwidth changes made while the sub had it.
When a re-init fails, the parent is failed and the remaining task_groups
still transfer uninited and get no cgroup ops - the same punting done for
tasks. The dying parent's own disable moves them onward.

The handover walks include dying but not yet offlined task_groups, the same
as root's bulk walks: a removed cgroup keeps hosting scheduling events until
its dying tasks finish their final context switches, and its
ops.cgroup_exit() must follow the last of them. tg on/offlining is excluded
through cgroup_lock(), so either ordering against an rmdir of a subtree
cgroup delivers balanced init/exit pairs.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Re-home tasks on cgroup migration

A task's sched (p->scx.sched) must match its cgroup's owner
(cgrp->scx_sched). cgroup migration breaks the invariant:
scx_cgroup_move_task() only fires r

sched_ext: Re-home tasks on cgroup migration

A task's sched (p->scx.sched) must match its cgroup's owner
(cgrp->scx_sched). cgroup migration breaks the invariant:
scx_cgroup_move_task() only fires root's ops.cgroup_move() and never
re-homes the task, leading to wrong-sched scheduling and, once the stale
sched is freed, a use-after-free.

Hook into the new cgroup task migration events and re-home each task whose
destination cgroup is owned by a different sched. The events map naturally
to the transfer: MIGRATING runs the fallible init for the destination sched,
letting it reject the migration the same way ops.cgroup_prep_move() can,
MIGRATED does the re-home, which can't fail, and CANCELED undoes the init
when the migration falls through.

Pre-commit, the task's task_group still reflects the source, so
__scx_init_task() grows an explicit cgroup argument for the migration path
to hand ops.init_task() the destination cgroup.

Signed-off-by: Tejun Heo <tj@kernel.org>
Closes: https://lore.kernel.org/r/alnxrsexEe_nQwqL@gpd4
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

sched_ext: Factor out scx_rehome_task() and scx_punt_task()

Factor out scx_rehome_task() and scx_punt_task() from the sub-disable
re-home loop and scx_fail_parent(). The upcoming cgroup migration re

sched_ext: Factor out scx_rehome_task() and scx_punt_task()

Factor out scx_rehome_task() and scx_punt_task() from the sub-disable
re-home loop and scx_fail_parent(). The upcoming cgroup migration re-homing
also needs scx_rehome_task(). No functional changes.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...

Merge branch 'for-7.2-fixes' into for-7.3

Pull to receive:

477869bfafea ("sched_ext: Reject setting disallow from init_task outside the enable path")
5f8b69642d18 ("sched_ext: Take cgroup_lock()

Merge branch 'for-7.2-fixes' into for-7.3

Pull to receive:

477869bfafea ("sched_ext: Reject setting disallow from init_task outside the enable path")
5f8b69642d18 ("sched_ext: Take cgroup_lock() first in scx_cgroup_lock()")
8c13364db9c9 ("sched_ext: Skip sub-disable teardown for never-linked sub-schedulers")
5cdc92859809 ("sched_ext: Don't enable non-ext tasks in the sub-sched task loops")

as dependencies for the upcoming cgroup migration patchset and to
resolve the conflicts with the ext.c/sub.c split on for-7.3.

5f8b69642d18 comments scx_cgroup_lock() which for-7.3 exported for
sub.c. Resolved by keeping the exported version with the comment.

8c13364db9c9 and 5cdc92859809 patch the pre-split sub-sched enable and
disable paths in ext.c which for-7.3 moved to sub.c. Resolved by
applying the never-linked teardown skip and the class gates to sub.c.

Signed-off-by: Tejun Heo <tj@kernel.org>

show more ...

sched_ext: Add the scx_has_subs static key and gate sub-sched hot paths

With CONFIG_EXT_SUB_SCHED=y but no sub-scheduler attached - the common case
- hot paths still pay for sub-sched bookkeeping. G

sched_ext: Add the scx_has_subs static key and gate sub-sched hot paths

With CONFIG_EXT_SUB_SCHED=y but no sub-scheduler attached - the common case
- hot paths still pay for sub-sched bookkeeping. Gate it behind
__scx_has_subs, a static key counting live sub-schedulers, so that a
root-only system stops paying.

Most conversions are simple skip-if-no-sub tests. scx_idle_notify() is
special - it's a hierarchy walk, so give it a fast path which notifies the
root directly using the same tests as the walk. A pending
SCX_RQ_SUB_IDLE_RENOTIFY can be ignored as no sub can be owed one and the
caller clears the flag either way.

Suggested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

show more ...