| e7103e5e | 17-Aug-2026 |
George Guo <guodongtai@kylinos.cn> |
LoongArch: BPF: Add arch_bpf_stack_walk() implementation
Implement arch_bpf_stack_walk() on top of the ORC unwinder within
the LoongArch BPF JIT backend to provide generic BPF stack walking
capabili
LoongArch: BPF: Add arch_bpf_stack_walk() implementation
Implement arch_bpf_stack_walk() on top of the ORC unwinder within
the LoongArch BPF JIT backend to provide generic BPF stack walking
capabilities.
This function is required by advanced BPF features, including timed
may_goto timeout tracing and BPF exceptions. It will be invoked in
the BPF core subsystem unwinding paths: bpf_prog_find_from_stack(),
bpf_stream_stage_dump_stack(), and bpf_throw().
Co-developed-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: George Guo <guodongtai@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 3fbf3534 | 17-Aug-2026 |
George Guo <guodongtai@kylinos.cn> |
LoongArch: BPF: Add timed may_goto implementation
Implement arch_bpf_timed_may_goto() support and advertise it through
bpf_jit_supports_timed_may_goto() so the verifier lowers may_goto into
the time
LoongArch: BPF: Add timed may_goto implementation
Implement arch_bpf_timed_may_goto() support and advertise it through
bpf_jit_supports_timed_may_goto() so the verifier lowers may_goto into
the timed variant: instead of a fixed iteration counter, the loop is
bounded by a wall-clock timeout maintained in a per-loop stack slot.
arch_bpf_timed_may_goto() uses a custom calling convention: the verifier
passes the count/timestamp stack offset in BPF_REG_AX and expects the
updated count back in the same register. The JIT call path therefore can
skip the usual 'BPF_REG_0 = C return value' move for this helper.
Acked-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: George Guo <guodongtai@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 1db6d003 | 17-Aug-2026 |
George Guo <guodongtai@kylinos.cn> |
LoongArch: BPF: Resolve per-CPU addrs for internal-only MOV
Add support for the internal-only BPF_MOV instruction that resolves the
absolute addresses of the per-CPU data from their per-CPU offsets.
LoongArch: BPF: Resolve per-CPU addrs for internal-only MOV
Add support for the internal-only BPF_MOV instruction that resolves the
absolute addresses of the per-CPU data from their per-CPU offsets. This
instruction is used only for internal inlining optimizations between the
BPF verifier and the JITs (e.g. inlining bpf_get_smp_processor_id() and
per-CPU map lookups).
LoongArch keeps the per-CPU offset of the current CPU in $r21 register
(a.k.a. __my_cpu_offset), so resolving a per-CPU address only requires
adding $r21 to the source register holding the per-CPU offset. Advertise
the capability via bpf_jit_supports_percpu_insn().
Acked-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: George Guo <guodongtai@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 1fec6bbc | 17-Aug-2026 |
Chenguang Zhao <zhaochenguang@kylinos.cn> |
LoongArch: BPF: Advertise JIT support for kptr xchg inline
The BPF verifier can lower bpf_kptr_xchg() to BPF_XCHG when the JIT
advertises ptr xchg support. With ordered amswap_db.* emission from the
LoongArch: BPF: Advertise JIT support for kptr xchg inline
The BPF verifier can lower bpf_kptr_xchg() to BPF_XCHG when the JIT
advertises ptr xchg support. With ordered amswap_db.* emission from the
previous patch, declare that LoongArch bpf JIT supports this inlining.
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
Acked-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Chenguang Zhao <zhaochenguang@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 576412d9 | 17-Aug-2026 |
Chenguang Zhao <zhaochenguang@kylinos.cn> |
LoongArch: BPF: Align value-returning atomics with LKMM
Per the Linux Kernel Memory Model, value-returning atomic RMW operations
must provide sequentially consistent ordering (a full memory barrier)
LoongArch: BPF: Align value-returning atomics with LKMM
Per the Linux Kernel Memory Model, value-returning atomic RMW operations
must provide sequentially consistent ordering (a full memory barrier).
On LoongArch, plain AMO instructions and bare ll/sc loops do not satisfy
this requirement by themselves.
Update emit_atomic_rmw() to emit barrier-carrying instructions for all
value-returning BPF atomics:
- BPF_FETCH (ADD/AND/OR/XOR): use am*_db.{b,h,w,d}
- BPF_XCHG: use amswap_db.{b,h,w,d}
- BPF_CMPXCHG: emit dbar 0x700 after the ll/sc loop, matching
__WEAK_LLSC_MB in cmpxchg.h
Add the corresponding instruction encodings and emit helpers to inst.h.
Non-value-returning RMW ops (plain BPF_ADD, BPF_AND, etc.) are left as
weakly ordered, consistent with LKMM.
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
Acked-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Chenguang Zhao <zhaochenguang@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 9fcdba34 | 17-Aug-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Split unconditional branch JA paths statically
In build_insn(), both 32-bit and 64-bit unconditional branch JA paths
currently share a single case block. It relies on a runtime condi
LoongArch: BPF: Split unconditional branch JA paths statically
In build_insn(), both 32-bit and 64-bit unconditional branch JA paths
currently share a single case block. It relies on a runtime condition
check to multiplex between the 'off' and 'imm' fields.
Since the instruction classes are already resolved at compile-time via
distinct switch-case labels, this runtime check is redundant.
Split the two paths into individual case blocks to remove the redundant
runtime check.
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 252ae40b | 17-Aug-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Remove dead move_imm() call in BPF_NEG path
The BPF_NEG operation is a unary operator that performs `dst = -dst`.
The current code unconditionally executes a move_imm() call before t
LoongArch: BPF: Remove dead move_imm() call in BPF_NEG path
The BPF_NEG operation is a unary operator that performs `dst = -dst`.
The current code unconditionally executes a move_imm() call before the
subtraction, generating useless JITted instructions to load data into
the temporary register `t1`. This `t1` register is never used anywhere
else in the entire BPF_NEG path.
Remove this dead `move_imm()` call to avoid useless instructions.
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 32a5e3ea | 17-Aug-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Remove redundant zext jumping in move_imm()
In move_imm(), when an immediate hits the 12-bit unsigned range, an
`ori rd, $zero, imm` instruction is emitted.
According to the manual,
LoongArch: BPF: Remove redundant zext jumping in move_imm()
In move_imm(), when an immediate hits the 12-bit unsigned range, an
`ori rd, $zero, imm` instruction is emitted.
According to the manual, the `ori` instruction inherently performs a
logical or with zero-extended immediate operands against $zero, so the
upper 32 bits of the destination register `rd` are already 0.
However, the existing JIT code unconditionally executes `goto zext;`
after `ori`, forcing it to fallthrough into `emit_zext_32()` to clear
the upper 32 bits for 32-bit ALU operations.
Fix this redundancy by directly returning from the function inside the
`is_unsigned_imm12()` block.
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 43430888 | 17-Aug-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Implement branchless conditional move for TCC
The current implementation handles combined bpf2bpf and tail calls by
checking at runtime whether REG_TCC holds a scalar count or a poin
LoongArch: BPF: Implement branchless conditional move for TCC
The current implementation handles combined bpf2bpf and tail calls by
checking at runtime whether REG_TCC holds a scalar count or a pointer
address via a conditional jump. This adds branch prediction overhead
in the hot path of tail call execution.
To implement branchless conditional move, use an unsigned comparison
(sltui) combined with mask instructions (maskeqz/masknez) to achieve
branchless classification and blending of incoming scalar counts and
kernel pointers in REG_TCC.
This optimization refactors the inner logic of the helper function,
unifies the offset decrement at the function entry, and removes all
runtime branching from the prologue hot path completely.
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 37d545d1 | 17-Aug-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Refactor jump offset calculation in tail call
The old macro-based jmp_offset calculation derives the jump distance
from a stale prior-pass code stride, which can lead to wrong branch
LoongArch: BPF: Refactor jump offset calculation in tail call
The old macro-based jmp_offset calculation derives the jump distance
from a stale prior-pass code stride, which can lead to wrong branch
offsets and soft lockups under extra JIT passes.
Fix this by calculating the offset directly on the absolute target:
"ctx->offset[insn + 1] - ctx->idx".
To avoid a false 16-bit range check abort during size estimation, add
a "ctx->image == NULL" guard to inject a safe dummy offset.
Cc: stable@vger.kernel.org
Fixes: cd39d9e6b7e4 ("LoongArch: BPF: Fix jump offset calculation in tailcall")
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| cd7e356b | 17-Aug-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Move arena register slot below TCC context
Currently, the stack layout places the optional arena register slot
above the tail call counter context. When arena_vm_start is dynamically
LoongArch: BPF: Move arena register slot below TCC context
Currently, the stack layout places the optional arena register slot
above the tail call counter context. When arena_vm_start is dynamically
enabled, it shifts the relative offset of the tcc_ptr slot within the
stack frame, causing hardcoded tracking macros to mismatch and leading
to memory misalignment or corruption potentially.
To fix this, move the arena register save and restore sequences below
the tail call counter context slots in both build_prologue() and the
epilogue.
Update __build_epilogue() to insert a proper offset decrement to safely
skip the unneeded tcc_ptr reading block while accurately aligning with
the relocated arena slot at the very bottom.
With this patch, the tcc_ptr slot is always positioned at a fixed
distance directly underneath the base callee-saved registers that is
independent of whether the arena features are on.
Cc: stable@vger.kernel.org
Fixes: ef54c517a937 ("LoongArch: BPF: Implement PROBE_MEM32 pseudo instructions")
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 47e20d4b | 24-Jul-2026 |
Pu Lehui <pulehui@huawei.com> |
LoongArch: BPF: Fix memory leak in bpf_jit_free()
When bpf_int_jit_compile() is called for subprograms, it returns early
during the first pass (!prog->is_func || extra_pass is false), keeping
ctx->o
LoongArch: BPF: Fix memory leak in bpf_jit_free()
When bpf_int_jit_compile() is called for subprograms, it returns early
during the first pass (!prog->is_func || extra_pass is false), keeping
ctx->offset alive for the subsequent extra pass.
If JIT compilation fails for a later subprogram, the BPF core aborts and
calls bpf_jit_free() to clean up the first subprogram. However,
bpf_jit_free() fails to free jit_data->ctx.offset, which causes a memory
leak of the JIT context offsets array.
So fix this by adding the missing kvfree(jit_data->ctx.offset) in
bpf_jit_free().
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 4ab17e762b34 ("LoongArch: BPF: Use BPF prog pack allocator")
Acked-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 68571a63 | 25-Jun-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Inline bpf_get_smp_processor_id() helper
The pointer to thread_info is always available in the $tp register, so
the call to bpf_get_smp_processor_id() can be inlined into a single lo
LoongArch: BPF: Inline bpf_get_smp_processor_id() helper
The pointer to thread_info is always available in the $tp register, so
the call to bpf_get_smp_processor_id() can be inlined into a single load
instruction.
(1) Here is the sample test.c:
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("raw_tp/sys_enter")
int test_cpuid(void *ctx)
{
return bpf_get_smp_processor_id();
}
char _license[] SEC("license") = "GPL";
(2) Here are the test steps:
sudo yum install libbpf-devel kernel-devel bpftool
clang -target bpf -O2 -c test.c -o test.o
sudo sysctl -w net.core.bpf_jit_enable=1
sudo bpftool prog show name test_cpuid
sudo rm -f /sys/fs/bpf/test_cpuid
sudo bpftool prog load test.o /sys/fs/bpf/test_cpuid
ID=$(sudo bpftool prog show pinned /sys/fs/bpf/test_cpuid | grep -oE '^[0-9]+')
sudo bpftool prog dump jited id $ID
(3) Here are the test results:
Before: 6 instructions
...
64: lu12i.w $t1, 1213
68: ori $t1, $t1, 1680
6c: lu32i.d $t1, 0
70: lu52i.d $t1, $t1, -1792
74: jirl $ra, $t1, 0
78: move $a5, $a0
...
After: 1 instruction
...
64: ld.wu $a5, $tp, 16
...
This is similar with commit 2ddec2c80b44 ("riscv, bpf: inline
bpf_get_smp_processor_id()").
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 7f5c21fc | 25-Jun-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Inline bpf_get_current_task/_btf() helpers
The pointer to task_struct is always available in the $tp register, so
the calls to bpf_get_current_task() and bpf_get_current_task_btf() c
LoongArch: BPF: Inline bpf_get_current_task/_btf() helpers
The pointer to task_struct is always available in the $tp register, so
the calls to bpf_get_current_task() and bpf_get_current_task_btf() can
be inlined into a single move instruction.
(1) Here is the sample test.c:
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("raw_tp/sys_enter")
long test_task(void *ctx)
{
return (long)bpf_get_current_task();
}
char _license[] SEC("license") = "GPL";
(2) Here are the test steps:
sudo yum install libbpf-devel kernel-devel bpftool
clang -target bpf -O2 -c test.c -o test.o
sudo sysctl -w net.core.bpf_jit_enable=1
sudo bpftool prog show name test_task
sudo rm -f /sys/fs/bpf/test_task
sudo bpftool prog load test.o /sys/fs/bpf/test_task
ID=$(sudo bpftool prog show pinned /sys/fs/bpf/test_task | grep -oE '^[0-9]+')
sudo bpftool prog dump jited id $ID
(3) Here are the test results:
Before: 6 instructions
...
64: lu12i.w $t1, 1093
68: ori $t1, $t1, 3320
6c: lu32i.d $t1, 0
70: lu52i.d $t1, $t1, -1792
74: jirl $ra, $t1, 0
78: move $a5, $a0
...
After: 1 instruction
...
64: move $a5, $tp
...
This is similar with commit 2bb138cb20a6 ("bpf, arm64: Inline
bpf_get_current_task/_btf() helpers").
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| 0379d10f | 25-Jun-2026 |
Tiezhu Yang <yangtiezhu@loongson.cn> |
LoongArch: BPF: Fix off-by-one error in tail call
The current code updates the tail call counter (TCC) using a pre-increment
approach, it stores the incremented value back to memory before performin
LoongArch: BPF: Fix off-by-one error in tail call
The current code updates the tail call counter (TCC) using a pre-increment
approach, it stores the incremented value back to memory before performing
any boundary or target validation checks.
This causes two major issues:
1. When a tail call fails because the target program is NULL, the TCC is
incorrectly incremented and saved in memory anyway.
2. This dummy increment implicitly consumes one slot of the allowed tail
call budget. As a result, the subsequent loop reaches the maximum limit
prematurely, leading to a test failure where the actual loop count is
32 instead of the expected 33.
Fix this by deferring the counter update. Change the branch condition to
BPF_JSGE (greater or equal) so that we check the boundary first. The TCC
is only incremented and stored back to memory after the boundary check
and the NULL-target check both pass.
Before:
$ sudo ./test_progs -t tailcalls/tailcall_3
...
test_tailcall_count:FAIL:tailcall count unexpected tailcall count: actual 32 != expected 33
...
#465/3 tailcalls/tailcall_3:FAIL
#465 tailcalls:FAIL
After:
$ sudo ./test_progs -t tailcalls/tailcall_3
#465/3 tailcalls/tailcall_3:OK
#465 tailcalls:OK
Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Fixes: c0fcc955ff82 ("LoongArch: BPF: Fix the tailcall hierarchy")
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|
| e815df29 | 23-Apr-2026 |
Hengqi Chen <hengqi.chen@gmail.com> |
LoongArch: BPF: Add fsession support for trampolines
Implement BPF_TRACE_FSESSION support in LoongArch BPF JIT. The logic
here is almost identical to what has been done in RISC-V JIT.
The key chang
LoongArch: BPF: Add fsession support for trampolines
Implement BPF_TRACE_FSESSION support in LoongArch BPF JIT. The logic
here is almost identical to what has been done in RISC-V JIT.
The key changes are:
- Allocate stack space for function meta and session cookies
- Introduce invoke_bpf() as a wrapper around invoke_bpf_prog() that
populates session cookies before each invocation
- Implement bpf_jit_supports_fsession() callback
Tested-by: Vincent Li <vincent.mc.li@gmail.com>
Reviewed-by: Menglong Dong <menglong8.dong@gmail.com>
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
show more ...
|