| d9ef13f7 | 16-Apr-2026 |
Xu Kuohai <xukuohai@huawei.com> |
bpf: Pass bpf_verifier_env to JIT
Pass bpf_verifier_env to bpf_int_jit_compile(). The follow-up patch will
use env->insn_aux_data in the JIT stage to detect indirect jump targets.
Since bpf_prog_se
bpf: Pass bpf_verifier_env to JIT
Pass bpf_verifier_env to bpf_int_jit_compile(). The follow-up patch will
use env->insn_aux_data in the JIT stage to detect indirect jump targets.
Since bpf_prog_select_runtime() can be called by cbpf and lib/test_bpf.c
code without verifier, introduce helper __bpf_prog_select_runtime()
to accept the env parameter.
Remove the call to bpf_prog_select_runtime() in bpf_prog_load(), and
switch to call __bpf_prog_select_runtime() in the verifier, with env
variable passed. The original bpf_prog_select_runtime() is preserved for
cbpf and lib/test_bpf.c, where env is NULL.
Now all constants blinding calls are moved into the verifier, except
the cbpf and lib/test_bpf.c cases. The instructions arrays are adjusted
by bpf_patch_insn_data() function for normal cases, so there is no need
to call adjust_insn_arrays() in bpf_jit_blind_constants(). Remove it.
Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com> # v8
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com> # v12
Acked-by: Hengqi Chen <hengqi.chen@gmail.com> # v14
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/r/20260416064341.151802-3-xukuohai@huaweicloud.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
show more ...
|
| d3e94522 | 16-Apr-2026 |
Xu Kuohai <xukuohai@huawei.com> |
bpf: Move constants blinding out of arch-specific JITs
During the JIT stage, constants blinding rewrites instructions but only
rewrites the private instruction copy of the JITed subprog, leaving the
bpf: Move constants blinding out of arch-specific JITs
During the JIT stage, constants blinding rewrites instructions but only
rewrites the private instruction copy of the JITed subprog, leaving the
global env->prog->insnsi and env->insn_aux_data untouched. This causes a
mismatch between subprog instructions and the global state, making it
difficult to use the global data in the JIT.
To avoid this mismatch, and given that all arch-specific JITs already
support constants blinding, move it to the generic verifier code, and
switch to rewrite the global env->prog->insnsi with the global states
adjusted, as other rewrites in the verifier do.
This removes the constants blinding calls in each JIT, which are largely
duplicated code across architectures.
Since constants blinding is only required for JIT, and there are two
JIT entry functions, jit_subprogs() for BPF programs with multiple
subprogs and bpf_prog_select_runtime() for programs with no subprogs,
move the constants blinding invocation into these two functions.
In the verifier path, bpf_patch_insn_data() is used to keep global
verifier auxiliary data in sync with patched instructions. A key
question is whether this global auxiliary data should be restored
on the failure path.
Besides instructions, bpf_patch_insn_data() adjusts:
- prog->aux->poke_tab
- env->insn_array_maps
- env->subprog_info
- env->insn_aux_data
For prog->aux->poke_tab, it is only used by JIT or only meaningful after
JIT succeeds, so it does not need to be restored on the failure path.
For env->insn_array_maps, when JIT fails, programs using insn arrays
are rejected by bpf_insn_array_ready() due to missing JIT addresses.
Hence, env->insn_array_maps is only meaningful for JIT and does not need
to be restored.
For subprog_info, if jit_subprogs fails and CONFIG_BPF_JIT_ALWAYS_ON
is not enabled, kernel falls back to interpreter. In this case,
env->subprog_info is used to determine subprogram stack depth. So it
must be restored on failure.
For env->insn_aux_data, it is freed by clear_insn_aux_data() at the
end of bpf_check(). Before freeing, clear_insn_aux_data() loops over
env->insn_aux_data to release jump targets recorded in it. The loop
uses env->prog->len as the array length, but this length no longer
matches the actual size of the adjusted env->insn_aux_data array after
constants blinding.
To address it, a simple approach is to keep insn_aux_data as adjusted
after failure, since it will be freed shortly, and record its actual size
for the loop in clear_insn_aux_data(). But since clear_insn_aux_data()
uses the same index to loop over both env->prog->insnsi and env->insn_aux_data,
this approach results in incorrect index for the insnsi array. So an
alternative approach is adopted: clone the original env->insn_aux_data
before blinding and restore it after failure, similar to env->prog.
For classic BPF programs, constants blinding works as before since it
is still invoked from bpf_prog_select_runtime().
Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com> # v8
Reviewed-by: Hari Bathini <hbathini@linux.ibm.com> # powerpc jit
Reviewed-by: Pu Lehui <pulehui@huawei.com> # riscv jit
Acked-by: Hengqi Chen <hengqi.chen@gmail.com> # loongarch jit
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/r/20260416064341.151802-2-xukuohai@huaweicloud.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
show more ...
|
| 35b3515b | 08-Feb-2026 |
Menglong Dong <menglong8.dong@gmail.com> |
bpf, riscv: add fsession support for trampolines
Implement BPF_TRACE_FSESSION support in the RISC-V trampoline JIT. The
logic here is similar to what we did in x86_64.
In order to simply the logic,
bpf, riscv: add fsession support for trampolines
Implement BPF_TRACE_FSESSION support in the RISC-V trampoline JIT. The
logic here is similar to what we did in x86_64.
In order to simply the logic, we factor out the function invoke_bpf() for
fentry and fexit.
Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
Tested-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/r/20260208053311.698352-3-dongml2@chinatelecom.cn
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
show more ...
|
| ae4a3160 | 18-Nov-2025 |
Menglong Dong <menglong8.dong@gmail.com> |
bpf: specify the old and new poke_type for bpf_arch_text_poke
In the origin logic, the bpf_arch_text_poke() assume that the old and new
instructions have the same opcode. However, they can have diff
bpf: specify the old and new poke_type for bpf_arch_text_poke
In the origin logic, the bpf_arch_text_poke() assume that the old and new
instructions have the same opcode. However, they can have different opcode
if we want to replace a "call" insn with a "jmp" insn.
Therefore, add the new function parameter "old_t" along with the "new_t",
which are used to indicate the old and new poke type. Meanwhile, adjust
the implement of bpf_arch_text_poke() for all the archs.
"BPF_MOD_NOP" is added to make the code more readable. In
bpf_arch_text_poke(), we still check if the new and old address is NULL to
determine if nop insn should be used, which I think is more safe.
Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
Link: https://lore.kernel.org/r/20251118123639.688444-6-dongml2@chinatelecom.cn
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
show more ...
|
| d0bf7cd5 | 22-Sep-2025 |
Chenghao Duan <duanchenghao@kylinos.cn> |
riscv: bpf: Fix uninitialized symbol 'retval_off'
In the __arch_prepare_bpf_trampoline() function, retval_off is only
meaningful when save_ret is true, so the current logic is correct.
However, in t
riscv: bpf: Fix uninitialized symbol 'retval_off'
In the __arch_prepare_bpf_trampoline() function, retval_off is only
meaningful when save_ret is true, so the current logic is correct.
However, in the original logic, retval_off is only initialized under
certain conditions; for example, in the fmod_ret logic, the compiler is
not aware that the flags of the fmod_ret program (prog) have set
BPF_TRAMP_F_CALL_ORIG, which results in an uninitialized symbol
compilation warning.
So initialize retval_off unconditionally to fix it.
Signed-off-by: Chenghao Duan <duanchenghao@kylinos.cn>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/r/20250922062244.822937-2-duanchenghao@kylinos.cn
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
show more ...
|
| fd2e0812 | 08-Sep-2025 |
Hengqi Chen <hengqi.chen@gmail.com> |
riscv, bpf: Sign extend struct ops return values properly
The ns_bpf_qdisc selftest triggers a kernel panic:
Unable to handle kernel paging request at virtual address ffffffffa38dbf58
Curre
riscv, bpf: Sign extend struct ops return values properly
The ns_bpf_qdisc selftest triggers a kernel panic:
Unable to handle kernel paging request at virtual address ffffffffa38dbf58
Current test_progs pgtable: 4K pagesize, 57-bit VAs, pgdp=0x00000001109cc000
[ffffffffa38dbf58] pgd=000000011fffd801, p4d=000000011fffd401, pud=000000011fffd001, pmd=0000000000000000
Oops [#1]
Modules linked in: bpf_testmod(OE) xt_conntrack nls_iso8859_1 [...] [last unloaded: bpf_testmod(OE)]
CPU: 1 UID: 0 PID: 23584 Comm: test_progs Tainted: G W OE 6.17.0-rc1-g2465bb83e0b4 #1 NONE
Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Unknown Unknown Product/Unknown Product, BIOS 2024.01+dfsg-1ubuntu5.1 01/01/2024
epc : __qdisc_run+0x82/0x6f0
ra : __qdisc_run+0x6e/0x6f0
epc : ffffffff80bd5c7a ra : ffffffff80bd5c66 sp : ff2000000eecb550
gp : ffffffff82472098 tp : ff60000096895940 t0 : ffffffff8001f180
t1 : ffffffff801e1664 t2 : 0000000000000000 s0 : ff2000000eecb5d0
s1 : ff60000093a6a600 a0 : ffffffffa38dbee8 a1 : 0000000000000001
a2 : ff2000000eecb510 a3 : 0000000000000001 a4 : 0000000000000000
a5 : 0000000000000010 a6 : 0000000000000000 a7 : 0000000000735049
s2 : ffffffffa38dbee8 s3 : 0000000000000040 s4 : ff6000008bcda000
s5 : 0000000000000008 s6 : ff60000093a6a680 s7 : ff60000093a6a6f0
s8 : ff60000093a6a6ac s9 : ff60000093140000 s10: 0000000000000000
s11: ff2000000eecb9d0 t3 : 0000000000000000 t4 : 0000000000ff0000
t5 : 0000000000000000 t6 : ff60000093a6a8b6
status: 0000000200000120 badaddr: ffffffffa38dbf58 cause: 000000000000000d
[<ffffffff80bd5c7a>] __qdisc_run+0x82/0x6f0
[<ffffffff80b6fe58>] __dev_queue_xmit+0x4c0/0x1128
[<ffffffff80b80ae0>] neigh_resolve_output+0xd0/0x170
[<ffffffff80d2daf6>] ip6_finish_output2+0x226/0x6c8
[<ffffffff80d31254>] ip6_finish_output+0x10c/0x2a0
[<ffffffff80d31446>] ip6_output+0x5e/0x178
[<ffffffff80d2e232>] ip6_xmit+0x29a/0x608
[<ffffffff80d6f4c6>] inet6_csk_xmit+0xe6/0x140
[<ffffffff80c985e4>] __tcp_transmit_skb+0x45c/0xaa8
[<ffffffff80c995fe>] tcp_connect+0x9ce/0xd10
[<ffffffff80d66524>] tcp_v6_connect+0x4ac/0x5e8
[<ffffffff80cc19b8>] __inet_stream_connect+0xd8/0x318
[<ffffffff80cc1c36>] inet_stream_connect+0x3e/0x68
[<ffffffff80b42b20>] __sys_connect_file+0x50/0x88
[<ffffffff80b42bee>] __sys_connect+0x96/0xc8
[<ffffffff80b42c40>] __riscv_sys_connect+0x20/0x30
[<ffffffff80e5bcae>] do_trap_ecall_u+0x256/0x378
[<ffffffff80e69af2>] handle_exception+0x14a/0x156
Code: 892a 0363 1205 489c 8bc1 c7e5 2d03 084a 2703 080a (2783) 0709
---[ end trace 0000000000000000 ]---
The bpf_fifo_dequeue prog returns a skb which is a pointer. The pointer
is treated as a 32bit value and sign extend to 64bit in epilogue. This
behavior is right for most bpf prog types but wrong for struct ops which
requires RISC-V ABI.
So let's sign extend struct ops return values according to the function
model and RISC-V ABI([0]).
[0]: https://riscv.org/wp-content/uploads/2024/12/riscv-calling.pdf
Fixes: 25ad10658dc1 ("riscv, bpf: Adapt bpf trampoline to optimized riscv ftrace framework")
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/bpf/20250908012448.1695-1-hengqi.chen@gmail.com
show more ...
|
| 8a16586f | 12-Aug-2025 |
Radim Krčmář <rkrcmar@ventanamicro.com> |
riscv, bpf: use lw when reading int cpu in bpf_get_smp_processor_id
emit_ld is wrong, because thread_info.cpu is 32-bit, not xlen-bit wide.
The struct currently has a hole after cpu, so little endia
riscv, bpf: use lw when reading int cpu in bpf_get_smp_processor_id
emit_ld is wrong, because thread_info.cpu is 32-bit, not xlen-bit wide.
The struct currently has a hole after cpu, so little endian accesses
seemed fine.
Fixes: 2ddec2c80b44 ("riscv, bpf: inline bpf_get_smp_processor_id()")
Cc: stable@vger.kernel.org
Signed-off-by: Radim Krčmář <rkrcmar@ventanamicro.com>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/r/20250812090256.757273-4-rkrcmar@ventanamicro.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
show more ...
|
| fb7cefab | 19-Jul-2025 |
Pu Lehui <pulehui@huawei.com> |
riscv, bpf: Add support arena atomics for RV64
Add arena atomics support for RMW atomics and load-acquire and
store-release instructions. Non-Zacas cmpxchg is implemented via loop,
which is not curr
riscv, bpf: Add support arena atomics for RV64
Add arena atomics support for RMW atomics and load-acquire and
store-release instructions. Non-Zacas cmpxchg is implemented via loop,
which is not currently supported because it requires more complex
extable and loop logic.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Björn Töpel <bjorn@rivosinc.com>
Reviewed-by: Björn Töpel <bjorn@rivosinc.com>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20250719091730.2660197-10-pulehui@huaweicloud.com
show more ...
|
| b18f4aae | 19-Jul-2025 |
Pu Lehui <pulehui@huawei.com> |
riscv, bpf: Add ex_insn_off and ex_jmp_off for exception table handling
Add ex_insn_off and ex_jmp_off fields to struct rv_jit_context so that
add_exception_handler() does not need to be immediately
riscv, bpf: Add ex_insn_off and ex_jmp_off for exception table handling
Add ex_insn_off and ex_jmp_off fields to struct rv_jit_context so that
add_exception_handler() does not need to be immediately followed by the
instruction to add the exception table. ex_insn_off indicates the offset
of the instruction to add the exception table, and ex_jmp_off indicates
the offset to jump over the faulting instruction. This is to prepare for
adding the exception table to atomic instructions later, because some
atomic instructions need to perform zext or other operations.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Björn Töpel <bjorn@rivosinc.com>
Reviewed-by: Björn Töpel <bjorn@rivosinc.com>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20250719091730.2660197-9-pulehui@huaweicloud.com
show more ...
|
| 1c0196b8 | 19-Jul-2025 |
Pu Lehui <pulehui@huawei.com> |
riscv, bpf: Optimize cmpxchg insn with Zacas support
Optimize cmpxchg instruction with amocas.w and amocas.d
Zacas instructions.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Daniel B
riscv, bpf: Optimize cmpxchg insn with Zacas support
Optimize cmpxchg instruction with amocas.w and amocas.d
Zacas instructions.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Björn Töpel <bjorn@rivosinc.com>
Reviewed-by: Björn Töpel <bjorn@rivosinc.com>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20250719091730.2660197-8-pulehui@huaweicloud.com
show more ...
|
| de39d2c4 | 19-Jul-2025 |
Pu Lehui <pulehui@huawei.com> |
riscv, bpf: Add Zacas instructions
Add Zacas instructions introduced by [0] to reduce code size and
improve performance of RV64 JIT.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Dani
riscv, bpf: Add Zacas instructions
Add Zacas instructions introduced by [0] to reduce code size and
improve performance of RV64 JIT.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Björn Töpel <bjorn@rivosinc.com>
Reviewed-by: Björn Töpel <bjorn@rivosinc.com>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://github.com/riscvarchive/riscv-zacas/releases/download/v1.0/riscv-zacas.pdf [0]
Link: https://lore.kernel.org/bpf/20250719091730.2660197-7-pulehui@huaweicloud.com
show more ...
|
| 5090b339 | 19-Jul-2025 |
Pu Lehui <pulehui@huawei.com> |
riscv, bpf: Add rv_ext_enabled macro for runtime detection extentsion
Add rv_ext_enabled macro to check whether the runtime detection
extension is enabled.
Signed-off-by: Pu Lehui <pulehui@huawei.c
riscv, bpf: Add rv_ext_enabled macro for runtime detection extentsion
Add rv_ext_enabled macro to check whether the runtime detection
extension is enabled.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Björn Töpel <bjorn@rivosinc.com>
Reviewed-by: Björn Töpel <bjorn@rivosinc.com>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20250719091730.2660197-6-pulehui@huaweicloud.com
show more ...
|