| 23f851ac | 10-Jul-2026 |
Pu Hu <hupu@transsion.com> |
arm64: kprobes: Allow reentering kprobes while single-stepping
A kprobe can be hit while another kprobe is in KPROBE_HIT_SS state. This can happen when tracing or perf code runs from the debug excep
arm64: kprobes: Allow reentering kprobes while single-stepping
A kprobe can be hit while another kprobe is in KPROBE_HIT_SS state. This can happen when tracing or perf code runs from the debug exception path while the first kprobe is preparing or executing its out-of-line single-step instruction.
Currently arm64 treats a kprobe hit in KPROBE_HIT_SS as unrecoverable, the same as a hit in KPROBE_REENTER. This is too strict. A hit in KPROBE_HIT_SS is still a one-level reentry and can be handled by saving the current kprobe state and setting up single-step for the new probe, just like reentry from KPROBE_HIT_ACTIVE or KPROBE_HIT_SSDONE.
The truly unrecoverable case is hitting another kprobe while already in KPROBE_REENTER, because the reentry save area has already been consumed.
Move KPROBE_HIT_SS to the recoverable reentry cases and leave KPROBE_REENTER as the unrecoverable nested reentry case.
This change also requires saving saved_irqflag in struct prev_kprobe. When a nested kprobe calls kprobes_save_local_irqflag(), it overwrites kcb->saved_irqflag with the currently masked DAIF value, losing the outer kprobe's original DAIF state. Without this fix, when the outer kprobe's single-step finishes, kprobes_restore_local_irqflag() applies the wrong DAIF mask and leaves interrupts permanently disabled.
Extend struct prev_kprobe with a saved_irqflag field and save/restore it alongside kp and status. This ensures the outer kprobe's original interrupt state is preserved across reentry.
This mirrors the x86 fix in commit 6a5022a56ac3 ("kprobes/x86: Allow to handle reentered kprobe on single-stepping").
Signed-off-by: Pu Hu <hupu@transsion.com> Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com> Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
show more ...
|
| ea87c553 | 18-Sep-2025 |
Jeremy Linton <jeremy.linton@arm.com> |
arm64: probes: Fix incorrect bl/blr address and register usage
The pt_regs registers are 64-bit on arm64, and should be u64 when manipulated. Correct this so that we aren't truncating the address du
arm64: probes: Fix incorrect bl/blr address and register usage
The pt_regs registers are 64-bit on arm64, and should be u64 when manipulated. Correct this so that we aren't truncating the address during br/blr sequences.
Fixes: efb07ac534e2 ("arm64: probes: Add GCS support to bl/blr/ret") Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> Signed-off-by: Will Deacon <will@kernel.org>
show more ...
|
| 4a601714 | 25-Aug-2025 |
Jeremy Linton <jeremy.linton@arm.com> |
arm64: uprobes: Add GCS support to uretprobes
Ret probes work by changing the value in the link register at the probe location to return to the probe rather than the calling routine. Thus the GCS ne
arm64: uprobes: Add GCS support to uretprobes
Ret probes work by changing the value in the link register at the probe location to return to the probe rather than the calling routine. Thus the GCS needs to be updated with this address as well.
Since its possible to insert probes at locations where the current value of the LR doesn't match the GCS state this needs to be detected and handled in order to maintain the existing no-fault behavior.
Co-developed-by: Steve Capper <steve.capper@arm.com> Signed-off-by: Steve Capper <steve.capper@arm.com> Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> [will: Add '__force' to gcspr casts in arch_uretprobe_hijack_return_addr()] Signed-off-by: Will Deacon <will@kernel.org>
show more ...
|
| efb07ac5 | 25-Aug-2025 |
Jeremy Linton <jeremy.linton@arm.com> |
arm64: probes: Add GCS support to bl/blr/ret
The arm64 probe simulation doesn't currently have logic in place to deal with GCS and this results in core dumps if probes are inserted at control flow l
arm64: probes: Add GCS support to bl/blr/ret
The arm64 probe simulation doesn't currently have logic in place to deal with GCS and this results in core dumps if probes are inserted at control flow locations. Fix-up bl, blr and ret to manipulate the shadow stack as needed.
While we manipulate and validate the shadow stack correctly, the hardware provides additional security by only allowing GCS operations against pages which are marked to support GCS. For writing there is gcssttr() which enforces this, but there isn't an equivalent for reading. This means that uprobe users should be aware that probing on control flow instructions which require reading the shadow stack (ex: ret) offers lower security guarantees than what is achieved without the uprobe active.
Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will@kernel.org>
show more ...
|
| 403b48aa | 07-Jul-2025 |
Ada Couprie Diaz <ada.coupriediaz@arm.com> |
arm64: debug: call step handlers statically
Software stepping checks for the correct handler by iterating over a list of dynamically registered handlers and calling all of them until one handles the
arm64: debug: call step handlers statically
Software stepping checks for the correct handler by iterating over a list of dynamically registered handlers and calling all of them until one handles the exception.
This is the only generic way to handle software stepping handlers in arm64 as the exception does not provide an immediate that could be checked, contrary to software breakpoints.
However, the registration mechanism is not exported and has only two current users : the KGDB stepping handler, and the uprobe single step handler. Given that one comes from user mode and the other from kernel mode, call the appropriate one by checking the source EL of the exception. Add a stand-in that returns DBG_HOOK_ERROR when the configuration options are not enabled.
Remove `arch_init_uprobes()` as it is not useful anymore and is specific to arm64.
Unify the naming of the handler to XXX_single_step_handler(), making it clear they are related.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com> Tested-by: Luis Claudio R. Goncalves <lgoncalv@redhat.com> Reviewed-by: Will Deacon <will@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20250707114109.35672-5-ada.coupriediaz@arm.com Signed-off-by: Will Deacon <will@kernel.org>
show more ...
|
| ac4ad5c0 | 09-Sep-2024 |
Liao Chang <liaochang1@huawei.com> |
arm64: insn: Simulate nop instruction for better uprobe performance
v2->v1: 1. Remove the simuation of STP and the related bits. 2. Use arm64_skip_faulting_instruction for single-stepping or FEAT_BT
arm64: insn: Simulate nop instruction for better uprobe performance
v2->v1: 1. Remove the simuation of STP and the related bits. 2. Use arm64_skip_faulting_instruction for single-stepping or FEAT_BTI scenario.
As Andrii pointed out, the uprobe/uretprobe selftest bench run into a counterintuitive result that nop and push variants are much slower than ret variant [0]. The root cause lies in the arch_probe_analyse_insn(), which excludes 'nop' and 'stp' from the emulatable instructions list. This force the kernel returns to userspace and execute them out-of-line, then trapping back to kernel for running uprobe callback functions. This leads to a significant performance overhead compared to 'ret' variant, which is already emulated.
Typicall uprobe is installed on 'nop' for USDT and on function entry which starts with the instrucion 'stp x29, x30, [sp, #imm]!' to push lr and fp into stack regardless kernel or userspace binary. In order to improve the performance of handling uprobe for common usecases. This patch supports the emulation of Arm64 equvialents instructions of 'nop' and 'push'. The benchmark results below indicates the performance gain of emulation is obvious.
On Kunpeng916 (Hi1616), 4 NUMA nodes, 64 Arm64 cores@2.4GHz.
xol (1 cpus) ------------ uprobe-nop: 0.916 ± 0.001M/s (0.916M/prod) uprobe-push: 0.908 ± 0.001M/s (0.908M/prod) uprobe-ret: 1.855 ± 0.000M/s (1.855M/prod) uretprobe-nop: 0.640 ± 0.000M/s (0.640M/prod) uretprobe-push: 0.633 ± 0.001M/s (0.633M/prod) uretprobe-ret: 0.978 ± 0.003M/s (0.978M/prod)
emulation (1 cpus) ------------------- uprobe-nop: 1.862 ± 0.002M/s (1.862M/prod) uprobe-push: 1.743 ± 0.006M/s (1.743M/prod) uprobe-ret: 1.840 ± 0.001M/s (1.840M/prod) uretprobe-nop: 0.964 ± 0.004M/s (0.964M/prod) uretprobe-push: 0.936 ± 0.004M/s (0.936M/prod) uretprobe-ret: 0.940 ± 0.001M/s (0.940M/prod)
As shown above, the performance gap between 'nop/push' and 'ret' variants has been significantly reduced. Due to the emulation of 'push' instruction needs to access userspace memory, it spent more cycles than the other.
As Mark suggested [1], it is painful to emulate the correct atomicity and ordering properties of STP, especially when it interacts with MTE, POE, etc. So this patch just focus on the simuation of 'nop'. The simluation of STP and related changes will be addressed in a separate patch.
[0] https://lore.kernel.org/all/CAEf4BzaO4eG6hr2hzXYpn+7Uer4chS0R99zLn02ezZ5YruVuQw@mail.gmail.com/ [1] https://lore.kernel.org/all/Zr3RN4zxF5XPgjEB@J2N7QTR9R3/
CC: Andrii Nakryiko <andrii.nakryiko@gmail.com> CC: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Liao Chang <liaochang1@huawei.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20240909071114.1150053-1-liaochang1@huawei.com [catalin.marinas@arm.com: small tweaks following MarkR's comments] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
show more ...
|
| 14762109 | 08-Oct-2024 |
Mark Rutland <mark.rutland@arm.com> |
arm64: probes: Remove probe_opcode_t
The probe_opcode_t typedef for u32 isn't necessary, and is a source of confusion as it is easily confused with kprobe_opcode_t, which is a typedef for __le32.
T
arm64: probes: Remove probe_opcode_t
The probe_opcode_t typedef for u32 isn't necessary, and is a source of confusion as it is easily confused with kprobe_opcode_t, which is a typedef for __le32.
The typedef is only used within arch/arm64, and all of arm64's commn insn code uses u32 for the endian-agnostic value of an instruction, so it'd be clearer to use u32 consistently.
Remove probe_opcode_t and use u32 directly.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: Catalin Marinas <catalin.marnias@arm.com> Cc: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20241008155851.801546-7-mark.rutland@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
show more ...
|
| dd0eb50e | 08-Oct-2024 |
Mark Rutland <mark.rutland@arm.com> |
arm64: probes: Cleanup kprobes endianness conversions
The core kprobes code uses kprobe_opcode_t for the in-memory representation of an instruction, using 'kprobe_opcode_t *' for XOL slots. As arm64
arm64: probes: Cleanup kprobes endianness conversions
The core kprobes code uses kprobe_opcode_t for the in-memory representation of an instruction, using 'kprobe_opcode_t *' for XOL slots. As arm64 instructions are always little-endian 32-bit values, kprobes_opcode_t should be __le32, but at the moment kprobe_opcode_t is typedef'd to u32.
Today there is no functional issue as we convert values via cpu_to_le32() and le32_to_cpu() where necessary, but these conversions are inconsistent with the types used, causing sparse warnings:
| CHECK arch/arm64/kernel/probes/kprobes.c | arch/arm64/kernel/probes/kprobes.c:102:21: warning: cast to restricted __le32 | CHECK arch/arm64/kernel/probes/decode-insn.c | arch/arm64/kernel/probes/decode-insn.c:122:46: warning: cast to restricted __le32 | arch/arm64/kernel/probes/decode-insn.c:124:50: warning: cast to restricted __le32 | arch/arm64/kernel/probes/decode-insn.c:136:31: warning: cast to restricted __le32
Improve this by making kprobes_opcode_t a typedef for __le32 and consistently using this for pointers to executable instructions. With this change we can rely on the type system to tell us where conversions are necessary.
Since kprobe::opcode is changed from u32 to __le32, the existing le32_to_cpu() converion moves from the point this is initialized (in arch_prepare_kprobe()) to the points this is consumed when passed to a handler or text patching function. As kprobe::opcode isn't altered or consumed elsewhere, this shouldn't result in a functional change.
Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20241008155851.801546-6-mark.rutland@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
show more ...
|
| 6105c5d4 | 08-Oct-2024 |
Mark Rutland <mark.rutland@arm.com> |
arm64: probes: Move kprobes-specific fields
We share struct arch_probe_insn between krpboes and uprobes, but most of its fields aren't necessary for uprobes:
* The 'insn' field is only used by kpro
arm64: probes: Move kprobes-specific fields
We share struct arch_probe_insn between krpboes and uprobes, but most of its fields aren't necessary for uprobes:
* The 'insn' field is only used by kprobes as a pointer to the XOL slot.
* The 'restore' field is only used by probes as the PC to restore after stepping an instruction in the XOL slot.
* The 'pstate_cc' field isn't used by kprobes or uprobes, and seems to only exist as a result of copy-pasting the 32-bit arm implementation of kprobes.
As these fields live in struct arch_probe_insn they cannot use definitions that only exist when CONFIG_KPROBES=y, such as the kprobe_opcode_t typedef, which we'd like to use in subsequent patches.
Clean this up by removing the 'pstate_cc' field, and moving the kprobes-specific fields into the kprobes-specific struct arch_specific_insn. To make it clear that the fields are related to stepping instructions in the XOL slot, 'insn' is renamed to 'xol_insn' and 'restore' is renamed to 'xol_restore'
At the same time, remove the misleading and useless comment above struct arch_probe_insn.
The should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20241008155851.801546-5-mark.rutland@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
show more ...
|