1.. SPDX-License-Identifier: GPL-2.0 2 3.. _kfuncs-header-label: 4 5============================= 6BPF Kernel Functions (kfuncs) 7============================= 8 91. Introduction 10=============== 11 12BPF Kernel Functions or more commonly known as kfuncs are functions in the Linux 13kernel which are exposed for use by BPF programs. Unlike normal BPF helpers, 14kfuncs do not have a stable interface and can change from one kernel release to 15another. Hence, BPF programs need to be updated in response to changes in the 16kernel. See :ref:`BPF_kfunc_lifecycle_expectations` for more information. 17 182. Defining a kfunc 19=================== 20 21There are two ways to expose a kernel function to BPF programs, either make an 22existing function in the kernel visible, or add a new wrapper for BPF. In both 23cases, care must be taken that BPF program can only call such function in a 24valid context. To enforce this, visibility of a kfunc can be per program type. 25 26If you are not creating a BPF wrapper for existing kernel function, skip ahead 27to :ref:`BPF_kfunc_nodef`. 28 292.1 Creating a wrapper kfunc 30---------------------------- 31 32When defining a wrapper kfunc, the wrapper function should have extern linkage. 33This prevents the compiler from optimizing away dead code, as this wrapper kfunc 34is not invoked anywhere in the kernel itself. It is not necessary to provide a 35prototype in a header for the wrapper kfunc. 36 37An example is given below:: 38 39 /* Disables missing prototype warnings */ 40 __diag_push(); 41 __diag_ignore_all("-Wmissing-prototypes", 42 "Global kfuncs as their definitions will be in BTF"); 43 44 __bpf_kfunc struct task_struct *bpf_find_get_task_by_vpid(pid_t nr) 45 { 46 return find_get_task_by_vpid(nr); 47 } 48 49 __diag_pop(); 50 51A wrapper kfunc is often needed when we need to annotate parameters of the 52kfunc. Otherwise one may directly make the kfunc visible to the BPF program by 53registering it with the BPF subsystem. See :ref:`BPF_kfunc_nodef`. 54 552.2 Annotating kfunc parameters 56------------------------------- 57 58Similar to BPF helpers, there is sometime need for additional context required 59by the verifier to make the usage of kernel functions safer and more useful. 60Hence, we can annotate a parameter by suffixing the name of the argument of the 61kfunc with a __tag, where tag may be one of the supported annotations. 62 632.2.1 __sz Annotation 64--------------------- 65 66This annotation is used to indicate a memory and size pair in the argument list. 67An example is given below:: 68 69 __bpf_kfunc void bpf_memzero(void *mem, int mem__sz) 70 { 71 ... 72 } 73 74Here, the verifier will treat first argument as a PTR_TO_MEM, and second 75argument as its size. By default, without __sz annotation, the size of the type 76of the pointer is used. Without __sz annotation, a kfunc cannot accept a void 77pointer. 78 792.2.2 __k Annotation 80-------------------- 81 82This annotation is only understood for scalar arguments, where it indicates that 83the verifier must check the scalar argument to be a known constant, which does 84not indicate a size parameter, and the value of the constant is relevant to the 85safety of the program. 86 87An example is given below:: 88 89 __bpf_kfunc void *bpf_obj_new(u32 local_type_id__k, ...) 90 { 91 ... 92 } 93 94Here, bpf_obj_new uses local_type_id argument to find out the size of that type 95ID in program's BTF and return a sized pointer to it. Each type ID will have a 96distinct size, hence it is crucial to treat each such call as distinct when 97values don't match during verifier state pruning checks. 98 99Hence, whenever a constant scalar argument is accepted by a kfunc which is not a 100size parameter, and the value of the constant matters for program safety, __k 101suffix should be used. 102 1032.2.2 __uninit Annotation 104------------------------- 105 106This annotation is used to indicate that the argument will be treated as 107uninitialized. 108 109An example is given below:: 110 111 __bpf_kfunc int bpf_dynptr_from_skb(..., struct bpf_dynptr_kern *ptr__uninit) 112 { 113 ... 114 } 115 116Here, the dynptr will be treated as an uninitialized dynptr. Without this 117annotation, the verifier will reject the program if the dynptr passed in is 118not initialized. 119 120.. _BPF_kfunc_nodef: 121 1222.3 Using an existing kernel function 123------------------------------------- 124 125When an existing function in the kernel is fit for consumption by BPF programs, 126it can be directly registered with the BPF subsystem. However, care must still 127be taken to review the context in which it will be invoked by the BPF program 128and whether it is safe to do so. 129 1302.4 Annotating kfuncs 131--------------------- 132 133In addition to kfuncs' arguments, verifier may need more information about the 134type of kfunc(s) being registered with the BPF subsystem. To do so, we define 135flags on a set of kfuncs as follows:: 136 137 BTF_SET8_START(bpf_task_set) 138 BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL) 139 BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE) 140 BTF_SET8_END(bpf_task_set) 141 142This set encodes the BTF ID of each kfunc listed above, and encodes the flags 143along with it. Ofcourse, it is also allowed to specify no flags. 144 145kfunc definitions should also always be annotated with the ``__bpf_kfunc`` 146macro. This prevents issues such as the compiler inlining the kfunc if it's a 147static kernel function, or the function being elided in an LTO build as it's 148not used in the rest of the kernel. Developers should not manually add 149annotations to their kfunc to prevent these issues. If an annotation is 150required to prevent such an issue with your kfunc, it is a bug and should be 151added to the definition of the macro so that other kfuncs are similarly 152protected. An example is given below:: 153 154 __bpf_kfunc struct task_struct *bpf_get_task_pid(s32 pid) 155 { 156 ... 157 } 158 1592.4.1 KF_ACQUIRE flag 160--------------------- 161 162The KF_ACQUIRE flag is used to indicate that the kfunc returns a pointer to a 163refcounted object. The verifier will then ensure that the pointer to the object 164is eventually released using a release kfunc, or transferred to a map using a 165referenced kptr (by invoking bpf_kptr_xchg). If not, the verifier fails the 166loading of the BPF program until no lingering references remain in all possible 167explored states of the program. 168 1692.4.2 KF_RET_NULL flag 170---------------------- 171 172The KF_RET_NULL flag is used to indicate that the pointer returned by the kfunc 173may be NULL. Hence, it forces the user to do a NULL check on the pointer 174returned from the kfunc before making use of it (dereferencing or passing to 175another helper). This flag is often used in pairing with KF_ACQUIRE flag, but 176both are orthogonal to each other. 177 1782.4.3 KF_RELEASE flag 179--------------------- 180 181The KF_RELEASE flag is used to indicate that the kfunc releases the pointer 182passed in to it. There can be only one referenced pointer that can be passed 183in. All copies of the pointer being released are invalidated as a result of 184invoking kfunc with this flag. KF_RELEASE kfuncs automatically receive the 185protection afforded by the KF_TRUSTED_ARGS flag described below. 186 1872.4.4 KF_TRUSTED_ARGS flag 188-------------------------- 189 190The KF_TRUSTED_ARGS flag is used for kfuncs taking pointer arguments. It 191indicates that the all pointer arguments are valid, and that all pointers to 192BTF objects have been passed in their unmodified form (that is, at a zero 193offset, and without having been obtained from walking another pointer, with one 194exception described below). 195 196There are two types of pointers to kernel objects which are considered "valid": 197 1981. Pointers which are passed as tracepoint or struct_ops callback arguments. 1992. Pointers which were returned from a KF_ACQUIRE kfunc. 200 201Pointers to non-BTF objects (e.g. scalar pointers) may also be passed to 202KF_TRUSTED_ARGS kfuncs, and may have a non-zero offset. 203 204The definition of "valid" pointers is subject to change at any time, and has 205absolutely no ABI stability guarantees. 206 207As mentioned above, a nested pointer obtained from walking a trusted pointer is 208no longer trusted, with one exception. If a struct type has a field that is 209guaranteed to be valid as long as its parent pointer is trusted, the 210``BTF_TYPE_SAFE_NESTED`` macro can be used to express that to the verifier as 211follows: 212 213.. code-block:: c 214 215 BTF_TYPE_SAFE_NESTED(struct task_struct) { 216 const cpumask_t *cpus_ptr; 217 }; 218 219In other words, you must: 220 2211. Wrap the trusted pointer type in the ``BTF_TYPE_SAFE_NESTED`` macro. 222 2232. Specify the type and name of the trusted nested field. This field must match 224 the field in the original type definition exactly. 225 2262.4.5 KF_SLEEPABLE flag 227----------------------- 228 229The KF_SLEEPABLE flag is used for kfuncs that may sleep. Such kfuncs can only 230be called by sleepable BPF programs (BPF_F_SLEEPABLE). 231 2322.4.6 KF_DESTRUCTIVE flag 233-------------------------- 234 235The KF_DESTRUCTIVE flag is used to indicate functions calling which is 236destructive to the system. For example such a call can result in system 237rebooting or panicking. Due to this additional restrictions apply to these 238calls. At the moment they only require CAP_SYS_BOOT capability, but more can be 239added later. 240 2412.4.7 KF_RCU flag 242----------------- 243 244The KF_RCU flag is a weaker version of KF_TRUSTED_ARGS. The kfuncs marked with 245KF_RCU expect either PTR_TRUSTED or MEM_RCU arguments. The verifier guarantees 246that the objects are valid and there is no use-after-free. The pointers are not 247NULL, but the object's refcount could have reached zero. The kfuncs need to 248consider doing refcnt != 0 check, especially when returning a KF_ACQUIRE 249pointer. Note as well that a KF_ACQUIRE kfunc that is KF_RCU should very likely 250also be KF_RET_NULL. 251 252.. _KF_deprecated_flag: 253 2542.4.8 KF_DEPRECATED flag 255------------------------ 256 257The KF_DEPRECATED flag is used for kfuncs which are scheduled to be 258changed or removed in a subsequent kernel release. A kfunc that is 259marked with KF_DEPRECATED should also have any relevant information 260captured in its kernel doc. Such information typically includes the 261kfunc's expected remaining lifespan, a recommendation for new 262functionality that can replace it if any is available, and possibly a 263rationale for why it is being removed. 264 265Note that while on some occasions, a KF_DEPRECATED kfunc may continue to be 266supported and have its KF_DEPRECATED flag removed, it is likely to be far more 267difficult to remove a KF_DEPRECATED flag after it's been added than it is to 268prevent it from being added in the first place. As described in 269:ref:`BPF_kfunc_lifecycle_expectations`, users that rely on specific kfuncs are 270encouraged to make their use-cases known as early as possible, and participate 271in upstream discussions regarding whether to keep, change, deprecate, or remove 272those kfuncs if and when such discussions occur. 273 2742.5 Registering the kfuncs 275-------------------------- 276 277Once the kfunc is prepared for use, the final step to making it visible is 278registering it with the BPF subsystem. Registration is done per BPF program 279type. An example is shown below:: 280 281 BTF_SET8_START(bpf_task_set) 282 BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL) 283 BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE) 284 BTF_SET8_END(bpf_task_set) 285 286 static const struct btf_kfunc_id_set bpf_task_kfunc_set = { 287 .owner = THIS_MODULE, 288 .set = &bpf_task_set, 289 }; 290 291 static int init_subsystem(void) 292 { 293 return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_task_kfunc_set); 294 } 295 late_initcall(init_subsystem); 296 2972.6 Specifying no-cast aliases with ___init 298-------------------------------------------- 299 300The verifier will always enforce that the BTF type of a pointer passed to a 301kfunc by a BPF program, matches the type of pointer specified in the kfunc 302definition. The verifier, does, however, allow types that are equivalent 303according to the C standard to be passed to the same kfunc arg, even if their 304BTF_IDs differ. 305 306For example, for the following type definition: 307 308.. code-block:: c 309 310 struct bpf_cpumask { 311 cpumask_t cpumask; 312 refcount_t usage; 313 }; 314 315The verifier would allow a ``struct bpf_cpumask *`` to be passed to a kfunc 316taking a ``cpumask_t *`` (which is a typedef of ``struct cpumask *``). For 317instance, both ``struct cpumask *`` and ``struct bpf_cpmuask *`` can be passed 318to bpf_cpumask_test_cpu(). 319 320In some cases, this type-aliasing behavior is not desired. ``struct 321nf_conn___init`` is one such example: 322 323.. code-block:: c 324 325 struct nf_conn___init { 326 struct nf_conn ct; 327 }; 328 329The C standard would consider these types to be equivalent, but it would not 330always be safe to pass either type to a trusted kfunc. ``struct 331nf_conn___init`` represents an allocated ``struct nf_conn`` object that has 332*not yet been initialized*, so it would therefore be unsafe to pass a ``struct 333nf_conn___init *`` to a kfunc that's expecting a fully initialized ``struct 334nf_conn *`` (e.g. ``bpf_ct_change_timeout()``). 335 336In order to accommodate such requirements, the verifier will enforce strict 337PTR_TO_BTF_ID type matching if two types have the exact same name, with one 338being suffixed with ``___init``. 339 340.. _BPF_kfunc_lifecycle_expectations: 341 3423. kfunc lifecycle expectations 343=============================== 344 345kfuncs provide a kernel <-> kernel API, and thus are not bound by any of the 346strict stability restrictions associated with kernel <-> user UAPIs. This means 347they can be thought of as similar to EXPORT_SYMBOL_GPL, and can therefore be 348modified or removed by a maintainer of the subsystem they're defined in when 349it's deemed necessary. 350 351Like any other change to the kernel, maintainers will not change or remove a 352kfunc without having a reasonable justification. Whether or not they'll choose 353to change a kfunc will ultimately depend on a variety of factors, such as how 354widely used the kfunc is, how long the kfunc has been in the kernel, whether an 355alternative kfunc exists, what the norm is in terms of stability for the 356subsystem in question, and of course what the technical cost is of continuing 357to support the kfunc. 358 359There are several implications of this: 360 361a) kfuncs that are widely used or have been in the kernel for a long time will 362 be more difficult to justify being changed or removed by a maintainer. In 363 other words, kfuncs that are known to have a lot of users and provide 364 significant value provide stronger incentives for maintainers to invest the 365 time and complexity in supporting them. It is therefore important for 366 developers that are using kfuncs in their BPF programs to communicate and 367 explain how and why those kfuncs are being used, and to participate in 368 discussions regarding those kfuncs when they occur upstream. 369 370b) Unlike regular kernel symbols marked with EXPORT_SYMBOL_GPL, BPF programs 371 that call kfuncs are generally not part of the kernel tree. This means that 372 refactoring cannot typically change callers in-place when a kfunc changes, 373 as is done for e.g. an upstreamed driver being updated in place when a 374 kernel symbol is changed. 375 376 Unlike with regular kernel symbols, this is expected behavior for BPF 377 symbols, and out-of-tree BPF programs that use kfuncs should be considered 378 relevant to discussions and decisions around modifying and removing those 379 kfuncs. The BPF community will take an active role in participating in 380 upstream discussions when necessary to ensure that the perspectives of such 381 users are taken into account. 382 383c) A kfunc will never have any hard stability guarantees. BPF APIs cannot and 384 will not ever hard-block a change in the kernel purely for stability 385 reasons. That being said, kfuncs are features that are meant to solve 386 problems and provide value to users. The decision of whether to change or 387 remove a kfunc is a multivariate technical decision that is made on a 388 case-by-case basis, and which is informed by data points such as those 389 mentioned above. It is expected that a kfunc being removed or changed with 390 no warning will not be a common occurrence or take place without sound 391 justification, but it is a possibility that must be accepted if one is to 392 use kfuncs. 393 3943.1 kfunc deprecation 395--------------------- 396 397As described above, while sometimes a maintainer may find that a kfunc must be 398changed or removed immediately to accommodate some changes in their subsystem, 399usually kfuncs will be able to accommodate a longer and more measured 400deprecation process. For example, if a new kfunc comes along which provides 401superior functionality to an existing kfunc, the existing kfunc may be 402deprecated for some period of time to allow users to migrate their BPF programs 403to use the new one. Or, if a kfunc has no known users, a decision may be made 404to remove the kfunc (without providing an alternative API) after some 405deprecation period so as to provide users with a window to notify the kfunc 406maintainer if it turns out that the kfunc is actually being used. 407 408It's expected that the common case will be that kfuncs will go through a 409deprecation period rather than being changed or removed without warning. As 410described in :ref:`KF_deprecated_flag`, the kfunc framework provides the 411KF_DEPRECATED flag to kfunc developers to signal to users that a kfunc has been 412deprecated. Once a kfunc has been marked with KF_DEPRECATED, the following 413procedure is followed for removal: 414 4151. Any relevant information for deprecated kfuncs is documented in the kfunc's 416 kernel docs. This documentation will typically include the kfunc's expected 417 remaining lifespan, a recommendation for new functionality that can replace 418 the usage of the deprecated function (or an explanation as to why no such 419 replacement exists), etc. 420 4212. The deprecated kfunc is kept in the kernel for some period of time after it 422 was first marked as deprecated. This time period will be chosen on a 423 case-by-case basis, and will typically depend on how widespread the use of 424 the kfunc is, how long it has been in the kernel, and how hard it is to move 425 to alternatives. This deprecation time period is "best effort", and as 426 described :ref:`above<BPF_kfunc_lifecycle_expectations>`, circumstances may 427 sometimes dictate that the kfunc be removed before the full intended 428 deprecation period has elapsed. 429 4303. After the deprecation period the kfunc will be removed. At this point, BPF 431 programs calling the kfunc will be rejected by the verifier. 432 4334. Core kfuncs 434============== 435 436The BPF subsystem provides a number of "core" kfuncs that are potentially 437applicable to a wide variety of different possible use cases and programs. 438Those kfuncs are documented here. 439 4404.1 struct task_struct * kfuncs 441------------------------------- 442 443There are a number of kfuncs that allow ``struct task_struct *`` objects to be 444used as kptrs: 445 446.. kernel-doc:: kernel/bpf/helpers.c 447 :identifiers: bpf_task_acquire bpf_task_release 448 449These kfuncs are useful when you want to acquire or release a reference to a 450``struct task_struct *`` that was passed as e.g. a tracepoint arg, or a 451struct_ops callback arg. For example: 452 453.. code-block:: c 454 455 /** 456 * A trivial example tracepoint program that shows how to 457 * acquire and release a struct task_struct * pointer. 458 */ 459 SEC("tp_btf/task_newtask") 460 int BPF_PROG(task_acquire_release_example, struct task_struct *task, u64 clone_flags) 461 { 462 struct task_struct *acquired; 463 464 acquired = bpf_task_acquire(task); 465 if (acquired) 466 /* 467 * In a typical program you'd do something like store 468 * the task in a map, and the map will automatically 469 * release it later. Here, we release it manually. 470 */ 471 bpf_task_release(acquired); 472 return 0; 473 } 474 475 476References acquired on ``struct task_struct *`` objects are RCU protected. 477Therefore, when in an RCU read region, you can obtain a pointer to a task 478embedded in a map value without having to acquire a reference: 479 480.. code-block:: c 481 482 #define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8))) 483 private(TASK) static struct task_struct *global; 484 485 /** 486 * A trivial example showing how to access a task stored 487 * in a map using RCU. 488 */ 489 SEC("tp_btf/task_newtask") 490 int BPF_PROG(task_rcu_read_example, struct task_struct *task, u64 clone_flags) 491 { 492 struct task_struct *local_copy; 493 494 bpf_rcu_read_lock(); 495 local_copy = global; 496 if (local_copy) 497 /* 498 * We could also pass local_copy to kfuncs or helper functions here, 499 * as we're guaranteed that local_copy will be valid until we exit 500 * the RCU read region below. 501 */ 502 bpf_printk("Global task %s is valid", local_copy->comm); 503 else 504 bpf_printk("No global task found"); 505 bpf_rcu_read_unlock(); 506 507 /* At this point we can no longer reference local_copy. */ 508 509 return 0; 510 } 511 512---- 513 514A BPF program can also look up a task from a pid. This can be useful if the 515caller doesn't have a trusted pointer to a ``struct task_struct *`` object that 516it can acquire a reference on with bpf_task_acquire(). 517 518.. kernel-doc:: kernel/bpf/helpers.c 519 :identifiers: bpf_task_from_pid 520 521Here is an example of it being used: 522 523.. code-block:: c 524 525 SEC("tp_btf/task_newtask") 526 int BPF_PROG(task_get_pid_example, struct task_struct *task, u64 clone_flags) 527 { 528 struct task_struct *lookup; 529 530 lookup = bpf_task_from_pid(task->pid); 531 if (!lookup) 532 /* A task should always be found, as %task is a tracepoint arg. */ 533 return -ENOENT; 534 535 if (lookup->pid != task->pid) { 536 /* bpf_task_from_pid() looks up the task via its 537 * globally-unique pid from the init_pid_ns. Thus, 538 * the pid of the lookup task should always be the 539 * same as the input task. 540 */ 541 bpf_task_release(lookup); 542 return -EINVAL; 543 } 544 545 /* bpf_task_from_pid() returns an acquired reference, 546 * so it must be dropped before returning from the 547 * tracepoint handler. 548 */ 549 bpf_task_release(lookup); 550 return 0; 551 } 552 5534.2 struct cgroup * kfuncs 554-------------------------- 555 556``struct cgroup *`` objects also have acquire and release functions: 557 558.. kernel-doc:: kernel/bpf/helpers.c 559 :identifiers: bpf_cgroup_acquire bpf_cgroup_release 560 561These kfuncs are used in exactly the same manner as bpf_task_acquire() and 562bpf_task_release() respectively, so we won't provide examples for them. 563 564---- 565 566Other kfuncs available for interacting with ``struct cgroup *`` objects are 567bpf_cgroup_ancestor() and bpf_cgroup_from_id(), allowing callers to access 568the ancestor of a cgroup and find a cgroup by its ID, respectively. Both 569return a cgroup kptr. 570 571.. kernel-doc:: kernel/bpf/helpers.c 572 :identifiers: bpf_cgroup_ancestor 573 574.. kernel-doc:: kernel/bpf/helpers.c 575 :identifiers: bpf_cgroup_from_id 576 577Eventually, BPF should be updated to allow this to happen with a normal memory 578load in the program itself. This is currently not possible without more work in 579the verifier. bpf_cgroup_ancestor() can be used as follows: 580 581.. code-block:: c 582 583 /** 584 * Simple tracepoint example that illustrates how a cgroup's 585 * ancestor can be accessed using bpf_cgroup_ancestor(). 586 */ 587 SEC("tp_btf/cgroup_mkdir") 588 int BPF_PROG(cgrp_ancestor_example, struct cgroup *cgrp, const char *path) 589 { 590 struct cgroup *parent; 591 592 /* The parent cgroup resides at the level before the current cgroup's level. */ 593 parent = bpf_cgroup_ancestor(cgrp, cgrp->level - 1); 594 if (!parent) 595 return -ENOENT; 596 597 bpf_printk("Parent id is %d", parent->self.id); 598 599 /* Return the parent cgroup that was acquired above. */ 600 bpf_cgroup_release(parent); 601 return 0; 602 } 603 6044.3 struct cpumask * kfuncs 605--------------------------- 606 607BPF provides a set of kfuncs that can be used to query, allocate, mutate, and 608destroy struct cpumask * objects. Please refer to :ref:`cpumasks-header-label` 609for more details. 610