LinuxKPI: move __kmalloc from slab.h to slab.c

In order to allow the allocator to change in the future move it into
the implementation file from being an inline function in the header.

While here f

LinuxKPI: move __kmalloc from slab.h to slab.c

In order to allow the allocator to change in the future move it into
the implementation file from being an inline function in the header.

While here factor out the size calculation and add a comment as-to why
this is done. We will need the size (_s) in the future to make a
decision on how to allocate.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D45815

show more ...

LinuxKPI: Add kmalloc_size_roundup function

kmalloc_size_roundup reports allocation bucket size for the given size.

Sponsored by: Serenity CyberSecurity, LLC
MFC after: 1 week
Reviewed by: manu, em

LinuxKPI: Add kmalloc_size_roundup function

kmalloc_size_roundup reports allocation bucket size for the given size.

Sponsored by: Serenity CyberSecurity, LLC
MFC after: 1 week
Reviewed by: manu, emaste
Differential Revision: https://reviews.freebsd.org/D45841

show more ...

LinuxKPI: Add kvrealloc to linux/slab.h

Sponsored by: Serenity Cyber Security, LLC
MFC after: 1 week
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D45616

LinuxKPI: Allow kmalloc to be called when FPU protection is enabled

Amdgpu driver does a lot of memory allocations in FPU-protected sections
of code for certain display cores, e.g. for DCN30. This d

LinuxKPI: Allow kmalloc to be called when FPU protection is enabled

Amdgpu driver does a lot of memory allocations in FPU-protected sections
of code for certain display cores, e.g. for DCN30. This does not work
on FreeBSD as its malloc function can not be run within a critical
section. Check this condition and temporally exit from FPU-protected
context to workaround issue and reduce source code patching.

Sponsored by: Serenity Cyber Security, LLC
Reviewed by: manu (previous version)
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D42822

show more ...

Revert "linuxkpi: `GFP_KERNEL` equals `M_NOWAIT` now"

This change seems to break some drivers such as the mlx5*(4) drivers.

As kib@ says:
> According to the 'official' Linux kernel documentation, t

Revert "linuxkpi: `GFP_KERNEL` equals `M_NOWAIT` now"

This change seems to break some drivers such as the mlx5*(4) drivers.

As kib@ says:
> According to the 'official' Linux kernel documentation, the GFP_KERNEL
> flag implies sleepable context.

It was introduced while working on the new vt(4)/DRM integration [1].
During this work, doing sleepable allocations broke vt(4) and the DRM
drivers. However, I made further improvements and some locking-related
fixed to the new integration without revisiting the need for it.

After more testing, the improvements to the integration mentionned above
seems to make the change to `GFP_KERNEL` unneeded now. I can thus
revert it to restore expectations of other drivers.

This reverts commit 14dcd40983748596d116d91acb934a8a95ac76bc.

[1] https://github.com/freebsd/drm-kmod/pull/243

Reviewed by: kib
Approved by: kib
Differential Revision: https://reviews.freebsd.org/D42962

show more ...

linuxkpi: `GFP_KERNEL` equals `M_NOWAIT` now

... instead of `M_WAITOK`.

[Why]
The reason is that in some places in the DRM drivers (in particular, the
framebuffer management code), kmalloc() is cal

linuxkpi: `GFP_KERNEL` equals `M_NOWAIT` now

... instead of `M_WAITOK`.

[Why]
The reason is that in some places in the DRM drivers (in particular, the
framebuffer management code), kmalloc() is called from a non-sleepable
context, such as after a call to mtx_lock(8) with an MTX_DEF mutex.

If `GFP_KERNEL` is defined as `M_WAITOK`, we hit an assertion from
witness(4).

[How]
The definition of `GFP_KERNEL` is changed to `M_NOWAIT`. This means that
callers should verify the return value of kmalloc(). Fortunately, this
is always the case in Linux.

Reviewed by: bz, emaste, manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D42054

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

linuxkpi: Support non-NULL zero-size pointers

DRM drivers set some pointers to `ZERO_SIZE_PTR` directly (without
allocating anything), to treat pointers which were "initialized" (set to
`ZERO_SIZE_P

linuxkpi: Support non-NULL zero-size pointers

DRM drivers set some pointers to `ZERO_SIZE_PTR` directly (without
allocating anything), to treat pointers which were "initialized" (set to
`ZERO_SIZE_PTR`) with no memory allocation like really allocated
pointers. NULL isn't used because it represents a third state.

Reviewed by: emaste, manu
Approved by: emaste, manu
Differential Revision: https://reviews.freebsd.org/D39055

show more ...

linuxkpi: Define `ZERO_OR_NULL_PTR()` in <linux/slab.h>

On Linux, the `kmalloc()` family of functions returns a special value if
the size of the allocation is zero. This macro verifies if the pointe

linuxkpi: Define `ZERO_OR_NULL_PTR()` in <linux/slab.h>

On Linux, the `kmalloc()` family of functions returns a special value if
the size of the allocation is zero. This macro verifies if the pointer
is NULL (the allocation failed) or the size is 0 (the allocation was not
performed AFAIU). This special value can be passed to `kfree()`.

On FreeBSD, our `malloc(9)` functions don't return a special value for
0-size allocations. Therefore we can simply compare the result against
NULL.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D37367

show more ...

linuxkpi: Add `krealloc_array()`

In FreeBSD, this is a wrapper on top of `realloc()`.

V2: Check if `n * size` would overflow and return `NULL` if that's the
case. Suggested by hselasky@ and ema

linuxkpi: Add `krealloc_array()`

In FreeBSD, this is a wrapper on top of `realloc()`.

V2: Check if `n * size` would overflow and return `NULL` if that's the
case. Suggested by hselasky@ and emaste@.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D36959

show more ...

linuxkpi: Add stub kmem_cache_shrink

Needed by drm-kmod.

Reviewed by: bz
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D36108

LinuxKPI: Constantly use _LINUXKPI_ prefix in include guards

MFC after: 1 week
Reviewed by: bz, emaste, hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33562

LinuxKPI: Move kfree_async() functionality in to kfree()

Obsolete it usage but keep for a while for drm-kmod 5.4 compatibility

MFC after: 1 week
Reviewed by: hselasky, manu
Differential Revision: h

LinuxKPI: Move kfree_async() functionality in to kfree()

Obsolete it usage but keep for a while for drm-kmod 5.4 compatibility

MFC after: 1 week
Reviewed by: hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33298

show more ...

LinuxKPI: Add some typical header pollution

To reduce amount of drm-kmod patching

MFC after: 1 week
Reviewed by: hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33297

LinuxKPI: Allow kmem_cache_free() to be called from critical sections

as it is required by i915kms driver from Linux kernel v 5.5.
This is done with asynchronous freeing of requested memory areas fr

LinuxKPI: Allow kmem_cache_free() to be called from critical sections

as it is required by i915kms driver from Linux kernel v 5.5.
This is done with asynchronous freeing of requested memory areas from
taskqueue thread. As memory to be freed is reused to store linked list
entry, backing UMA zone item size is rounded up to pointer size.

While here, make struct linux_kmem_cache private to LKPI to reduce amount
of BSD headers included by linux/slab.h and switch RCU code to usage of
LKPI's linux_irq_work_tq taskqueue to avoid injection of current into
system-wide taskqueue_fast thread context.

Submitted by: nc (initial version for drm-kmod)
Reviewed by: manu, nc
Differential revision: https://reviews.freebsd.org/D30760

show more ...

LinuxKPI: add kfree_sensitive() using zfree().

Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Reviewed by: hselasky
Differential Revision: https://reviews.freebsd.org/D30437

Implement basic support for allocating memory from a specific numa node
in the LinuxKPI.

Differential Revision: https://reviews.freebsd.org/D29077
Reviewed by: markj@ and kib@
MFC after: 1 week
Spon

Implement basic support for allocating memory from a specific numa node
in the LinuxKPI.

Differential Revision: https://reviews.freebsd.org/D29077
Reviewed by: markj@ and kib@
MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

show more ...

LinuxKPI: Reimplement irq_work queue on top of fast taskqueue

Summary:
Linux's irq_work queue was created for asynchronous execution of code from contexts where spin_lock's are not available like "h

LinuxKPI: Reimplement irq_work queue on top of fast taskqueue

Summary:
Linux's irq_work queue was created for asynchronous execution of code from contexts where spin_lock's are not available like "hardware interrupt context". FreeBSD's fast taskqueues was created for the same purposes.

Drm-kmod 5.4 uses irq_work_queue() at least in one place to schedule execution of task/work from the critical section that triggers following INVARIANTS-induced panic:

```
panic: acquiring blockable sleep lock with spinlock or critical section held (sleep mutex) linuxkpi_short_wq @ /usr/src/sys/kern/subr_taskqueue.c:281
cpuid = 6
time = 1605048416
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe006b538c90
vpanic() at vpanic+0x182/frame 0xfffffe006b538ce0
panic() at panic+0x43/frame 0xfffffe006b538d40
witness_checkorder() at witness_checkorder+0xf3e/frame 0xfffffe006b538f00
__mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xfffffe006b538f50
taskqueue_enqueue() at taskqueue_enqueue+0x42/frame 0xfffffe006b538f70
linux_queue_work_on() at linux_queue_work_on+0xe9/frame 0xfffffe006b538fb0
irq_work_queue() at irq_work_queue+0x21/frame 0xfffffe006b538fd0
semaphore_notify() at semaphore_notify+0xb2/frame 0xfffffe006b539020
__i915_sw_fence_notify() at __i915_sw_fence_notify+0x2e/frame 0xfffffe006b539050
__i915_sw_fence_complete() at __i915_sw_fence_complete+0x63/frame 0xfffffe006b539080
i915_sw_fence_complete() at i915_sw_fence_complete+0x8e/frame 0xfffffe006b5390c0
dma_i915_sw_fence_wake() at dma_i915_sw_fence_wake+0x4f/frame 0xfffffe006b539100
dma_fence_signal_locked() at dma_fence_signal_locked+0x105/frame 0xfffffe006b539180
dma_fence_signal() at dma_fence_signal+0x72/frame 0xfffffe006b5391c0
dma_fence_is_signaled() at dma_fence_is_signaled+0x80/frame 0xfffffe006b539200
dma_resv_add_shared_fence() at dma_resv_add_shared_fence+0xb3/frame 0xfffffe006b539270
i915_vma_move_to_active() at i915_vma_move_to_active+0x18a/frame 0xfffffe006b5392b0
eb_move_to_gpu() at eb_move_to_gpu+0x3ad/frame 0xfffffe006b539320
eb_submit() at eb_submit+0x15/frame 0xfffffe006b539350
i915_gem_do_execbuffer() at i915_gem_do_execbuffer+0x7d4/frame 0xfffffe006b539570
i915_gem_execbuffer2_ioctl() at i915_gem_execbuffer2_ioctl+0x1c1/frame 0xfffffe006b539600
drm_ioctl_kernel() at drm_ioctl_kernel+0xd9/frame 0xfffffe006b539670
drm_ioctl() at drm_ioctl+0x5cd/frame 0xfffffe006b539820
linux_file_ioctl() at linux_file_ioctl+0x323/frame 0xfffffe006b539880
kern_ioctl() at kern_ioctl+0x1f4/frame 0xfffffe006b5398f0
sys_ioctl() at sys_ioctl+0x12a/frame 0xfffffe006b5399c0
amd64_syscall() at amd64_syscall+0x121/frame 0xfffffe006b539af0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe006b539af0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800a6f09a, rsp = 0x7fffffffe588, rbp = 0x7fffffffe640 ---
KDB: enter: panic
```
Here, the dma_resv_add_shared_fence() performs a critical_enter() and following call of schedule_work() from semaphore_notify() triggers 'acquiring blockable sleep lock with spinlock or critical section held' panic.

Switching irq_work implementation to fast taskqueue fixes the panic for me.

Other report with the similar bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247166

Reviewed By: hselasky
Differential Revision: https://reviews.freebsd.org/D27171

show more ...

LinuxKPI: Implement ksize() function.

In Linux, ksize() gets the actual amount of memory allocated for a given
object. This commit adds malloc_usable_size() to FreeBSD KPI which does
the same. It al

LinuxKPI: Implement ksize() function.

In Linux, ksize() gets the actual amount of memory allocated for a given
object. This commit adds malloc_usable_size() to FreeBSD KPI which does
the same. It also maps LinuxKPI ksize() to newly created function.

ksize() function is used by drm-kmod.

Reviewed by: hselasky, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26215

show more ...

MFHead@r345275

Implement more malloc function macros in the LinuxKPI.
Fix arguments for currently unused kvmalloc().

Submitted by: Johannes Lundberg <johalun0@gmail.com>
MFC after: 1 week
Sponsored by: Limeligh

Implement more malloc function macros in the LinuxKPI.
Fix arguments for currently unused kvmalloc().

Submitted by: Johannes Lundberg <johalun0@gmail.com>
MFC after: 1 week
Sponsored by: Limelight Networks
Sponsored by: Mellanox Technologies

show more ...

Define ARCH_KMALLOC_MINALIGN in the LinuxKPI.

Submitted by: Johannes Lundberg <johalun0@gmail.com>
MFC after: 1 week
Sponsored by: Mellanox Technologies
Sponsored by: Limelight Networks

Implement support for the kvmalloc_array() function in the LinuxKPI.

Submitted by: Johannes Lundberg <johalun0@gmail.com>
MFC after: 1 week
Sponsored by: Mellanox Technologies

linuxkpi whitespace cleanup

Reviewed by: hselasky, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D14807

Keep the old SLAB_DESTROY_BY_RCU macro definition around in the LinuxKPI
to avoid compilation breakage in external kernel modules.

MFC after: 1 week
Sponsored by: Mellanox Technologies