LinuxKPI; cleanup slab.h a bit; move more free() into slab.c

Move kfree() into slab.c as an implementation and hide the private
function linux_kfree_async() entirely.

Remove a ; at the end of a def

LinuxKPI; cleanup slab.h a bit; move more free() into slab.c

Move kfree() into slab.c as an implementation and hide the private
function linux_kfree_async() entirely.

Remove a ; at the end of a define and sort some defines into place.

Remove extern from function declarations and move the closer to where
they belong.

Sort the functions into "base allocator/free" functions--these have
an implementation in slab.c and are ensuring contiguous physical
memory allocations.

Followed by inline functions using these base allocators to implement
their functionality; vmalloc/kvalloc, and misc functions.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: dumbbell
Differential Revision: https://reviews.freebsd.org/D49572

show more ...

LinuxKPI: implement krealloc() for memory being contiguous

Implement krealloc_array() using krealloc(). Implement krealloc()
doing the various size checks ourselves and use realloc() or kmalloc()
d

LinuxKPI: implement krealloc() for memory being contiguous

Implement krealloc_array() using krealloc(). Implement krealloc()
doing the various size checks ourselves and use realloc() or kmalloc()
depending on old and new allocation sizes.
This way we can ensure that allocated memory stays physically contiguous.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Suggested by: jhb (see D46657)
Reviewed by: jhb, markj
Differential Revision: https://reviews.freebsd.org/D49571

show more ...

LinuxKPI: switch mallocarray to an lkpi implementation using __kmalloc()

With mallocarray() we cannot guarantee that any size larger than
PAGE_SIZE will be contiguous. Switch kmalloc_array() and
km

LinuxKPI: switch mallocarray to an lkpi implementation using __kmalloc()

With mallocarray() we cannot guarantee that any size larger than
PAGE_SIZE will be contiguous. Switch kmalloc_array() and
kmalloc_array_node() to use __kmalloc()/lkpi___kmalloc_node() as their
underlying implementation which now does provide that guarantee.
Likewise adjust kcalloc_node() to use kmalloc_array_node().
This means we only have two (plain + _node) underlying allocation
routines for the entire category of functions.

Also adjust kvmalloc() and kvmalloc_array() to be a "mirrored"
implementation to their non-v counterparts. These may return
non-contiguous memory so can use malloc().

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: jhb
Extra thanks to: jhb for helping sorting this out
Differential Revision: https://reviews.freebsd.org/D46657

show more ...

LinuxKPi: lkpi___kmalloc change a < into <= for proper boundry check

lkpi___kmalloc() would use contigmalloc if the requested size is exactly
PAGE_SIZE. But we can still fullfill this request from

LinuxKPi: lkpi___kmalloc change a < into <= for proper boundry check

lkpi___kmalloc() would use contigmalloc if the requested size is exactly
PAGE_SIZE. But we can still fullfill this request from malloc() without
losing the guarantee of physical contiguous memory.

Sponsored by: The FreeBSD Foundation
Reported by: emaste (in D46656)
Fixes: 19df0c5abcb9d
MFC after: 3 days
Reviewed by: dumbbell, emaste
Differential Revision: https://reviews.freebsd.org/D49570

show more ...

LinuxKPI: make __kmalloc() play by the rules

According to Documentation/core-api/dma-api.rst kmalloc() is supposd
to provide physically contiguous memory. [1]

In order to guarantee that allocations

LinuxKPI: make __kmalloc() play by the rules

According to Documentation/core-api/dma-api.rst kmalloc() is supposd
to provide physically contiguous memory. [1]

In order to guarantee that allocations are contiguous even if using
PAGE_SIZE or larger check the size and use contigmalloc if needed.
This makes use of 9e6544dd6e02 (and following) allowing free(9) to
also work for contigmalloced memory.

Sponsored by: The FreeBSD Foundation
Pointed out by: jhb [1]
Reviewed by: jhb, emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D46656

show more ...

LinuxKPI: move __kmalloc from slab.h to slab.c

In order to allow the allocator to change in the future move it into
the implementation file from being an inline function in the header.

While here f

LinuxKPI: move __kmalloc from slab.h to slab.c

In order to allow the allocator to change in the future move it into
the implementation file from being an inline function in the header.

While here factor out the size calculation and add a comment as-to why
this is done. We will need the size (_s) in the future to make a
decision on how to allocate.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D45815

show more ...

LinuxKPI: Allow kmalloc to be called when FPU protection is enabled

Amdgpu driver does a lot of memory allocations in FPU-protected sections
of code for certain display cores, e.g. for DCN30. This d

LinuxKPI: Allow kmalloc to be called when FPU protection is enabled

Amdgpu driver does a lot of memory allocations in FPU-protected sections
of code for certain display cores, e.g. for DCN30. This does not work
on FreeBSD as its malloc function can not be run within a critical
section. Check this condition and temporally exit from FPU-protected
context to workaround issue and reduce source code patching.

Sponsored by: Serenity Cyber Security, LLC
Reviewed by: manu (previous version)
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D42822

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

linuxkpi: Handle a NULL cache pointer in kmem_cache_destroy()

This is compatible with Linux, and some driver error paths depend on it.

Reviewed by: bz, emaste
MFC after: 1 week
Sponsored by: The Fr

linuxkpi: Handle a NULL cache pointer in kmem_cache_destroy()

This is compatible with Linux, and some driver error paths depend on it.

Reviewed by: bz, emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32337

show more ...

LinuxKPI: Allow kmem_cache_free() to be called from critical sections

as it is required by i915kms driver from Linux kernel v 5.5.
This is done with asynchronous freeing of requested memory areas fr

LinuxKPI: Allow kmem_cache_free() to be called from critical sections

as it is required by i915kms driver from Linux kernel v 5.5.
This is done with asynchronous freeing of requested memory areas from
taskqueue thread. As memory to be freed is reused to store linked list
entry, backing UMA zone item size is rounded up to pointer size.

While here, make struct linux_kmem_cache private to LKPI to reduce amount
of BSD headers included by linux/slab.h and switch RCU code to usage of
LKPI's linux_irq_work_tq taskqueue to avoid injection of current into
system-wide taskqueue_fast thread context.

Submitted by: nc (initial version for drm-kmod)
Reviewed by: manu, nc
Differential revision: https://reviews.freebsd.org/D30760

show more ...

LinuxKPI: Reimplement irq_work queue on top of fast taskqueue

Summary:
Linux's irq_work queue was created for asynchronous execution of code from contexts where spin_lock's are not available like "h

LinuxKPI: Reimplement irq_work queue on top of fast taskqueue

Summary:
Linux's irq_work queue was created for asynchronous execution of code from contexts where spin_lock's are not available like "hardware interrupt context". FreeBSD's fast taskqueues was created for the same purposes.

Drm-kmod 5.4 uses irq_work_queue() at least in one place to schedule execution of task/work from the critical section that triggers following INVARIANTS-induced panic:

```
panic: acquiring blockable sleep lock with spinlock or critical section held (sleep mutex) linuxkpi_short_wq @ /usr/src/sys/kern/subr_taskqueue.c:281
cpuid = 6
time = 1605048416
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe006b538c90
vpanic() at vpanic+0x182/frame 0xfffffe006b538ce0
panic() at panic+0x43/frame 0xfffffe006b538d40
witness_checkorder() at witness_checkorder+0xf3e/frame 0xfffffe006b538f00
__mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xfffffe006b538f50
taskqueue_enqueue() at taskqueue_enqueue+0x42/frame 0xfffffe006b538f70
linux_queue_work_on() at linux_queue_work_on+0xe9/frame 0xfffffe006b538fb0
irq_work_queue() at irq_work_queue+0x21/frame 0xfffffe006b538fd0
semaphore_notify() at semaphore_notify+0xb2/frame 0xfffffe006b539020
__i915_sw_fence_notify() at __i915_sw_fence_notify+0x2e/frame 0xfffffe006b539050
__i915_sw_fence_complete() at __i915_sw_fence_complete+0x63/frame 0xfffffe006b539080
i915_sw_fence_complete() at i915_sw_fence_complete+0x8e/frame 0xfffffe006b5390c0
dma_i915_sw_fence_wake() at dma_i915_sw_fence_wake+0x4f/frame 0xfffffe006b539100
dma_fence_signal_locked() at dma_fence_signal_locked+0x105/frame 0xfffffe006b539180
dma_fence_signal() at dma_fence_signal+0x72/frame 0xfffffe006b5391c0
dma_fence_is_signaled() at dma_fence_is_signaled+0x80/frame 0xfffffe006b539200
dma_resv_add_shared_fence() at dma_resv_add_shared_fence+0xb3/frame 0xfffffe006b539270
i915_vma_move_to_active() at i915_vma_move_to_active+0x18a/frame 0xfffffe006b5392b0
eb_move_to_gpu() at eb_move_to_gpu+0x3ad/frame 0xfffffe006b539320
eb_submit() at eb_submit+0x15/frame 0xfffffe006b539350
i915_gem_do_execbuffer() at i915_gem_do_execbuffer+0x7d4/frame 0xfffffe006b539570
i915_gem_execbuffer2_ioctl() at i915_gem_execbuffer2_ioctl+0x1c1/frame 0xfffffe006b539600
drm_ioctl_kernel() at drm_ioctl_kernel+0xd9/frame 0xfffffe006b539670
drm_ioctl() at drm_ioctl+0x5cd/frame 0xfffffe006b539820
linux_file_ioctl() at linux_file_ioctl+0x323/frame 0xfffffe006b539880
kern_ioctl() at kern_ioctl+0x1f4/frame 0xfffffe006b5398f0
sys_ioctl() at sys_ioctl+0x12a/frame 0xfffffe006b5399c0
amd64_syscall() at amd64_syscall+0x121/frame 0xfffffe006b539af0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe006b539af0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800a6f09a, rsp = 0x7fffffffe588, rbp = 0x7fffffffe640 ---
KDB: enter: panic
```
Here, the dma_resv_add_shared_fence() performs a critical_enter() and following call of schedule_work() from semaphore_notify() triggers 'acquiring blockable sleep lock with spinlock or critical section held' panic.

Switching irq_work implementation to fast taskqueue fixes the panic for me.

Other report with the similar bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247166

Reviewed By: hselasky
Differential Revision: https://reviews.freebsd.org/D27171

show more ...

Rename the SLAB_DESTROY_BY_RCU flag into SLAB_TYPESAFE_BY_RCU in the LinuxKPI
to be compatible with Linux.

MFC after: 1 week
Requested by: Johannes Lundberg <johalun0@gmail.com>
Sponsored by: Mellan

Rename the SLAB_DESTROY_BY_RCU flag into SLAB_TYPESAFE_BY_RCU in the LinuxKPI
to be compatible with Linux.

MFC after: 1 week
Requested by: Johannes Lundberg <johalun0@gmail.com>
Sponsored by: Mellanox Technologies

show more ...

Cleanup the LinuxKPI slab implementation.

Put large functions into linux_slab.c instead of declaring them static
inline.

Add support for more memory allocation wrappers like kmalloc_array()
and __v

Cleanup the LinuxKPI slab implementation.

Put large functions into linux_slab.c instead of declaring them static
inline.

Add support for more memory allocation wrappers like kmalloc_array()
and __vmalloc().

Make sure either the M_WAITOK or the M_NOWAIT flag is set and mask
away unused memory allocation flags before calling FreeBSD's malloc()
routine.

Move kmalloc_node() definition to slab.h where it belongs.

Implement support for the SLAB_DESTROY_BY_RCU feature when creating a
kmem_cache which basically means kmem_cache memory is freed using
call_rcu().

MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...