linuxkpi: Add `folio` and `folio_batch` APIs

They are used by the i915 DRM driver in Linux 6.6 (although this change
was only backported with Linux 6.7 DRM drivers).

`struct folio` simply wraps `st

linuxkpi: Add `folio` and `folio_batch` APIs

They are used by the i915 DRM driver in Linux 6.6 (although this change
was only backported with Linux 6.7 DRM drivers).

`struct folio` simply wraps `struct page` for now.

`struct folio_batch` is the same as `struct pagevec` but it works with
`struct folio` instead of `struct page` directly.

Reviewed by: bz, kib, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48743

show more ...

linuxkpi: Make "free page" code paths closer to Linux

There is basically one code path to free pages on Linux. In particular,
`free_pages()` is used for other pages than those returned by
`alloc_pag

linuxkpi: Make "free page" code paths closer to Linux

There is basically one code path to free pages on Linux. In particular,
`free_pages()` is used for other pages than those returned by
`alloc_pages()`.

Also on Linux, `release_pages()` takes either `struct page` or `struct
folio`. `struct folio` support will be added in a followup commit.
Regardless, because pages come from several sources,
`linux_free_pages()` need to accept managed and unmanaged pages.

Reviewed by: bz, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D49813

show more ...

linuxkpi: Constify `struct page` arg to `linux_page_address()`

Reviewed by: emaste, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D49812

linuxkpi: Move `release_pages()` to linux_page.c

We want to avoid putting code managing memory in inline functions in the
future.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differentia

linuxkpi: Move `release_pages()` to linux_page.c

We want to avoid putting code managing memory in inline functions in the
future.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D49810

show more ...

vm_page: make iter_insert() public

In places where vm_page_insert() is used after lookups, or for
consecutive pages, use vm_page_iter_insert instead, to exploit
locality.

Reviewed by: kib, markj
Di

vm_page: make iter_insert() public

In places where vm_page_insert() is used after lookups, or for
consecutive pages, use vm_page_iter_insert instead, to exploit
locality.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D50248

show more ...

LinuxKPI: make linux_alloc_pages() honor __GFP_NORETRY

This is to fix slowdowns with drm-kmod that get worse over time as
physical memory become more fragmented (and probably also depending on
other

LinuxKPI: make linux_alloc_pages() honor __GFP_NORETRY

This is to fix slowdowns with drm-kmod that get worse over time as
physical memory become more fragmented (and probably also depending on
other factors).

Based on information posted in this bug report:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277476

By default, linux_alloc_pages() retries failed allocations by calling
vm_page_reclaim_contig() to attempt to free contiguous physical memory
pages. vm_page_reclaim_contig() does not always succeed and calling it
can be very slow even when it fails. When physical memory is very
fragmented, vm_page_reclaim_contig() can end up being called (and
failing) after every allocation attempt. This could cause very
noticeable graphical desktop hangs (which could last seconds).

The drm-kmod code in question attempts to allocate multiple contiguous
pages at once but does not actually require them to be contiguous. It
can fallback to doing multiple smaller allocations when larger
allocations fail. It passes alloc_pages() the __GFP_NORETRY flag in this
case.

This patch makes linux_alloc_pages() fail early (without retrying) when
this flag is passed.

[olce: The problem this patch fixes is longer and longer GUI freezes as
a machine's memory gets filled and becomes fragmented, when using amdgpu
from DRM kmod 5.15 and DRM kmod 6.1 (DRM kmod 5.10 is unaffected; newer
Linux kernel introduced an "optimization" by which a pool of pages is
filled preferentially with contiguous pages, which triggered the problem
for us). The original commit message above evokes freezes lasting
seconds, but I occasionally witnessed some lasting tens of minutes,
rendering a machine completely useless.

The patch has been reviewed for its potential impacts to other LinuxKPI
parts and our existing DRM kmods' code. In particular, there is no
other user of __GFP_NORETRY/GFP_NORETRY with Linux's alloc_pages*()
functions in our tree or DRM kmod ports.

It has also been tested extensively, by me for months against 14-STABLE
and sporadically on -CURRENT on a RX580, and by several others as
reported below and as is visible in more details in the quoted bugzilla
PR and in the initial drm-kmod issue at
https://github.com/freebsd/drm-kmod/issues/302, on a variety of other
AMD GPUs (several RX580, RX570, Radeon Pro WX5100, Green Sardine 5600G,
Ryzen 9 4900H with embedded Renoir).]

PR: 277476
Reported by: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>
Reviewed by: olce
Tested by: many (olce, Pierre Pronchery, Evgenii Khramtsov, chaplina, rk)
MFC after: 2 weeks
Relnotes: yes
Sponsored by: The FreeBSD Foundation (review and part of testing)

show more ...

LinuxKPI: always use contig allocations in linux_alloc_kmem()

In linux_alloc_kmem() [used by *get_page*()] we always at least allocate
PAGE_SIZE and we want the allocation to be contiguous so it can

LinuxKPI: always use contig allocations in linux_alloc_kmem()

In linux_alloc_kmem() [used by *get_page*()] we always at least allocate
PAGE_SIZE and we want the allocation to be contiguous so it can be passed
to DMA. Always use kmem_alloc_contig() and only change the low argument
depending on the GFP_DMA32 flag being given or not.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: jhb, dumbbell
Differential Revision: https://reviews.freebsd.org/D46661

show more ...

device_pager: user iterators to free device pages

Change cdev_mgtdev_page_free_page to take an iterator, rather than an
object and page, so that removing the page from the object radix tree
can take

device_pager: user iterators to free device pages

Change cdev_mgtdev_page_free_page to take an iterator, rather than an
object and page, so that removing the page from the object radix tree
can take advantage of locality with iterators. Define a
general-purpose function to free all pages, which can be used in
several places.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D47692

show more ...

dev_pager: define free_page for mgt devices

Callers of cdev_pager_free_page in the kernel always have object->type
== OBJT_MGTDEVICE. Define a function for them to call that skips the
runtime type c

dev_pager: define free_page for mgt devices

Callers of cdev_pager_free_page in the kernel always have object->type
== OBJT_MGTDEVICE. Define a function for them to call that skips the
runtime type check in cdev_pager_free.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46389

show more ...

LinuxKPI: Remove vmas argument from get_user_pages on KBI layer

To chase Linux kernel 6.5

Sponsored by: Serenity Cyber Security, LLC
Reviewed by: manu
Differential Revision: https://reviews.freebsd

LinuxKPI: Remove vmas argument from get_user_pages on KBI layer

To chase Linux kernel 6.5

Sponsored by: Serenity Cyber Security, LLC
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D45850

show more ...

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM i

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM indicates
lack of available memory to reclaim, with any other error (currently
only ERANGE) indicating that reclamation is impossible for the
specified address range. Change all callers to only follow
up with vm_page_wait* in the ENOMEM case.

- Introduce vm_domainset_iter_ignore(), which marks the specified
domain as unavailable for further use by the iterator. Use this
function to ignore domains that can't possibly satisfy a physical
allocation request. Since WAITOK allocations run the iterators
repeatedly, this avoids the possibility of infinitely spinning
in domain iteration if no available domain can satisfy the
allocation request.

PR: 274252
Reported by: kevans
Tested by: kevans
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D42706

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

linuxkpi: Fix an assertion in linux_free_kmem()

In the LinuxKPI, PAGE_MASK is the logical negation of FreeBSD's
PAGE_MASK, so the original assertion was simply incorrect.

Reported by: trasz
Tested

linuxkpi: Fix an assertion in linux_free_kmem()

In the LinuxKPI, PAGE_MASK is the logical negation of FreeBSD's
PAGE_MASK, so the original assertion was simply incorrect.

Reported by: trasz
Tested by: trasz
Fixes: 6223d0b67af9 ("linuxkpi: Handle direct-mapped addresses in linux_free_kmem()")

show more ...

linuxkpi: Handle direct-mapped addresses in linux_free_kmem()

See the analysis in PR 271333. It is possible for driver code to
allocate a page, store its address as returned by page_address(), then

linuxkpi: Handle direct-mapped addresses in linux_free_kmem()

See the analysis in PR 271333. It is possible for driver code to
allocate a page, store its address as returned by page_address(), then
call free_page() on that address. On most systems that'll result in the
LinuxKPI calling kmem_free() with a direct-mapped address, which is not
legal.

Fix the problem by making linux_free_kmem() check the address to see
whether it's direct-mapped or not, and handling it appropriately.

PR: 271333, 274515
Reviewed by: hselasky, bz
Tested by: trasz
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40028

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

LinuxKPI: replace vtophys+PHYS_TO_VM_PAGE with virt_to_page

Rather than using the FreeBSD internal calls vtophys() and
PHYS_TO_VM_PAGE() use the already existing LinuxKPI abstraction for
this called

LinuxKPI: replace vtophys+PHYS_TO_VM_PAGE with virt_to_page

Rather than using the FreeBSD internal calls vtophys() and
PHYS_TO_VM_PAGE() use the already existing LinuxKPI abstraction for
this called virt_to_page(). This reduces the amount of compat code
to maintain and will allow further work on struct [vm_]page.

This should be a NOP.

Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41257

show more ...

LinuxKPI: cleanup internal calls to VM_PAGE_TO_PHYS

Replace FreeBSD native VM_PAGE_TO_PHYS() calls with page_to_phys()
allowing us to work on a struct page in the future using the one
single public

LinuxKPI: cleanup internal calls to VM_PAGE_TO_PHYS

Replace FreeBSD native VM_PAGE_TO_PHYS() calls with page_to_phys()
allowing us to work on a struct page in the future using the one
single public Linux KPI interface to map to a native FreeBSD vm_page.

This should be a NOP.

Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41256

show more ...

LinuxKPI: reduce usage of struct vm_page and vm_page_t

We currently define (Linux) page to (FreeBSD) vm_page.
Cleanup some of the direct struct vm_page and vm_page_t declarations
and usages in the L

LinuxKPI: reduce usage of struct vm_page and vm_page_t

We currently define (Linux) page to (FreeBSD) vm_page.
Cleanup some of the direct struct vm_page and vm_page_t declarations
and usages in the Linux KPI and make them 'struct page' or
'struct page *' to prepare for more upcoming work.

This should be a NOP.

Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41255

show more ...

linuxkpi: Add `freeram` and `freehigh` to `struct sysinfo`

The struct layout is modified with this commit because new fields are
added in the middle, keeping original Linux order.

Reviewed by: manu

linuxkpi: Add `freeram` and `freehigh` to `struct sysinfo`

The struct layout is modified with this commit because new fields are
added in the middle, keeping original Linux order.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D37932

show more ...

linuxkpi: Add `pin_user_pages*()` functions

They were defined in the i915 DRM driver. I move the code in linuxkpi so
it can benefit other drivers.

Reviewed by: manu
Approved by: manu
Differential R

linuxkpi: Add `pin_user_pages*()` functions

They were defined in the i915 DRM driver. I move the code in linuxkpi so
it can benefit other drivers.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D38078

show more ...

linuxkpi: Add `io_mapping_map_user()` and `remap_pfn_range()`

The code comes from the i915 DRM driver.

In Linux commits b739f125e4ebd73d10ed30a856574e13649119ed and
b12d691ea5e01db42ccf3b4207e57cb3

linuxkpi: Add `io_mapping_map_user()` and `remap_pfn_range()`

The code comes from the i915 DRM driver.

In Linux commits b739f125e4ebd73d10ed30a856574e13649119ed and
b12d691ea5e01db42ccf3b4207e57cb3ce7cfe91 (Linux 5.13), the i915 DRM
driver dropped specific implementations to use Linux generic functions.
Therefore I moved the FreeBSD code from that i915 driver to linuxkpi.

However, these commits were later reverted (also in Linux 5.13) so the
i915 driver doesn't use these functions. But perhaps it will help in the
future.

To sum up, the code comes from the i915 DRM driver but it doesn't use it
(i.e. it continues to use its internal implementation).

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D38088

show more ...

LinuxKPI: fix possible NULL dereference in linuxkpi_page_frag_alloc()

Fix a possible NULL pointer deref in case alloc_pages() fails.
This is theoretical so far as up to now no code in the tree uses

LinuxKPI: fix possible NULL dereference in linuxkpi_page_frag_alloc()

Fix a possible NULL pointer deref in case alloc_pages() fails.
This is theoretical so far as up to now no code in the tree uses
linuxkpi_page_frag_alloc().

Reported by: Coverity via emaste
Coverity ID: 1502345
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
X-MFC-with: 55038a6306a570c9f2df89f5ad076de0f7d98152

show more ...

LinuxKPI: add simplified vesion of page_frag_cache

For the moment and the currently only consumer (mt76) add a simplified
version of the page_frag_cache. We will only accept fragement sizes up
to 1

LinuxKPI: add simplified vesion of page_frag_cache

For the moment and the currently only consumer (mt76) add a simplified
version of the page_frag_cache. We will only accept fragement sizes up
to 1 PAGE_SIZE (KASSERT) and we will always return a full page.
Should we add more consumers or small (or large) objects would become a
problem we can always add a more elaborate version.

Discussed with: markj
Reviewed by: markj (,hselasky commented as well)
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D37595

show more ...

kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36549

LinuxKPI: Import MTRR support functions from drm-kmod

They are superseded by PAT and mostly useless nowadays but still can be
used on Pentium III/IV era processors. Unlike drm-kmod version, this one

LinuxKPI: Import MTRR support functions from drm-kmod

They are superseded by PAT and mostly useless nowadays but still can be
used on Pentium III/IV era processors. Unlike drm-kmod version, this one
ignores MTRR if PAT is available that fixes confusing "Failed to add WC
MTRR for [0xXXXX-0xYYYY]: 22; performance may suffer" message often
appearing during drm-kmod initialization process.

MFC after: 1 week
Reviewed by: hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33561

show more ...