device_pager: use VM_RADIX_FOREACH

In cdev_mgtdev_pager_free_pages, use VM_RADIX_FOREACH instead of a
loop header that does the same thing, in the same way, with more
words.

No functional change.

device_pager: use VM_RADIX_FOREACH

In cdev_mgtdev_pager_free_pages, use VM_RADIX_FOREACH instead of a
loop header that does the same thing, in the same way, with more
words.

No functional change.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49916

show more ...

device_pager: Add cdev_pager_get_path to retrieve the "path" for an object

This wraps a new optional cdev_pg_path method in struct
cdev_pager_ops. If the method pointer is NULL, the function does
n

device_pager: Add cdev_pager_get_path to retrieve the "path" for an object

This wraps a new optional cdev_pg_path method in struct
cdev_pager_ops. If the method pointer is NULL, the function does
nothing. The old device pager reads the pathname of the cdev stored
in the object handle to match the existing code.

Retire the OBJ_CDEVH flag as it is no longer needed.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49335

show more ...

device_pager: Assert that the handle is not NULL

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49333

vm_object: don't reset new iterator

Don't call pctrie_iter_reset on a just-initialized iterator. Let the
code jumping back up do the resetting. Add an assertion at the jump destination to check tha

vm_object: don't reset new iterator

Don't call pctrie_iter_reset on a just-initialized iterator. Let the
code jumping back up do the resetting. Add an assertion at the jump destination to check that a reset happened.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49207

show more ...

vm_page: drop page_iter_lookup

The functions vm_page_iter_lookup and vm_page_iter_lookup_ge are just
wrappers around vm_radix_iter_lookup and vm_radix_iter_lookup_ge,
respectively. They server no re

vm_page: drop page_iter_lookup

The functions vm_page_iter_lookup and vm_page_iter_lookup_ge are just
wrappers around vm_radix_iter_lookup and vm_radix_iter_lookup_ge,
respectively. They server no real purpose, so drop them and use the
vm_radix versions everywhere.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49203

show more ...

vm_page: pass page to iter_remove

Pass the to-be-freed page to vm_page_iter_remove as a parameter,
rather than computing it from the iterator parameter, to improve
performance.

Reviewed by: alc
Dif

vm_page: pass page to iter_remove

Pass the to-be-freed page to vm_page_iter_remove as a parameter,
rather than computing it from the iterator parameter, to improve
performance.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D47730

show more ...

device_pager: user iterators to free device pages

Change cdev_mgtdev_page_free_page to take an iterator, rather than an
object and page, so that removing the page from the object radix tree
can take

device_pager: user iterators to free device pages

Change cdev_mgtdev_page_free_page to take an iterator, rather than an
object and page, so that removing the page from the object radix tree
can take advantage of locality with iterators. Define a
general-purpose function to free all pages, which can be used in
several places.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D47692

show more ...

device_pager: rename the un_pager.devp.dev field to handle

because it is not neccessary struct cdev *.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revisio

device_pager: rename the un_pager.devp.dev field to handle

because it is not neccessary struct cdev *.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D47443

show more ...

dev_pager: define free_page for mgt devices

Callers of cdev_pager_free_page in the kernel always have object->type
== OBJT_MGTDEVICE. Define a function for them to call that skips the
runtime type c

dev_pager: define free_page for mgt devices

Callers of cdev_pager_free_page in the kernel always have object->type
== OBJT_MGTDEVICE. Define a function for them to call that skips the
runtime type check in cdev_pager_free.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46389

show more ...

cdev_pager_allocate(): ensure that the cdev_pager_ops ctr is called only once

per allocated vm_object. Otherwise, since constructors are not
idempotent, we e.g. leak device reference in case of non

cdev_pager_allocate(): ensure that the cdev_pager_ops ctr is called only once

per allocated vm_object. Otherwise, since constructors are not
idempotent, we e.g. leak device reference in case of non-managed pager.

PR: 278826
Reported by: Austin Zhang <austin.zhang@dell.com>
Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D45113

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

vm_object_kvme_type(): reimplement by embedding kvme_type into pagerops

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://review

vm_object_kvme_type(): reimplement by embedding kvme_type into pagerops

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D30168

show more ...

Constify vm_pager-related virtual tables.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D30070

Merge ^/head r356848 through r356919.

Don't hold the object lock while calling getpages.

The vnode pager does not want the object lock held. Moving this out allows
further object lock scope reduction in callers. While here add some mi

Don't hold the object lock while calling getpages.

The vnode pager does not want the object lock held. Moving this out allows
further object lock scope reduction in callers. While here add some missing
paging in progress calls and an assert. The object handle is now protected
explicitly with pip.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D23033

show more ...

Make page busy state deterministic on free. Pages must be xbusy when
removed from objects including calls to free. Pages must not be xbusy
when freed and not on an object. Strengthen assertions to

Make page busy state deterministic on free. Pages must be xbusy when
removed from objects including calls to free. Pages must not be xbusy
when freed and not on an object. Strengthen assertions to match these
expectations. In practice very little code had to change busy handling
to meet these rules but we can now make stronger guarantees to busy
holders and avoid conditionally dropping busy in free.

Refine vm_page_remove() and vm_page_replace() semantics now that we have
stronger guarantees about busy state. This removes redundant and
potentially problematic code that has proliferated.

Discussed with: markj
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D22822

show more ...

(4/6) Protect page valid with the busy lock.

Atomics are used for page busy and valid state when the shared busy is
held. The details of the locking protocol and valid and dirty
synchronization are

(4/6) Protect page valid with the busy lock.

Atomics are used for page busy and valid state when the shared busy is
held. The details of the locking protocol and valid and dirty
synchronization are in the updated vm_page.h comments.

Reviewed by: kib, markj
Tested by: pho
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D21594

show more ...

(2/6) Don't release xbusy in vm_page_remove(), defer to vm_page_free_prep().

This persists busy state across operations like rename and replace.

Reviewed by: kib, markj
Tested by: pho
Spons

(2/6) Don't release xbusy in vm_page_remove(), defer to vm_page_free_prep().

This persists busy state across operations like rename and replace.

Reviewed by: kib, markj
Tested by: pho
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D21549

show more ...

Merge ^/head r352105 through r352307.

Change synchonization rules for vm_page reference counting.

There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
pa

Change synchonization rules for vm_page reference counting.

There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.

Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.

The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.

The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.

__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.

Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486

show more ...

MFHead @349476

Sponsored by: The FreeBSD Foundation

Add a return value to vm_page_remove().

Use it to indicate whether the page may be safely freed following
its removal from the object. Also change vm_page_remove() to assume
that the page's object

Add a return value to vm_page_remove().

Use it to indicate whether the page may be safely freed following
its removal from the object. Also change vm_page_remove() to assume
that the page's object pointer is non-NULL, and have callers perform
this check instead.

This is a step towards an implementation of an atomic reference counter
for each physical page structure.

Reviewed by: alc, dougm, kib
MFC after: 1 week
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20758

show more ...

Merge ^/head r340918 through r341763.