dev_pager: define free_page for mgt devices

Callers of cdev_pager_free_page in the kernel always have object->type
== OBJT_MGTDEVICE. Define a function for them to call that skips the
runtime type c

dev_pager: define free_page for mgt devices

Callers of cdev_pager_free_page in the kernel always have object->type
== OBJT_MGTDEVICE. Define a function for them to call that skips the
runtime type check in cdev_pager_free.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46389

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

xen: bump used interface version

This is required for a further change that will make use of a field
that was added in version 0x00040d00.

No functional change expected.

Sponsored by: Citrix Syste

xen: bump used interface version

This is required for a further change that will make use of a field
that was added in version 0x00040d00.

No functional change expected.

Sponsored by: Citrix Systems R&D

show more ...

xen: Remove unused devclass arguments to DRIVER_MODULE.

xen: plug some of set-but-not-used vars

Sponsored by: Rubicon Communications, LLC ("Netgate")

xen/privcmd: fix MMAP_RESOURCE ioctl to copy out results

The current definition for the MMAP_RESOURCE ioctl was wrong as it
didn't copy back the result to the caller. Fix the definition and also
rem

xen/privcmd: fix MMAP_RESOURCE ioctl to copy out results

The current definition for the MMAP_RESOURCE ioctl was wrong as it
didn't copy back the result to the caller. Fix the definition and also
remove the bogus attempt to copy the result in the implementation.

Note such copy back is only needed when querying the size of a
resource.

Sponsored by: Citrix Systems R&D

show more ...

xen/privcmd: implement the restrict ioctl

Use an interface compatible with the Linux one so that the user-space
libraries already using the Linux interface can be used without much
modifications.

T

xen/privcmd: implement the restrict ioctl

Use an interface compatible with the Linux one so that the user-space
libraries already using the Linux interface can be used without much
modifications.

This allows an open privcmd instance to limit against which domains it
can act upon.

Sponsored by: Citrix Systems R&D

show more ...

xen/privcmd: implement the dm op ioctl

Use an interface compatible with the Linux one so that the user-space
libraries already using the Linux interface can be used without much
modifications.

This

xen/privcmd: implement the dm op ioctl

Use an interface compatible with the Linux one so that the user-space
libraries already using the Linux interface can be used without much
modifications.

This allows user-space to make use of the dm_op family of hypercalls,
which are used by device models.

Sponsored by: Citrix Systems R&D

show more ...

xen/privcmd: implement the map resource ioctl

The interface is mostly the same as the Linux ioctl, so that we don't
need to modify the user-space libraries that make use of it.

The ioctl is just a

xen/privcmd: implement the map resource ioctl

The interface is mostly the same as the Linux ioctl, so that we don't
need to modify the user-space libraries that make use of it.

The ioctl is just a proxy for the XENMEM_acquire_resource hypercall.

Sponsored by: Citrix Systems R&D

show more ...

xen/privcmd: split setup of virtual address range into helper

Preparatory change for further additions that will also make use of
the same code. No functional change.

Sponsored by: Citrix Systems R

xen/privcmd: split setup of virtual address range into helper

Preparatory change for further additions that will also make use of
the same code. No functional change.

Sponsored by: Citrix Systems R&D

show more ...

xen/privcmd: make some integers unsigned

There's no reason for them to be signed. No functional change.

Sponsored by: Citrix Systems R&D

Make page busy state deterministic on free. Pages must be xbusy when
removed from objects including calls to free. Pages must not be xbusy
when freed and not on an object. Strengthen assertions to

Make page busy state deterministic on free. Pages must be xbusy when
removed from objects including calls to free. Pages must not be xbusy
when freed and not on an object. Strengthen assertions to match these
expectations. In practice very little code had to change busy handling
to meet these rules but we can now make stronger guarantees to busy
holders and avoid conditionally dropping busy in free.

Refine vm_page_remove() and vm_page_replace() semantics now that we have
stronger guarantees about busy state. This removes redundant and
potentially problematic code that has proliferated.

Discussed with: markj
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D22822

show more ...

(4/6) Protect page valid with the busy lock.

Atomics are used for page busy and valid state when the shared busy is
held. The details of the locking protocol and valid and dirty
synchronization are

(4/6) Protect page valid with the busy lock.

Atomics are used for page busy and valid state when the shared busy is
held. The details of the locking protocol and valid and dirty
synchronization are in the updated vm_page.h comments.

Reviewed by: kib, markj
Tested by: pho
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D21594

show more ...

(1/6) Replace busy checks with acquires where it is trival to do so.

This is the first in a series of patches that promotes the page busy field
to a first class lock that no longer requires the obje

(1/6) Replace busy checks with acquires where it is trival to do so.

This is the first in a series of patches that promotes the page busy field
to a first class lock that no longer requires the object lock for
consistency.

Reviewed by: kib, markj
Tested by: pho
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D21548

show more ...

Merge ^/head r352105 through r352307.

Replace redundant code with a few new vm_page_grab facilities:
- VM_ALLOC_NOCREAT will grab without creating a page.
- vm_page_grab_valid() will grab and page in if necessary.
- vm_page_busy_acqui

Replace redundant code with a few new vm_page_grab facilities:
- VM_ALLOC_NOCREAT will grab without creating a page.
- vm_page_grab_valid() will grab and page in if necessary.
- vm_page_busy_acquire() automates some busy acquire loops.

Discussed with: alc, kib, markj
Tested by: pho (part of larger branch)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D21546

show more ...

Change synchonization rules for vm_page reference counting.

There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
pa

Change synchonization rules for vm_page reference counting.

There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.

Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.

The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.

The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.

__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.

Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486

show more ...

Merge ^/head r338595 through r338689, and resolve conflicts.

xen: temporary disable SMAP when forwarding hypercalls from user-space

The Xen page-table walker used to resolve the virtual addresses in the
hypercalls will refuse to access user-space pages when S

xen: temporary disable SMAP when forwarding hypercalls from user-space

The Xen page-table walker used to resolve the virtual addresses in the
hypercalls will refuse to access user-space pages when SMAP is enabled
unless the AC flag in EFLAGS is set (just like normal hardware with
SMAP support would do).

Since privcmd allows forwarding hypercalls (and buffers) from
user-space into Xen make sure SMAP is temporary disabled for the
duration of the hypercall from user-space.

Approved by: re (gjb)
Sponsored by: Citrix Systems R&D

show more ...

Merge ^/head r316992 through r317215.

Remove unneeded include of vm_phys.h.

xen/privcmd: fix integer truncation in IOCTL_PRIVCMD_MMAPBATCH

The size field in the XENMEM_add_to_physmap_range is an uint16_t, and the
privcmd driver was doing an implicit truncation of an int int

xen/privcmd: fix integer truncation in IOCTL_PRIVCMD_MMAPBATCH

The size field in the XENMEM_add_to_physmap_range is an uint16_t, and the
privcmd driver was doing an implicit truncation of an int into an uint16_t
when filling the hypercall parameters.

Fix this by adding a loop and making sure privcmd splits ioctl request into
2^16 chunks when issuing the hypercalls.

Reported and tested by: Marcin Cieslak <saper@saper.info>
Sponsored by: Citrix Systems R&D

show more ...

Catch up with HEAD (r280229-r284686).

MFH: r282615-r283655

Sponsored by: The FreeBSD Foundation