iommu: extend iommu_map_entry to store the list of associated freed page table pages

The pages are inserted into the added slist if the entry parameter is
passed to iommu_pgfree(). For now it is no

iommu: extend iommu_map_entry to store the list of associated freed page table pages

The pages are inserted into the added slist if the entry parameter is
passed to iommu_pgfree(). For now it is nop.

Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

iommu: change iommu_domain_map_ops to take iommu_map_entry

instead of base/size.

Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

iommu_gas_remove(): consolidate places where decision is made to keep the entry

Move all flag checks into iommu_gas_remove_unmap().

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

iommu_gas: make placeholder entry at the start of the GAS zero size

same as the placeholder at the end. This is required to allow GAS to
start at zero, for integration with vmm.

Also, in iommu_gas

iommu_gas: make placeholder entry at the start of the GAS zero size

same as the placeholder at the end. This is required to allow GAS to
start at zero, for integration with vmm.

Also, in iommu_gas_remove(), accept placeholders after remove op.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

iommu: add iommu_gas_remove_locked()

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

IOMMU: add GAS map entry flag IOMMU_MAP_ENTRY_FAKE

to allow to shut down assert in iommu_gas_cmp_entries() when used
against fake entry to search for specific place in the tree.

Sponsored by: The F

IOMMU: add GAS map entry flag IOMMU_MAP_ENTRY_FAKE

to allow to shut down assert in iommu_gas_cmp_entries() when used
against fake entry to search for specific place in the tree.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

iommu_gas: zero fake on-stack map entry used in iommu_gas_remove_clip_left()

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

iommu_gas: print entries flags and domains if an overlap detected

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

iommu_gas: add ddb 'show iommu_domain' command

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

iommu_gas_match_one(): check for underflow

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

iommu: remove leftover sys/cdefs.h includes

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

iommu_gas: initialize start_gap as first node

In iommu_gas.c, domain->start_gap points to one of the nodes on either
side of the first free, unallocated range. In iommu_gas_init_domain,
it is initia

iommu_gas: initialize start_gap as first node

In iommu_gas.c, domain->start_gap points to one of the nodes on either
side of the first free, unallocated range. In iommu_gas_init_domain,
it is initialized to point to the node after the single free
range. Change it to point to the node before that free range, so that,
when 'lowaddr' is within the initial free range, the first allocation
search for free space below 'lowaddr' does not begin and end at an
address above 'lowaddr'. This fixes problems on a machine with Intel
DMAR enabled.

Reported by: jah
Reviewed by: dougm
Tested by: jah
Obtained from: jah
Fixes: commit db151ca0c343531256b8839f938a4ecbd8b4fd7e iommu_gas: start space search from 1st free space
MFC after: 1 day

show more ...

iommu_gas: avoid overflow in bounds check

Change the range test in iommu_gas_match_one from '< ubound' to '<=
ubound', and pass a smaller-by-one ubound parameter to it, to avoid
overflow in ubound c

iommu_gas: avoid overflow in bounds check

Change the range test in iommu_gas_match_one from '< ubound' to '<=
ubound', and pass a smaller-by-one ubound parameter to it, to avoid
overflow in ubound calculation.

Reported by: andrew
Reviewed by: andrew (previous version)
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D37764

show more ...

iommu_gas: drop fini cleanup loop

Assertions suggest that the loop in iommu_gas_fini_domain is executed
zero times, so remove it.

Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: htt

iommu_gas: drop fini cleanup loop

Assertions suggest that the loop in iommu_gas_fini_domain is executed
zero times, so remove it.

Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D37204

show more ...

iommu_gas: start space search from 1st free space

Maintain a pointer to an element in the domain map that is left of any
sufficiently large free gap in the tree and start the search for free
space t

iommu_gas: start space search from 1st free space

Maintain a pointer to an element in the domain map that is left of any
sufficiently large free gap in the tree and start the search for free
space there, rather than at the root of the tree. On find_space, move
that pointer to the leftmost leaf in the subtree of nodes with
free_down greater than or equal to the minimum allocation size before
starting the search for space from that pointer. On removal of a node
with address less than that pointer, update that pointer to point to
the predecessor or successor of the removed node.

In experiments with netperf streaming, this reduces by about 40% the
number of map entries examined in first-fit allocation.

Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D36624

show more ...

rb_tree: let insert search start from next node

When the node to insert in the rb_tree is known to precede or follow a
particular node, new methods RB_INSERT_PREV and RB_INSERT_NEXT,
defined here, a

rb_tree: let insert search start from next node

When the node to insert in the rb_tree is known to precede or follow a
particular node, new methods RB_INSERT_PREV and RB_INSERT_NEXT,
defined here, allow the search for where to insert the new node begin
with that particular node, rather than at the root, to save a bit of
time.

Using those methods, instead of RB_INSERT, in managing a tree in
iommu_gas.c, saves a little time.

Reviewed by: kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35516

show more ...

rb_tree: augmentation shortcut

RB-tree augmentation maintains data in each node of the tree that
represents the product of some associative operator applied to all the
nodes of the subtree rooted at

rb_tree: augmentation shortcut

RB-tree augmentation maintains data in each node of the tree that
represents the product of some associative operator applied to all the
nodes of the subtree rooted at that node. If a node in the tree
changes, augmentation data for the node is updated for that node and
all nodes on the path from that node to the tree root. However,
sometimes, augmenting a node changes no data in that node,
particularly if the associated operation is something involving 'max'
or 'min'. If augmentation changes nothing in a node, then the work of
walking to the tree root from that point is pointless, because
augmentation will change nothing in those nodes either. This change
makes it possible to avoid that wasted work.

Define RB_AUGMENT_CHECK as a macro much like RB_AUGMENT, but which
returns a value 'true' when augmentation changes the augmentation data
of a node, and false otherwise. Change code that unconditionally walks
and augments to the top of tree to code that stops once an
augmentation has no effect. In the case of rebalancing the tree after
insertion or deletion, where previously a node rotated into the path
was inevitably augmented on the march to the tree root, now check to
see if it needs augmentation because the march to the tree root
stopped before reaching it.

Change the augmentation function in iommu_gas.c so that it returns
true/false to indicate whether the augmentation had any effect.

Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D36509

show more ...

iommu_gas: don't remove and insert boundary entry

Avoid removing an item in iommu_gas_free_region only to reinsert it,
by avoiding removing an entry that is either first_place or
last_place.

Review

iommu_gas: don't remove and insert boundary entry

Avoid removing an item in iommu_gas_free_region only to reinsert it,
by avoiding removing an entry that is either first_place or
last_place.

Reviewed by: kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D36597

show more ...

iommu_gas: add iommu_gas_remove()

The function removes the range of addresses from GAS. Right now it is
unused.

The initial version of the patch was extracted from the stalled work for
integration

iommu_gas: add iommu_gas_remove()

The function removes the range of addresses from GAS. Right now it is
unused.

The initial version of the patch was extracted from the stalled work for
integration of DMAR and bhyve, see D25672. Having the function in the
main tree would allow it to co-evolve with other active changes to the
IOMMU driver.

Requested by: alc
Reviewed by: alc, dougm
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D36001

show more ...

iommu_gas: Eliminate redundant parameters and push down lock acquisition

Since IOMMU map entries store a reference to the domain in which they
reside, there is no need to pass the domain to iommu_ga

iommu_gas: Eliminate redundant parameters and push down lock acquisition

Since IOMMU map entries store a reference to the domain in which they
reside, there is no need to pass the domain to iommu_gas_free_entry(),
iommu_gas_free_space(), and iommu_gas_free_region().

Push down the acquisition and release of the IOMMU domain lock into
iommu_gas_free_space() and iommu_gas_free_region().

Both of these changes allow for simplifications in the callers of the
functions without really complicating the functions themselves.
Moreover, the latter change eliminates the direct use of the IOMMU
domain lock from the x86-specific DMAR code.

Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35995

show more ...

x86/iommu: Reduce DMAR lock contention

Replace the DMAR unit's tlb_flush TAILQ by a custom list implementation
that enables dmar_qi_task() to dequeue entries without holding the DMAR
lock.

Reviewed

x86/iommu: Reduce DMAR lock contention

Replace the DMAR unit's tlb_flush TAILQ by a custom list implementation
that enables dmar_qi_task() to dequeue entries without holding the DMAR
lock.

Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35951

show more ...

iommu_gas: Eliminate a possible case of use-after-free

Eliminate a possible case of use-after-free in an error handling path
after a mapping failure. Specifically, eliminate IOMMU_MAP_ENTRY_QI_NF
a

iommu_gas: Eliminate a possible case of use-after-free

Eliminate a possible case of use-after-free in an error handling path
after a mapping failure. Specifically, eliminate IOMMU_MAP_ENTRY_QI_NF
and instead perform the IOTLB invalidation synchronously. Otherwise,
when iommu_domain_unload_entry() is called and told not to free the
IOMMU map entry, the caller could free the entry before dmar_qi_task()
is finished with it.

Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35878

show more ...

iommu_gas: consolidate find_space helpers

Merge lowermatch and uppermatch into find_space. Eliminate uppermatch
recursion. Merge match_insert into match_one and eliminate some
redundant calculatio

iommu_gas: consolidate find_space helpers

Merge lowermatch and uppermatch into find_space. Eliminate uppermatch
recursion. Merge match_insert into match_one and eliminate some
redundant calculation. Move some initialization out of find_space and
into map (and out from under a lock).

Reviewed by: kib (previous version), alc
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35440

show more ...