swap_pager: fix seek_data with invalid first page

Correct swap_pager_seek_data so that, when the first lookup finds no
valid pages, second and subsequent lookups are attempted anyway.

This was brok

swap_pager: fix seek_data with invalid first page

Correct swap_pager_seek_data so that, when the first lookup finds no
valid pages, second and subsequent lookups are attempted anyway.

This was broken by db08b0b04deced766c3b5f07bcfb82333666226c.

Reported by: marklmi@yahoo.com
Reviewed by: kib
Tested by: marklmi@yahoo.com
Fixes: db08b0b04deced76 tmpfs_vnops: move swap work to swap_pager
Differential Revision: https://reviews.freebsd.org/D47767

show more ...

swap_pager: Ensure that swapoff puts swapped-in pages in page queues

Readahead/behind pages are handled by the swap pager, but the get_pages
caller is responsible for putting fetched pages into queu

swap_pager: Ensure that swapoff puts swapped-in pages in page queues

Readahead/behind pages are handled by the swap pager, but the get_pages
caller is responsible for putting fetched pages into queues (or wiring
them beforehand).

Note that the VM object lock prevents the newly queued page from being
immediately reclaimed in the window before it is marked dirty by
swap_pager_swapoff_object().

Reported by: pho
Tested by: pho
Reviewed by: dougm, alc, kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D47526

show more ...

swap_pager: iter in haspage, lookup, getpages

Use pctrie iterators for swblk traversal in more swap_pager
functions: swap_pager_haspage, swp_pager_meta_lookup, and
swap_pager_getpages.

Reported by:

swap_pager: iter in haspage, lookup, getpages

Use pctrie iterators for swblk traversal in more swap_pager
functions: swap_pager_haspage, swp_pager_meta_lookup, and
swap_pager_getpages.

Reported by: markj
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D47232

show more ...

swap_pager: fix seek_hole assert

Moving code from tmpfs to swap_pager introduced another WLOCKED object
assert that should have been an RLOCKED object assert. Fix it.

swap_pager: fix assert in seek_data

An assertion that an object was write-locked should be instead an
assertion that the object is read locked.

Reported by: Jenkins
Fixes: db08b0b04deced tmpfs_vno

swap_pager: fix assert in seek_data

An assertion that an object was write-locked should be instead an
assertion that the object is read locked.

Reported by: Jenkins
Fixes: db08b0b04deced tmpfs_vnops: move swap work to swap_pager
Differential Revision: https://reviews.freebsd.org/D47278

show more ...

tmpfs_vnops: move swap work to swap_pager

Two functions in tmpfs_vnops.c use an interface provided by
swap_pager.c. Move most of the implementation of those functions to
swap_pager.c so that they ca

tmpfs_vnops: move swap work to swap_pager

Two functions in tmpfs_vnops.c use an interface provided by
swap_pager.c. Move most of the implementation of those functions to
swap_pager.c so that they can be implemented more effectively, with
access to implementation details of the swap pager.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D47212

show more ...

swap_pager: move scan_all_shadowed, use iterators

Move vm_object_scan_all_shadowed from vm_object.c to swap_pager.c, and
rename it. In the moved function, use vm_page and swblk iterators to
advance

swap_pager: move scan_all_shadowed, use iterators

Move vm_object_scan_all_shadowed from vm_object.c to swap_pager.c, and
rename it. In the moved function, use vm_page and swblk iterators to
advance through the objects. Avoid checking a backing page for
busyness or validity more than once, or when it is beyond the upper
bound of the scan.

Reviewed by: kib, markj
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D47150

show more ...

swap_pager: swapoff detecting object death

In swap_pager_swapoff_object, the object is initially not dead, and
can only become dead while the object lock is not held. Move the test
for object-death

swap_pager: swapoff detecting object death

In swap_pager_swapoff_object, the object is initially not dead, and
can only become dead while the object lock is not held. Move the test
for object-death so that it is right after the early loop-break that
happens after lock re-acquisition, and before the iterator is
re-initialized, which fails an assertion when the object is dead,
and not swap.

Addresses a problem produced in testing by @pho and diagnosed by @kib.

Reported by: pho
Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D47064

show more ...

swap_pager: use iterators in swp_pager_meta_build

Add a method to use an iterator for pctrie insertion; this should
improve performance when the last search ended near the place where
the new item w

swap_pager: use iterators in swp_pager_meta_build

Add a method to use an iterator for pctrie insertion; this should
improve performance when the last search ended near the place where
the new item will be inserted.

Add an iterator argument to swp_pager_meta_build, so that the lookups
and insertions it does can be faster in the common case when keys are
bunched close together, or appear in sequence.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D46848

show more ...

swap_pager: rename iter init functions

Add _init to the function names of the functions that initialize
iterators for swblks.

Reported by: alc, markj
Reviewed by: alc
Differential Revision: https:/

swap_pager: rename iter init functions

Add _init to the function names of the functions that initialize
iterators for swblks.

Reported by: alc, markj
Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D46974

show more ...

swap_pager: examine swblks with pctrie iterators

Replace calls to pctrie lookup and remove functions, which always
begin with a search from the pctrie root, with the use of pctrie
iterators to trave

swap_pager: examine swblks with pctrie iterators

Replace calls to pctrie lookup and remove functions, which always
begin with a search from the pctrie root, with the use of pctrie
iterators to traverse and remove items from the pctrie of swapblks
without duplicating searches. Take care to reset iterators after
reacquiring an object lock, since with the lock released they could be
invalidated.

Reviewed by: alc, markj, kib
Tested by: pho (previous versions)
Differential Revision: https://reviews.freebsd.org/D46620

show more ...

swap_pager: use vm_page_iterators for lookup

Replace the use of page lookups and page next pointers in swap_pager.c
with vm_page iterators.

Reviewed by: markj
Differential Revision: https://reviews

swap_pager: use vm_page_iterators for lookup

Replace the use of page lookups and page next pointers in swap_pager.c
with vm_page iterators.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D46704

show more ...

swap_pager: update meta_free comment

Clarify the comment that introduces swp_page_meta_free.

Reviewed by: markj, alc
Differential Revision: https://reviews.freebsd.org/D46754

swap_pager: deoptimize swapoff_object

Undo a change in swap_pager_swapoff_object that assumed a swap block
would be valid after reacquiring a lock, when that validity cannot be
assumed.

Reviewed by

swap_pager: deoptimize swapoff_object

Undo a change in swap_pager_swapoff_object that assumed a swap block
would be valid after reacquiring a lock, when that validity cannot be
assumed.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D46753

show more ...

swap_pager: enhance meta_transfer comments

Clarify comments about what happens to source blocks in
swp_pager_meta_transfer. No functional changes.

Reviewed by: alc
Differential Revision: https://r

swap_pager: enhance meta_transfer comments

Clarify comments about what happens to source blocks in
swp_pager_meta_transfer. No functional changes.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D46682

show more ...

swap_pager: avoid meta_transfer race

Function swp_pager_meta_transfer expects that after dropping and
reacquiring an object lock, the swap block it's processing still
exists, and has not been remove

swap_pager: avoid meta_transfer race

Function swp_pager_meta_transfer expects that after dropping and
reacquiring an object lock, the swap block it's processing still
exists, and has not been removed from the trie and freed. Rewrite to
avoid depending on that, by scrubbing, removing and freeing it before
releasing the lock.

Reviewed by: alc, markj
Differential Revision: https://reviews.freebsd.org/D46629

show more ...

swap_pager: clean up pctrie usage

Define wrapper functions for the pctrie operations specific to
swap_pager, to hide some verbose details. Separate the meta_transfer
and meta_free functions into se

swap_pager: clean up pctrie usage

Define wrapper functions for the pctrie operations specific to
swap_pager, to hide some verbose details. Separate the meta_transfer
and meta_free functions into separate functions.

Reviewed by: kib (previous version)
Tested by: pho (previous version)
Differential Revision: https://reviews.freebsd.org/D46315

show more ...

swap_pager_freespace: fix freed count

Function swp_pager_meta_transfer uses 'pindex' as the start address of
the swblk in calculating which page to lookup in order to count freed
pages. However, th

swap_pager_freespace: fix freed count

Function swp_pager_meta_transfer uses 'pindex' as the start address of
the swblk in calculating which page to lookup in order to count freed
pages. However, the lookup for a swblk at 'pindex' or greater may
produce one greater than 'pindex', given by sb->p, and that's the
value that should be used to compute a page adddress.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46234

show more ...

swap_pager: speedup meta_transfer

Add a parameter to swp_pager_meta_build, for the benefit of
swp_pager_meta_transfer.

swp_pager_meta_transfer calls swp_pager_xfer_source, which may look up
the sam

swap_pager: speedup meta_transfer

Add a parameter to swp_pager_meta_build, for the benefit of
swp_pager_meta_transfer.

swp_pager_meta_transfer calls swp_pager_xfer_source, which may look up
the same trie entry twice - first, by calling sw_pager_meta_lookup,
and then as the first step in swp_pager_meta_build. A boolean
parameter to swp_pager_meta_build tells that function not to replace a
previously assigned swapblk with a new one, and setting it in this
call makes the first meta_lookup call unnecessary.

swp_pager_meta_transfer calls swp_pager_xfer_source, which may release
and reacquire the source object write lock, because the call to
swp_pager_meta_build may acquire and then release the destination
object write block. But it probably doesn't, so fiddling with the
source object write block was probably unnecessary. This boolean
parameter to swp_pager_meta_build tells it to return immediately if
memory allocation problems are about to require a lock
release/reacquisitiion, so that the caller can release/reacquire the
source object write lock only if truly necessary, around a second call
the swp_pager_meta_build with that boolean parameter not set. This
should make manipulation of the source object write lock rarer.

Reviewed by: alc, kib (previous version)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D45781

show more ...

swap_pager: cleanup swapoff_object

Function swap_pager_swapoff_object calls vm_pager_unswapped (via
swp_pager_force_dirty) for every page that must be unswapped. That
means that there's an unneeded

swap_pager: cleanup swapoff_object

Function swap_pager_swapoff_object calls vm_pager_unswapped (via
swp_pager_force_dirty) for every page that must be unswapped. That
means that there's an unneeded check for lock ownership (the caller
always owns it), a needless PCTRIE_LOOKUP (the caller has already
found it), a call to free one page of swap space only, and a check to
see if all blocks are empty, when the caller usually knows that the
check is useless.

Isolate the essential part, needed however swap_pager_unswapped is
invoked, into a smaller function swap_pager_unswapped_acct. From
swapoff_object, invoke swp_pager_update_freerange for each appropriate
page, so that there are potentially fewer calls to
swp_pager_freeswapspace. Consider freeing a set of blocks (a struct
swblk) only after having invalidated all those blocks.

Replace the doubly-nested loops with a single loop, and refetch and
rescan a swblk only when the object write lock has been released and
reacquired.

After getting a page from swap, dirty it immediately to address a race
condition observed by @kib.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45668

show more ...

Revert "swap_pager: small improvement to find_least"

This reverts commit dd0e5c02ab13b9eb240d42a71a8f41a8b036bd33.

pctrie: avoid typecast

Have PCTRIE_RECLAIM_CALLBACK typecast one function pointer type to
another, to relieve the writer of the call back function from having
to cast its first argument from void* t

pctrie: avoid typecast

Have PCTRIE_RECLAIM_CALLBACK typecast one function pointer type to
another, to relieve the writer of the call back function from having
to cast its first argument from void* to member type.

Reviewed by: rlibby
Differential Revision: https://reviews.freebsd.org/D45586

show more ...

swap_pager: pctrie_reclaim_cb in meta_free_all

Replace the lookup-remove loop in swp_pager_meta_free_all with a call
to SWAP_PCTRIE_RECLAIM_CALLBACK, to eliminate repeated trie searches.

Reviewed b

swap_pager: pctrie_reclaim_cb in meta_free_all

Replace the lookup-remove loop in swp_pager_meta_free_all with a call
to SWAP_PCTRIE_RECLAIM_CALLBACK, to eliminate repeated trie searches.

Reviewed by: rlibby
Differential Revision: https://reviews.freebsd.org/D45583

show more ...

swap_pager: add new page range struct

Define a page_range struct to pair up the two values passed to
freerange functions. Have swp_pager_freeswapspace also take a
page_range argument rather than a p

swap_pager: add new page range struct

Define a page_range struct to pair up the two values passed to
freerange functions. Have swp_pager_freeswapspace also take a
page_range argument rather than a pair of arguments.

In swp_pager_meta_free_all, drop a needless test and use a new
helper function to do the cleanup for each swap block.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45562

show more ...

swap_pager: small improvement to find_least

Drop an unneeded test, a branch and a needless computation to save a
few instructions.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.or

swap_pager: small improvement to find_least

Drop an unneeded test, a branch and a needless computation to save a
few instructions.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45558

show more ...