unionfs: do not create a new status object during vop_close()

Split the portion of unionfs_get_node_status() that searches for an
existing status object into a new helper function,
unionfs_find_node

unionfs: do not create a new status object during vop_close()

Split the portion of unionfs_get_node_status() that searches for an
existing status object into a new helper function,
unionfs_find_node_status(), and use that in unionfs_close().

Additionally, modify unionfs_close() to accept a NULL status object
if unionfs_find_node_status() does not find a matching status
object. This can happen due to the unconditional VOP_CLOSE()
operation issued by vgonel().

Differential Revision: https://reviews.freebsd.org/D45398
Reviewed by: olce
Tested by: pho

show more ...

unionfs: rework locking scheme to only lock a single vnode

Instead of locking both the lower and upper vnodes, which is both
complex and deadlock-prone, only lock the upper vnode, or the lower
vnode

unionfs: rework locking scheme to only lock a single vnode

Instead of locking both the lower and upper vnodes, which is both
complex and deadlock-prone, only lock the upper vnode, or the lower
vnode if no upper vnode is present.

In most cases this is all that is needed; for the cases in which
both vnodes do need to be locked, this change also employs deadlock-
avoiding techniques such as LK_NOWAIT and vn_lock_pair().

There are still some corner cases in which the current implementation
ends up taking multiple vnode locks across different filesystems
without taking special steps to avoid deadlock; those cases have
been noted in the comments.

Differential Revision: https://reviews.freebsd.org/D45398
Reviewed by: olce
Tested by: pho

show more ...

unionfs: accommodate underlying FS calls that may re-lock

Since non-doomed unionfs vnodes always share their primary lock with
either the lower or upper vnode, any forwarded call to the base FS
whic

unionfs: accommodate underlying FS calls that may re-lock

Since non-doomed unionfs vnodes always share their primary lock with
either the lower or upper vnode, any forwarded call to the base FS
which transiently drops that upper or lower vnode lock may result in
the unionfs vnode becoming completely unlocked during that transient
window. The unionfs vnode may then become doomed by a concurrent
forced unmount, which can lead to either or both of the following:

--Complete loss of the unionfs lock: in the process of being
doomed, the unionfs vnode switches back to the default vnode lock,
so even if the base FS VOP reacquires the upper/lower vnode lock,
that no longer translates into the unionfs vnode being relocked.
This will then violate that caller's locking assumptions as well
as various assertions that are enabled with DEBUG_VFS_LOCKS.

--Complete less of reference on the upper/lower vnode: the caller
normally holds a reference on the unionfs vnode, while the unionfs
vnode in turn holds references on the upper/lower vnodes. But in
the course of being doomed, the unionfs vnode will drop the latter
set of references, which can effectively lead to the base FS VOP
executing with no references at all on its vnode, violating the
assumption that vnodes can't be recycled during these calls and
(if lucky) violating various assertions in the base FS.

Fix this by adding two new functions, unionfs_forward_vop_start_pair()
and unionfs_forward_vop_finish_pair(), which are intended to bookend
any forwarded VOP which may transiently unlock the relevant vnode(s).
These functions are currently only applied to VOPs that modify file
state (and require vnode reference and lock state to be identical at
call entry and exit), as the common reason for transiently dropping
locks is to update filesystem metadata.

Reviewed by: olce
Tested by: pho
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D44076

show more ...

unionfs: cache upper/lower mount objects

Store the upper/lower FS mount objects in unionfs per-mount data and
use these instead of the v_mount field of the upper/lower root
vnodes. As described in

unionfs: cache upper/lower mount objects

Store the upper/lower FS mount objects in unionfs per-mount data and
use these instead of the v_mount field of the upper/lower root
vnodes. As described in the referenced PR, it is unsafe to access this
field on the unionfs unmount path as ZFS rollback may have obliterated
the v_mount field of the upper or lower root vnode. Use these stored
objects to slightly simplify other code that needs access to the
upper/lower mount objects as well.

PR: 275870
Reported by: Karlo Miličević <karlo98.m@gmail.com>
Tested by: Karlo Miličević <karlo98.m@gmail.com>
Reviewed by: kib (prior version), olce
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43815

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .h pattern

Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/

Remove unnecessary thread argument from unionfs_nodeget() and _noderem()

Also remove a couple of write-only variables found by the recent clang
update. No functional change intended.

Discussed wit

Remove unnecessary thread argument from unionfs_nodeget() and _noderem()

Also remove a couple of write-only variables found by the recent clang
update. No functional change intended.

Discussed with: kib
Differential Revision: https://reviews.freebsd.org/D33008

show more ...

unionfs: Improve vnode validation

Instead of validating that a vnode belongs to unionfs only when the
caller attempts to extract the upper or lower vnode pointers, do this
validation any time the ca

unionfs: Improve vnode validation

Instead of validating that a vnode belongs to unionfs only when the
caller attempts to extract the upper or lower vnode pointers, do this
validation any time the caller tries to extract a unionfs_node from
the vnode private data.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D32629

show more ...

unionfs: assorted style fixes

No functional change intended, beyond slightly different panic strings

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D32629

unionfs: implement vnode-based cache lookup

unionfs uses a per-directory hashtable to cache subdirectory nodes.
Currently this hashtable is looked up using the directory name, but
since unionfs node

unionfs: implement vnode-based cache lookup

unionfs uses a per-directory hashtable to cache subdirectory nodes.
Currently this hashtable is looked up using the directory name, but
since unionfs nodes aren't removed from the cache until they're
reclaimed, this poses some problems. For example, if a directory is
created on a unionfs mount shortly after deleting a previous directory
with the same path, the cache may end up reusing the node for the
previous directory, including its upper/lower FS vnodes. Operations
against those vnodes with then likely fail because the vnodes
represent deleted files; for example UFS will reject VOP_MKDIR()
against such a vnode because its effective link count is 0. This may
then manifest as e.g. mkdir(2) or open(2) returning ENOENT for an
attempt to create a file under the re-created directory.

While it would be possible to fix this by explicitly managing the
name-based cache during delete or rename operations, or by rejecting
cache hits if the underlying FS vnodes don't match those passed to
unionfs_nodeget(), it seems cleaner to instead hash the unionfs nodes
based on their underlying FS vnodes. Since unionfs prefers to operate
against the upper vnode if one is present, the lower vnode will only
be used for hashing as long as the upper vnode is NULL. This should
also make hashing faster by eliminating string traversal and using
the already-computed hash index stored in each vnode.

While here, fix a couple of other cache-related issues:

--Remove 8 bytes of unnecessary baggage from each unionfs node by
getting rid of the stored hash mask field. The mask is knowable
at compile time.

--When a matching node is found in the cache, reference its vnode
using vrefl() while still holding the vnode interlock. Previously
unionfs_nodeget() would vref() the vnode after the interlock was
dropped, but the vnode may be reclaimed during that window. This
caused intermittent panics from vn_lock(9) during unionfs stress
testing.

Reviewed by: kib, markj
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D32533

show more ...

unionfs: style

Fix the more egregious style(9) violations in unionfs.
No functional change intended.

unionfs: rework pathname handling

Running stress2 unionfs tests reliably produces a namei_zone corruption
panic due to unionfs_relookup() attempting to NUL-terminate a newly-
allocate pathname buffe

unionfs: rework pathname handling

Running stress2 unionfs tests reliably produces a namei_zone corruption
panic due to unionfs_relookup() attempting to NUL-terminate a newly-
allocate pathname buffer without first validating the buffer length.

Instead, avoid allocating new pathname buffers in unionfs entirely,
using already-provided buffers while ensuring the the correct flags
are set in struct componentname to prevent freeing or manipulation
of those buffers at lower layers.

While here, also compute and store the path length once in the unionfs
node instead of constantly invoking strlen() on it.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D31728

show more ...

Allow stacked filesystems to be recursively unmounted

In certain emergency cases such as media failure or removal, UFS will
initiate a forced unmount in order to prevent dirty buffers from
accumulat

Allow stacked filesystems to be recursively unmounted

In certain emergency cases such as media failure or removal, UFS will
initiate a forced unmount in order to prevent dirty buffers from
accumulating against the no-longer-usable filesystem. The presence
of a stacked filesystem such as nullfs or unionfs above the UFS mount
will prevent this forced unmount from succeeding.

This change addreses the situation by allowing stacked filesystems to
be recursively unmounted on a taskqueue thread when the MNT_RECURSE
flag is specified to dounmount(). This call will block until all upper
mounts have been removed unless the caller specifies the MNT_DEFERRED
flag to indicate the base filesystem should also be unmounted from the
taskqueue.

To achieve this, the recently-added vfs_pin_from_vp()/vfs_unpin() KPIs
have been combined with the existing 'mnt_uppers' list used by nullfs
and renamed to vfs_register_upper_from_vp()/vfs_unregister_upper().
The format of the mnt_uppers list has also been changed to accommodate
filesystems such as unionfs in which a given mount may be stacked atop
more than one lower mount. Additionally, management of lower FS
reclaim/unlink notifications has been split into a separate list
managed by a separate set of KPIs, as registration of an upper FS no
longer implies interest in these notifications.

Reviewed by: kib, mckusick
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D31016

show more ...

unionfs: release parent vnodes in deferred context

Each unionfs node holds a reference to its parent directory vnode.
A single open file reference can therefore end up keeping an
arbitrarily deep vn

unionfs: release parent vnodes in deferred context

Each unionfs node holds a reference to its parent directory vnode.
A single open file reference can therefore end up keeping an
arbitrarily deep vnode hierarchy in place. When that reference is
released, the resulting VOP_RECLAIM call chain can then exhaust the
kernel stack.

This is easily reproducible by running the unionfs.sh stress2 test.
Fix it by deferring recursive unionfs vnode release to taskqueue
context.

PR: 238883
Reviewed By: kib (earlier version), markj
Differential Revision: https://reviews.freebsd.org/D30748

show more ...

Merge ^/head r325999 through r326131.

sys: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for

sys: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

show more ...

Merge ^/head r314420 through r314481.

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

show more ...

integrate from head@185615

Fixes Unionfs socket issue reported as kern/118346.

PR: 118346
Submitted by: Masanori OZAWA <ozawa@ongs.co.jp>
Discussed at: devsummit Strassburg, EuroBSDCon2008
Discussed with: rwatson, gnn, hrs
M

Fixes Unionfs socket issue reported as kern/118346.

PR: 118346
Submitted by: Masanori OZAWA <ozawa@ongs.co.jp>
Discussed at: devsummit Strassburg, EuroBSDCon2008
Discussed with: rwatson, gnn, hrs
MFC after: 2 week

show more ...

o Fixed multi thread access issue reported by Alexander V. Chernikov
(admin@su29.net)
fixed: kern/109950

PR: kern/109950
Submitted by: Alexander V. Chernikov (admin@su29.net)
Reviewed by: Mas

o Fixed multi thread access issue reported by Alexander V. Chernikov
(admin@su29.net)
fixed: kern/109950

PR: kern/109950
Submitted by: Alexander V. Chernikov (admin@su29.net)
Reviewed by: Masanori OZAWA (ozawa@ongs.co.jp)
MFC after: 1 week

show more ...

o Fixed inaccessible issue especially including devfs on unionfs case.
fixed also: kern/117829

PR: kern/117829
Submitted by: Masanori OZAWA (ozawa@ongs.co.jp)
MFC after: 1 week

Added whiteout behavior option. ``-o whiteout=always'' is default mode
(it is established practice) and ``-o whiteout=whenneeded'' is less
disk-space using mode especially for resource restricted env

Added whiteout behavior option. ``-o whiteout=always'' is default mode
(it is established practice) and ``-o whiteout=whenneeded'' is less
disk-space using mode especially for resource restricted environments
like embedded environments. (Contributed by Ed Schouten. Thanks)

Submitted by: Masanori Ozawa <ozawa@ongs.co.jp> (unionfs developer)
Reviewed by: jeff, kensmith
Approved by: re (kensmith)
MFC after: 1 week

show more ...

- Added support for vfs_cache on unionfs. As a result, you can use
applications that use procfs on unionfs.
- Removed unionfs internal cache mechanism because it has
vfs_cache support instead. As

- Added support for vfs_cache on unionfs. As a result, you can use
applications that use procfs on unionfs.
- Removed unionfs internal cache mechanism because it has
vfs_cache support instead. As a result, it just simplified code of
unionfs.
- Fixed kern/111262 issue.

Submitted by: Masanori Ozawa <ozawa@ongs.co.jp> (unionfs developer)
Reviewed by: jeff, kensmith
Approved by: re (kensmith)
MFC after: 1 week

show more ...

Revert UF_OPENING workaround for CURRENT.
Change the VOP_OPEN(), vn_open() vnode operation and d_fdopen() cdev operation
argument from being file descriptor index into the pointer to struct file.

Pr

Revert UF_OPENING workaround for CURRENT.
Change the VOP_OPEN(), vn_open() vnode operation and d_fdopen() cdev operation
argument from being file descriptor index into the pointer to struct file.

Proposed and reviewed by: jhb
Reviewed by: daichi (unionfs)
Approved by: re (kensmith)

show more ...