Fix backward compatibility with UFS1 filesystems created before June 2002

The code to handle compatibility with old filesystems exists and does
the right things. The loss of compatibility was introd

Fix backward compatibility with UFS1 filesystems created before June 2002

The code to handle compatibility with old filesystems exists and does
the right things. The loss of compatibility was introduced 2.5 years ago
when the superblock integrity checks were added. The problem is that the
compatibility code was being run after the new integrity checks rather
than before them.

The reason that it has taken so long to show up is because it only
affected filesystems created before UFS2 got added in June 2002 and
that had never been mounted read-write (as their superblocks would
be updated by the compatibility code if they could be written).
Hence the problem was not apparent until someone tried to mount a
pre-2002 virgin UFS1 filesystem image.

See the Pahabricator review for more details.

Reviewed-by: kib
Tested-by: Peter Holm
MFC-after: 1 week
Differential-Revision: https://reviews.freebsd.org/D48472

show more ...

SU+J: all writes to SU journal must be exempt from runningbufspace throttling

regardless whether they come from the system thread or initiated from a
normal thread helping the system. If we block w

SU+J: all writes to SU journal must be exempt from runningbufspace throttling

regardless whether they come from the system thread or initiated from a
normal thread helping the system. If we block waiting for other writes,
that writes might not finish because our journal updates block that.

Set TDP_NORUNNINGBUF around softdep_process_journal().

Note: Another solution might be to use bwrite() instead of bawrite() if the
current thread is subject to the runningbufspace limit. The exempt
approach is used to be same as the bufdaemon.

PR: 282449
Noted and reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

softdep_mount: report failure of the softdepflush thread creation

and do not hang waiting forever for an ack from non-existing thread.

PR: 281511
Reported and tested by: pho
Sponsored by: The FreeB

softdep_mount: report failure of the softdepflush thread creation

and do not hang waiting forever for an ack from non-existing thread.

PR: 281511
Reported and tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

softdep_mount: on journal start failure, report the mount point path

Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

proc: Remove kernel stack swapping support, part 7

Remove some uses of PHOLD which were there only to prevent the process'
threads from being swapped out.

Tested by: pho
Reviewed by: imp, kib
Diffe

proc: Remove kernel stack swapping support, part 7

Remove some uses of PHOLD which were there only to prevent the process'
threads from being swapped out.

Tested by: pho
Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D46118

show more ...

Increase UFS/FFS maximum link count from 32767 to 65530.

The link count for a UFS/FFS inode is stored in a signed 16-bit
integer. Thus the maximum link count has been 32767.

This limit has been rec

Increase UFS/FFS maximum link count from 32767 to 65530.

The link count for a UFS/FFS inode is stored in a signed 16-bit
integer. Thus the maximum link count has been 32767.

This limit has been recently hit by the poudriere build system when
doing a ports build as it needs one directory per port and the
number of ports recently passed 32767.

A long-term solution would be to use one of the spare 32-bit fields
in the inode to store the link count. However, the UFS1 format does
not have a spare and adding the spare in UFS2 would make it hard
to make it compatible when running on older kernels that use the
original link count field. So this patch uses the much simpler
approach of changing the existing link count field from a signed
16-bit value to an unsigned 16-bit value. It has the fewest lines
of code changes. The only thing that changes is the type in the
dinode and inode structures and the definition of UFS_LINK_MAX. It
has the added benefit that it works with both UFS1 and UFS2.

It allows easy backward compatibility. Indeed it is backward
compatibility that is the primary reason to go with this approach.
If a filesystem with the new organization is mounted on an older
kernel, it still needs to work. Thus if we move the new link count
to a new field, we still need to maintain the old link count as
best as possible even when running on a kernel that knows about the
larger link counts. And we would have to carry this overhead for
the indefinite future.

If we have a new link-count field, we will have to add a new
filesystem flag to indicate that we are running with larger link
counts. We will also need to add of one of the new-feature flags
to say that we have larger link counts. Older kernels clear the
new-feature flags that they do not know about, so when a filesystem
is used on an older kernel and then moved back to a newer one, the
newer one will know that the new link counts have not been maintained
and that it will be necessary to run a full fsck on the filesystem
to correct the link counts before it can be mounted.

With this change, older kernels will generally work with the bigger
counts. While it will not itself allow the link count to exceed
32767, it will have no problem working with inodes that have a link
count greater than 32767. Since it tests that i_nlink <= UFS_LINK_MAX,
counts that are bigger than 32767 will appear negative, so will
still pass the test. Of course, if they ever drop below 32767, they
will no longer be able to exceed 32767. The one issue is if the
link count ever exceeds 65535 then it will wrap to zero and the
older kernel will be none the wiser. But this corner case is likely
to be very rare since these kernels and the applications running
on them do not expect to be able to get link counts over 32767. And
over time, the use of new filesystems on older kernels will become
rarer and rarer.

Reported-by: Mark Millard running poudriere on the ports tree
Reviewed-by: kib, olce.freebsd_certner.fr
Tested-by: Peter Holm, Mark Millard
MFC-after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D42767

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Handle UFS/FFS file deletion from cylinder groups with check-hash failure.

When a file is deleted, its blocks need to be put back in the free
block list and its inode needs to be put back in the ino

Handle UFS/FFS file deletion from cylinder groups with check-hash failure.

When a file is deleted, its blocks need to be put back in the free
block list and its inode needs to be put back in the inode free list.
These lists reside in cylinder-group maps. If either some of its blocks
or its inode reside in a cylinder-group map with a bad check hash
it is not possible to free the associated resource. Since the cylinder
group cannot be repaired until the filesystem is unmounted these
resources cannot be freed. They simply accumulate in memory. And
any attempt to unmount the filesystem loops forever trying to flush them.

With this change, the resource update claims to succeed so that the
file deletion can successfully complete. The filesystem is marked as
requiring an fsck so that before the next time that the filesystem is
mounted, the offending cylinder groups are reconstructed causing the
lost resources to be reclaimed.

A better solution would be to downgrade the filesystem to read-only,
but that capability is not currently implemented.

Reported-by: Peter Holm
Tested-by: Peter Holm
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

show more ...

UFS/FFS: Migrate to modern uintXX_t from u_intXX_t.

As per https://lists.freebsd.org/archives/freebsd-scsi/2023-July/000257.html
move to the modern uintXX_t. While here also migrate u_char to uint8_

UFS/FFS: Migrate to modern uintXX_t from u_intXX_t.

As per https://lists.freebsd.org/archives/freebsd-scsi/2023-July/000257.html
move to the modern uintXX_t. While here also migrate u_char to uint8_t.
Where other kernel interfaces allow, migrate u_long to uint64_t.

No functional changes intended.

MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

show more ...

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

vn_lock_pair(): allow to request shared locking

If either of vnodes is shared locked, lock must not be recursed.

Requested by: rmacklem
Reviewed by: markj, rmacklem
Tested by: pho
Sponsored by: The

vn_lock_pair(): allow to request shared locking

If either of vnodes is shared locked, lock must not be recursed.

Requested by: rmacklem
Reviewed by: markj, rmacklem
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D39444

show more ...

Improvement in UFS/FFS directory placement when doing mkdir(2).

The algorithm for laying out new directories was devised in the 1980s
and markedly improved the performance of the filesystem. In thos

Improvement in UFS/FFS directory placement when doing mkdir(2).

The algorithm for laying out new directories was devised in the 1980s
and markedly improved the performance of the filesystem. In those days
large disks had at most 100 cylinder groups and often as few as 10-20.
Modern multi-terrabyte disks have thousands of cylinder groups. The
original algorithm does not handle these large sizes well. This change
attempts to expand the scope of the original algorithm to work well
with these much larger disks while still retaining the properties
of the original algorithm for small disks.

The filesystem implementation is divided into policy routines and
implementation routines. The policy routines can be changed in any
way desired without risk of corrupting the filesystem. The policy
requests are handled by the implementation layer. If the policy
asks for an available resource, it is granted. But if it asks for
an already in-use resource, then the implementation will provide
an available one nearby the request. Thus it is impossible for a
policy to double allocate. This change is limited to the policy
implementation.

This change updates the ffs_dirpref() routine which is responsible
for selecting the cylinder group into which a new directory should
be placed. If we are near the root of the filesystem we aim to
spread them out as much as possible. As we descend deeper from the
root we cluster them closer together around their parent as we
expect them to be more closely interactive. Higher-level directories
like usr/src/sys and usr/src/bin should be separated while the
directories in these areas are more likely to be accessed together
so should be closer. And directories within commands or kernel
subsystems should be closer still.

We pick a range of cylinder groups around the cylinder group of the
directory in which we are being created. The size of the range for
our search is based on our depth from the root of our filesystem.
We then probe that range based on how many directories are already
present. The first new directory is at 1/2 (middle) of the range;
the second is in the first 1/4 of the range, then at 3/4, 1/8, 3/8,
5/8, 7/8, 1/16, 3/16, 5/16, etc.

It is desirable to store the depth of a directory in its on-disk
inode so that it is available when we need it. We add a new field
di_dirdepth to track the depth of each directory. Because there are
few spare fields left in the inode, we choose to share an existing
field in the inode rather than having one of our own. Specifically
we create a union with the di_freelink field. The di_freelink field
is used to track inodes that have been unlinked but remain referenced.
It is not needed until a rmdir(2) operation has been done on a
directory. At that point, the directory has no contents and even
if it is kept active as a current directory is no longer able to
have any new directories or files created in it. Thus the use of
di_dirdepth and di_freelink will never coincide.

Reported by: Timo Voelker
Reviewed by: kib
Tested by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39246

show more ...

ffs_syncvnode(): avoid a LoR for SU

There is another case where SU code does ffs_syncvnode(dvp) for the
parent directory dvp while the child vnode vp is locked. Avoid the
issue by relocking and ret

ffs_syncvnode(): avoid a LoR for SU

There is another case where SU code does ffs_syncvnode(dvp) for the
parent directory dvp while the child vnode vp is locked. Avoid the
issue by relocking and returning ERELOOKUP to indicate the need of
resync.

Reported by: jkim
Reviewed by: mckusick
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D37997

show more ...

AST: rework

Make most AST handlers dynamically registered. This allows to have
subsystem-specific handler source located in the subsystem files,
instead of making subr_trap.c aware of it. For inst

AST: rework

Make most AST handlers dynamically registered. This allows to have
subsystem-specific handler source located in the subsystem files,
instead of making subr_trap.c aware of it. For instance, signal
delivery code on return to userspace is now moved to kern_sig.c.

Also, it allows to have some handlers designated as the cleanup (kclear)
type, which are called both at AST and on thread/process exit. For
instance, ast(), exit1(), and NFS server no longer need to be aware
about UFS softdep processing.

The dynamic registration also allows third-party modules to register AST
handlers if needed. There is one caveat with loadable modules: the
code does not make any effort to ensure that the module is not unloaded
before all threads processed through AST handler in it. In fact, this
is already present behavior for hwpmc.ko and ufs.ko. I do not think it
is worth the efforts and the runtime overhead to try to fix it.

Reviewed by: markj
Tested by: emaste (arm64), pho
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D35888

show more ...

Rewrite function definitions in the UFS/FFS code base with identifier lists.

The K&R style in UFS and other places in the tree's days are numbered
as this syntax is removed in C2x proposal N2432:

Rewrite function definitions in the UFS/FFS code base with identifier lists.

The K&R style in UFS and other places in the tree's days are numbered
as this syntax is removed in C2x proposal N2432:

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2432.pdf

Though running to nearly 6000 lines of diffs this update should cause
no functional change to the code.

Requested by: Warner Losh
MFC after: 2 weeks

show more ...

ffs: plug a set-but-not-used var

ufs: Fix a typo in a source code comment

- s/explicitely/explicitly/

MFC after: 3 days

ufs: be more persistent with finishing some operations

when the vnode is doomed after relock. The mere fact that the vnode is
doomed does not prevent us from doing UFS operations on it while it is

ufs: be more persistent with finishing some operations

when the vnode is doomed after relock. The mere fact that the vnode is
doomed does not prevent us from doing UFS operations on it while it is
still belongs to UFS, which is determined by non-NULL v_data. Not
finishing some operations, e.g. not syncing the inode block only because
the vnode started reclamation, is not correct.

Add macro IS_UFS() which incapsulates the v_data != NULL, and use it
instead of VN_IS_DOOMED() for places where the operation completion is
important.

Reviewed by: markj, mckusick
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D34072

show more ...

ffs_softdep: Fix a typo in a source code comment

- s/conditonally/conditionally/

MFC after: 3 days

ufs: remove write-only variables

Mark variables as __diagused for invariant-only vars

Reviewed by: imp, mjg
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://rev

ufs: remove write-only variables

Mark variables as __diagused for invariant-only vars

Reviewed by: imp, mjg
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32577

show more ...

vfs: remove cn_thread

It is always curthread.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D32453

ddb: do not assume that ffs is mounted with softdep

Avoid a panic when debugging with "show ffs" in ddb.

Reviewed By: kib, markj, mckusick
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differenti

ddb: do not assume that ffs is mounted with softdep

Avoid a panic when debugging with "show ffs" in ddb.

Reviewed By: kib, markj, mckusick
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D31622

show more ...

Clean up orphaned indirdep dependency structures after disk failure.

During forcible unmount after a disk failure there is a bug that
causes one or more indirdep dependency structures to fail to be

Clean up orphaned indirdep dependency structures after disk failure.

During forcible unmount after a disk failure there is a bug that
causes one or more indirdep dependency structures to fail to be
deallocated. Until we manage to track down why they fail to get
cleaned up, this code tracks them down and eliminates them so that
the unmount can succeed.

Reported by: Peter Holm
Help from: kib
Reviewed by: Chuck Silvers
Tested by: Peter Holm
MFC after: 7 days
Sponsored by: Netflix

show more ...

Diagnotic improvement to soft dependency structure management.

The soft updates diagnotic code keeps a list for each type of soft
update dependency. When a new block is allocated for a file it is
in

Diagnotic improvement to soft dependency structure management.

The soft updates diagnotic code keeps a list for each type of soft
update dependency. When a new block is allocated for a file it is
initially tracked by a "newblk" dependency. The "newblk" dependency
eventually becomes either an "allocdirect" dependency or an "indiralloc"
dependency. The diagnotic code failed to move the "newblk" from the list
of "newblk"s to its new type list.

No functional change intended.

Reviewed by: Chuck Silvers (as part of a larger change)
Tested by: Peter Holm (as part of a larger change)
Sponsored by: Netflix

show more ...