Defer the January 19, 2038 date limit in UFS1 filesystems to February 7, 2106

UFS1 uses a signed 32-bit value for its times. Zero is
January 1, 1970 UTC. Negative values of 32-bit time predate
Janua

Defer the January 19, 2038 date limit in UFS1 filesystems to February 7, 2106

UFS1 uses a signed 32-bit value for its times. Zero is
January 1, 1970 UTC. Negative values of 32-bit time predate
January 1, 1970 back to December 13, 1901. The maximum positive
value for 32-bit time is on January 19, 2038 (my 84th birthday).
On that date, time will go negative and start registering from
December 13, 1901. Note that this issue only affects UFS1 filesystems
since UFS2 has 64-bit times. This fix changes UFS1 times from
signed to unsigned 32-bit values. With this change it will no longer
be possible to represent time from before January 1, 1970, but it
will accurately track time until February 7, 2106. Hopefully there
will not be any FreeBSD systems using UFS1 still in existence by
that time (and by then I will have been dead long enough that no-one
will know at whom to yell :-).

It is possible that some existing UFS1 systems will have set times
predating January 1, 1970. With this commit they will appear as
later than the current time. This commit checks inode times when
they are read into memory and if they are greater than the current
time resets them to the current time. By default this reset happens
silently, but setting the sysctl vfs.ffs.prttimechgs=1 will cause
console messages to be printed whenever a future time is changed.

Reviewed-by: kib
Tested-by: Peter Holm
MFC-after: 1 week
Differential Revision: https://reviews.freebsd.org/D48472

show more ...

Fix build from aa90fbed151de5 by eliminating an unused variable.

Standardize the definition of a UFS dinode.

Each program that operates on UFS on-disk inodes defines its own
version of a dinode. They all (of necessity) define the same
layout but use different nam

Standardize the definition of a UFS dinode.

Each program that operates on UFS on-disk inodes defines its own
version of a dinode. They all (of necessity) define the same
layout but use different names. This change adds a definition of
a dinode (a union of a UFS1 on-disk inode and a UFS2 on-disk inode)
as well as a dinodep (a union of a pointer to a UFS1 on-disk inode
and a pointer to a UFS2 on-disk inode) in sys/ufs/ufs/dinode.h.
It then deletes the definitions of dinode and dinodep in all the
programs that operate on them and instead uses these standard
definitions.

No functional change intended.

MFC-after: 1 week

show more ...

sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remo

sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl

sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

Ensure I/O buffers in libufs(3) are 128-byte aligned.

Various disk controllers require their buffers to be aligned to a
cache-line size (128 bytes). For buffers allocated in structures,
ensure that

Ensure I/O buffers in libufs(3) are 128-byte aligned.

Various disk controllers require their buffers to be aligned to a
cache-line size (128 bytes). For buffers allocated in structures,
ensure that they are 128-byte aligned. Use aligned_malloc to allocate
memory to ensure that the returned memory is 128-byte aligned.

While we are here, we replace the dynamically allocated inode buffer
with a buffer allocated in the uufsd structure just as the superblock
and cylinder group buffers do.

This can be removed if/when the kernel is fixed. Because this problem
has existed on one I/O subsystem or another since the 1990's, we
are probably stuck with dealing with it forever.

The problem most recent showed up in Azure, see:
https://reviews.freebsd.org/D41728
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267654
Before these fixes were applied, it was confirmed that the changes
in this commit also fixed the issue in Azure.

Reviewed-by: Warner Losh, kib
Tested-by: Souradeep Chakrabarti of Microsoft (earlier version)
PR: 267654
Differential Revision: https://reviews.freebsd.org/D41724

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Cleanups to fsck_ffs(8).

When checking an inode ensure that it does not have a negative size.
Stop scaning a directory when an unallocated block is found.
Fully clear an inode when it is first alloc

Cleanups to fsck_ffs(8).

When checking an inode ensure that it does not have a negative size.
Stop scaning a directory when an unallocated block is found.
Fully clear an inode when it is first allocated.
Ensure that an inode is marked dirty whenever it is updated and that
it has a correct check hash when it is released.

MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

show more ...

Fix a bug in fsck_ffs(8) triggered by corrupted filesystems.

The last valid inode in the filesystem is maxino - 1, not maxino.
Thus validity checks should ino < maxino, not ino <= maxino.

Reported-

Fix a bug in fsck_ffs(8) triggered by corrupted filesystems.

The last valid inode in the filesystem is maxino - 1, not maxino.
Thus validity checks should ino < maxino, not ino <= maxino.

Reported-by: Robert Morris
PR: 271312
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

show more ...

Fix off-by-one error in fsck_ffs(8) chkrange() block-number check.

On an amd64-CURRENT machine with an i-node that refers to a block
number that is one too large will cause a core dump, due to writi

Fix off-by-one error in fsck_ffs(8) chkrange() block-number check.

On an amd64-CURRENT machine with an i-node that refers to a block
number that is one too large will cause a core dump, due to writing
beyond the end of blockmap[] and corrupting the next heap block,
which happens to contain a struct inoinfo in inphash[]. Note that
valgrind catches the blockmap[] access.

Reported by: Robert Morris
PR: 271289
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

show more ...

fsck_ffs: fix the previous change that skipped pass 5 in some cases

The previous change involved calling check_cgmagic() twice in a row
for the same CG in order to differentiate when the CG was alre

fsck_ffs: fix the previous change that skipped pass 5 in some cases

The previous change involved calling check_cgmagic() twice in a row
for the same CG in order to differentiate when the CG was already ok vs.
when the CG was rebuilt, but that doesn't work because the second call
(which was supposed to rebuild the CG) returns 0 (indicating that
the CG was not rebuilt) due to the prevfailcg check causing an early
failure return. Fix this by moving the rebuild part of check_cgmagic()
out into a separate function which is called by pass1() when it wants to
rebuild a CG.

Fixes: da86e7a20dc4a4b17e8d9e7630ed9b675cf71702
Reported by: pho
Discussed with: mckusick
Sponsored by: Netflix

show more ...

Skip Pass 5 in fsck_ffs(8) when corrupt cylinder groups remain unfixed.

Pass 1 of fsck_ffs checks the integrity of all the cylinder groups.
If any are found to have been corrupted it offers to rebui

Skip Pass 5 in fsck_ffs(8) when corrupt cylinder groups remain unfixed.

Pass 1 of fsck_ffs checks the integrity of all the cylinder groups.
If any are found to have been corrupted it offers to rebuild them.
Pass 5 then makes a second pass over the cylinder groups to validate
their block and inode maps. Pass 5 assumes that the cylinder groups
are not corrupted and can segment fault if they are corrupted. Rather
than rerunning the corruption checks a second time in pass 5, this
fix keeps track whether any corrupt cylinder groups were found but not
fixed in pass 1 either due to running with the -n flag or by explicitly
answering `no' when asked whether to fix a corrupted cylinder group.
If any corrupted cylinder groups remain after pass 1, fsck_ffs will
decline to run pass 5. Instead it marks the filesystem as unclean
so that fsck_ffs will need to be run again before the filesystem can
be mounted.

This patch cleans up and documents the return value from check_cgmagic().
It also renames the variable / parameter "rebuildcg" to "rebuiltcg".
This parameter describes whether the cylinder group has been rebuilt
rather than whether it should be rebuilt.

Reported by: Chuck Silvers
Reviewed by: Chuck Silvers
MFC after: 1 week

show more ...

Bug fixes for fsck_ffs(8).

Increment a reference count when returning a zero'ed out buffer
after a failed read.

Zero out a structure before using it.

Only dirty a buffer that has been modified.

S

Bug fixes for fsck_ffs(8).

Increment a reference count when returning a zero'ed out buffer
after a failed read.

Zero out a structure before using it.

Only dirty a buffer that has been modified.

Submitted by: Chuck Silvers
Sponsored by: Netflix
MFC after: 1 week

show more ...

Improvement in UFS/FFS directory placement when doing mkdir(2).

The algorithm for laying out new directories was devised in the 1980s
and markedly improved the performance of the filesystem. In thos

Improvement in UFS/FFS directory placement when doing mkdir(2).

The algorithm for laying out new directories was devised in the 1980s
and markedly improved the performance of the filesystem. In those days
large disks had at most 100 cylinder groups and often as few as 10-20.
Modern multi-terrabyte disks have thousands of cylinder groups. The
original algorithm does not handle these large sizes well. This change
attempts to expand the scope of the original algorithm to work well
with these much larger disks while still retaining the properties
of the original algorithm for small disks.

The filesystem implementation is divided into policy routines and
implementation routines. The policy routines can be changed in any
way desired without risk of corrupting the filesystem. The policy
requests are handled by the implementation layer. If the policy
asks for an available resource, it is granted. But if it asks for
an already in-use resource, then the implementation will provide
an available one nearby the request. Thus it is impossible for a
policy to double allocate. This change is limited to the policy
implementation.

This change updates the ffs_dirpref() routine which is responsible
for selecting the cylinder group into which a new directory should
be placed. If we are near the root of the filesystem we aim to
spread them out as much as possible. As we descend deeper from the
root we cluster them closer together around their parent as we
expect them to be more closely interactive. Higher-level directories
like usr/src/sys and usr/src/bin should be separated while the
directories in these areas are more likely to be accessed together
so should be closer. And directories within commands or kernel
subsystems should be closer still.

We pick a range of cylinder groups around the cylinder group of the
directory in which we are being created. The size of the range for
our search is based on our depth from the root of our filesystem.
We then probe that range based on how many directories are already
present. The first new directory is at 1/2 (middle) of the range;
the second is in the first 1/4 of the range, then at 3/4, 1/8, 3/8,
5/8, 7/8, 1/16, 3/16, 5/16, etc.

It is desirable to store the depth of a directory in its on-disk
inode so that it is available when we need it. We add a new field
di_dirdepth to track the depth of each directory. Because there are
few spare fields left in the inode, we choose to share an existing
field in the inode rather than having one of our own. Specifically
we create a union with the di_freelink field. The di_freelink field
is used to track inodes that have been unlinked but remain referenced.
It is not needed until a rmdir(2) operation has been done on a
directory. At that point, the directory has no contents and even
if it is kept active as a current directory is no longer able to
have any new directories or files created in it. Thus the use of
di_dirdepth and di_freelink will never coincide.

Reported by: Timo Voelker
Reviewed by: kib
Tested by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39246

show more ...

Rewrite function definitions with identifier lists.

A few functions snuck in with K&R style definitions.

Also add some missing memory frees.

MFC after: 1 week

Correct several bugs in fsck_ffs(8) triggered by corrupted filesystems.

If a directory entry has an illegal inode number (less than zero
or greater than the last inode in the filesystem) the entry i

Correct several bugs in fsck_ffs(8) triggered by corrupted filesystems.

If a directory entry has an illegal inode number (less than zero
or greater than the last inode in the filesystem) the entry is removed.
If a directory '.' or '..' entry had an illegal inode number they
were being removed. Since fsck_ffs knows what the correct value is
for these two entries fix them rather deleting them.

Add much more extensive cylinder group checks and use them to be
more careful about rebuilding a cylinder group.

Check for out-of-range block numbers before trying to free them.

When a directory is deleted also remove its cache entry created
in pass1 so that later passes do not try to operate on a deleted
directory.

Check for ctime(3) returning NULL before trying to use its return.

When freeing a directory inode, do not try to interpret it as a
directory.

Reserve space in the inostatlist to have room to allocate a
lost+found directory.

If an invalid block number is found past the end of an inode simply
remove it rather than clearing and removing the inode.

Modernize the inoinfo structure to use queue(3) LIST rather than a
handrolled linked list implementation.

Reported by: Bob Prohaska, John-Mark Gurney, and Mark Millard
Tested by: Peter Holm
Reviewed by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D38668

show more ...

Fix printfs for fsck_ffs(8) i386 build.

Reported by: jenkins
Sponsored by: The FreeBSD Foundation

Fix types for fsck_ffs(8) i386 build.

Reported by: jenkins
Reported by: Cy Schubert
Sponsored by: The FreeBSD Foundation

Add support for managing UFS/FFS snapshots to fsck_ffs(8).

The kernel handles the managment of UFS/FFS snapshots. Since UFS/FFS
updates filesystem data (rather than always writing changes to new
loc

Add support for managing UFS/FFS snapshots to fsck_ffs(8).

The kernel handles the managment of UFS/FFS snapshots. Since UFS/FFS
updates filesystem data (rather than always writing changes to new
locations like ZFS), the kernel must check every filesystem write
to see if the block being written is part of a snapshot. If it is
part of a snapshot, then the kernel must make a copy of the old
block value into a newly allocated block for the snapshot before
allowing the write to be done. Similarly, if a block is being freed,
the kernel must check to see if it is part of a snapshot and let
the snapshot claim the block rather than freeing it for future use.
When a snapshot is freed, its blocks need to be offered to older
snapshots and freed only if no older snapshots wish to claim them.

When snapshots were added to UFS/FFS they were integrated into soft
updates and just a small part of the management of snapshots needed
to be added to fsck_ffs(8) as soft updates minimized the set of
snapshot changes that might need correction. When journaling was
added to soft updates a much more complete knowledge of snapshots
needed to be added to fsck_ffs(8) for it to be able to properly
handle the filesystem changes that a journal rollback needs to do
(specifically the freeing and allocation of blocks). Since this
functionality was unavailable, the use of snapshots was disabled
when running with journaled soft updates.

This set of changes imports the kernel code for the management of
snapshots to fsck_ffs(8). With this code in place it will become
possible to enable snapshots when running with journalled soft
updates. The most immediate benefit will be the ability to use
snapshots to take consistent filesystem dumps on live filesystems.
Future work will be done to update fsck_ffs(8) to be able to use
snapshots to run in background on live filesystems running with
journaled soft updates.

Reviewed by: kib
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36491

show more ...

Fix printf formating.

Fix for f4fc389.

Reported by: Jenkins
Sponsored by: The FreeBSD Foundation

Properly handle the replacement of a partially allocated root directory.

If the root directory exists but has a bad block number Pass1 will
accept it and setup an inoinfo structure for it. When Pass

Properly handle the replacement of a partially allocated root directory.

If the root directory exists but has a bad block number Pass1 will
accept it and setup an inoinfo structure for it. When Pass2 runs
and cannot read the root inode's content because of a bad (or
duplicate) block number, it removes the bad root inode and replaces
it. As part of creating the replacement root inode, it creates an
inoinfo entry for it. But Pass2 did delete the inoinfo entry that
Pass1 had set up for the root inode so ended up with two inoinfo
structures for it. The final step of Pass2 checks that all the ".."
entries are correct adding them if they are missing which resulted
in a second ".." entry being added to the root directory which
definitely did not go over well in the kernel name cache!

Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation

show more ...

Correct calculation of inode location in getnextino cache.

Fix for 345bfec.

Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation

Correct calculation of inode location in getnextino cache.

Fix for 345bfec.

Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation

Provide cache coherency between getnextinode() and ginode()

The fsck_ffs(8) utility has two subsystems for reading and writing
inodes. The getnextinode() interface is used in Pass 1 (and Pass
1b if

Provide cache coherency between getnextinode() and ginode()

The fsck_ffs(8) utility has two subsystems for reading and writing
inodes. The getnextinode() interface is used in Pass 1 (and Pass
1b if needed) to sequentially walk through all the inodes in the
filesystem. The ginode() interface is used to read and write
individual inodes. Pass 1 uses a mix of both interfaces. This
change ensures that ginode() returns a pointer to the inode in the
cache maintained by getnextinode() when that interface holds the
requested inode so that all modifications to the inode are made in
a single place and are all written to the disk together.

Reported by: Peter Holm
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation

show more ...

Correctness cleanups in fsck_ffs(8).

Allocation or I/O failures in fsck_ffs(8) could cause segment
faults because of missing checks or not-yet-initialized data
structures. Correct these issues.

Rep

Correctness cleanups in fsck_ffs(8).

Allocation or I/O failures in fsck_ffs(8) could cause segment
faults because of missing checks or not-yet-initialized data
structures. Correct these issues.

Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation

show more ...