zfs: merge openzfs/zfs@3ee08abd2

Notable upstream pull request merges:
#17000 -multiple cmd/zfs: support creating parents with canmount=no
#17926 b481a8bbb Make zpool status dedup table support ra

zfs: merge openzfs/zfs@3ee08abd2

Notable upstream pull request merges:
#17000 -multiple cmd/zfs: support creating parents with canmount=no
#17926 b481a8bbb Make zpool status dedup table support raw bytes -p output
#18238 -multiple Report space metrics per allocation class
#18240 5dad9459d Add --no-preserve-encryption flag
#18257 1eace5906 libzfs: use mount_setattr for selective remount including
legacy mounts
#18259 1e2c94a04 More consistent use of TREE_* macros in AVL comparators
#18262 c58b8b7dc Fix check for .cfi_negate_ra_state on aarch64
#18264 f109c7bb9 Add the --file-layout (-f) option to zdb(8)
#18268 304de7f19 libzfs: handle EDOM error in zpool_create
#18275 753f1e1e2 zstream: add a drop_record subcommand
#18276 0f90a797d Fix vdev_rebuild_range() tx commit
#18277 1e1d64d66 Fix log vdev removal issues
#18278 06b0abfe6 Fix the send --exclude option to work with encryption
#18281 e73ada771 libzpool: lift zfs_file ops out to separate source file
#18284 d97945776 zstream: consolidate shared code
#18285 -multiple Separate SIMD compile checks for userspace and kernel
#18288 c5905b2cb Implement lzc_send_progress
#18289 -multiple L2ARC: Add depth cap and write budget fairness for
persistent markers
#18290 d45c8d648 FreeBSD: Improve dmesg kernel message prefix
#18294 b403040c4 draid: fix data corruption after disk clear
#18296 -multiple libzfs: mnttab cleanup
#18300 d35951b18 zpool clear: remove undocumented rewind flags
#18301 ae7fcd5f9 fix libzfs diff mem leak in an error path
#18303 65165df12 zilstat: add man page
#18314 7f65e04ab libzfs: scrub: only include start and end nv pairs if
needed for scrub
#18315 -multiple Sync ZVOLs block cloning conditions with file systems
#18330 f80338fac zarcsummary: add man page
#18337 f259a47c7 zpool-iostat.8: clarify first report shows per-second
averages

Obtained from: OpenZFS
OpenZFS commit: 3ee08abd2fcef3123188e9ad61a9cd80b9b7831c

show more ...

zfs: merge openzfs/zfs@f8e5af53e

Notable upstream pull request merges:
#17358 4975430cf Add vdev property to disable vdev scheduler
#18031 c77f17b75 Add snapshots_changed_nsecs dataset property
#

zfs: merge openzfs/zfs@f8e5af53e

Notable upstream pull request merges:
#17358 4975430cf Add vdev property to disable vdev scheduler
#18031 c77f17b75 Add snapshots_changed_nsecs dataset property
#18080 dbb3f247e cmd/zfs: clone: accept `-u` to not mount newly created
datasets
#18089 -multiple Zstd: Update bundled library to version 1.5.7
#18091 2301755df Fix zfs_open() to skip zil_async_to_sync() for the
snapshot
#18093 -multiple L2ARC: Rework write throttling with DWPD rate limiting
and parallel writes
#18095 2dbd6af5e Rename several printf attributes declarations to
__printf__
#18096 8605bdfdd FreeBSD: unbreak compilation on i386
#18105 794f1587d When receiving a stream with the large block flag,
activate feature
#18115 765929cb4 DDT: Add locking for table ZAP destruction
#18118 09e4e01e9 Fix history logging for `zpool create -t`
#18119 2f1f25217 icp: emit .note.GNU-stack section for all ELF targets
#18131 3fffe4e70 Fix --enable-invariants on FreeBSD
#18133 d2f5cb3a5 Move range_tree, btree, highbit64 to common code
#18136 54b141fab FreeBSD: Remove references to DEBUG_VFS_LOCKS
#18138 cdf89f413 Flush RRD only when TXGs contain data
#18139 a157ef62a Make sure we can still write data to txg
#18140 cd895f0e5 remove thread unsafe debug code causing FreeBSD double
free panic
#18144 4f180e095 Fix activating large_microzap on receive
#18146 35b2d3970 Lock db_mtx around arc_release() in couple places
#18154 b36472052 nvpair: chase FreeBSD xdrproc_t definition
#18160 21bbe7cb6 Improve caching for dbuf prefetches
#18177 -multiple Multihost Improvements
#18179 2646bd558 Allow rewrite skip cloned and snapshotted blocks
#18180 aa29455dd Restrict cloning with different properties
#18184 040ba7a7c libzfs: improve error message for zpool create with
ENXIO
#18188 1412bdc6c zfs_vnops_os.c: Move a vput() to after
zfs_setattr_dir()
#18198 cc184fe98 Fix `send:raw` permission for send `-w -I`
#18208 ba970eb20 Cleanup allocation class selection
#18212 0f9564e85 Simplify dnode_level_is_l2cacheable()
#18214 370570890 Remove parent ZIO from dbuf_prefetch()
#18218 bfb276e55 freebsd: Fix TIMESPEC_OVERFLOW for PowerPC
#18222 d06a1d9ac Fix available space accounting for special/dedup
#18225 d48967728 ICP: AES-GCM VAES-AVX2: fix typos and document
source files
#18226 c8a72a27e ICP: AES-GCM assembly: remove unused Gmul functions
#18230 -multiple Fix zdb --key crash for unencrypted datasets, and
teach tests to understand this better
#18233 -multiple icp: add SHA-512 implementation using Intel SHA512
extension
#18245 991fc56fa Introduce dedupused/dedupsaved pool properties
#18251 6a717f31e Improve misleading error messages for
ZPOOL_STATUS_CORRUPT_POOL
#18254 7744f0496 SIMD: libspl: test the correct CPUID bit for AVX512VL
#18255 6495dafd5 range_tree: use zfs_panic_recover() for
partial-overlap remov
#18256 3408332d7 zhack: Fix importing large allocation profiles on
small pools
#18258 f8457fbdc Fix deadlock on dmu_tx_assign() from vdev_rebuild()
#18263 f8e5af53e Fix redundant declaration of dsl_pool_t

Obtained from: OpenZFS
OpenZFS commit: f8e5af53e92fa7c03393fbd4922cb9c1d0c15920

show more ...

zfs: world changes after 89f729dcc merge

Remove, unbind and obsolete libuutil and libtpool
Update zfs_configh and zfs_gitrev.h

Use ZFSTOP more broadly

Reviewed by: brooks
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D53791

packages: Use LIB_PACKAGE for zfs and ufs packages

Commit f9513c334ff7 moved the shared libraries for ZFS and UFS into
separate packages (libzfs and libufs), which resulted in a rather
large number

packages: Use LIB_PACKAGE for zfs and ufs packages

Commit f9513c334ff7 moved the shared libraries for ZFS and UFS into
separate packages (libzfs and libufs), which resulted in a rather
large number of packages being created, e.g. for ZFS:

FreeBSD-libzfs
FreeBSD-libzfs-dbg
FreeBSD-libzfs-dbg-lib32
FreeBSD-libzfs-dev
FreeBSD-libzfs-dev-lib32
FreeBSD-libzfs-lib32
FreeBSD-zfs
FreeBSD-zfs-dbg
FreeBSD-zfs-dbg-lib32
FreeBSD-zfs-dev
FreeBSD-zfs-dev-lib32
FreeBSD-zfs-lib32
FreeBSD-zfs-man

Use LIB_PACKAGE instead, which significantly reduces the number of
packages:

FreeBSD-zfs
FreeBSD-zfs-dbg
FreeBSD-zfs-dbg-lib32
FreeBSD-zfs-dev
FreeBSD-zfs-dev-lib32
FreeBSD-zfs-lib
FreeBSD-zfs-lib32
FreeBSD-zfs-man

MFC after: 3 days
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D52416

show more ...

add packages for libufs, libzfs

currently FreeBSD-utilities depends on both FreeBSD-ufs and FreeBSD-zfs.
this is not desirable, because those are both relatively large packages
and the user may want

add packages for libufs, libzfs

currently FreeBSD-utilities depends on both FreeBSD-ufs and FreeBSD-zfs.
this is not desirable, because those are both relatively large packages
and the user may want to remove one or the other – or perhaps both, e.g.
in a jail or embedded system.

the reason for this dependency is that fstyp(8), which is in
FreeBSD-utilities, links both libufs and libzfs. FreeBSD-utilities is
the correct place for fstyp, so we don't want to move that.

instead, add two new packages: libufs contains libufs, and libzfs
contains libzfs plus the ZFS libraries it depends on: libavl, libnvpair,
libspl, libtpool, libumem, libuutil, libzfs_core and libzutil.

with this change, it is possible to remove FreeBSD-ufs and/or
FreeBSD-zfs while leaving FreeBSD-libufs, FreeBSD-libzfs and
FreeBSD-utilities installed.

Reviewed by: manu, des
Approved by: des (mentor)
Differential Revision: https://reviews.freebsd.org/D50148

show more ...

zfs: merge openzfs/zfs@b10992582

Notable upstream pull request merges:
#15892 -multiple Fast Dedup: Introduce the FDT on-disk format and feature flag
#15893 -multiple Fast Dedup: “flat” DDT entry

zfs: merge openzfs/zfs@b10992582

Notable upstream pull request merges:
#15892 -multiple Fast Dedup: Introduce the FDT on-disk format and feature flag
#15893 -multiple Fast Dedup: “flat” DDT entry format
#15895 -multiple Fast Dedup: FDT-log feature
#16239 6be8bf555 zpool: Provide GUID to zpool-reguid(8) with -g
#16277 -multiple Fast Dedup: prune unique entries
#16316 5807de90a Fix null ptr deref when renaming a zvol with snaps and snapdev=visible
#16343 77a797a38 Enable L2 cache of all (MRU+MFU) metadata but MFU data only
#16446 83f359245 FreeBSD: fix build without kernel option MAC
#16449 963e6c9f3 Fix incorrect error report on vdev attach/replace
#16505 b10992582 spa_prop_get: require caller to supply output nvlist

Obtained from: OpenZFS
OpenZFS commit: b109925820fb79db3e37670c159977f03edd950f

show more ...

Remove residual blank line at start of Makefile

This is a residual of the $FreeBSD$ removal.

MFC After: 3 days (though I'll just run the command on the branches)
Sponsored by: Netflix

libc: expose execvpe for Linux compat

We already implemented execvpe internally with an _ prefix in libc so
go ahead and expose it for compatibility with Linux.

This reverts c605eea952146348e5e1ad5

libc: expose execvpe for Linux compat

We already implemented execvpe internally with an _ prefix in libc so
go ahead and expose it for compatibility with Linux.

This reverts c605eea952146348e5e1ad5cab6c127d7a1bd164.

Bump __FreeBSD_version for the addition and add definitions to supress
compat shims in libzfs (zfs changes were merged from upstream).

PR: 275370 (request and exp-run (thanks antoine!))
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D42846

show more ...

zfs: merge openzfs/zfs@2e2a46e0a

Notable upstream pull request merges:
#15243 b53077a9e Add zfs_prepare_disk script for disk firmware install
#15298 2e2a46e0a Invoke zdb by guid to avoid import er

zfs: merge openzfs/zfs@2e2a46e0a

Notable upstream pull request merges:
#15243 b53077a9e Add zfs_prepare_disk script for disk firmware install
#15298 2e2a46e0a Invoke zdb by guid to avoid import errors
#15301 e5d70f467 ZIL: Avoid dbuf_read() in ztest_get_data()

Obtained from: OpenZFS
OpenZFS commit: 2e2a46e0a597b3ee606ea7dc5bc527459077322f

show more ...

zfs: merge openzfs/zfs@804414aad

Notable upstream pull request merges:
#15024 Add missed DMU_PROJECTUSED_OBJECT prefetch
#15029 Do not request data L1 buffers on scan prefetch
#15036 FreeBSD:

zfs: merge openzfs/zfs@804414aad

Notable upstream pull request merges:
#15024 Add missed DMU_PROJECTUSED_OBJECT prefetch
#15029 Do not request data L1 buffers on scan prefetch
#15036 FreeBSD: catch up to __FreeBSD_version 1400093
#15039 Fix raw receive with different indirect block size
#15047 FreeBSD: Fix build on stable/13 after 1302506
#15049 Fix the ZFS checksum error histograms with larger record sizes
#15052 Reduce bloat in ereport.fs.zfs.checksum events
#15056 Avoid extra snprintf() in dsl_deadlist_merge()
#15061 Ignore pool ashift property during vdev attachment
#15063 Don't panic if setting vdev properties is unsupported for this vdev type
#15067 spa_min_alloc should be GCD, not min
#15071 Add explicit prefetches to bpobj_iterate()
#15072 Adjust prefetch parameters
#15076 Refactor dmu_prefetch()
#15079 set autotrim default to 'off' everywhere
#15080 ZIL: Fix config lock deadlock
#15088 metaslab: tuneable to better control force ganging
#15096 Avoid waiting in dmu_sync_late_arrival()
#15097 BRT should return EOPNOTSUPP
#15103 Remove zl_issuer_lock from zil_suspend()
#15107 Remove fastwrite mechanism
#15113 libzfs: sendrecv: send_progress_thread: handle SIGINFO/SIGUSR1
#15122 ZIL: Second attempt to reduce scope of zl_issuer_lock
#15129 zpool_vdev_remove() should handle EALREADY error return
#15132 ZIL: Replay blocks without next block pointer
#15148 zfs_clone_range should return descriptive error codes
#15153 ZIL: Avoid dbuf_read() before dmu_sync()
#15172 copy_file_range: fix fallback when source create on same txg
#15180 Update outdated assertion from zio_write_compress

Obtained from: OpenZFS
OpenZFS commit: 804414aad224b432590afe3f9ec114ffb49e0f13

show more ...

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

zfs: merge openzfs/zfs@431083f75

Notable upstream pull request merges:
#12194 Fix short-lived txg caused by autotrim
#13368 ZFS_IOC_COUNT_FILLED does unnecessary txg_wait_synced()
#13392 Imple

zfs: merge openzfs/zfs@431083f75

Notable upstream pull request merges:
#12194 Fix short-lived txg caused by autotrim
#13368 ZFS_IOC_COUNT_FILLED does unnecessary txg_wait_synced()
#13392 Implementation of block cloning for ZFS
#13741 SHA2 reworking and API for iterating over multiple implementations
#14282 Sync thread should avoid holding the spa config write lock
when possible
#14283 txg_sync should handle write errors in ZIL
#14359 More adaptive ARC eviction
#14469 Fix NULL pointer dereference in zio_ready()
#14479 zfs redact fails when dnodesize=auto
#14496 improve error message of zfs redact
#14500 Skip memory allocation when compressing holes
#14501 FreeBSD: don't verify recycled vnode for zfs control directory
#14502 partially revert PR 14304 (eee9362a7)
#14509 Fix per-jail zfs.mount_snapshot setting
#14514 Fix data race between zil_commit() and zil_suspend()
#14516 System-wide speculative prefetch limit
#14517 Use rw_tryupgrade() in dmu_bonus_hold_by_dnode()
#14519 Do not hold spa_config in ZIL while blocked on IO
#14523 Move dmu_buf_rele() after dsl_dataset_sync_done()
#14524 Ignore too large stack in case of dsl_deadlist_merge
#14526 Use .section .rodata instead of .rodata on FreeBSD
#14528 ICP: AES-GCM: Refactor gcm_clear_ctx()
#14529 ICP: AES-GCM: Unify gcm_init_ctx() and gmac_init_ctx()
#14532 Handle unexpected errors in zil_lwb_commit() without ASSERT()
#14544 icp: Prevent compilers from optimizing away memset()
in gcm_clear_ctx()
#14546 Revert zfeature_active() to static
#14556 Remove bad kmem_free() oversight from previous zfsdev_state_list
patch
#14563 Optimize the is_l2cacheable functions
#14565 FreeBSD: zfs_znode_alloc: lock the vnode earlier
#14566 FreeBSD: fix false assert in cache_vop_rmdir when replaying ZIL
#14567 spl: Add cmn_err_once() to log a message only on the first call
#14568 Fix incremental receive silently failing for recursive sends
#14569 Restore ASMABI and other Unify work
#14576 Fix detection of IBM Power8 machines (ISA 2.07)
#14577 Better handling for future crypto parameters
#14600 zcommon: Refactor FPU state handling in fletcher4
#14603 Fix prefetching of indirect blocks while destroying
#14633 Fixes in persistent error log
#14639 FreeBSD: Remove extra arc_reduce_target_size() call
#14641 Additional limits on hole reporting
#14649 Drop lying to the compiler in the fletcher4 code
#14652 panic loop when removing slog device
#14653 Update vdev state for spare vdev
#14655 Fix cloning into already dirty dbufs
#14678 Revert "Do not hold spa_config in ZIL while blocked on IO"

Obtained from: OpenZFS
OpenZFS commit: 431083f75bdd3efaee992bdd672625ec7240d252

show more ...

pkgbase: Put zfs utilities and lib in their own package

It is useful to have zfs utilities and lib in a separate package as
it allow users to create image that can support ZFS (i.e. not with
WITHOUT

pkgbase: Put zfs utilities and lib in their own package

It is useful to have zfs utilities and lib in a separate package as
it allow users to create image that can support ZFS (i.e. not with
WITHOUT_ZFS in src.conf set) without bloating the default image with
all zfs tools (for example for jails).

Differential Revision: https://reviews.freebsd.org/D36225

show more ...

zfs: merge openzfs/zfs@ec64fdb93 (master) into main

Notable upstream pull request merges:
#12392 Avoid panic in case of pool errors and missing L2ARC
#12448 skip snapshot in zfs_iter_mounted()

zfs: merge openzfs/zfs@ec64fdb93 (master) into main

Notable upstream pull request merges:
#12392 Avoid panic in case of pool errors and missing L2ARC
#12448 skip snapshot in zfs_iter_mounted()
#12516 Fix NFS and large reads on older kernels
#12533 Fail invalid incremental recursive send gracefully
#12569 FreeBSD: Really zero the zero page
#12575 Reject zfs send -RI with nonexistent fromsnap
#12602 Correct refcount_add in dmu_zfetch
#12650 zpool should call zfs_nicestrtonum() with non-NULL handle

Obtained from: OpenZFS
OpenZFS commit: ec64fdb93d144ab1884097cfd36e18b62a2db848

show more ...

zfs: unbreak build broken in 160388166

zfs: bump version and install new share files

- bump version to 2.0.0-FreeBSD_gbf156c966
- install definition files for the new "-o compatibility" option
to "zpool create"

MFC after: 2 weeks

zfs: merge OpenZFS master-436ab35a5

- speed up writing to ZFS pools without ZIL devices (aa755b3)
- speed up importing ZFS pools (2d8f72d, a0e0199, cf0977a)
...

MFC after: 2 weeks
Reviewed by: mj

zfs: merge OpenZFS master-436ab35a5

- speed up writing to ZFS pools without ZIL devices (aa755b3)
- speed up importing ZFS pools (2d8f72d, a0e0199, cf0977a)
...

MFC after: 2 weeks
Reviewed by: mjg (partial)
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D28677

show more ...

Use MACHINE_CPUARCH when checking for arm64

Use MACHINE_CPUARCH with arm64 (aarch64) when we build code that could run
on any 64-bit Arm instruction set. This will simplify checks in downstream
cons

Use MACHINE_CPUARCH when checking for arm64

Use MACHINE_CPUARCH with arm64 (aarch64) when we build code that could run
on any 64-bit Arm instruction set. This will simplify checks in downstream
consumers targeting prototype instruction sets.

The only place we check for MACHINE_ARCH == aarch64 is when building the
device tree blobs. As these are targeting current generation ISAs.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D26370

show more ...

MFH

Sponsored by: Rubicon Communications, LLC (netgate.com)

libzfs: Add missing crypto dependency

libzfs_crypto.c uses PKCS5_PBKDF2_HMAC_SHA1 from libcrypto.

Reported by: John Kennedy
Sponsored by: iXsystems, Inc.

Move zstd sources from libzfs to libzpool

zstd is kernel code that was not supposed to be in libzfs.

libzpool provides userland shims for kernel code and is where the
zstd code needs to be included

Move zstd sources from libzfs to libzpool

zstd is kernel code that was not supposed to be in libzfs.

libzpool provides userland shims for kernel code and is where the
zstd code needs to be included.

Reported by: John Kennedy
Discussed with: mmacy
Sponsored by: iXsystems, Inc.

show more ...

Fix libzfs build failures for some

In some environments adding zstd/include to the include path
would cause zstd's stdlib.h to hide the system one.

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would a

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872

show more ...

Merge ^/head r351732 through r352104.