1.. SPDX-License-Identifier: GPL-2.0 2 3.. _fsverity: 4 5======================================================= 6fs-verity: read-only file-based authenticity protection 7======================================================= 8 9Introduction 10============ 11 12fs-verity (``fs/verity/``) is a support layer that filesystems can 13hook into to support transparent integrity and authenticity protection 14of read-only files. Currently, it is supported by the ext4, f2fs, and 15btrfs filesystems. Like fscrypt, not too much filesystem-specific 16code is needed to support fs-verity. 17 18fs-verity is similar to `dm-verity 19<https://www.kernel.org/doc/Documentation/admin-guide/device-mapper/verity.rst>`_ 20but works on files rather than block devices. On regular files on 21filesystems supporting fs-verity, userspace can execute an ioctl that 22causes the filesystem to build a Merkle tree for the file and persist 23it to a filesystem-specific location associated with the file. 24 25After this, the file is made readonly, and all reads from the file are 26automatically verified against the file's Merkle tree. Reads of any 27corrupted data, including mmap reads, will fail. 28 29Userspace can use another ioctl to retrieve the root hash (actually 30the "fs-verity file digest", which is a hash that includes the Merkle 31tree root hash) that fs-verity is enforcing for the file. This ioctl 32executes in constant time, regardless of the file size. 33 34fs-verity is essentially a way to hash a file in constant time, 35subject to the caveat that reads which would violate the hash will 36fail at runtime. 37 38Use cases 39========= 40 41By itself, fs-verity only provides integrity protection, i.e. 42detection of accidental (non-malicious) corruption. 43 44However, because fs-verity makes retrieving the file hash extremely 45efficient, it's primarily meant to be used as a tool to support 46authentication (detection of malicious modifications) or auditing 47(logging file hashes before use). 48 49A standard file hash could be used instead of fs-verity. However, 50this is inefficient if the file is large and only a small portion may 51be accessed. This is often the case for Android application package 52(APK) files, for example. These typically contain many translations, 53classes, and other resources that are infrequently or even never 54accessed on a particular device. It would be slow and wasteful to 55read and hash the entire file before starting the application. 56 57Unlike an ahead-of-time hash, fs-verity also re-verifies data each 58time it's paged in. This ensures that malicious disk firmware can't 59undetectably change the contents of the file at runtime. 60 61fs-verity does not replace or obsolete dm-verity. dm-verity should 62still be used on read-only filesystems. fs-verity is for files that 63must live on a read-write filesystem because they are independently 64updated and potentially user-installed, so dm-verity cannot be used. 65 66fs-verity does not mandate a particular scheme for authenticating its 67file hashes. (Similarly, dm-verity does not mandate a particular 68scheme for authenticating its block device root hashes.) Options for 69authenticating fs-verity file hashes include: 70 71- Trusted userspace code. Often, the userspace code that accesses 72 files can be trusted to authenticate them. Consider e.g. an 73 application that wants to authenticate data files before using them, 74 or an application loader that is part of the operating system (which 75 is already authenticated in a different way, such as by being loaded 76 from a read-only partition that uses dm-verity) and that wants to 77 authenticate applications before loading them. In these cases, this 78 trusted userspace code can authenticate a file's contents by 79 retrieving its fs-verity digest using `FS_IOC_MEASURE_VERITY`_, then 80 verifying a signature of it using any userspace cryptographic 81 library that supports digital signatures. 82 83- Integrity Measurement Architecture (IMA). IMA supports fs-verity 84 file digests as an alternative to its traditional full file digests. 85 "IMA appraisal" enforces that files contain a valid, matching 86 signature in their "security.ima" extended attribute, as controlled 87 by the IMA policy. For more information, see the IMA documentation. 88 89- Integrity Policy Enforcement (IPE). IPE supports enforcing access 90 control decisions based on immutable security properties of files, 91 including those protected by fs-verity's built-in signatures. 92 "IPE policy" specifically allows for the authorization of fs-verity 93 files using properties ``fsverity_digest`` for identifying 94 files by their verity digest, and ``fsverity_signature`` to authorize 95 files with a verified fs-verity's built-in signature. For 96 details on configuring IPE policies and understanding its operational 97 modes, please refer to :doc:`IPE admin guide </admin-guide/LSM/ipe>`. 98 99- Trusted userspace code in combination with `Built-in signature 100 verification`_. This approach should be used only with great care. 101 102User API 103======== 104 105FS_IOC_ENABLE_VERITY 106-------------------- 107 108The FS_IOC_ENABLE_VERITY ioctl enables fs-verity on a file. It takes 109in a pointer to a struct fsverity_enable_arg, defined as 110follows:: 111 112 struct fsverity_enable_arg { 113 __u32 version; 114 __u32 hash_algorithm; 115 __u32 block_size; 116 __u32 salt_size; 117 __u64 salt_ptr; 118 __u32 sig_size; 119 __u32 __reserved1; 120 __u64 sig_ptr; 121 __u64 __reserved2[11]; 122 }; 123 124This structure contains the parameters of the Merkle tree to build for 125the file. It must be initialized as follows: 126 127- ``version`` must be 1. 128- ``hash_algorithm`` must be the identifier for the hash algorithm to 129 use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256. See 130 ``include/uapi/linux/fsverity.h`` for the list of possible values. 131- ``block_size`` is the Merkle tree block size, in bytes. In Linux 132 v6.3 and later, this can be any power of 2 between (inclusively) 133 1024 and the minimum of the system page size and the filesystem 134 block size. In earlier versions, the page size was the only allowed 135 value. 136- ``salt_size`` is the size of the salt in bytes, or 0 if no salt is 137 provided. The salt is a value that is prepended to every hashed 138 block; it can be used to personalize the hashing for a particular 139 file or device. Currently the maximum salt size is 32 bytes. 140- ``salt_ptr`` is the pointer to the salt, or NULL if no salt is 141 provided. 142- ``sig_size`` is the size of the builtin signature in bytes, or 0 if no 143 builtin signature is provided. Currently the builtin signature is 144 (somewhat arbitrarily) limited to 16128 bytes. 145- ``sig_ptr`` is the pointer to the builtin signature, or NULL if no 146 builtin signature is provided. A builtin signature is only needed 147 if the `Built-in signature verification`_ feature is being used. It 148 is not needed for IMA appraisal, and it is not needed if the file 149 signature is being handled entirely in userspace. 150- All reserved fields must be zeroed. 151 152FS_IOC_ENABLE_VERITY causes the filesystem to build a Merkle tree for 153the file and persist it to a filesystem-specific location associated 154with the file, then mark the file as a verity file. This ioctl may 155take a long time to execute on large files, and it is interruptible by 156fatal signals. 157 158FS_IOC_ENABLE_VERITY checks for write access to the inode. However, 159it must be executed on an O_RDONLY file descriptor and no processes 160can have the file open for writing. Attempts to open the file for 161writing while this ioctl is executing will fail with ETXTBSY. (This 162is necessary to guarantee that no writable file descriptors will exist 163after verity is enabled, and to guarantee that the file's contents are 164stable while the Merkle tree is being built over it.) 165 166On success, FS_IOC_ENABLE_VERITY returns 0, and the file becomes a 167verity file. On failure (including the case of interruption by a 168fatal signal), no changes are made to the file. 169 170FS_IOC_ENABLE_VERITY can fail with the following errors: 171 172- ``EACCES``: the process does not have write access to the file 173- ``EBADMSG``: the builtin signature is malformed 174- ``EBUSY``: this ioctl is already running on the file 175- ``EEXIST``: the file already has verity enabled 176- ``EFAULT``: the caller provided inaccessible memory 177- ``EFBIG``: the file is too large to enable verity on 178- ``EINTR``: the operation was interrupted by a fatal signal 179- ``EINVAL``: unsupported version, hash algorithm, or block size; or 180 reserved bits are set; or the file descriptor refers to neither a 181 regular file nor a directory. 182- ``EISDIR``: the file descriptor refers to a directory 183- ``EKEYREJECTED``: the builtin signature doesn't match the file 184- ``EMSGSIZE``: the salt or builtin signature is too long 185- ``ENOKEY``: the ".fs-verity" keyring doesn't contain the certificate 186 needed to verify the builtin signature 187- ``ENOPKG``: fs-verity recognizes the hash algorithm, but it's not 188 available in the kernel as currently configured 189- ``ENOTTY``: this type of filesystem does not implement fs-verity 190- ``EOPNOTSUPP``: the kernel was not configured with fs-verity 191 support; or the filesystem superblock has not had the 'verity' 192 feature enabled on it; or the filesystem does not support fs-verity 193 on this file. (See `Filesystem support`_.) 194- ``EPERM``: the file is append-only; or, a builtin signature is 195 required and one was not provided. 196- ``EROFS``: the filesystem is read-only 197- ``ETXTBSY``: someone has the file open for writing. This can be the 198 caller's file descriptor, another open file descriptor, or the file 199 reference held by a writable memory map. 200 201FS_IOC_MEASURE_VERITY 202--------------------- 203 204The FS_IOC_MEASURE_VERITY ioctl retrieves the digest of a verity file. 205The fs-verity file digest is a cryptographic digest that identifies 206the file contents that are being enforced on reads; it is computed via 207a Merkle tree and is different from a traditional full-file digest. 208 209This ioctl takes in a pointer to a variable-length structure:: 210 211 struct fsverity_digest { 212 __u16 digest_algorithm; 213 __u16 digest_size; /* input/output */ 214 __u8 digest[]; 215 }; 216 217``digest_size`` is an input/output field. On input, it must be 218initialized to the number of bytes allocated for the variable-length 219``digest`` field. 220 221On success, 0 is returned and the kernel fills in the structure as 222follows: 223 224- ``digest_algorithm`` will be the hash algorithm used for the file 225 digest. It will match ``fsverity_enable_arg::hash_algorithm``. 226- ``digest_size`` will be the size of the digest in bytes, e.g. 32 227 for SHA-256. (This can be redundant with ``digest_algorithm``.) 228- ``digest`` will be the actual bytes of the digest. 229 230FS_IOC_MEASURE_VERITY is guaranteed to execute in constant time, 231regardless of the size of the file. 232 233FS_IOC_MEASURE_VERITY can fail with the following errors: 234 235- ``EFAULT``: the caller provided inaccessible memory 236- ``ENODATA``: the file is not a verity file 237- ``ENOTTY``: this type of filesystem does not implement fs-verity 238- ``EOPNOTSUPP``: the kernel was not configured with fs-verity 239 support, or the filesystem superblock has not had the 'verity' 240 feature enabled on it. (See `Filesystem support`_.) 241- ``EOVERFLOW``: the digest is longer than the specified 242 ``digest_size`` bytes. Try providing a larger buffer. 243 244FS_IOC_READ_VERITY_METADATA 245--------------------------- 246 247The FS_IOC_READ_VERITY_METADATA ioctl reads verity metadata from a 248verity file. This ioctl is available since Linux v5.12. 249 250This ioctl is useful for cases where the verity verification should be 251performed somewhere other than the currently running kernel. 252 253One example is a server program that takes a verity file and serves it 254to a client program, such that the client can do its own fs-verity 255compatible verification of the file. This only makes sense if the 256client doesn't trust the server and if the server needs to provide the 257storage for the client. 258 259Another example is copying verity metadata when creating filesystem 260images in userspace (such as with ``mkfs.ext4 -d``). 261 262This is a fairly specialized use case, and most fs-verity users won't 263need this ioctl. 264 265This ioctl takes in a pointer to the following structure:: 266 267 #define FS_VERITY_METADATA_TYPE_MERKLE_TREE 1 268 #define FS_VERITY_METADATA_TYPE_DESCRIPTOR 2 269 #define FS_VERITY_METADATA_TYPE_SIGNATURE 3 270 271 struct fsverity_read_metadata_arg { 272 __u64 metadata_type; 273 __u64 offset; 274 __u64 length; 275 __u64 buf_ptr; 276 __u64 __reserved; 277 }; 278 279``metadata_type`` specifies the type of metadata to read: 280 281- ``FS_VERITY_METADATA_TYPE_MERKLE_TREE`` reads the blocks of the 282 Merkle tree. The blocks are returned in order from the root level 283 to the leaf level. Within each level, the blocks are returned in 284 the same order that their hashes are themselves hashed. 285 See `Merkle tree`_ for more information. 286 287- ``FS_VERITY_METADATA_TYPE_DESCRIPTOR`` reads the fs-verity 288 descriptor. See `fs-verity descriptor`_. 289 290- ``FS_VERITY_METADATA_TYPE_SIGNATURE`` reads the builtin signature 291 which was passed to FS_IOC_ENABLE_VERITY, if any. See `Built-in 292 signature verification`_. 293 294The semantics are similar to those of ``pread()``. ``offset`` 295specifies the offset in bytes into the metadata item to read from, and 296``length`` specifies the maximum number of bytes to read from the 297metadata item. ``buf_ptr`` is the pointer to the buffer to read into, 298cast to a 64-bit integer. ``__reserved`` must be 0. On success, the 299number of bytes read is returned. 0 is returned at the end of the 300metadata item. The returned length may be less than ``length``, for 301example if the ioctl is interrupted. 302 303The metadata returned by FS_IOC_READ_VERITY_METADATA isn't guaranteed 304to be authenticated against the file digest that would be returned by 305`FS_IOC_MEASURE_VERITY`_, as the metadata is expected to be used to 306implement fs-verity compatible verification anyway (though absent a 307malicious disk, the metadata will indeed match). E.g. to implement 308this ioctl, the filesystem is allowed to just read the Merkle tree 309blocks from disk without actually verifying the path to the root node. 310 311FS_IOC_READ_VERITY_METADATA can fail with the following errors: 312 313- ``EFAULT``: the caller provided inaccessible memory 314- ``EINTR``: the ioctl was interrupted before any data was read 315- ``EINVAL``: reserved fields were set, or ``offset + length`` 316 overflowed 317- ``ENODATA``: the file is not a verity file, or 318 FS_VERITY_METADATA_TYPE_SIGNATURE was requested but the file doesn't 319 have a builtin signature 320- ``ENOTTY``: this type of filesystem does not implement fs-verity, or 321 this ioctl is not yet implemented on it 322- ``EOPNOTSUPP``: the kernel was not configured with fs-verity 323 support, or the filesystem superblock has not had the 'verity' 324 feature enabled on it. (See `Filesystem support`_.) 325 326FS_IOC_GETFLAGS 327--------------- 328 329The existing ioctl FS_IOC_GETFLAGS (which isn't specific to fs-verity) 330can also be used to check whether a file has fs-verity enabled or not. 331To do so, check for FS_VERITY_FL (0x00100000) in the returned flags. 332 333The verity flag is not settable via FS_IOC_SETFLAGS. You must use 334FS_IOC_ENABLE_VERITY instead, since parameters must be provided. 335 336statx 337----- 338 339Since Linux v5.5, the statx() system call sets STATX_ATTR_VERITY if 340the file has fs-verity enabled. This can perform better than 341FS_IOC_GETFLAGS and FS_IOC_MEASURE_VERITY because it doesn't require 342opening the file, and opening verity files can be expensive. 343 344FS_IOC_FSGETXATTR 345----------------- 346 347Since Linux v7.0, the FS_IOC_FSGETXATTR ioctl sets FS_XFLAG_VERITY (0x00020000) 348in the returned flags when the file has verity enabled. Note that this attribute 349cannot be set with FS_IOC_FSSETXATTR as enabling verity requires input 350parameters. See FS_IOC_ENABLE_VERITY. 351 352file_getattr 353------------ 354 355Since Linux v7.0, the file_getattr() syscall sets FS_XFLAG_VERITY (0x00020000) 356in the returned flags when the file has verity enabled. Note that this attribute 357cannot be set with file_setattr() as enabling verity requires input parameters. 358See FS_IOC_ENABLE_VERITY. 359 360.. _accessing_verity_files: 361 362Accessing verity files 363====================== 364 365Applications can transparently access a verity file just like a 366non-verity one, with the following exceptions: 367 368- Verity files are readonly. They cannot be opened for writing or 369 truncate()d, even if the file mode bits allow it. Attempts to do 370 one of these things will fail with EPERM. However, changes to 371 metadata such as owner, mode, timestamps, and xattrs are still 372 allowed, since these are not measured by fs-verity. Verity files 373 can also still be renamed, deleted, and linked to. 374 375- Direct I/O is not supported on verity files. Attempts to use direct 376 I/O on such files will fall back to buffered I/O. 377 378- DAX (Direct Access) is not supported on verity files, because this 379 would circumvent the data verification. 380 381- Reads of data that doesn't match the verity Merkle tree will fail 382 with EIO (for read()) or SIGBUS (for mmap() reads). 383 384- If the sysctl "fs.verity.require_signatures" is set to 1 and the 385 file is not signed by a key in the ".fs-verity" keyring, then 386 opening the file will fail. See `Built-in signature verification`_. 387 388Direct access to the Merkle tree is not supported. Therefore, if a 389verity file is copied, or is backed up and restored, then it will lose 390its "verity"-ness. fs-verity is primarily meant for files like 391executables that are managed by a package manager. 392 393File digest computation 394======================= 395 396This section describes how fs-verity hashes the file contents using a 397Merkle tree to produce the digest which cryptographically identifies 398the file contents. This algorithm is the same for all filesystems 399that support fs-verity. 400 401Userspace only needs to be aware of this algorithm if it needs to 402compute fs-verity file digests itself, e.g. in order to sign files. 403 404.. _fsverity_merkle_tree: 405 406Merkle tree 407----------- 408 409The file contents is divided into blocks, where the block size is 410configurable but is usually 4096 bytes. The end of the last block is 411zero-padded if needed. Each block is then hashed, producing the first 412level of hashes. Then, the hashes in this first level are grouped 413into 'blocksize'-byte blocks (zero-padding the ends as needed) and 414these blocks are hashed, producing the second level of hashes. This 415proceeds up the tree until only a single block remains. The hash of 416this block is the "Merkle tree root hash". 417 418If the file fits in one block and is nonempty, then the "Merkle tree 419root hash" is simply the hash of the single data block. If the file 420is empty, then the "Merkle tree root hash" is all zeroes. 421 422The "blocks" here are not necessarily the same as "filesystem blocks". 423 424If a salt was specified, then it's zero-padded to the closest multiple 425of the input size of the hash algorithm's compression function, e.g. 42664 bytes for SHA-256 or 128 bytes for SHA-512. The padded salt is 427prepended to every data or Merkle tree block that is hashed. 428 429The purpose of the block padding is to cause every hash to be taken 430over the same amount of data, which simplifies the implementation and 431keeps open more possibilities for hardware acceleration. The purpose 432of the salt padding is to make the salting "free" when the salted hash 433state is precomputed, then imported for each hash. 434 435Example: in the recommended configuration of SHA-256 and 4K blocks, 436128 hash values fit in each block. Thus, each level of the Merkle 437tree is approximately 128 times smaller than the previous, and for 438large files the Merkle tree's size converges to approximately 1/127 of 439the original file size. However, for small files, the padding is 440significant, making the space overhead proportionally more. 441 442.. _fsverity_descriptor: 443 444fs-verity descriptor 445-------------------- 446 447By itself, the Merkle tree root hash is ambiguous. For example, it 448can't a distinguish a large file from a small second file whose data 449is exactly the top-level hash block of the first file. Ambiguities 450also arise from the convention of padding to the next block boundary. 451 452To solve this problem, the fs-verity file digest is actually computed 453as a hash of the following structure, which contains the Merkle tree 454root hash as well as other fields such as the file size:: 455 456 struct fsverity_descriptor { 457 __u8 version; /* must be 1 */ 458 __u8 hash_algorithm; /* Merkle tree hash algorithm */ 459 __u8 log_blocksize; /* log2 of size of data and tree blocks */ 460 __u8 salt_size; /* size of salt in bytes; 0 if none */ 461 __le32 __reserved_0x04; /* must be 0 */ 462 __le64 data_size; /* size of file the Merkle tree is built over */ 463 __u8 root_hash[64]; /* Merkle tree root hash */ 464 __u8 salt[32]; /* salt prepended to each hashed block */ 465 __u8 __reserved[144]; /* must be 0's */ 466 }; 467 468Built-in signature verification 469=============================== 470 471CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y adds supports for in-kernel 472verification of fs-verity builtin signatures. 473 474**IMPORTANT**! Please take great care before using this feature. 475It is not the only way to do signatures with fs-verity, and the 476alternatives (such as userspace signature verification, and IMA 477appraisal) can be much better. It's also easy to fall into a trap 478of thinking this feature solves more problems than it actually does. 479 480Enabling this option adds the following: 481 4821. At boot time, the kernel creates a keyring named ".fs-verity". The 483 root user can add trusted X.509 certificates to this keyring using 484 the add_key() system call. 485 4862. `FS_IOC_ENABLE_VERITY`_ accepts a pointer to a PKCS#7 formatted 487 detached signature in DER format of the file's fs-verity digest. 488 On success, the ioctl persists the signature alongside the Merkle 489 tree. Then, any time the file is opened, the kernel verifies the 490 file's actual digest against this signature, using the certificates 491 in the ".fs-verity" keyring. This verification happens as long as the 492 file's signature exists, regardless of the state of the sysctl variable 493 "fs.verity.require_signatures" described in the next item. The IPE LSM 494 relies on this behavior to recognize and label fsverity files 495 that contain a verified built-in fsverity signature. 496 4973. A new sysctl "fs.verity.require_signatures" is made available. 498 When set to 1, the kernel requires that all verity files have a 499 correctly signed digest as described in (2). 500 501The data that the signature as described in (2) must be a signature of 502is the fs-verity file digest in the following format:: 503 504 struct fsverity_formatted_digest { 505 char magic[8]; /* must be "FSVerity" */ 506 __le16 digest_algorithm; 507 __le16 digest_size; 508 __u8 digest[]; 509 }; 510 511That's it. It should be emphasized again that fs-verity builtin 512signatures are not the only way to do signatures with fs-verity. See 513`Use cases`_ for an overview of ways in which fs-verity can be used. 514fs-verity builtin signatures have some major limitations that should 515be carefully considered before using them: 516 517- Builtin signature verification does *not* make the kernel enforce 518 that any files actually have fs-verity enabled. Thus, it is not a 519 complete authentication policy. Currently, if it is used, one 520 way to complete the authentication policy is for trusted userspace 521 code to explicitly check whether files have fs-verity enabled with a 522 signature before they are accessed. (With 523 fs.verity.require_signatures=1, just checking whether fs-verity is 524 enabled suffices.) But, in this case the trusted userspace code 525 could just store the signature alongside the file and verify it 526 itself using a cryptographic library, instead of using this feature. 527 528- Another approach is to utilize fs-verity builtin signature 529 verification in conjunction with the IPE LSM, which supports defining 530 a kernel-enforced, system-wide authentication policy that allows only 531 files with a verified fs-verity builtin signature to perform certain 532 operations, such as execution. Note that IPE doesn't require 533 fs.verity.require_signatures=1. 534 Please refer to :doc:`IPE admin guide </admin-guide/LSM/ipe>` for 535 more details. 536 537- A file's builtin signature can only be set at the same time that 538 fs-verity is being enabled on the file. Changing or deleting the 539 builtin signature later requires re-creating the file. 540 541- Builtin signature verification uses the same set of public keys for 542 all fs-verity enabled files on the system. Different keys cannot be 543 trusted for different files; each key is all or nothing. 544 545- The sysctl fs.verity.require_signatures applies system-wide. 546 Setting it to 1 only works when all users of fs-verity on the system 547 agree that it should be set to 1. This limitation can prevent 548 fs-verity from being used in cases where it would be helpful. 549 550- Builtin signature verification can only use signature algorithms 551 that are supported by the kernel. For example, the kernel does not 552 yet support Ed25519, even though this is often the signature 553 algorithm that is recommended for new cryptographic designs. 554 555- fs-verity builtin signatures are in PKCS#7 format, and the public 556 keys are in X.509 format. These formats are commonly used, 557 including by some other kernel features (which is why the fs-verity 558 builtin signatures use them), and are very feature rich. 559 Unfortunately, history has shown that code that parses and handles 560 these formats (which are from the 1990s and are based on ASN.1) 561 often has vulnerabilities as a result of their complexity. This 562 complexity is not inherent to the cryptography itself. 563 564 fs-verity users who do not need advanced features of X.509 and 565 PKCS#7 should strongly consider using simpler formats, such as plain 566 Ed25519 keys and signatures, and verifying signatures in userspace. 567 568 fs-verity users who choose to use X.509 and PKCS#7 anyway should 569 still consider that verifying those signatures in userspace is more 570 flexible (for other reasons mentioned earlier in this document) and 571 eliminates the need to enable CONFIG_FS_VERITY_BUILTIN_SIGNATURES 572 and its associated increase in kernel attack surface. In some cases 573 it can even be necessary, since advanced X.509 and PKCS#7 features 574 do not always work as intended with the kernel. For example, the 575 kernel does not check X.509 certificate validity times. 576 577 Note: IMA appraisal, which supports fs-verity, does not use PKCS#7 578 for its signatures, so it partially avoids the issues discussed 579 here. IMA appraisal does use X.509. 580 581Filesystem support 582================== 583 584fs-verity is supported by several filesystems, described below. The 585CONFIG_FS_VERITY kconfig option must be enabled to use fs-verity on 586any of these filesystems. 587 588``include/linux/fsverity.h`` declares the interface between the 589``fs/verity/`` support layer and filesystems. Briefly, filesystems 590must provide an ``fsverity_operations`` structure that provides 591methods to read and write the verity metadata to a filesystem-specific 592location, including the Merkle tree blocks and 593``fsverity_descriptor``. Filesystems must also call functions in 594``fs/verity/`` at certain times, such as when a file is opened or when 595pages have been read into the pagecache. (See `Verifying data`_.) 596 597ext4 598---- 599 600ext4 supports fs-verity since Linux v5.4 and e2fsprogs v1.45.2. 601 602To create verity files on an ext4 filesystem, the filesystem must have 603been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on 604it. "verity" is an RO_COMPAT filesystem feature, so once set, old 605kernels will only be able to mount the filesystem readonly, and old 606versions of e2fsck will be unable to check the filesystem. 607 608Originally, an ext4 filesystem with the "verity" feature could only be 609mounted when its block size was equal to the system page size 610(typically 4096 bytes). In Linux v6.3, this limitation was removed. 611 612ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files. It 613can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared. 614 615ext4 also supports encryption, which can be used simultaneously with 616fs-verity. In this case, the plaintext data is verified rather than 617the ciphertext. This is necessary in order to make the fs-verity file 618digest meaningful, since every file is encrypted differently. 619 620ext4 stores the verity metadata (Merkle tree and fsverity_descriptor) 621past the end of the file, starting at the first 64K boundary beyond 622i_size. This approach works because (a) verity files are readonly, 623and (b) pages fully beyond i_size aren't visible to userspace but can 624be read/written internally by ext4 with only some relatively small 625changes to ext4. This approach avoids having to depend on the 626EA_INODE feature and on rearchitecturing ext4's xattr support to 627support paging multi-gigabyte xattrs into memory, and to support 628encrypting xattrs. Note that the verity metadata *must* be encrypted 629when the file is, since it contains hashes of the plaintext data. 630 631ext4 only allows verity on extent-based files. 632 633f2fs 634---- 635 636f2fs supports fs-verity since Linux v5.4 and f2fs-tools v1.11.0. 637 638To create verity files on an f2fs filesystem, the filesystem must have 639been formatted with ``-O verity``. 640 641f2fs sets the FADVISE_VERITY_BIT on-disk inode flag on verity files. 642It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be 643cleared. 644 645Like ext4, f2fs stores the verity metadata (Merkle tree and 646fsverity_descriptor) past the end of the file, starting at the first 64764K boundary beyond i_size. See explanation for ext4 above. 648Moreover, f2fs supports at most 4096 bytes of xattr entries per inode 649which usually wouldn't be enough for even a single Merkle tree block. 650 651f2fs doesn't support enabling verity on files that currently have 652atomic or volatile writes pending. 653 654btrfs 655----- 656 657btrfs supports fs-verity since Linux v5.15. Verity-enabled inodes are 658marked with a RO_COMPAT inode flag, and the verity metadata is stored 659in separate btree items. 660 661Implementation details 662====================== 663 664Verifying data 665-------------- 666 667fs-verity ensures that all reads of a verity file's data are verified, 668regardless of which syscall is used to do the read (e.g. mmap(), 669read(), pread()) and regardless of whether it's the first read or a 670later read (unless the later read can return cached data that was 671already verified). Below, we describe how filesystems implement this. 672 673Pagecache 674~~~~~~~~~ 675 676For filesystems using Linux's pagecache, the ``->read_folio()`` and 677``->readahead()`` methods must be modified to verify folios before 678they are marked Uptodate. Merely hooking ``->read_iter()`` would be 679insufficient, since ``->read_iter()`` is not used for memory maps. 680 681Therefore, fs/verity/ provides the function fsverity_verify_blocks() 682which verifies data that has been read into the pagecache of a verity 683inode. The containing folio must still be locked and not Uptodate, so 684it's not yet readable by userspace. As needed to do the verification, 685fsverity_verify_blocks() will call back into the filesystem to read 686hash blocks via fsverity_operations::read_merkle_tree_page(). 687 688fsverity_verify_blocks() returns false if verification failed; in this 689case, the filesystem must not set the folio Uptodate. Following this, 690as per the usual Linux pagecache behavior, attempts by userspace to 691read() from the part of the file containing the folio will fail with 692EIO, and accesses to the folio within a memory map will raise SIGBUS. 693 694In principle, verifying a data block requires verifying the entire 695path in the Merkle tree from the data block to the root hash. 696However, for efficiency the filesystem may cache the hash blocks. 697Therefore, fsverity_verify_blocks() only ascends the tree reading hash 698blocks until an already-verified hash block is seen. It then verifies 699the path to that block. 700 701This optimization, which is also used by dm-verity, results in 702excellent sequential read performance. This is because usually (e.g. 703127 in 128 times for 4K blocks and SHA-256) the hash block from the 704bottom level of the tree will already be cached and checked from 705reading a previous data block. However, random reads perform worse. 706 707Block device based filesystems 708~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 709 710Block device based filesystems (e.g. ext4 and f2fs) in Linux also use 711the pagecache, so the above subsection applies too. However, they 712also usually read many data blocks from a file at once, grouped into a 713structure called a "bio". To make it easier for these types of 714filesystems to support fs-verity, fs/verity/ also provides a function 715fsverity_verify_bio() which verifies all data blocks in a bio. 716 717ext4 and f2fs also support encryption. If a verity file is also 718encrypted, the data must be decrypted before being verified. To 719support this, these filesystems allocate a "post-read context" for 720each bio and store it in ``->bi_private``:: 721 722 struct bio_post_read_ctx { 723 struct bio *bio; 724 struct work_struct work; 725 unsigned int cur_step; 726 unsigned int enabled_steps; 727 }; 728 729``enabled_steps`` is a bitmask that specifies whether decryption, 730verity, or both is enabled. After the bio completes, for each needed 731postprocessing step the filesystem enqueues the bio_post_read_ctx on a 732workqueue, and then the workqueue work does the decryption or 733verification. Finally, folios where no decryption or verity error 734occurred are marked Uptodate, and the folios are unlocked. 735 736On many filesystems, files can contain holes. Normally, 737``->readahead()`` simply zeroes hole blocks and considers the 738corresponding data to be up-to-date; no bios are issued. To prevent 739this case from bypassing fs-verity, filesystems use 740fsverity_verify_blocks() to verify hole blocks. 741 742Filesystems also disable direct I/O on verity files, since otherwise 743direct I/O would bypass fs-verity. 744 745Userspace utility 746================= 747 748This document focuses on the kernel, but a userspace utility for 749fs-verity can be found at: 750 751 https://git.kernel.org/pub/scm/fs/fsverity/fsverity-utils.git 752 753See the README.md file in the fsverity-utils source tree for details, 754including examples of setting up fs-verity protected files. 755 756Tests 757===== 758 759To test fs-verity, use xfstests. For example, using `kvm-xfstests 760<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_:: 761 762 kvm-xfstests -c ext4,f2fs,btrfs -g verity 763 764FAQ 765=== 766 767This section answers frequently asked questions about fs-verity that 768weren't already directly answered in other parts of this document. 769 770:Q: Why isn't fs-verity part of IMA? 771:A: fs-verity and IMA (Integrity Measurement Architecture) have 772 different focuses. fs-verity is a filesystem-level mechanism for 773 hashing individual files using a Merkle tree. In contrast, IMA 774 specifies a system-wide policy that specifies which files are 775 hashed and what to do with those hashes, such as log them, 776 authenticate them, or add them to a measurement list. 777 778 IMA supports the fs-verity hashing mechanism as an alternative 779 to full file hashes, for those who want the performance and 780 security benefits of the Merkle tree based hash. However, it 781 doesn't make sense to force all uses of fs-verity to be through 782 IMA. fs-verity already meets many users' needs even as a 783 standalone filesystem feature, and it's testable like other 784 filesystem features e.g. with xfstests. 785 786:Q: Isn't fs-verity useless because the attacker can just modify the 787 hashes in the Merkle tree, which is stored on-disk? 788:A: To verify the authenticity of an fs-verity file you must verify 789 the authenticity of the "fs-verity file digest", which 790 incorporates the root hash of the Merkle tree. See `Use cases`_. 791 792:Q: Isn't fs-verity useless because the attacker can just replace a 793 verity file with a non-verity one? 794:A: See `Use cases`_. In the initial use case, it's really trusted 795 userspace code that authenticates the files; fs-verity is just a 796 tool to do this job efficiently and securely. The trusted 797 userspace code will consider non-verity files to be inauthentic. 798 799:Q: Why does the Merkle tree need to be stored on-disk? Couldn't you 800 store just the root hash? 801:A: If the Merkle tree wasn't stored on-disk, then you'd have to 802 compute the entire tree when the file is first accessed, even if 803 just one byte is being read. This is a fundamental consequence of 804 how Merkle tree hashing works. To verify a leaf node, you need to 805 verify the whole path to the root hash, including the root node 806 (the thing which the root hash is a hash of). But if the root 807 node isn't stored on-disk, you have to compute it by hashing its 808 children, and so on until you've actually hashed the entire file. 809 810 That defeats most of the point of doing a Merkle tree-based hash, 811 since if you have to hash the whole file ahead of time anyway, 812 then you could simply do sha256(file) instead. That would be much 813 simpler, and a bit faster too. 814 815 It's true that an in-memory Merkle tree could still provide the 816 advantage of verification on every read rather than just on the 817 first read. However, it would be inefficient because every time a 818 hash page gets evicted (you can't pin the entire Merkle tree into 819 memory, since it may be very large), in order to restore it you 820 again need to hash everything below it in the tree. This again 821 defeats most of the point of doing a Merkle tree-based hash, since 822 a single block read could trigger re-hashing gigabytes of data. 823 824:Q: But couldn't you store just the leaf nodes and compute the rest? 825:A: See previous answer; this really just moves up one level, since 826 one could alternatively interpret the data blocks as being the 827 leaf nodes of the Merkle tree. It's true that the tree can be 828 computed much faster if the leaf level is stored rather than just 829 the data, but that's only because each level is less than 1% the 830 size of the level below (assuming the recommended settings of 831 SHA-256 and 4K blocks). For the exact same reason, by storing 832 "just the leaf nodes" you'd already be storing over 99% of the 833 tree, so you might as well simply store the whole tree. 834 835:Q: Can the Merkle tree be built ahead of time, e.g. distributed as 836 part of a package that is installed to many computers? 837:A: This isn't currently supported. It was part of the original 838 design, but was removed to simplify the kernel UAPI and because it 839 wasn't a critical use case. Files are usually installed once and 840 used many times, and cryptographic hashing is somewhat fast on 841 most modern processors. 842 843:Q: Why doesn't fs-verity support writes? 844:A: Write support would be very difficult and would require a 845 completely different design, so it's well outside the scope of 846 fs-verity. Write support would require: 847 848 - A way to maintain consistency between the data and hashes, 849 including all levels of hashes, since corruption after a crash 850 (especially of potentially the entire file!) is unacceptable. 851 The main options for solving this are data journalling, 852 copy-on-write, and log-structured volume. But it's very hard to 853 retrofit existing filesystems with new consistency mechanisms. 854 Data journalling is available on ext4, but is very slow. 855 856 - Rebuilding the Merkle tree after every write, which would be 857 extremely inefficient. Alternatively, a different authenticated 858 dictionary structure such as an "authenticated skiplist" could 859 be used. However, this would be far more complex. 860 861 Compare it to dm-verity vs. dm-integrity. dm-verity is very 862 simple: the kernel just verifies read-only data against a 863 read-only Merkle tree. In contrast, dm-integrity supports writes 864 but is slow, is much more complex, and doesn't actually support 865 full-device authentication since it authenticates each sector 866 independently, i.e. there is no "root hash". It doesn't really 867 make sense for the same device-mapper target to support these two 868 very different cases; the same applies to fs-verity. 869 870:Q: Since verity files are immutable, why isn't the immutable bit set? 871:A: The existing "immutable" bit (FS_IMMUTABLE_FL) already has a 872 specific set of semantics which not only make the file contents 873 read-only, but also prevent the file from being deleted, renamed, 874 linked to, or having its owner or mode changed. These extra 875 properties are unwanted for fs-verity, so reusing the immutable 876 bit isn't appropriate. 877 878:Q: Why does the API use ioctls instead of setxattr() and getxattr()? 879:A: Abusing the xattr interface for basically arbitrary syscalls is 880 heavily frowned upon by most of the Linux filesystem developers. 881 An xattr should really just be an xattr on-disk, not an API to 882 e.g. magically trigger construction of a Merkle tree. 883 884:Q: Does fs-verity support remote filesystems? 885:A: So far all filesystems that have implemented fs-verity support are 886 local filesystems, but in principle any filesystem that can store 887 per-file verity metadata can support fs-verity, regardless of 888 whether it's local or remote. Some filesystems may have fewer 889 options of where to store the verity metadata; one possibility is 890 to store it past the end of the file and "hide" it from userspace 891 by manipulating i_size. The data verification functions provided 892 by ``fs/verity/`` also assume that the filesystem uses the Linux 893 pagecache, but both local and remote filesystems normally do so. 894 895:Q: Why is anything filesystem-specific at all? Shouldn't fs-verity 896 be implemented entirely at the VFS level? 897:A: There are many reasons why this is not possible or would be very 898 difficult, including the following: 899 900 - To prevent bypassing verification, folios must not be marked 901 Uptodate until they've been verified. Currently, each 902 filesystem is responsible for marking folios Uptodate via 903 ``->readahead()``. Therefore, currently it's not possible for 904 the VFS to do the verification on its own. Changing this would 905 require significant changes to the VFS and all filesystems. 906 907 - It would require defining a filesystem-independent way to store 908 the verity metadata. Extended attributes don't work for this 909 because (a) the Merkle tree may be gigabytes, but many 910 filesystems assume that all xattrs fit into a single 4K 911 filesystem block, and (b) ext4 and f2fs encryption doesn't 912 encrypt xattrs, yet the Merkle tree *must* be encrypted when the 913 file contents are, because it stores hashes of the plaintext 914 file contents. 915 916 So the verity metadata would have to be stored in an actual 917 file. Using a separate file would be very ugly, since the 918 metadata is fundamentally part of the file to be protected, and 919 it could cause problems where users could delete the real file 920 but not the metadata file or vice versa. On the other hand, 921 having it be in the same file would break applications unless 922 filesystems' notion of i_size were divorced from the VFS's, 923 which would be complex and require changes to all filesystems. 924 925 - It's desirable that FS_IOC_ENABLE_VERITY uses the filesystem's 926 transaction mechanism so that either the file ends up with 927 verity enabled, or no changes were made. Allowing intermediate 928 states to occur after a crash may cause problems. 929