1.. SPDX-License-Identifier: GPL-2.0 2 3=================================== 4Cache on Already Mounted Filesystem 5=================================== 6 7.. Contents: 8 9 (*) Overview. 10 11 (*) Requirements. 12 13 (*) Configuration. 14 15 (*) Starting the cache. 16 17 (*) Things to avoid. 18 19 (*) Cache culling. 20 21 (*) Cache structure. 22 23 (*) Security model and SELinux. 24 25 (*) A note on security. 26 27 (*) Statistical information. 28 29 (*) Debugging. 30 31 32Overview 33======== 34 35CacheFiles is a caching backend that's meant to use as a cache a directory on 36an already mounted filesystem of a local type (such as Ext3). 37 38CacheFiles uses a userspace daemon to do some of the cache management - such as 39reaping stale nodes and culling. This is called cachefilesd and lives in 40/sbin. 41 42The filesystem and data integrity of the cache are only as good as those of the 43filesystem providing the backing services. Note that CacheFiles does not 44attempt to journal anything since the journalling interfaces of the various 45filesystems are very specific in nature. 46 47CacheFiles creates a misc character device - "/dev/cachefiles" - that is used 48to communication with the daemon. Only one thing may have this open at once, 49and while it is open, a cache is at least partially in existence. The daemon 50opens this and sends commands down it to control the cache. 51 52CacheFiles is currently limited to a single cache. 53 54CacheFiles attempts to maintain at least a certain percentage of free space on 55the filesystem, shrinking the cache by culling the objects it contains to make 56space if necessary - see the "Cache Culling" section. This means it can be 57placed on the same medium as a live set of data, and will expand to make use of 58spare space and automatically contract when the set of data requires more 59space. 60 61 62 63Requirements 64============ 65 66The use of CacheFiles and its daemon requires the following features to be 67available in the system and in the cache filesystem: 68 69 - dnotify. 70 71 - extended attributes (xattrs). 72 73 - openat() and friends. 74 75 - bmap() support on files in the filesystem (FIBMAP ioctl). 76 77 - The use of bmap() to detect a partial page at the end of the file. 78 79It is strongly recommended that the "dir_index" option is enabled on Ext3 80filesystems being used as a cache. 81 82 83Configuration 84============= 85 86The cache is configured by a script in /etc/cachefilesd.conf. These commands 87set up cache ready for use. The following script commands are available: 88 89 brun <N>%, bcull <N>%, bstop <N>%, frun <N>%, fcull <N>%, fstop <N>% 90 Configure the culling limits. Optional. See the section on culling 91 The defaults are 7% (run), 5% (cull) and 1% (stop) respectively. 92 93 The commands beginning with a 'b' are file space (block) limits, those 94 beginning with an 'f' are file count limits. 95 96 dir <path> 97 Specify the directory containing the root of the cache. Mandatory. 98 99 tag <name> 100 Specify a tag to FS-Cache to use in distinguishing multiple caches. 101 Optional. The default is "CacheFiles". 102 103 debug <mask> 104 Specify a numeric bitmask to control debugging in the kernel module. 105 Optional. The default is zero (all off). The following values can be 106 OR'd into the mask to collect various information: 107 108 == ================================================= 109 1 Turn on trace of function entry (_enter() macros) 110 2 Turn on trace of function exit (_leave() macros) 111 4 Turn on trace of internal debug points (_debug()) 112 == ================================================= 113 114 This mask can also be set through sysfs, eg:: 115 116 echo 5 > /sys/module/cachefiles/parameters/debug 117 118 119Starting the Cache 120================== 121 122The cache is started by running the daemon. The daemon opens the cache device, 123configures the cache and tells it to begin caching. At that point the cache 124binds to fscache and the cache becomes live. 125 126The daemon is run as follows:: 127 128 /sbin/cachefilesd [-d]* [-s] [-n] [-f <configfile>] 129 130The flags are: 131 132 ``-d`` 133 Increase the debugging level. This can be specified multiple times and 134 is cumulative with itself. 135 136 ``-s`` 137 Send messages to stderr instead of syslog. 138 139 ``-n`` 140 Don't daemonise and go into background. 141 142 ``-f <configfile>`` 143 Use an alternative configuration file rather than the default one. 144 145 146Things to Avoid 147=============== 148 149Do not mount other things within the cache as this will cause problems. The 150kernel module contains its own very cut-down path walking facility that ignores 151mountpoints, but the daemon can't avoid them. 152 153Do not create, rename or unlink files and directories in the cache while the 154cache is active, as this may cause the state to become uncertain. 155 156Renaming files in the cache might make objects appear to be other objects (the 157filename is part of the lookup key). 158 159Do not change or remove the extended attributes attached to cache files by the 160cache as this will cause the cache state management to get confused. 161 162Do not create files or directories in the cache, lest the cache get confused or 163serve incorrect data. 164 165Do not chmod files in the cache. The module creates things with minimal 166permissions to prevent random users being able to access them directly. 167 168 169Cache Culling 170============= 171 172The cache may need culling occasionally to make space. This involves 173discarding objects from the cache that have been used less recently than 174anything else. Culling is based on the access time of data objects. Empty 175directories are culled if not in use. 176 177Cache culling is done on the basis of the percentage of blocks and the 178percentage of files available in the underlying filesystem. There are six 179"limits": 180 181 brun, frun 182 If the amount of free space and the number of available files in the cache 183 rises above both these limits, then culling is turned off. 184 185 bcull, fcull 186 If the amount of available space or the number of available files in the 187 cache falls below either of these limits, then culling is started. 188 189 bstop, fstop 190 If the amount of available space or the number of available files in the 191 cache falls below either of these limits, then no further allocation of 192 disk space or files is permitted until culling has raised things above 193 these limits again. 194 195These must be configured thusly:: 196 197 0 <= bstop < bcull < brun < 100 198 0 <= fstop < fcull < frun < 100 199 200Note that these are percentages of available space and available files, and do 201_not_ appear as 100 minus the percentage displayed by the "df" program. 202 203The userspace daemon scans the cache to build up a table of cullable objects. 204These are then culled in least recently used order. A new scan of the cache is 205started as soon as space is made in the table. Objects will be skipped if 206their atimes have changed or if the kernel module says it is still using them. 207 208 209Cache Structure 210=============== 211 212The CacheFiles module will create two directories in the directory it was 213given: 214 215 * cache/ 216 * graveyard/ 217 218The active cache objects all reside in the first directory. The CacheFiles 219kernel module moves any retired or culled objects that it can't simply unlink 220to the graveyard from which the daemon will actually delete them. 221 222The daemon uses dnotify to monitor the graveyard directory, and will delete 223anything that appears therein. 224 225 226The module represents index objects as directories with the filename "I..." or 227"J...". Note that the "cache/" directory is itself a special index. 228 229Data objects are represented as files if they have no children, or directories 230if they do. Their filenames all begin "D..." or "E...". If represented as a 231directory, data objects will have a file in the directory called "data" that 232actually holds the data. 233 234Special objects are similar to data objects, except their filenames begin 235"S..." or "T...". 236 237 238If an object has children, then it will be represented as a directory. 239Immediately in the representative directory are a collection of directories 240named for hash values of the child object keys with an '@' prepended. Into 241this directory, if possible, will be placed the representations of the child 242objects:: 243 244 /INDEX /INDEX /INDEX /DATA FILES 245 /=========/==========/=================================/================ 246 cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400 247 cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...DB1ry 248 cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...N22ry 249 cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...FP1ry 250 251 252If the key is so long that it exceeds NAME_MAX with the decorations added on to 253it, then it will be cut into pieces, the first few of which will be used to 254make a nest of directories, and the last one of which will be the objects 255inside the last directory. The names of the intermediate directories will have 256'+' prepended:: 257 258 J1223/@23/+xy...z/+kl...m/Epqr 259 260 261Note that keys are raw data, and not only may they exceed NAME_MAX in size, 262they may also contain things like '/' and NUL characters, and so they may not 263be suitable for turning directly into a filename. 264 265To handle this, CacheFiles will use a suitably printable filename directly and 266"base-64" encode ones that aren't directly suitable. The two versions of 267object filenames indicate the encoding: 268 269 =============== =============== =============== 270 OBJECT TYPE PRINTABLE ENCODED 271 =============== =============== =============== 272 Index "I..." "J..." 273 Data "D..." "E..." 274 Special "S..." "T..." 275 =============== =============== =============== 276 277Intermediate directories are always "@" or "+" as appropriate. 278 279 280Each object in the cache has an extended attribute label that holds the object 281type ID (required to distinguish special objects) and the auxiliary data from 282the netfs. The latter is used to detect stale objects in the cache and update 283or retire them. 284 285 286Note that CacheFiles will erase from the cache any file it doesn't recognise or 287any file of an incorrect type (such as a FIFO file or a device file). 288 289 290Security Model and SELinux 291========================== 292 293CacheFiles is implemented to deal properly with the LSM security features of 294the Linux kernel and the SELinux facility. 295 296One of the problems that CacheFiles faces is that it is generally acting on 297behalf of a process, and running in that process's context, and that includes a 298security context that is not appropriate for accessing the cache - either 299because the files in the cache are inaccessible to that process, or because if 300the process creates a file in the cache, that file may be inaccessible to other 301processes. 302 303The way CacheFiles works is to temporarily change the security context (fsuid, 304fsgid and actor security label) that the process acts as - without changing the 305security context of the process when it the target of an operation performed by 306some other process (so signalling and suchlike still work correctly). 307 308 309When the CacheFiles module is asked to bind to its cache, it: 310 311 (1) Finds the security label attached to the root cache directory and uses 312 that as the security label with which it will create files. By default, 313 this is:: 314 315 cachefiles_var_t 316 317 (2) Finds the security label of the process which issued the bind request 318 (presumed to be the cachefilesd daemon), which by default will be:: 319 320 cachefilesd_t 321 322 and asks LSM to supply a security ID as which it should act given the 323 daemon's label. By default, this will be:: 324 325 cachefiles_kernel_t 326 327 SELinux transitions the daemon's security ID to the module's security ID 328 based on a rule of this form in the policy:: 329 330 type_transition <daemon's-ID> kernel_t : process <module's-ID>; 331 332 For instance:: 333 334 type_transition cachefilesd_t kernel_t : process cachefiles_kernel_t; 335 336 337The module's security ID gives it permission to create, move and remove files 338and directories in the cache, to find and access directories and files in the 339cache, to set and access extended attributes on cache objects, and to read and 340write files in the cache. 341 342The daemon's security ID gives it only a very restricted set of permissions: it 343may scan directories, stat files and erase files and directories. It may 344not read or write files in the cache, and so it is precluded from accessing the 345data cached therein; nor is it permitted to create new files in the cache. 346 347 348There are policy source files available in: 349 350 https://people.redhat.com/~dhowells/fscache/cachefilesd-0.8.tar.bz2 351 352and later versions. In that tarball, see the files:: 353 354 cachefilesd.te 355 cachefilesd.fc 356 cachefilesd.if 357 358They are built and installed directly by the RPM. 359 360If a non-RPM based system is being used, then copy the above files to their own 361directory and run:: 362 363 make -f /usr/share/selinux/devel/Makefile 364 semodule -i cachefilesd.pp 365 366You will need checkpolicy and selinux-policy-devel installed prior to the 367build. 368 369 370By default, the cache is located in /var/fscache, but if it is desirable that 371it should be elsewhere, than either the above policy files must be altered, or 372an auxiliary policy must be installed to label the alternate location of the 373cache. 374 375For instructions on how to add an auxiliary policy to enable the cache to be 376located elsewhere when SELinux is in enforcing mode, please see:: 377 378 /usr/share/doc/cachefilesd-*/move-cache.txt 379 380When the cachefilesd rpm is installed; alternatively, the document can be found 381in the sources. 382 383 384A Note on Security 385================== 386 387CacheFiles makes use of the split security in the task_struct. It allocates 388its own task_security structure, and redirects current->cred to point to it 389when it acts on behalf of another process, in that process's context. 390 391The reason it does this is that it calls vfs_mkdir() and suchlike rather than 392bypassing security and calling inode ops directly. Therefore the VFS and LSM 393may deny the CacheFiles access to the cache data because under some 394circumstances the caching code is running in the security context of whatever 395process issued the original syscall on the netfs. 396 397Furthermore, should CacheFiles create a file or directory, the security 398parameters with that object is created (UID, GID, security label) would be 399derived from that process that issued the system call, thus potentially 400preventing other processes from accessing the cache - including CacheFiles's 401cache management daemon (cachefilesd). 402 403What is required is to temporarily override the security of the process that 404issued the system call. We can't, however, just do an in-place change of the 405security data as that affects the process as an object, not just as a subject. 406This means it may lose signals or ptrace events for example, and affects what 407the process looks like in /proc. 408 409So CacheFiles makes use of a logical split in the security between the 410objective security (task->real_cred) and the subjective security (task->cred). 411The objective security holds the intrinsic security properties of a process and 412is never overridden. This is what appears in /proc, and is what is used when a 413process is the target of an operation by some other process (SIGKILL for 414example). 415 416The subjective security holds the active security properties of a process, and 417may be overridden. This is not seen externally, and is used when a process 418acts upon another object, for example SIGKILLing another process or opening a 419file. 420 421LSM hooks exist that allow SELinux (or Smack or whatever) to reject a request 422for CacheFiles to run in a context of a specific security label, or to create 423files and directories with another security label. 424 425 426Statistical Information 427======================= 428 429If FS-Cache is compiled with the following option enabled:: 430 431 CONFIG_CACHEFILES_HISTOGRAM=y 432 433then it will gather certain statistics and display them through a proc file. 434 435 /proc/fs/cachefiles/histogram 436 437 :: 438 439 cat /proc/fs/cachefiles/histogram 440 JIFS SECS LOOKUPS MKDIRS CREATES 441 ===== ===== ========= ========= ========= 442 443 This shows the breakdown of the number of times each amount of time 444 between 0 jiffies and HZ-1 jiffies a variety of tasks took to run. The 445 columns are as follows: 446 447 ======= ======================================================= 448 COLUMN TIME MEASUREMENT 449 ======= ======================================================= 450 LOOKUPS Length of time to perform a lookup on the backing fs 451 MKDIRS Length of time to perform a mkdir on the backing fs 452 CREATES Length of time to perform a create on the backing fs 453 ======= ======================================================= 454 455 Each row shows the number of events that took a particular range of times. 456 Each step is 1 jiffy in size. The JIFS column indicates the particular 457 jiffy range covered, and the SECS field the equivalent number of seconds. 458 459 460Debugging 461========= 462 463If CONFIG_CACHEFILES_DEBUG is enabled, the CacheFiles facility can have runtime 464debugging enabled by adjusting the value in:: 465 466 /sys/module/cachefiles/parameters/debug 467 468This is a bitmask of debugging streams to enable: 469 470 ======= ======= =============================== ======================= 471 BIT VALUE STREAM POINT 472 ======= ======= =============================== ======================= 473 0 1 General Function entry trace 474 1 2 Function exit trace 475 2 4 General 476 ======= ======= =============================== ======================= 477 478The appropriate set of values should be OR'd together and the result written to 479the control file. For example:: 480 481 echo $((1|4|8)) >/sys/module/cachefiles/parameters/debug 482 483will turn on all function entry debugging. 484