1.\" $File: file.man,v 1.146 2022/10/26 16:56:14 christos Exp $ 2.Dd October 26, 2022 3.Dt FILE __CSECTION__ 4.Os 5.Sh NAME 6.Nm file 7.Nd determine file type 8.Sh SYNOPSIS 9.Nm 10.Bk -words 11.Op Fl bcdEhiklLNnprsSvzZ0 12.Op Fl Fl apple 13.Op Fl Fl exclude-quiet 14.Op Fl Fl extension 15.Op Fl Fl mime-encoding 16.Op Fl Fl mime-type 17.Op Fl e Ar testname 18.Op Fl F Ar separator 19.Op Fl f Ar namefile 20.Op Fl m Ar magicfiles 21.Op Fl P Ar name=value 22.Ar 23.Ek 24.Nm 25.Fl C 26.Op Fl m Ar magicfiles 27.Nm 28.Op Fl Fl help 29.Sh DESCRIPTION 30This manual page documents version __VERSION__ of the 31.Nm 32command. 33.Pp 34.Nm 35tests each argument in an attempt to classify it. 36There are three sets of tests, performed in this order: 37filesystem tests, magic tests, and language tests. 38The 39.Em first 40test that succeeds causes the file type to be printed. 41.Pp 42The type printed will usually contain one of the words 43.Em text 44(the file contains only 45printing characters and a few common control 46characters and is probably safe to read on an 47.Dv ASCII 48terminal), 49.Em executable 50(the file contains the result of compiling a program 51in a form understandable to some 52.Tn UNIX 53kernel or another), 54or 55.Em data 56meaning anything else (data is usually 57.Dq binary 58or non-printable). 59Exceptions are well-known file formats (core files, tar archives) 60that are known to contain binary data. 61When modifying magic files or the program itself, make sure to 62.Em preserve these keywords . 63Users depend on knowing that all the readable files in a directory 64have the word 65.Dq text 66printed. 67Don't do as Berkeley did and change 68.Dq shell commands text 69to 70.Dq shell script . 71.Pp 72The filesystem tests are based on examining the return from a 73.Xr stat 2 74system call. 75The program checks to see if the file is empty, 76or if it's some sort of special file. 77Any known file types appropriate to the system you are running on 78(sockets, symbolic links, or named pipes (FIFOs) on those systems that 79implement them) 80are intuited if they are defined in the system header file 81.In sys/stat.h . 82.Pp 83The magic tests are used to check for files with data in 84particular fixed formats. 85The canonical example of this is a binary executable (compiled program) 86.Dv a.out 87file, whose format is defined in 88.In elf.h , 89.In a.out.h 90and possibly 91.In exec.h 92in the standard include directory. 93These files have a 94.Dq magic number 95stored in a particular place 96near the beginning of the file that tells the 97.Tn UNIX 98operating system 99that the file is a binary executable, and which of several types thereof. 100The concept of a 101.Dq magic number 102has been applied by extension to data files. 103Any file with some invariant identifier at a small fixed 104offset into the file can usually be described in this way. 105The information identifying these files is read from the compiled 106magic file 107.Pa __MAGIC__.mgc , 108or the files in the directory 109.Pa __MAGIC__ 110if the compiled file does not exist. 111In addition, if 112.Pa $HOME/.magic.mgc 113or 114.Pa $HOME/.magic 115exists, it will be used in preference to the system magic files. 116.Pp 117If a file does not match any of the entries in the magic file, 118it is examined to see if it seems to be a text file. 119ASCII, ISO-8859-x, non-ISO 8-bit extended-ASCII character sets 120(such as those used on Macintosh and IBM PC systems), 121UTF-8-encoded Unicode, UTF-16-encoded Unicode, and EBCDIC 122character sets can be distinguished by the different 123ranges and sequences of bytes that constitute printable text 124in each set. 125If a file passes any of these tests, its character set is reported. 126ASCII, ISO-8859-x, UTF-8, and extended-ASCII files are identified 127as 128.Dq text 129because they will be mostly readable on nearly any terminal; 130UTF-16 and EBCDIC are only 131.Dq character data 132because, while 133they contain text, it is text that will require translation 134before it can be read. 135In addition, 136.Nm 137will attempt to determine other characteristics of text-type files. 138If the lines of a file are terminated by CR, CRLF, or NEL, instead 139of the Unix-standard LF, this will be reported. 140Files that contain embedded escape sequences or overstriking 141will also be identified. 142.Pp 143Once 144.Nm 145has determined the character set used in a text-type file, 146it will 147attempt to determine in what language the file is written. 148The language tests look for particular strings (cf. 149.In names.h ) 150that can appear anywhere in the first few blocks of a file. 151For example, the keyword 152.Em .br 153indicates that the file is most likely a 154.Xr troff 1 155input file, just as the keyword 156.Em struct 157indicates a C program. 158These tests are less reliable than the previous 159two groups, so they are performed last. 160The language test routines also test for some miscellany 161(such as 162.Xr tar 1 163archives, JSON files). 164.Pp 165Any file that cannot be identified as having been written 166in any of the character sets listed above is simply said to be 167.Dq data . 168.Sh OPTIONS 169.Bl -tag -width indent 170.It Fl Fl apple 171Causes the 172.Nm 173command to output the file type and creator code as 174used by older MacOS versions. 175The code consists of eight letters, 176the first describing the file type, the latter the creator. 177This option works properly only for file formats that have the 178apple-style output defined. 179.It Fl b , Fl Fl brief 180Do not prepend filenames to output lines (brief mode). 181.It Fl C , Fl Fl compile 182Write a 183.Pa magic.mgc 184output file that contains a pre-parsed version of the magic file or directory. 185.It Fl c , Fl Fl checking-printout 186Cause a checking printout of the parsed form of the magic file. 187This is usually used in conjunction with the 188.Fl m 189option to debug a new magic file before installing it. 190.It Fl d 191Prints internal debugging information to stderr. 192.It Fl E 193On filesystem errors (file not found etc), instead of handling the error 194as regular output as POSIX mandates and keep going, issue an error message 195and exit. 196.It Fl e , Fl Fl exclude Ar testname 197Exclude the test named in 198.Ar testname 199from the list of tests made to determine the file type. 200Valid test names are: 201.Bl -tag -width compress 202.It apptype 203.Dv EMX 204application type (only on EMX). 205.It ascii 206Various types of text files (this test will try to guess the text 207encoding, irrespective of the setting of the 208.Sq encoding 209option). 210.It encoding 211Different text encodings for soft magic tests. 212.It tokens 213Ignored for backwards compatibility. 214.It cdf 215Prints details of Compound Document Files. 216.It compress 217Checks for, and looks inside, compressed files. 218.It csv 219Checks Comma Separated Value files. 220.It elf 221Prints ELF file details, provided soft magic tests are enabled and the 222elf magic is found. 223.It json 224Examines JSON (RFC-7159) files by parsing them for compliance. 225.It soft 226Consults magic files. 227.It tar 228Examines tar files by verifying the checksum of the 512 byte tar header. 229Excluding this test can provide more detailed content description by using 230the soft magic method. 231.It text 232A synonym for 233.Sq ascii . 234.El 235.It Fl Fl exclude-quiet 236Like 237.Fl Fl exclude 238but ignore tests that 239.Nm 240does not know about. 241This is intended for compatibility with older versions of 242.Nm . 243.It Fl Fl extension 244Print a slash-separated list of valid extensions for the file type found. 245.It Fl F , Fl Fl separator Ar separator 246Use the specified string as the separator between the filename and the 247file result returned. 248Defaults to 249.Sq \&: . 250.It Fl f , Fl Fl files-from Ar namefile 251Read the names of the files to be examined from 252.Ar namefile 253(one per line) 254before the argument list. 255Either 256.Ar namefile 257or at least one filename argument must be present; 258to test the standard input, use 259.Sq - 260as a filename argument. 261Please note that 262.Ar namefile 263is unwrapped and the enclosed filenames are processed when this option is 264encountered and before any further options processing is done. 265This allows one to process multiple lists of files with different command line 266arguments on the same 267.Nm 268invocation. 269Thus if you want to set the delimiter, you need to do it before you specify 270the list of files, like: 271.Dq Fl F Ar @ Fl f Ar namefile , 272instead of: 273.Dq Fl f Ar namefile Fl F Ar @ . 274.It Fl h , Fl Fl no-dereference 275This option causes symlinks not to be followed 276(on systems that support symbolic links). 277This is the default if the environment variable 278.Dv POSIXLY_CORRECT 279is not defined. 280.It Fl i , Fl Fl mime 281Causes the 282.Nm 283command to output mime type strings rather than the more 284traditional human readable ones. 285Thus it may say 286.Sq text/plain; charset=us-ascii 287rather than 288.Dq ASCII text . 289.It Fl Fl mime-type , Fl Fl mime-encoding 290Like 291.Fl i , 292but print only the specified element(s). 293.It Fl k , Fl Fl keep-going 294Don't stop at the first match, keep going. 295Subsequent matches will be 296have the string 297.Sq "\[rs]012\- " 298prepended. 299(If you want a newline, see the 300.Fl r 301option.) 302The magic pattern with the highest strength (see the 303.Fl l 304option) comes first. 305.It Fl l , Fl Fl list 306Shows a list of patterns and their strength sorted descending by 307.Xr magic __FSECTION__ 308strength 309which is used for the matching (see also the 310.Fl k 311option). 312.It Fl L , Fl Fl dereference 313This option causes symlinks to be followed, as the like-named option in 314.Xr ls 1 315(on systems that support symbolic links). 316This is the default if the environment variable 317.Ev POSIXLY_CORRECT 318is defined. 319.It Fl m , Fl Fl magic-file Ar magicfiles 320Specify an alternate list of files and directories containing magic. 321This can be a single item, or a colon-separated list. 322If a compiled magic file is found alongside a file or directory, 323it will be used instead. 324.It Fl N , Fl Fl no-pad 325Don't pad filenames so that they align in the output. 326.It Fl n , Fl Fl no-buffer 327Force stdout to be flushed after checking each file. 328This is only useful if checking a list of files. 329It is intended to be used by programs that want filetype output from a pipe. 330.It Fl p , Fl Fl preserve-date 331On systems that support 332.Xr utime 3 333or 334.Xr utimes 2 , 335attempt to preserve the access time of files analyzed, to pretend that 336.Nm 337never read them. 338.It Fl P , Fl Fl parameter Ar name=value 339Set various parameter limits. 340.Bl -column "elf_phnum" "Default" "XXXXXXXXXXXXXXXXXXXXXXXXXXX" -offset indent 341.It Sy "Name" Ta Sy "Default" Ta Sy "Explanation" 342.It Li bytes Ta 1048576 Ta max number of bytes to read from file 343.It Li elf_notes Ta 256 Ta max ELF notes processed 344.It Li elf_phnum Ta 2048 Ta max ELF program sections processed 345.It Li elf_shnum Ta 32768 Ta max ELF sections processed 346.It Li encoding Ta 65536 Ta max number of bytes to scan for encoding evaluation 347.It Li indir Ta 50 Ta recursion limit for indirect magic 348.It Li name Ta 50 Ta use count limit for name/use magic 349.It Li regex Ta 8192 Ta length limit for regex searches 350.El 351.It Fl r , Fl Fl raw 352Don't translate unprintable characters to \eooo. 353Normally 354.Nm 355translates unprintable characters to their octal representation. 356.It Fl s , Fl Fl special-files 357Normally, 358.Nm 359only attempts to read and determine the type of argument files which 360.Xr stat 2 361reports are ordinary files. 362This prevents problems, because reading special files may have peculiar 363consequences. 364Specifying the 365.Fl s 366option causes 367.Nm 368to also read argument files which are block or character special files. 369This is useful for determining the filesystem types of the data in raw 370disk partitions, which are block special files. 371This option also causes 372.Nm 373to disregard the file size as reported by 374.Xr stat 2 375since on some systems it reports a zero size for raw disk partitions. 376.It Fl S , Fl Fl no-sandbox 377On systems where libseccomp 378.Pa ( https://github.com/seccomp/libseccomp ) 379is available, the 380.Fl S 381option disables sandboxing which is enabled by default. 382This option is needed for 383.Nm 384to execute external decompressing programs, 385i.e. when the 386.Fl z 387option is specified and the built-in decompressors are not available. 388On systems where sandboxing is not available, this option has no effect. 389.It Fl v , Fl Fl version 390Print the version of the program and exit. 391.It Fl z , Fl Fl uncompress 392Try to look inside compressed files. 393.It Fl Z , Fl Fl uncompress-noreport 394Try to look inside compressed files, but report information about the contents 395only not the compression. 396.It Fl 0 , Fl Fl print0 397Output a null character 398.Sq \e0 399after the end of the filename. 400Nice to 401.Xr cut 1 402the output. 403This does not affect the separator, which is still printed. 404.Pp 405If this option is repeated more than once, then 406.Nm 407prints just the filename followed by a NUL followed by the description 408(or ERROR: text) followed by a second NUL for each entry. 409.It Fl -help 410Print a help message and exit. 411.El 412.Sh ENVIRONMENT 413The environment variable 414.Ev MAGIC 415can be used to set the default magic file name. 416If that variable is set, then 417.Nm 418will not attempt to open 419.Pa $HOME/.magic . 420.Nm 421adds 422.Dq Pa .mgc 423to the value of this variable as appropriate. 424The environment variable 425.Ev POSIXLY_CORRECT 426controls (on systems that support symbolic links), whether 427.Nm 428will attempt to follow symlinks or not. 429If set, then 430.Nm 431follows symlink, otherwise it does not. 432This is also controlled by the 433.Fl L 434and 435.Fl h 436options. 437.Sh FILES 438.Bl -tag -width __MAGIC__.mgc -compact 439.It Pa __MAGIC__.mgc 440Default compiled list of magic. 441.It Pa __MAGIC__ 442Directory containing default magic files. 443.El 444.Sh EXIT STATUS 445.Nm 446will exit with 447.Dv 0 448if the operation was successful or 449.Dv >0 450if an error was encountered. 451The following errors cause diagnostic messages, but don't affect the program 452exit code (as POSIX requires), unless 453.Fl E 454is specified: 455.Bl -bullet -compact -offset indent 456.It 457A file cannot be found 458.It 459There is no permission to read a file 460.It 461The file type cannot be determined 462.El 463.Sh EXAMPLES 464.Bd -literal -offset indent 465$ file file.c file /dev/{wd0a,hda} 466file.c: C program text 467file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 468 dynamically linked (uses shared libs), stripped 469/dev/wd0a: block special (0/0) 470/dev/hda: block special (3/0) 471 472$ file -s /dev/wd0{b,d} 473/dev/wd0b: data 474/dev/wd0d: x86 boot sector 475 476$ file -s /dev/hda{,1,2,3,4,5,6,7,8,9,10} 477/dev/hda: x86 boot sector 478/dev/hda1: Linux/i386 ext2 filesystem 479/dev/hda2: x86 boot sector 480/dev/hda3: x86 boot sector, extended partition table 481/dev/hda4: Linux/i386 ext2 filesystem 482/dev/hda5: Linux/i386 swap file 483/dev/hda6: Linux/i386 swap file 484/dev/hda7: Linux/i386 swap file 485/dev/hda8: Linux/i386 swap file 486/dev/hda9: empty 487/dev/hda10: empty 488 489$ file -i file.c file /dev/{wd0a,hda} 490file.c: text/x-c 491file: application/x-executable 492/dev/hda: application/x-not-regular-file 493/dev/wd0a: application/x-not-regular-file 494 495.Ed 496.Sh SEE ALSO 497.Xr hexdump 1 , 498.Xr od 1 , 499.Xr strings 1 , 500.Xr magic __FSECTION__ , 501.Xr fstyp 8 502.Sh STANDARDS CONFORMANCE 503This program is believed to exceed the System V Interface Definition 504of FILE(CMD), as near as one can determine from the vague language 505contained therein. 506Its behavior is mostly compatible with the System V program of the same name. 507This version knows more magic, however, so it will produce 508different (albeit more accurate) output in many cases. 509.\" URL: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html 510.Pp 511The one significant difference 512between this version and System V 513is that this version treats any white space 514as a delimiter, so that spaces in pattern strings must be escaped. 515For example, 516.Bd -literal -offset indent 517\*[Gt]10 string language impress\ (imPRESS data) 518.Ed 519.Pp 520in an existing magic file would have to be changed to 521.Bd -literal -offset indent 522\*[Gt]10 string language\e impress (imPRESS data) 523.Ed 524.Pp 525In addition, in this version, if a pattern string contains a backslash, 526it must be escaped. 527For example 528.Bd -literal -offset indent 5290 string \ebegindata Andrew Toolkit document 530.Ed 531.Pp 532in an existing magic file would have to be changed to 533.Bd -literal -offset indent 5340 string \e\ebegindata Andrew Toolkit document 535.Ed 536.Pp 537SunOS releases 3.2 and later from Sun Microsystems include a 538.Nm 539command derived from the System V one, but with some extensions. 540This version differs from Sun's only in minor ways. 541It includes the extension of the 542.Sq \*[Am] 543operator, used as, 544for example, 545.Bd -literal -offset indent 546\*[Gt]16 long\*[Am]0x7fffffff \*[Gt]0 not stripped 547.Ed 548.Sh SECURITY 549On systems where libseccomp 550.Pa ( https://github.com/seccomp/libseccomp ) 551is available, 552.Nm 553is enforces limiting system calls to only the ones necessary for the 554operation of the program. 555This enforcement does not provide any security benefit when 556.Nm 557is asked to decompress input files running external programs with 558the 559.Fl z 560option. 561To enable execution of external decompressors, one needs to disable 562sandboxing using the 563.Fl S 564option. 565.Sh MAGIC DIRECTORY 566The magic file entries have been collected from various sources, 567mainly USENET, and contributed by various authors. 568Christos Zoulas (address below) will collect additional 569or corrected magic file entries. 570A consolidation of magic file entries 571will be distributed periodically. 572.Pp 573The order of entries in the magic file is significant. 574Depending on what system you are using, the order that 575they are put together may be incorrect. 576If your old 577.Nm 578command uses a magic file, 579keep the old magic file around for comparison purposes 580(rename it to 581.Pa __MAGIC__.orig ) . 582.Sh HISTORY 583There has been a 584.Nm 585command in every 586.Dv UNIX since at least Research Version 4 587(man page dated November, 1973). 588The System V version introduced one significant major change: 589the external list of magic types. 590This slowed the program down slightly but made it a lot more flexible. 591.Pp 592This program, based on the System V version, 593was written by Ian Darwin 594.Aq ian@darwinsys.com 595without looking at anybody else's source code. 596.Pp 597John Gilmore revised the code extensively, making it better than 598the first version. 599Geoff Collyer found several inadequacies 600and provided some magic file entries. 601Contributions of the 602.Sq \*[Am] 603operator by Rob McMahon, 604.Aq cudcv@warwick.ac.uk , 6051989. 606.Pp 607Guy Harris, 608.Aq guy@netapp.com , 609made many changes from 1993 to the present. 610.Pp 611Primary development and maintenance from 1990 to the present by 612Christos Zoulas 613.Aq christos@astron.com . 614.Pp 615Altered by Chris Lowth 616.Aq chris@lowth.com , 6172000: handle the 618.Fl i 619option to output mime type strings, using an alternative 620magic file and internal logic. 621.Pp 622Altered by Eric Fischer 623.Aq enf@pobox.com , 624July, 2000, 625to identify character codes and attempt to identify the languages 626of non-ASCII files. 627.Pp 628Altered by Reuben Thomas 629.Aq rrt@sc3d.org , 6302007-2011, to improve MIME support, merge MIME and non-MIME magic, 631support directories as well as files of magic, apply many bug fixes, 632update and fix a lot of magic, improve the build system, improve the 633documentation, and rewrite the Python bindings in pure Python. 634.Pp 635The list of contributors to the 636.Sq magic 637directory (magic files) 638is too long to include here. 639You know who you are; thank you. 640Many contributors are listed in the source files. 641.Sh LEGAL NOTICE 642Copyright (c) Ian F. Darwin, Toronto, Canada, 1986-1999. 643Covered by the standard Berkeley Software Distribution copyright; see the file 644COPYING in the source distribution. 645.Pp 646The files 647.Pa tar.h 648and 649.Pa is_tar.c 650were written by John Gilmore from his public-domain 651.Xr tar 1 652program, and are not covered by the above license. 653.Sh BUGS 654Please report bugs and send patches to the bug tracker at 655.Pa https://bugs.astron.com/ 656or the mailing list at 657.Aq file@astron.com 658(visit 659.Pa https://mailman.astron.com/mailman/listinfo/file 660first to subscribe). 661.Sh TODO 662Fix output so that tests for MIME and APPLE flags are not needed all 663over the place, and actual output is only done in one place. 664This needs a design. 665Suggestion: push possible outputs on to a list, then pick the 666last-pushed (most specific, one hopes) value at the end, or 667use a default if the list is empty. 668This should not slow down evaluation. 669.Pp 670The handling of 671.Dv MAGIC_CONTINUE 672and printing \e012- between entries is clumsy and complicated; refactor 673and centralize. 674.Pp 675Some of the encoding logic is hard-coded in encoding.c and can be moved 676to the magic files if we had a !:charset annotation. 677.Pp 678Continue to squash all magic bugs. 679See Debian BTS for a good source. 680.Pp 681Store arbitrarily long strings, for example for %s patterns, so that 682they can be printed out. 683Fixes Debian bug #271672. 684This can be done by allocating strings in a string pool, storing the 685string pool at the end of the magic file and converting all the string 686pointers to relative offsets from the string pool. 687.Pp 688Add syntax for relative offsets after current level (Debian bug #466037). 689.Pp 690Make file -ki work, i.e. give multiple MIME types. 691.Pp 692Add a zip library so we can peek inside Office2007 documents to 693print more details about their contents. 694.Pp 695Add an option to print URLs for the sources of the file descriptions. 696.Pp 697Combine script searches and add a way to map executable names to MIME 698types (e.g. have a magic value for !:mime which causes the resulting 699string to be looked up in a table). 700This would avoid adding the same magic repeatedly for each new 701hash-bang interpreter. 702.Pp 703When a file descriptor is available, we can skip and adjust the buffer 704instead of the hacky buffer management we do now. 705.Pp 706Fix 707.Dq name 708and 709.Dq use 710to check for consistency at compile time (duplicate 711.Dq name , 712.Dq use 713pointing to undefined 714.Dq name 715). 716Make 717.Dq name 718/ 719.Dq use 720more efficient by keeping a sorted list of names. 721Special-case ^ to flip endianness in the parser so that it does not 722have to be escaped, and document it. 723.Pp 724If the offsets specified internally in the file exceed the buffer size 725( 726.Dv HOWMANY 727variable in file.h), then we don't seek to that offset, but we give up. 728It would be better if buffer managements was done when the file descriptor 729is available so we can seek around the file. 730One must be careful though because this has performance and thus security 731considerations, because one can slow down things by repeateadly seeking. 732.Pp 733There is support now for keeping separate buffers and having offsets from 734the end of the file, but the internal buffer management still needs an 735overhaul. 736.Sh AVAILABILITY 737You can obtain the original author's latest version by anonymous FTP 738on 739.Pa ftp.astron.com 740in the directory 741.Pa /pub/file/file-X.YZ.tar.gz . 742