1.\" $File: magic.man,v 1.97 2019/11/15 21:03:14 christos Exp $ 2.Dd January 21, 2019 3.Dt MAGIC __FSECTION__ 4.Os 5.\" install as magic.4 on USG, magic.5 on V7, Berkeley and Linux systems. 6.Sh NAME 7.Nm magic 8.Nd file command's magic pattern file 9.Sh DESCRIPTION 10This manual page documents the format of magic files as 11used by the 12.Xr file __CSECTION__ 13command, version __VERSION__. 14The 15.Xr file __CSECTION__ 16command identifies the type of a file using, 17among other tests, 18a test for whether the file contains certain 19.Dq "magic patterns" . 20The database of these 21.Dq "magic patterns" 22is usually located in a binary file in 23.Pa __MAGIC__.mgc 24or a directory of source text magic pattern fragment files in 25.Pa __MAGIC__ . 26The database specifies what patterns are to be tested for, what message or 27MIME type to print if a particular pattern is found, 28and additional information to extract from the file. 29.Pp 30The format of the source fragment files that are used to build this database 31is as follows: 32Each line of a fragment file specifies a test to be performed. 33A test compares the data starting at a particular offset 34in the file with a byte value, a string or a numeric value. 35If the test succeeds, a message is printed. 36The line consists of the following fields: 37.Bl -tag -width ".Dv message" 38.It Dv offset 39A number specifying the offset (in bytes) into the file of the data 40which is to be tested. 41This offset can be a negative number if it is: 42.Bl -bullet -compact 43.It 44The first direct offset of the magic entry (at continuation level 0), 45in which case it is interpreted an offset from end end of the file 46going backwards. 47This works only when a file descriptor to the file is available and it 48is a regular file. 49.It 50A continuation offset relative to the end of the last up-level field 51.Dv ( \*[Am] ) . 52.El 53.It Dv type 54The type of the data to be tested. 55The possible values are: 56.Bl -tag -width ".Dv lestring16" 57.It Dv byte 58A one-byte value. 59.It Dv short 60A two-byte value in this machine's native byte order. 61.It Dv long 62A four-byte value in this machine's native byte order. 63.It Dv quad 64An eight-byte value in this machine's native byte order. 65.It Dv float 66A 32-bit single precision IEEE floating point number in this machine's native byte order. 67.It Dv double 68A 64-bit double precision IEEE floating point number in this machine's native byte order. 69.It Dv string 70A string of bytes. 71The string type specification can be optionally followed 72by /[WwcCtbT]*. 73The 74.Dq W 75flag compacts whitespace in the target, which must 76contain at least one whitespace character. 77If the magic has 78.Dv n 79consecutive blanks, the target needs at least 80.Dv n 81consecutive blanks to match. 82The 83.Dq w 84flag treats every blank in the magic as an optional blank. 85The 86.Dq c 87flag specifies case insensitive matching: lower case 88characters in the magic match both lower and upper case characters in the 89target, whereas upper case characters in the magic only match upper case 90characters in the target. 91The 92.Dq C 93flag specifies case insensitive matching: upper case 94characters in the magic match both lower and upper case characters in the 95target, whereas lower case characters in the magic only match upper case 96characters in the target. 97To do a complete case insensitive match, specify both 98.Dq c 99and 100.Dq C . 101The 102.Dq t 103flag forces the test to be done for text files, while the 104.Dq b 105flag forces the test to be done for binary files. 106The 107.Dq T 108flag causes the string to be trimmed, i.e. leading and trailing whitespace 109is deleted before the string is printed. 110.It Dv pstring 111A Pascal-style string where the first byte/short/int is interpreted as the 112unsigned length. 113The length defaults to byte and can be specified as a modifier. 114The following modifiers are supported: 115.Bl -tag -compact -width B 116.It B 117A byte length (default). 118.It H 119A 2 byte big endian length. 120.It h 121A 2 byte little endian length. 122.It L 123A 4 byte big endian length. 124.It l 125A 4 byte little endian length. 126.It J 127The length includes itself in its count. 128.El 129The string is not NUL terminated. 130.Dq J 131is used rather than the more 132valuable 133.Dq I 134because this type of length is a feature of the JPEG 135format. 136.It Dv date 137A four-byte value interpreted as a UNIX date. 138.It Dv qdate 139An eight-byte value interpreted as a UNIX date. 140.It Dv ldate 141A four-byte value interpreted as a UNIX-style date, but interpreted as 142local time rather than UTC. 143.It Dv qldate 144An eight-byte value interpreted as a UNIX-style date, but interpreted as 145local time rather than UTC. 146.It Dv qwdate 147An eight-byte value interpreted as a Windows-style date. 148.It Dv beid3 149A 32-bit ID3 length in big-endian byte order. 150.It Dv beshort 151A two-byte value in big-endian byte order. 152.It Dv belong 153A four-byte value in big-endian byte order. 154.It Dv bequad 155An eight-byte value in big-endian byte order. 156.It Dv befloat 157A 32-bit single precision IEEE floating point number in big-endian byte order. 158.It Dv bedouble 159A 64-bit double precision IEEE floating point number in big-endian byte order. 160.It Dv bedate 161A four-byte value in big-endian byte order, 162interpreted as a Unix date. 163.It Dv beqdate 164An eight-byte value in big-endian byte order, 165interpreted as a Unix date. 166.It Dv beldate 167A four-byte value in big-endian byte order, 168interpreted as a UNIX-style date, but interpreted as local time rather 169than UTC. 170.It Dv beqldate 171An eight-byte value in big-endian byte order, 172interpreted as a UNIX-style date, but interpreted as local time rather 173than UTC. 174.It Dv beqwdate 175An eight-byte value in big-endian byte order, 176interpreted as a Windows-style date. 177.It Dv bestring16 178A two-byte unicode (UCS16) string in big-endian byte order. 179.It Dv leid3 180A 32-bit ID3 length in little-endian byte order. 181.It Dv leshort 182A two-byte value in little-endian byte order. 183.It Dv lelong 184A four-byte value in little-endian byte order. 185.It Dv lequad 186An eight-byte value in little-endian byte order. 187.It Dv lefloat 188A 32-bit single precision IEEE floating point number in little-endian byte order. 189.It Dv ledouble 190A 64-bit double precision IEEE floating point number in little-endian byte order. 191.It Dv ledate 192A four-byte value in little-endian byte order, 193interpreted as a UNIX date. 194.It Dv leqdate 195An eight-byte value in little-endian byte order, 196interpreted as a UNIX date. 197.It Dv leldate 198A four-byte value in little-endian byte order, 199interpreted as a UNIX-style date, but interpreted as local time rather 200than UTC. 201.It Dv leqldate 202An eight-byte value in little-endian byte order, 203interpreted as a UNIX-style date, but interpreted as local time rather 204than UTC. 205.It Dv leqwdate 206An eight-byte value in little-endian byte order, 207interpreted as a Windows-style date. 208.It Dv lestring16 209A two-byte unicode (UCS16) string in little-endian byte order. 210.It Dv melong 211A four-byte value in middle-endian (PDP-11) byte order. 212.It Dv medate 213A four-byte value in middle-endian (PDP-11) byte order, 214interpreted as a UNIX date. 215.It Dv meldate 216A four-byte value in middle-endian (PDP-11) byte order, 217interpreted as a UNIX-style date, but interpreted as local time rather 218than UTC. 219.It Dv indirect 220Starting at the given offset, consult the magic database again. 221The offset of the 222.Dv indirect 223magic is by default absolute in the file, but one can specify 224.Dv /r 225to indicate that the offset is relative from the beginning of the entry. 226.It Dv name 227Define a 228.Dq named 229magic instance that can be called from another 230.Dv use 231magic entry, like a subroutine call. 232Named instance direct magic offsets are relative to the offset of the 233previous matched entry, but indirect offsets are relative to the beginning 234of the file as usual. 235Named magic entries always match. 236.It Dv use 237Recursively call the named magic starting from the current offset. 238If the name of the referenced begins with a 239.Dv ^ 240then the endianness of the magic is switched; if the magic mentioned 241.Dv leshort 242for example, 243it is treated as 244.Dv beshort 245and vice versa. 246This is useful to avoid duplicating the rules for different endianness. 247.It Dv regex 248A regular expression match in extended POSIX regular expression syntax 249(like egrep). 250Regular expressions can take exponential time to process, and their 251performance is hard to predict, so their use is discouraged. 252When used in production environments, their performance 253should be carefully checked. 254The size of the string to search should also be limited by specifying 255.Dv /<length> , 256to avoid performance issues scanning long files. 257The type specification can also be optionally followed by 258.Dv /[c][s][l] . 259The 260.Dq c 261flag makes the match case insensitive, while the 262.Dq s 263flag update the offset to the start offset of the match, rather than the end. 264The 265.Dq l 266modifier, changes the limit of length to mean number of lines instead of a 267byte count. 268Lines are delimited by the platforms native line delimiter. 269When a line count is specified, an implicit byte count also computed assuming 270each line is 80 characters long. 271If neither a byte or line count is specified, the search is limited automatically 272to 8KiB. 273.Dv ^ 274and 275.Dv $ 276match the beginning and end of individual lines, respectively, 277not beginning and end of file. 278.It Dv search 279A literal string search starting at the given offset. 280The same modifier flags can be used as for string patterns. 281The search expression must contain the range in the form 282.Dv /number, 283that is the number of positions at which the match will be 284attempted, starting from the start offset. 285This is suitable for 286searching larger binary expressions with variable offsets, using 287.Dv \e 288escapes for special characters. 289The order of modifier and number is not relevant. 290.It Dv default 291This is intended to be used with the test 292.Em x 293(which is always true) and it has no type. 294It matches when no other test at that continuation level has matched before. 295Clearing that matched tests for a continuation level, can be done using the 296.Dv clear 297test. 298.It Dv clear 299This test is always true and clears the match flag for that continuation level. 300It is intended to be used with the 301.Dv default 302test. 303.El 304.Pp 305For compatibility with the Single 306.Ux 307Standard, the type specifiers 308.Dv dC 309and 310.Dv d1 311are equivalent to 312.Dv byte , 313the type specifiers 314.Dv uC 315and 316.Dv u1 317are equivalent to 318.Dv ubyte , 319the type specifiers 320.Dv dS 321and 322.Dv d2 323are equivalent to 324.Dv short , 325the type specifiers 326.Dv uS 327and 328.Dv u2 329are equivalent to 330.Dv ushort , 331the type specifiers 332.Dv dI , 333.Dv dL , 334and 335.Dv d4 336are equivalent to 337.Dv long , 338the type specifiers 339.Dv uI , 340.Dv uL , 341and 342.Dv u4 343are equivalent to 344.Dv ulong , 345the type specifier 346.Dv d8 347is equivalent to 348.Dv quad , 349the type specifier 350.Dv u8 351is equivalent to 352.Dv uquad , 353and the type specifier 354.Dv s 355is equivalent to 356.Dv string . 357In addition, the type specifier 358.Dv dQ 359is equivalent to 360.Dv quad 361and the type specifier 362.Dv uQ 363is equivalent to 364.Dv uquad . 365.Pp 366Each top-level magic pattern (see below for an explanation of levels) 367is classified as text or binary according to the types used. 368Types 369.Dq regex 370and 371.Dq search 372are classified as text tests, unless non-printable characters are used 373in the pattern. 374All other tests are classified as binary. 375A top-level 376pattern is considered to be a test text when all its patterns are text 377patterns; otherwise, it is considered to be a binary pattern. 378When 379matching a file, binary patterns are tried first; if no match is 380found, and the file looks like text, then its encoding is determined 381and the text patterns are tried. 382.Pp 383The numeric types may optionally be followed by 384.Dv \*[Am] 385and a numeric value, 386to specify that the value is to be AND'ed with the 387numeric value before any comparisons are done. 388Prepending a 389.Dv u 390to the type indicates that ordered comparisons should be unsigned. 391.It Dv test 392The value to be compared with the value from the file. 393If the type is 394numeric, this value 395is specified in C form; if it is a string, it is specified as a C string 396with the usual escapes permitted (e.g. \en for new-line). 397.Pp 398Numeric values 399may be preceded by a character indicating the operation to be performed. 400It may be 401.Dv = , 402to specify that the value from the file must equal the specified value, 403.Dv \*[Lt] , 404to specify that the value from the file must be less than the specified 405value, 406.Dv \*[Gt] , 407to specify that the value from the file must be greater than the specified 408value, 409.Dv \*[Am] , 410to specify that the value from the file must have set all of the bits 411that are set in the specified value, 412.Dv ^ , 413to specify that the value from the file must have clear any of the bits 414that are set in the specified value, or 415.Dv ~ , 416the value specified after is negated before tested. 417.Dv x , 418to specify that any value will match. 419If the character is omitted, it is assumed to be 420.Dv = . 421Operators 422.Dv \*[Am] , 423.Dv ^ , 424and 425.Dv ~ 426don't work with floats and doubles. 427The operator 428.Dv !\& 429specifies that the line matches if the test does 430.Em not 431succeed. 432.Pp 433Numeric values are specified in C form; e.g. 434.Dv 13 435is decimal, 436.Dv 013 437is octal, and 438.Dv 0x13 439is hexadecimal. 440.Pp 441Numeric operations are not performed on date types, instead the numeric 442value is interpreted as an offset. 443.Pp 444For string values, the string from the 445file must match the specified string. 446The operators 447.Dv = , 448.Dv \*[Lt] 449and 450.Dv \*[Gt] 451(but not 452.Dv \*[Am] ) 453can be applied to strings. 454The length used for matching is that of the string argument 455in the magic file. 456This means that a line can match any non-empty string (usually used to 457then print the string), with 458.Em \*[Gt]\e0 459(because all non-empty strings are greater than the empty string). 460.Pp 461Dates are treated as numerical values in the respective internal 462representation. 463.Pp 464The special test 465.Em x 466always evaluates to true. 467.It Dv message 468The message to be printed if the comparison succeeds. 469If the string contains a 470.Xr printf 3 471format specification, the value from the file (with any specified masking 472performed) is printed using the message as the format string. 473If the string begins with 474.Dq \eb , 475the message printed is the remainder of the string with no whitespace 476added before it: multiple matches are normally separated by a single 477space. 478.El 479.Pp 480An APPLE 4+4 character APPLE creator and type can be specified as: 481.Bd -literal -offset indent 482!:apple CREATYPE 483.Ed 484.Pp 485A MIME type is given on a separate line, which must be the next 486non-blank or comment line after the magic line that identifies the 487file type, and has the following format: 488.Bd -literal -offset indent 489!:mime MIMETYPE 490.Ed 491.Pp 492i.e. the literal string 493.Dq !:mime 494followed by the MIME type. 495.Pp 496An optional strength can be supplied on a separate line which refers to 497the current magic description using the following format: 498.Bd -literal -offset indent 499!:strength OP VALUE 500.Ed 501.Pp 502The operand 503.Dv OP 504can be: 505.Dv + , 506.Dv - , 507.Dv * , 508or 509.Dv / 510and 511.Dv VALUE 512is a constant between 0 and 255. 513This constant is applied using the specified operand 514to the currently computed default magic strength. 515.Pp 516Some file formats contain additional information which is to be printed 517along with the file type or need additional tests to determine the true 518file type. 519These additional tests are introduced by one or more 520.Em \*[Gt] 521characters preceding the offset. 522The number of 523.Em \*[Gt] 524on the line indicates the level of the test; a line with no 525.Em \*[Gt] 526at the beginning is considered to be at level 0. 527Tests are arranged in a tree-like hierarchy: 528if the test on a line at level 529.Em n 530succeeds, all following tests at level 531.Em n+1 532are performed, and the messages printed if the tests succeed, until a line 533with level 534.Em n 535(or less) appears. 536For more complex files, one can use empty messages to get just the 537"if/then" effect, in the following way: 538.Bd -literal -offset indent 5390 string MZ 540\*[Gt]0x18 leshort \*[Lt]0x40 MS-DOS executable 541\*[Gt]0x18 leshort \*[Gt]0x3f extended PC executable (e.g., MS Windows) 542.Ed 543.Pp 544Offsets do not need to be constant, but can also be read from the file 545being examined. 546If the first character following the last 547.Em \*[Gt] 548is a 549.Em \&( 550then the string after the parenthesis is interpreted as an indirect offset. 551That means that the number after the parenthesis is used as an offset in 552the file. 553The value at that offset is read, and is used again as an offset 554in the file. 555Indirect offsets are of the form: 556.Em (( x [[.,][bBcCeEfFgGhHiIlmsSqQ]][+\-][ y ]) . 557The value of 558.Em x 559is used as an offset in the file. 560A byte, id3 length, short or long is read at that offset depending on the 561.Em [bBcCeEfFgGhHiIlmsSqQ] 562type specifier. 563The value is treated as signed if 564.Dq , 565is specified or unsigned if 566.Dq . 567is specified. 568The capitalized types interpret the number as a big endian 569value, whereas the small letter versions interpret the number as a little 570endian value; 571the 572.Em m 573type interprets the number as a middle endian (PDP-11) value. 574To that number the value of 575.Em y 576is added and the result is used as an offset in the file. 577The default type if one is not specified is long. 578The following types are recognized: 579.Bl -column -offset indent "Type" "Half/Short" "Little" "Size" 580.It Sy Type Sy Mnemonic Sy Endian Sy Size 581.It bcBc Byte/Char N/A 1 582.It efg Double Little 8 583.It EFG Double Big 8 584.It hs Half/Short Little 2 585.It HS Half/Short Big 2 586.It i ID3 Little 4 587.It I ID3 Big 4 588.It m Middle Middle 4 589.It q Quad Little 8 590.It Q Quad Big 8 591.El 592.Pp 593That way variable length structures can be examined: 594.Bd -literal -offset indent 595# MS Windows executables are also valid MS-DOS executables 5960 string MZ 597\*[Gt]0x18 leshort \*[Lt]0x40 MZ executable (MS-DOS) 598# skip the whole block below if it is not an extended executable 599\*[Gt]0x18 leshort \*[Gt]0x3f 600\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 601\*[Gt]\*[Gt](0x3c.l) string LX\e0\e0 LX executable (OS/2) 602.Ed 603.Pp 604This strategy of examining has a drawback: you must make sure that you 605eventually print something, or users may get empty output (such as when 606there is neither PE\e0\e0 nor LE\e0\e0 in the above example). 607.Pp 608If this indirect offset cannot be used directly, simple calculations are 609possible: appending 610.Em [+-*/%\*[Am]|^]number 611inside parentheses allows one to modify 612the value read from the file before it is used as an offset: 613.Bd -literal -offset indent 614# MS Windows executables are also valid MS-DOS executables 6150 string MZ 616# sometimes, the value at 0x18 is less that 0x40 but there's still an 617# extended executable, simply appended to the file 618\*[Gt]0x18 leshort \*[Lt]0x40 619\*[Gt]\*[Gt](4.s*512) leshort 0x014c COFF executable (MS-DOS, DJGPP) 620\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 621.Ed 622.Pp 623Sometimes you do not know the exact offset as this depends on the length or 624position (when indirection was used before) of preceding fields. 625You can specify an offset relative to the end of the last up-level 626field using 627.Sq \*[Am] 628as a prefix to the offset: 629.Bd -literal -offset indent 6300 string MZ 631\*[Gt]0x18 leshort \*[Gt]0x3f 632\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 633# immediately following the PE signature is the CPU type 634\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x14c for Intel 80386 635\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x184 for DEC Alpha 636.Ed 637.Pp 638Indirect and relative offsets can be combined: 639.Bd -literal -offset indent 6400 string MZ 641\*[Gt]0x18 leshort \*[Lt]0x40 642\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 643# if it's not COFF, go back 512 bytes and add the offset taken 644# from byte 2/3, which is yet another way of finding the start 645# of the extended executable 646\*[Gt]\*[Gt]\*[Gt]\*[Am](2.s-514) string LE LE executable (MS Windows VxD driver) 647.Ed 648.Pp 649Or the other way around: 650.Bd -literal -offset indent 6510 string MZ 652\*[Gt]0x18 leshort \*[Gt]0x3f 653\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 654# at offset 0x80 (-4, since relative offsets start at the end 655# of the up-level match) inside the LE header, we find the absolute 656# offset to the code area, where we look for a specific signature 657\*[Gt]\*[Gt]\*[Gt](\*[Am]0x7c.l+0x26) string UPX \eb, UPX compressed 658.Ed 659.Pp 660Or even both! 661.Bd -literal -offset indent 6620 string MZ 663\*[Gt]0x18 leshort \*[Gt]0x3f 664\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 665# at offset 0x58 inside the LE header, we find the relative offset 666# to a data area where we look for a specific signature 667\*[Gt]\*[Gt]\*[Gt]\*[Am](\*[Am]0x54.l-3) string UNACE \eb, ACE self-extracting archive 668.Ed 669.Pp 670If you have to deal with offset/length pairs in your file, even the 671second value in a parenthesized expression can be taken from the file itself, 672using another set of parentheses. 673Note that this additional indirect offset is always relative to the 674start of the main indirect offset. 675.Bd -literal -offset indent 6760 string MZ 677\*[Gt]0x18 leshort \*[Gt]0x3f 678\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 679# search for the PE section called ".idata"... 680\*[Gt]\*[Gt]\*[Gt]\*[Am]0xf4 search/0x140 .idata 681# ...and go to the end of it, calculated from start+length; 682# these are located 14 and 10 bytes after the section name 683\*[Gt]\*[Gt]\*[Gt]\*[Gt](\*[Am]0xe.l+(-4)) string PK\e3\e4 \eb, ZIP self-extracting archive 684.Ed 685.Pp 686If you have a list of known values at a particular continuation level, 687and you want to provide a switch-like default case: 688.Bd -literal -offset indent 689# clear that continuation level match 690\*[Gt]18 clear 691\*[Gt]18 lelong 1 one 692\*[Gt]18 lelong 2 two 693\*[Gt]18 default x 694# print default match 695\*[Gt]\*[Gt]18 lelong x unmatched 0x%x 696.Ed 697.Sh SEE ALSO 698.Xr file __CSECTION__ 699\- the command that reads this file. 700.Sh BUGS 701The formats 702.Dv long , 703.Dv belong , 704.Dv lelong , 705.Dv melong , 706.Dv short , 707.Dv beshort , 708and 709.Dv leshort 710do not depend on the length of the C data types 711.Dv short 712and 713.Dv long 714on the platform, even though the Single 715.Ux 716Specification implies that they do. However, as OS X Mountain Lion has 717passed the Single 718.Ux 719Specification validation suite, and supplies a version of 720.Xr file __CSECTION__ 721in which they do not depend on the sizes of the C data types and that is 722built for a 64-bit environment in which 723.Dv long 724is 8 bytes rather than 4 bytes, presumably the validation suite does not 725test whether, for example 726.Dv long 727refers to an item with the same size as the C data type 728.Dv long . 729There should probably be 730.Dv type 731names 732.Dv int8 , 733.Dv uint8 , 734.Dv int16 , 735.Dv uint16 , 736.Dv int32 , 737.Dv uint32 , 738.Dv int64 , 739and 740.Dv uint64 , 741and specified-byte-order variants of them, 742to make it clearer that those types have specified widths. 743.\" 744.\" From: guy@sun.uucp (Guy Harris) 745.\" Newsgroups: net.bugs.usg 746.\" Subject: /etc/magic's format isn't well documented 747.\" Message-ID: <2752@sun.uucp> 748.\" Date: 3 Sep 85 08:19:07 GMT 749.\" Organization: Sun Microsystems, Inc. 750.\" Lines: 136 751.\" 752.\" Here's a manual page for the format accepted by the "file" made by adding 753.\" the changes I posted to the S5R2 version. 754.\" 755.\" Modified for Ian Darwin's version of the file command. 756