1.\" $File: magic.man,v 1.88 2016/07/27 09:42:49 rrt Exp $ 2.Dd July 20, 2016 3.Dt MAGIC __FSECTION__ 4.Os 5.\" install as magic.4 on USG, magic.5 on V7, Berkeley and Linux systems. 6.Sh NAME 7.Nm magic 8.Nd file command's magic pattern file 9.Sh DESCRIPTION 10This manual page documents the format of the magic file as 11used by the 12.Xr file __CSECTION__ 13command, version __VERSION__. 14The 15.Xr file __CSECTION__ 16command identifies the type of a file using, 17among other tests, 18a test for whether the file contains certain 19.Dq "magic patterns" . 20The file 21.Pa __MAGIC__ 22specifies what patterns are to be tested for, what message or 23MIME type to print if a particular pattern is found, 24and additional information to extract from the file. 25.Pp 26Each line of the file specifies a test to be performed. 27A test compares the data starting at a particular offset 28in the file with a byte value, a string or a numeric value. 29If the test succeeds, a message is printed. 30The line consists of the following fields: 31.Bl -tag -width ".Dv message" 32.It Dv offset 33A number specifying the offset, in bytes, into the file of the data 34which is to be tested. 35.It Dv type 36The type of the data to be tested. 37The possible values are: 38.Bl -tag -width ".Dv lestring16" 39.It Dv byte 40A one-byte value. 41.It Dv short 42A two-byte value in this machine's native byte order. 43.It Dv long 44A four-byte value in this machine's native byte order. 45.It Dv quad 46An eight-byte value in this machine's native byte order. 47.It Dv float 48A 32-bit single precision IEEE floating point number in this machine's native byte order. 49.It Dv double 50A 64-bit double precision IEEE floating point number in this machine's native byte order. 51.It Dv string 52A string of bytes. 53The string type specification can be optionally followed 54by /[WwcCtbT]*. 55The 56.Dq W 57flag compacts whitespace in the target, which must 58contain at least one whitespace character. 59If the magic has 60.Dv n 61consecutive blanks, the target needs at least 62.Dv n 63consecutive blanks to match. 64The 65.Dq w 66flag treats every blank in the magic as an optional blank. 67The 68.Dq c 69flag specifies case insensitive matching: lower case 70characters in the magic match both lower and upper case characters in the 71target, whereas upper case characters in the magic only match upper case 72characters in the target. 73The 74.Dq C 75flag specifies case insensitive matching: upper case 76characters in the magic match both lower and upper case characters in the 77target, whereas lower case characters in the magic only match upper case 78characters in the target. 79To do a complete case insensitive match, specify both 80.Dq c 81and 82.Dq C . 83The 84.Dq t 85flag forces the test to be done for text files, while the 86.Dq b 87flag forces the test to be done for binary files. 88The 89.Dq T 90flag causes the string to be trimmed, i.e. leading and trailing whitespace 91is deleted before the string is printed. 92.It Dv pstring 93A Pascal-style string where the first byte/short/int is interpreted as the 94unsigned length. 95The length defaults to byte and can be specified as a modifier. 96The following modifiers are supported: 97.Bl -tag -compact -width B 98.It B 99A byte length (default). 100.It H 101A 2 byte big endian length. 102.It h 103A 2 byte big little length. 104.It L 105A 4 byte big endian length. 106.It l 107A 4 byte big little length. 108.It J 109The length includes itself in its count. 110.El 111The string is not NUL terminated. 112.Dq J 113is used rather than the more 114valuable 115.Dq I 116because this type of length is a feature of the JPEG 117format. 118.It Dv date 119A four-byte value interpreted as a UNIX date. 120.It Dv qdate 121A eight-byte value interpreted as a UNIX date. 122.It Dv ldate 123A four-byte value interpreted as a UNIX-style date, but interpreted as 124local time rather than UTC. 125.It Dv qldate 126An eight-byte value interpreted as a UNIX-style date, but interpreted as 127local time rather than UTC. 128.It Dv qwdate 129An eight-byte value interpreted as a Windows-style date. 130.It Dv beid3 131A 32-bit ID3 length in big-endian byte order. 132.It Dv beshort 133A two-byte value in big-endian byte order. 134.It Dv belong 135A four-byte value in big-endian byte order. 136.It Dv bequad 137An eight-byte value in big-endian byte order. 138.It Dv befloat 139A 32-bit single precision IEEE floating point number in big-endian byte order. 140.It Dv bedouble 141A 64-bit double precision IEEE floating point number in big-endian byte order. 142.It Dv bedate 143A four-byte value in big-endian byte order, 144interpreted as a Unix date. 145.It Dv beqdate 146An eight-byte value in big-endian byte order, 147interpreted as a Unix date. 148.It Dv beldate 149A four-byte value in big-endian byte order, 150interpreted as a UNIX-style date, but interpreted as local time rather 151than UTC. 152.It Dv beqldate 153An eight-byte value in big-endian byte order, 154interpreted as a UNIX-style date, but interpreted as local time rather 155than UTC. 156.It Dv beqwdate 157An eight-byte value in big-endian byte order, 158interpreted as a Windows-style date. 159.It Dv bestring16 160A two-byte unicode (UCS16) string in big-endian byte order. 161.It Dv leid3 162A 32-bit ID3 length in little-endian byte order. 163.It Dv leshort 164A two-byte value in little-endian byte order. 165.It Dv lelong 166A four-byte value in little-endian byte order. 167.It Dv lequad 168An eight-byte value in little-endian byte order. 169.It Dv lefloat 170A 32-bit single precision IEEE floating point number in little-endian byte order. 171.It Dv ledouble 172A 64-bit double precision IEEE floating point number in little-endian byte order. 173.It Dv ledate 174A four-byte value in little-endian byte order, 175interpreted as a UNIX date. 176.It Dv leqdate 177An eight-byte value in little-endian byte order, 178interpreted as a UNIX date. 179.It Dv leldate 180A four-byte value in little-endian byte order, 181interpreted as a UNIX-style date, but interpreted as local time rather 182than UTC. 183.It Dv leqldate 184An eight-byte value in little-endian byte order, 185interpreted as a UNIX-style date, but interpreted as local time rather 186than UTC. 187.It Dv leqwdate 188An eight-byte value in little-endian byte order, 189interpreted as a Windows-style date. 190.It Dv lestring16 191A two-byte unicode (UCS16) string in little-endian byte order. 192.It Dv melong 193A four-byte value in middle-endian (PDP-11) byte order. 194.It Dv medate 195A four-byte value in middle-endian (PDP-11) byte order, 196interpreted as a UNIX date. 197.It Dv meldate 198A four-byte value in middle-endian (PDP-11) byte order, 199interpreted as a UNIX-style date, but interpreted as local time rather 200than UTC. 201.It Dv indirect 202Starting at the given offset, consult the magic database again. 203The offset of the 204.Dv indirect 205magic is by default absolute in the file, but one can specify 206.Dv /r 207to indicate that the offset is relative from the beginning of the entry. 208.It Dv name 209Define a 210.Dq named 211magic instance that can be called from another 212.Dv use 213magic entry, like a subroutine call. 214Named instance direct magic offsets are relative to the offset of the 215previous matched entry, but indirect offsets are relative to the beginning 216of the file as usual. 217Named magic entries always match. 218.It Dv use 219Recursively call the named magic starting from the current offset. 220If the name of the referenced begins with a 221.Dv ^ 222then the endianness of the magic is switched; if the magic mentioned 223.Dv leshort 224for example, 225it is treated as 226.Dv beshort 227and vice versa. 228This is useful to avoid duplicating the rules for different endianness. 229.It Dv regex 230A regular expression match in extended POSIX regular expression syntax 231(like egrep). 232Regular expressions can take exponential time to process, and their 233performance is hard to predict, so their use is discouraged. 234When used in production environments, their performance 235should be carefully checked. 236The size of the string to search should also be limited by specifying 237.Dv /<length> , 238to avoid performance issues scanning long files. 239The type specification can also be optionally followed by 240.Dv /[c][s][l] . 241The 242.Dq c 243flag makes the match case insensitive, while the 244.Dq s 245flag update the offset to the start offset of the match, rather than the end. 246The 247.Dq l 248modifier, changes the limit of length to mean number of lines instead of a 249byte count. 250Lines are delimited by the platforms native line delimiter. 251When a line count is specified, an implicit byte count also computed assuming 252each line is 80 characters long. 253If neither a byte or line count is specified, the search is limited automatically 254to 8KiB. 255.Dv ^ 256and 257.Dv $ 258match the beginning and end of individual lines, respectively, 259not beginning and end of file. 260.It Dv search 261A literal string search starting at the given offset. 262The same modifier flags can be used as for string patterns. 263The search expression must contain the range in the form 264.Dv /number, 265that is the number of positions at which the match will be 266attempted, starting from the start offset. 267This is suitable for 268searching larger binary expressions with variable offsets, using 269.Dv \e 270escapes for special characters. 271The order of modifier and number is not relevant. 272.It Dv default 273This is intended to be used with the test 274.Em x 275(which is always true) and it has no type. 276It matches when no other test at that continuation level has matched before. 277Clearing that matched tests for a continuation level, can be done using the 278.Dv clear 279test. 280.It Dv clear 281This test is always true and clears the match flag for that continuation level. 282It is intended to be used with the 283.Dv default 284test. 285.El 286.Pp 287For compatibility with the Single 288.Ux 289Standard, the type specifiers 290.Dv dC 291and 292.Dv d1 293are equivalent to 294.Dv byte , 295the type specifiers 296.Dv uC 297and 298.Dv u1 299are equivalent to 300.Dv ubyte , 301the type specifiers 302.Dv dS 303and 304.Dv d2 305are equivalent to 306.Dv short , 307the type specifiers 308.Dv uS 309and 310.Dv u2 311are equivalent to 312.Dv ushort , 313the type specifiers 314.Dv dI , 315.Dv dL , 316and 317.Dv d4 318are equivalent to 319.Dv long , 320the type specifiers 321.Dv uI , 322.Dv uL , 323and 324.Dv u4 325are equivalent to 326.Dv ulong , 327the type specifier 328.Dv d8 329is equivalent to 330.Dv quad , 331the type specifier 332.Dv u8 333is equivalent to 334.Dv uquad , 335and the type specifier 336.Dv s 337is equivalent to 338.Dv string . 339In addition, the type specifier 340.Dv dQ 341is equivalent to 342.Dv quad 343and the type specifier 344.Dv uQ 345is equivalent to 346.Dv uquad . 347.Pp 348Each top-level magic pattern (see below for an explanation of levels) 349is classified as text or binary according to the types used. 350Types 351.Dq regex 352and 353.Dq search 354are classified as text tests, unless non-printable characters are used 355in the pattern. 356All other tests are classified as binary. 357A top-level 358pattern is considered to be a test text when all its patterns are text 359patterns; otherwise, it is considered to be a binary pattern. 360When 361matching a file, binary patterns are tried first; if no match is 362found, and the file looks like text, then its encoding is determined 363and the text patterns are tried. 364.Pp 365The numeric types may optionally be followed by 366.Dv \*[Am] 367and a numeric value, 368to specify that the value is to be AND'ed with the 369numeric value before any comparisons are done. 370Prepending a 371.Dv u 372to the type indicates that ordered comparisons should be unsigned. 373.It Dv test 374The value to be compared with the value from the file. 375If the type is 376numeric, this value 377is specified in C form; if it is a string, it is specified as a C string 378with the usual escapes permitted (e.g. \en for new-line). 379.Pp 380Numeric values 381may be preceded by a character indicating the operation to be performed. 382It may be 383.Dv = , 384to specify that the value from the file must equal the specified value, 385.Dv \*[Lt] , 386to specify that the value from the file must be less than the specified 387value, 388.Dv \*[Gt] , 389to specify that the value from the file must be greater than the specified 390value, 391.Dv \*[Am] , 392to specify that the value from the file must have set all of the bits 393that are set in the specified value, 394.Dv ^ , 395to specify that the value from the file must have clear any of the bits 396that are set in the specified value, or 397.Dv ~ , 398the value specified after is negated before tested. 399.Dv x , 400to specify that any value will match. 401If the character is omitted, it is assumed to be 402.Dv = . 403Operators 404.Dv \*[Am] , 405.Dv ^ , 406and 407.Dv ~ 408don't work with floats and doubles. 409The operator 410.Dv !\& 411specifies that the line matches if the test does 412.Em not 413succeed. 414.Pp 415Numeric values are specified in C form; e.g. 416.Dv 13 417is decimal, 418.Dv 013 419is octal, and 420.Dv 0x13 421is hexadecimal. 422.Pp 423Numeric operations are not performed on date types, instead the numeric 424value is interpreted as an offset. 425.Pp 426For string values, the string from the 427file must match the specified string. 428The operators 429.Dv = , 430.Dv \*[Lt] 431and 432.Dv \*[Gt] 433(but not 434.Dv \*[Am] ) 435can be applied to strings. 436The length used for matching is that of the string argument 437in the magic file. 438This means that a line can match any non-empty string (usually used to 439then print the string), with 440.Em \*[Gt]\e0 441(because all non-empty strings are greater than the empty string). 442.Pp 443Dates are treated as numerical values in the respective internal 444representation. 445.Pp 446The special test 447.Em x 448always evaluates to true. 449.It Dv message 450The message to be printed if the comparison succeeds. 451If the string contains a 452.Xr printf 3 453format specification, the value from the file (with any specified masking 454performed) is printed using the message as the format string. 455If the string begins with 456.Dq \eb , 457the message printed is the remainder of the string with no whitespace 458added before it: multiple matches are normally separated by a single 459space. 460.El 461.Pp 462An APPLE 4+4 character APPLE creator and type can be specified as: 463.Bd -literal -offset indent 464!:apple CREATYPE 465.Ed 466.Pp 467A MIME type is given on a separate line, which must be the next 468non-blank or comment line after the magic line that identifies the 469file type, and has the following format: 470.Bd -literal -offset indent 471!:mime MIMETYPE 472.Ed 473.Pp 474i.e. the literal string 475.Dq !:mime 476followed by the MIME type. 477.Pp 478An optional strength can be supplied on a separate line which refers to 479the current magic description using the following format: 480.Bd -literal -offset indent 481!:strength OP VALUE 482.Ed 483.Pp 484The operand 485.Dv OP 486can be: 487.Dv + , 488.Dv - , 489.Dv * , 490or 491.Dv / 492and 493.Dv VALUE 494is a constant between 0 and 255. 495This constant is applied using the specified operand 496to the currently computed default magic strength. 497.Pp 498Some file formats contain additional information which is to be printed 499along with the file type or need additional tests to determine the true 500file type. 501These additional tests are introduced by one or more 502.Em \*[Gt] 503characters preceding the offset. 504The number of 505.Em \*[Gt] 506on the line indicates the level of the test; a line with no 507.Em \*[Gt] 508at the beginning is considered to be at level 0. 509Tests are arranged in a tree-like hierarchy: 510if the test on a line at level 511.Em n 512succeeds, all following tests at level 513.Em n+1 514are performed, and the messages printed if the tests succeed, until a line 515with level 516.Em n 517(or less) appears. 518For more complex files, one can use empty messages to get just the 519"if/then" effect, in the following way: 520.Bd -literal -offset indent 5210 string MZ 522\*[Gt]0x18 leshort \*[Lt]0x40 MS-DOS executable 523\*[Gt]0x18 leshort \*[Gt]0x3f extended PC executable (e.g., MS Windows) 524.Ed 525.Pp 526Offsets do not need to be constant, but can also be read from the file 527being examined. 528If the first character following the last 529.Em \*[Gt] 530is a 531.Em \&( 532then the string after the parenthesis is interpreted as an indirect offset. 533That means that the number after the parenthesis is used as an offset in 534the file. 535The value at that offset is read, and is used again as an offset 536in the file. 537Indirect offsets are of the form: 538.Em (( x [[.,][bislBISL]][+\-][ y ]) . 539The value of 540.Em x 541is used as an offset in the file. 542A byte, id3 length, short or long is read at that offset depending on the 543.Em [bislBISLm] 544type specifier. 545The value is treated as signed if 546.Dq , 547is specified or unsigned if 548.Dq . 549is specified. 550The capitalized types interpret the number as a big endian 551value, whereas the small letter versions interpret the number as a little 552endian value; 553the 554.Em m 555type interprets the number as a middle endian (PDP-11) value. 556To that number the value of 557.Em y 558is added and the result is used as an offset in the file. 559The default type if one is not specified is long. 560.Pp 561That way variable length structures can be examined: 562.Bd -literal -offset indent 563# MS Windows executables are also valid MS-DOS executables 5640 string MZ 565\*[Gt]0x18 leshort \*[Lt]0x40 MZ executable (MS-DOS) 566# skip the whole block below if it is not an extended executable 567\*[Gt]0x18 leshort \*[Gt]0x3f 568\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 569\*[Gt]\*[Gt](0x3c.l) string LX\e0\e0 LX executable (OS/2) 570.Ed 571.Pp 572This strategy of examining has a drawback: you must make sure that you 573eventually print something, or users may get empty output (such as when 574there is neither PE\e0\e0 nor LE\e0\e0 in the above example). 575.Pp 576If this indirect offset cannot be used directly, simple calculations are 577possible: appending 578.Em [+-*/%\*[Am]|^]number 579inside parentheses allows one to modify 580the value read from the file before it is used as an offset: 581.Bd -literal -offset indent 582# MS Windows executables are also valid MS-DOS executables 5830 string MZ 584# sometimes, the value at 0x18 is less that 0x40 but there's still an 585# extended executable, simply appended to the file 586\*[Gt]0x18 leshort \*[Lt]0x40 587\*[Gt]\*[Gt](4.s*512) leshort 0x014c COFF executable (MS-DOS, DJGPP) 588\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 589.Ed 590.Pp 591Sometimes you do not know the exact offset as this depends on the length or 592position (when indirection was used before) of preceding fields. 593You can specify an offset relative to the end of the last up-level 594field using 595.Sq \*[Am] 596as a prefix to the offset: 597.Bd -literal -offset indent 5980 string MZ 599\*[Gt]0x18 leshort \*[Gt]0x3f 600\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 601# immediately following the PE signature is the CPU type 602\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x14c for Intel 80386 603\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x184 for DEC Alpha 604.Ed 605.Pp 606Indirect and relative offsets can be combined: 607.Bd -literal -offset indent 6080 string MZ 609\*[Gt]0x18 leshort \*[Lt]0x40 610\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 611# if it's not COFF, go back 512 bytes and add the offset taken 612# from byte 2/3, which is yet another way of finding the start 613# of the extended executable 614\*[Gt]\*[Gt]\*[Gt]\*[Am](2.s-514) string LE LE executable (MS Windows VxD driver) 615.Ed 616.Pp 617Or the other way around: 618.Bd -literal -offset indent 6190 string MZ 620\*[Gt]0x18 leshort \*[Gt]0x3f 621\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 622# at offset 0x80 (-4, since relative offsets start at the end 623# of the up-level match) inside the LE header, we find the absolute 624# offset to the code area, where we look for a specific signature 625\*[Gt]\*[Gt]\*[Gt](\*[Am]0x7c.l+0x26) string UPX \eb, UPX compressed 626.Ed 627.Pp 628Or even both! 629.Bd -literal -offset indent 6300 string MZ 631\*[Gt]0x18 leshort \*[Gt]0x3f 632\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 633# at offset 0x58 inside the LE header, we find the relative offset 634# to a data area where we look for a specific signature 635\*[Gt]\*[Gt]\*[Gt]\*[Am](\*[Am]0x54.l-3) string UNACE \eb, ACE self-extracting archive 636.Ed 637.Pp 638If you have to deal with offset/length pairs in your file, even the 639second value in a parenthesized expression can be taken from the file itself, 640using another set of parentheses. 641Note that this additional indirect offset is always relative to the 642start of the main indirect offset. 643.Bd -literal -offset indent 6440 string MZ 645\*[Gt]0x18 leshort \*[Gt]0x3f 646\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 647# search for the PE section called ".idata"... 648\*[Gt]\*[Gt]\*[Gt]\*[Am]0xf4 search/0x140 .idata 649# ...and go to the end of it, calculated from start+length; 650# these are located 14 and 10 bytes after the section name 651\*[Gt]\*[Gt]\*[Gt]\*[Gt](\*[Am]0xe.l+(-4)) string PK\e3\e4 \eb, ZIP self-extracting archive 652.Ed 653.Pp 654If you have a list of known avalues at a particular continuation level, 655and you want to provide a switch-like default case: 656.Bd -literal -offset indent 657# clear that continuation level match 658\*[Gt]18 clear 659\*[Gt]18 lelong 1 one 660\*[Gt]18 lelong 2 two 661\*[Gt]18 default x 662# print default match 663\*[Gt]\*[Gt]18 lelong x unmatched 0x%x 664.Ed 665.Sh SEE ALSO 666.Xr file __CSECTION__ 667\- the command that reads this file. 668.Sh BUGS 669The formats 670.Dv long , 671.Dv belong , 672.Dv lelong , 673.Dv melong , 674.Dv short , 675.Dv beshort , 676and 677.Dv leshort 678do not depend on the length of the C data types 679.Dv short 680and 681.Dv long 682on the platform, even though the Single 683.Ux 684Specification implies that they do. However, as OS X Mountain Lion has 685passed the Single 686.Ux 687Specification validation suite, and supplies a version of 688.Xr file __CSECTION__ 689in which they do not depend on the sizes of the C data types and that is 690built for a 64-bit environment in which 691.Dv long 692is 8 bytes rather than 4 bytes, presumably the validation suite does not 693test whether, for example 694.Dv long 695refers to an item with the same size as the C data type 696.Dv long . 697There should probably be 698.Dv type 699names 700.Dv int8 , 701.Dv uint8 , 702.Dv int16 , 703.Dv uint16 , 704.Dv int32 , 705.Dv uint32 , 706.Dv int64 , 707and 708.Dv uint64 , 709and specified-byte-order variants of them, 710to make it clearer that those types have specified widths. 711.\" 712.\" From: guy@sun.uucp (Guy Harris) 713.\" Newsgroups: net.bugs.usg 714.\" Subject: /etc/magic's format isn't well documented 715.\" Message-ID: <2752@sun.uucp> 716.\" Date: 3 Sep 85 08:19:07 GMT 717.\" Organization: Sun Microsystems, Inc. 718.\" Lines: 136 719.\" 720.\" Here's a manual page for the format accepted by the "file" made by adding 721.\" the changes I posted to the S5R2 version. 722.\" 723.\" Modified for Ian Darwin's version of the file command. 724