1 2 3 4 5 6 7Network Working Group P. Deutsch 8Request for Comments: 1952 Aladdin Enterprises 9Category: Informational May 1996 10 11 12 GZIP file format specification version 4.3 13 14Status of This Memo 15 16 This memo provides information for the Internet community. This memo 17 does not specify an Internet standard of any kind. Distribution of 18 this memo is unlimited. 19 20IESG Note: 21 22 The IESG takes no position on the validity of any Intellectual 23 Property Rights statements contained in this document. 24 25Notices 26 27 Copyright (c) 1996 L. Peter Deutsch 28 29 Permission is granted to copy and distribute this document for any 30 purpose and without charge, including translations into other 31 languages and incorporation into compilations, provided that the 32 copyright notice and this notice are preserved, and that any 33 substantive changes or deletions from the original are clearly 34 marked. 35 36 A pointer to the latest version of this and related documentation in 37 HTML format can be found at the URL 38 <ftp://ftp.uu.net/graphics/png/documents/zlib/zdoc-index.html>. 39 40Abstract 41 42 This specification defines a lossless compressed data format that is 43 compatible with the widely used GZIP utility. The format includes a 44 cyclic redundancy check value for detecting data corruption. The 45 format presently uses the DEFLATE method of compression but can be 46 easily extended to use other compression methods. The format can be 47 implemented readily in a manner not covered by patents. 48 49 50 51 52 53 54 55 56 57 58Deutsch Informational [Page 1] 59 60RFC 1952 GZIP File Format Specification May 1996 61 62 63Table of Contents 64 65 1. Introduction ................................................... 2 66 1.1. Purpose ................................................... 2 67 1.2. Intended audience ......................................... 3 68 1.3. Scope ..................................................... 3 69 1.4. Compliance ................................................ 3 70 1.5. Definitions of terms and conventions used ................. 3 71 1.6. Changes from previous versions ............................ 3 72 2. Detailed specification ......................................... 4 73 2.1. Overall conventions ....................................... 4 74 2.2. File format ............................................... 5 75 2.3. Member format ............................................. 5 76 2.3.1. Member header and trailer ........................... 6 77 2.3.1.1. Extra field ................................... 8 78 2.3.1.2. Compliance .................................... 9 79 3. References .................................................. 9 80 4. Security Considerations .................................... 10 81 5. Acknowledgements ........................................... 10 82 6. Author's Address ........................................... 10 83 7. Appendix: Jean-Loup Gailly's gzip utility .................. 11 84 8. Appendix: Sample CRC Code .................................. 11 85 861. Introduction 87 88 1.1. Purpose 89 90 The purpose of this specification is to define a lossless 91 compressed data format that: 92 93 * Is independent of CPU type, operating system, file system, 94 and character set, and hence can be used for interchange; 95 * Can compress or decompress a data stream (as opposed to a 96 randomly accessible file) to produce another data stream, 97 using only an a priori bounded amount of intermediate 98 storage, and hence can be used in data communications or 99 similar structures such as Unix filters; 100 * Compresses data with efficiency comparable to the best 101 currently available general-purpose compression methods, 102 and in particular considerably better than the "compress" 103 program; 104 * Can be implemented readily in a manner not covered by 105 patents, and hence can be practiced freely; 106 * Is compatible with the file format produced by the current 107 widely used gzip utility, in that conforming decompressors 108 will be able to read data produced by the existing gzip 109 compressor. 110 111 112 113 114Deutsch Informational [Page 2] 115 116RFC 1952 GZIP File Format Specification May 1996 117 118 119 The data format defined by this specification does not attempt to: 120 121 * Provide random access to compressed data; 122 * Compress specialized data (e.g., raster graphics) as well as 123 the best currently available specialized algorithms. 124 125 1.2. Intended audience 126 127 This specification is intended for use by implementors of software 128 to compress data into gzip format and/or decompress data from gzip 129 format. 130 131 The text of the specification assumes a basic background in 132 programming at the level of bits and other primitive data 133 representations. 134 135 1.3. Scope 136 137 The specification specifies a compression method and a file format 138 (the latter assuming only that a file can store a sequence of 139 arbitrary bytes). It does not specify any particular interface to 140 a file system or anything about character sets or encodings 141 (except for file names and comments, which are optional). 142 143 1.4. Compliance 144 145 Unless otherwise indicated below, a compliant decompressor must be 146 able to accept and decompress any file that conforms to all the 147 specifications presented here; a compliant compressor must produce 148 files that conform to all the specifications presented here. The 149 material in the appendices is not part of the specification per se 150 and is not relevant to compliance. 151 152 1.5. Definitions of terms and conventions used 153 154 byte: 8 bits stored or transmitted as a unit (same as an octet). 155 (For this specification, a byte is exactly 8 bits, even on 156 machines which store a character on a number of bits different 157 from 8.) See below for the numbering of bits within a byte. 158 159 1.6. Changes from previous versions 160 161 There have been no technical changes to the gzip format since 162 version 4.1 of this specification. In version 4.2, some 163 terminology was changed, and the sample CRC code was rewritten for 164 clarity and to eliminate the requirement for the caller to do pre- 165 and post-conditioning. Version 4.3 is a conversion of the 166 specification to RFC style. 167 168 169 170Deutsch Informational [Page 3] 171 172RFC 1952 GZIP File Format Specification May 1996 173 174 1752. Detailed specification 176 177 2.1. Overall conventions 178 179 In the diagrams below, a box like this: 180 181 +---+ 182 | | <-- the vertical bars might be missing 183 +---+ 184 185 represents one byte; a box like this: 186 187 +==============+ 188 | | 189 +==============+ 190 191 represents a variable number of bytes. 192 193 Bytes stored within a computer do not have a "bit order", since 194 they are always treated as a unit. However, a byte considered as 195 an integer between 0 and 255 does have a most- and least- 196 significant bit, and since we write numbers with the most- 197 significant digit on the left, we also write bytes with the most- 198 significant bit on the left. In the diagrams below, we number the 199 bits of a byte so that bit 0 is the least-significant bit, i.e., 200 the bits are numbered: 201 202 +--------+ 203 |76543210| 204 +--------+ 205 206 This document does not address the issue of the order in which 207 bits of a byte are transmitted on a bit-sequential medium, since 208 the data format described here is byte- rather than bit-oriented. 209 210 Within a computer, a number may occupy multiple bytes. All 211 multi-byte numbers in the format described here are stored with 212 the least-significant byte first (at the lower memory address). 213 For example, the decimal number 520 is stored as: 214 215 0 1 216 +--------+--------+ 217 |00001000|00000010| 218 +--------+--------+ 219 ^ ^ 220 | | 221 | + more significant byte = 2 x 256 222 + less significant byte = 8 223 224 225 226Deutsch Informational [Page 4] 227 228RFC 1952 GZIP File Format Specification May 1996 229 230 231 2.2. File format 232 233 A gzip file consists of a series of "members" (compressed data 234 sets). The format of each member is specified in the following 235 section. The members simply appear one after another in the file, 236 with no additional information before, between, or after them. 237 238 2.3. Member format 239 240 Each member has the following structure: 241 242 +---+---+---+---+---+---+---+---+---+---+ 243 |ID1|ID2|CM |FLG| MTIME |XFL|OS | (more-->) 244 +---+---+---+---+---+---+---+---+---+---+ 245 246 (if FLG.FEXTRA set) 247 248 +---+---+=================================+ 249 | XLEN |...XLEN bytes of "extra field"...| (more-->) 250 +---+---+=================================+ 251 252 (if FLG.FNAME set) 253 254 +=========================================+ 255 |...original file name, zero-terminated...| (more-->) 256 +=========================================+ 257 258 (if FLG.FCOMMENT set) 259 260 +===================================+ 261 |...file comment, zero-terminated...| (more-->) 262 +===================================+ 263 264 (if FLG.FHCRC set) 265 266 +---+---+ 267 | CRC16 | 268 +---+---+ 269 270 +=======================+ 271 |...compressed blocks...| (more-->) 272 +=======================+ 273 274 0 1 2 3 4 5 6 7 275 +---+---+---+---+---+---+---+---+ 276 | CRC32 | ISIZE | 277 +---+---+---+---+---+---+---+---+ 278 279 280 281 282Deutsch Informational [Page 5] 283 284RFC 1952 GZIP File Format Specification May 1996 285 286 287 2.3.1. Member header and trailer 288 289 ID1 (IDentification 1) 290 ID2 (IDentification 2) 291 These have the fixed values ID1 = 31 (0x1f, \037), ID2 = 139 292 (0x8b, \213), to identify the file as being in gzip format. 293 294 CM (Compression Method) 295 This identifies the compression method used in the file. CM 296 = 0-7 are reserved. CM = 8 denotes the "deflate" 297 compression method, which is the one customarily used by 298 gzip and which is documented elsewhere. 299 300 FLG (FLaGs) 301 This flag byte is divided into individual bits as follows: 302 303 bit 0 FTEXT 304 bit 1 FHCRC 305 bit 2 FEXTRA 306 bit 3 FNAME 307 bit 4 FCOMMENT 308 bit 5 reserved 309 bit 6 reserved 310 bit 7 reserved 311 312 If FTEXT is set, the file is probably ASCII text. This is 313 an optional indication, which the compressor may set by 314 checking a small amount of the input data to see whether any 315 non-ASCII characters are present. In case of doubt, FTEXT 316 is cleared, indicating binary data. For systems which have 317 different file formats for ascii text and binary data, the 318 decompressor can use FTEXT to choose the appropriate format. 319 We deliberately do not specify the algorithm used to set 320 this bit, since a compressor always has the option of 321 leaving it cleared and a decompressor always has the option 322 of ignoring it and letting some other program handle issues 323 of data conversion. 324 325 If FHCRC is set, a CRC16 for the gzip header is present, 326 immediately before the compressed data. The CRC16 consists 327 of the two least significant bytes of the CRC32 for all 328 bytes of the gzip header up to and not including the CRC16. 329 [The FHCRC bit was never set by versions of gzip up to 330 1.2.4, even though it was documented with a different 331 meaning in gzip 1.2.4.] 332 333 If FEXTRA is set, optional extra fields are present, as 334 described in a following section. 335 336 337 338Deutsch Informational [Page 6] 339 340RFC 1952 GZIP File Format Specification May 1996 341 342 343 If FNAME is set, an original file name is present, 344 terminated by a zero byte. The name must consist of ISO 345 8859-1 (LATIN-1) characters; on operating systems using 346 EBCDIC or any other character set for file names, the name 347 must be translated to the ISO LATIN-1 character set. This 348 is the original name of the file being compressed, with any 349 directory components removed, and, if the file being 350 compressed is on a file system with case insensitive names, 351 forced to lower case. There is no original file name if the 352 data was compressed from a source other than a named file; 353 for example, if the source was stdin on a Unix system, there 354 is no file name. 355 356 If FCOMMENT is set, a zero-terminated file comment is 357 present. This comment is not interpreted; it is only 358 intended for human consumption. The comment must consist of 359 ISO 8859-1 (LATIN-1) characters. Line breaks should be 360 denoted by a single line feed character (10 decimal). 361 362 Reserved FLG bits must be zero. 363 364 MTIME (Modification TIME) 365 This gives the most recent modification time of the original 366 file being compressed. The time is in Unix format, i.e., 367 seconds since 00:00:00 GMT, Jan. 1, 1970. (Note that this 368 may cause problems for MS-DOS and other systems that use 369 local rather than Universal time.) If the compressed data 370 did not come from a file, MTIME is set to the time at which 371 compression started. MTIME = 0 means no time stamp is 372 available. 373 374 XFL (eXtra FLags) 375 These flags are available for use by specific compression 376 methods. The "deflate" method (CM = 8) sets these flags as 377 follows: 378 379 XFL = 2 - compressor used maximum compression, 380 slowest algorithm 381 XFL = 4 - compressor used fastest algorithm 382 383 OS (Operating System) 384 This identifies the type of file system on which compression 385 took place. This may be useful in determining end-of-line 386 convention for text files. The currently defined values are 387 as follows: 388 389 390 391 392 393 394Deutsch Informational [Page 7] 395 396RFC 1952 GZIP File Format Specification May 1996 397 398 399 0 - FAT filesystem (MS-DOS, OS/2, NT/Win32) 400 1 - Amiga 401 2 - VMS (or OpenVMS) 402 3 - Unix 403 4 - VM/CMS 404 5 - Atari TOS 405 6 - HPFS filesystem (OS/2, NT) 406 7 - Macintosh 407 8 - Z-System 408 9 - CP/M 409 10 - TOPS-20 410 11 - NTFS filesystem (NT) 411 12 - QDOS 412 13 - Acorn RISCOS 413 255 - unknown 414 415 XLEN (eXtra LENgth) 416 If FLG.FEXTRA is set, this gives the length of the optional 417 extra field. See below for details. 418 419 CRC32 (CRC-32) 420 This contains a Cyclic Redundancy Check value of the 421 uncompressed data computed according to CRC-32 algorithm 422 used in the ISO 3309 standard and in section 8.1.1.6.2 of 423 ITU-T recommendation V.42. (See http://www.iso.ch for 424 ordering ISO documents. See gopher://info.itu.ch for an 425 online version of ITU-T V.42.) 426 427 ISIZE (Input SIZE) 428 This contains the size of the original (uncompressed) input 429 data modulo 2^32. 430 431 2.3.1.1. Extra field 432 433 If the FLG.FEXTRA bit is set, an "extra field" is present in 434 the header, with total length XLEN bytes. It consists of a 435 series of subfields, each of the form: 436 437 +---+---+---+---+==================================+ 438 |SI1|SI2| LEN |... LEN bytes of subfield data ...| 439 +---+---+---+---+==================================+ 440 441 SI1 and SI2 provide a subfield ID, typically two ASCII letters 442 with some mnemonic value. Jean-Loup Gailly 443 <gzip@prep.ai.mit.edu> is maintaining a registry of subfield 444 IDs; please send him any subfield ID you wish to use. Subfield 445 IDs with SI2 = 0 are reserved for future use. The following 446 IDs are currently defined: 447 448 449 450Deutsch Informational [Page 8] 451 452RFC 1952 GZIP File Format Specification May 1996 453 454 455 SI1 SI2 Data 456 ---------- ---------- ---- 457 0x41 ('A') 0x70 ('P') Apollo file type information 458 459 LEN gives the length of the subfield data, excluding the 4 460 initial bytes. 461 462 2.3.1.2. Compliance 463 464 A compliant compressor must produce files with correct ID1, 465 ID2, CM, CRC32, and ISIZE, but may set all the other fields in 466 the fixed-length part of the header to default values (255 for 467 OS, 0 for all others). The compressor must set all reserved 468 bits to zero. 469 470 A compliant decompressor must check ID1, ID2, and CM, and 471 provide an error indication if any of these have incorrect 472 values. It must examine FEXTRA/XLEN, FNAME, FCOMMENT and FHCRC 473 at least so it can skip over the optional fields if they are 474 present. It need not examine any other part of the header or 475 trailer; in particular, a decompressor may ignore FTEXT and OS 476 and always produce binary output, and still be compliant. A 477 compliant decompressor must give an error indication if any 478 reserved bit is non-zero, since such a bit could indicate the 479 presence of a new field that would cause subsequent data to be 480 interpreted incorrectly. 481 4823. References 483 484 [1] "Information Processing - 8-bit single-byte coded graphic 485 character sets - Part 1: Latin alphabet No.1" (ISO 8859-1:1987). 486 The ISO 8859-1 (Latin-1) character set is a superset of 7-bit 487 ASCII. Files defining this character set are available as 488 iso_8859-1.* in ftp://ftp.uu.net/graphics/png/documents/ 489 490 [2] ISO 3309 491 492 [3] ITU-T recommendation V.42 493 494 [4] Deutsch, L.P.,"DEFLATE Compressed Data Format Specification", 495 available in ftp://ftp.uu.net/pub/archiving/zip/doc/ 496 497 [5] Gailly, J.-L., GZIP documentation, available as gzip-*.tar in 498 ftp://prep.ai.mit.edu/pub/gnu/ 499 500 [6] Sarwate, D.V., "Computation of Cyclic Redundancy Checks via Table 501 Look-Up", Communications of the ACM, 31(8), pp.1008-1013. 502 503 504 505 506Deutsch Informational [Page 9] 507 508RFC 1952 GZIP File Format Specification May 1996 509 510 511 [7] Schwaderer, W.D., "CRC Calculation", April 85 PC Tech Journal, 512 pp.118-133. 513 514 [8] ftp://ftp.adelaide.edu.au/pub/rocksoft/papers/crc_v3.txt, 515 describing the CRC concept. 516 5174. Security Considerations 518 519 Any data compression method involves the reduction of redundancy in 520 the data. Consequently, any corruption of the data is likely to have 521 severe effects and be difficult to correct. Uncompressed text, on 522 the other hand, will probably still be readable despite the presence 523 of some corrupted bytes. 524 525 It is recommended that systems using this data format provide some 526 means of validating the integrity of the compressed data, such as by 527 setting and checking the CRC-32 check value. 528 5295. Acknowledgements 530 531 Trademarks cited in this document are the property of their 532 respective owners. 533 534 Jean-Loup Gailly designed the gzip format and wrote, with Mark Adler, 535 the related software described in this specification. Glenn 536 Randers-Pehrson converted this document to RFC and HTML format. 537 5386. Author's Address 539 540 L. Peter Deutsch 541 Aladdin Enterprises 542 203 Santa Margarita Ave. 543 Menlo Park, CA 94025 544 545 Phone: (415) 322-0103 (AM only) 546 FAX: (415) 322-1734 547 EMail: <ghost@aladdin.com> 548 549 Questions about the technical content of this specification can be 550 sent by email to: 551 552 Jean-Loup Gailly <gzip@prep.ai.mit.edu> and 553 Mark Adler <madler@alumni.caltech.edu> 554 555 Editorial comments on this specification can be sent by email to: 556 557 L. Peter Deutsch <ghost@aladdin.com> and 558 Glenn Randers-Pehrson <randeg@alumni.rpi.edu> 559 560 561 562Deutsch Informational [Page 10] 563 564RFC 1952 GZIP File Format Specification May 1996 565 566 5677. Appendix: Jean-Loup Gailly's gzip utility 568 569 The most widely used implementation of gzip compression, and the 570 original documentation on which this specification is based, were 571 created by Jean-Loup Gailly <gzip@prep.ai.mit.edu>. Since this 572 implementation is a de facto standard, we mention some more of its 573 features here. Again, the material in this section is not part of 574 the specification per se, and implementations need not follow it to 575 be compliant. 576 577 When compressing or decompressing a file, gzip preserves the 578 protection, ownership, and modification time attributes on the local 579 file system, since there is no provision for representing protection 580 attributes in the gzip file format itself. Since the file format 581 includes a modification time, the gzip decompressor provides a 582 command line switch that assigns the modification time from the file, 583 rather than the local modification time of the compressed input, to 584 the decompressed output. 585 5868. Appendix: Sample CRC Code 587 588 The following sample code represents a practical implementation of 589 the CRC (Cyclic Redundancy Check). (See also ISO 3309 and ITU-T V.42 590 for a formal specification.) 591 592 The sample code is in the ANSI C programming language. Non C users 593 may find it easier to read with these hints: 594 595 & Bitwise AND operator. 596 ^ Bitwise exclusive-OR operator. 597 >> Bitwise right shift operator. When applied to an 598 unsigned quantity, as here, right shift inserts zero 599 bit(s) at the left. 600 ! Logical NOT operator. 601 ++ "n++" increments the variable n. 602 0xNNN 0x introduces a hexadecimal (base 16) constant. 603 Suffix L indicates a long value (at least 32 bits). 604 605 /* Table of CRCs of all 8-bit messages. */ 606 unsigned long crc_table[256]; 607 608 /* Flag: has the table been computed? Initially false. */ 609 int crc_table_computed = 0; 610 611 /* Make the table for a fast CRC. */ 612 void make_crc_table(void) 613 { 614 unsigned long c; 615 616 617 618Deutsch Informational [Page 11] 619 620RFC 1952 GZIP File Format Specification May 1996 621 622 623 int n, k; 624 for (n = 0; n < 256; n++) { 625 c = (unsigned long) n; 626 for (k = 0; k < 8; k++) { 627 if (c & 1) { 628 c = 0xedb88320L ^ (c >> 1); 629 } else { 630 c = c >> 1; 631 } 632 } 633 crc_table[n] = c; 634 } 635 crc_table_computed = 1; 636 } 637 638 /* 639 Update a running crc with the bytes buf[0..len-1] and return 640 the updated crc. The crc should be initialized to zero. Pre- and 641 post-conditioning (one's complement) is performed within this 642 function so it shouldn't be done by the caller. Usage example: 643 644 unsigned long crc = 0L; 645 646 while (read_buffer(buffer, length) != EOF) { 647 crc = update_crc(crc, buffer, length); 648 } 649 if (crc != original_crc) error(); 650 */ 651 unsigned long update_crc(unsigned long crc, 652 unsigned char *buf, int len) 653 { 654 unsigned long c = crc ^ 0xffffffffL; 655 int n; 656 657 if (!crc_table_computed) 658 make_crc_table(); 659 for (n = 0; n < len; n++) { 660 c = crc_table[(c ^ buf[n]) & 0xff] ^ (c >> 8); 661 } 662 return c ^ 0xffffffffL; 663 } 664 665 /* Return the CRC of the bytes buf[0..len-1]. */ 666 unsigned long crc(unsigned char *buf, int len) 667 { 668 return update_crc(0L, buf, len); 669 } 670 671 672 673 674Deutsch Informational [Page 12] 675 676