1 2XZ data compression in Linux 3============================ 4 5Introduction 6 7 XZ is a general purpose data compression format with high compression 8 ratio and relatively fast decompression. The primary compression 9 algorithm (filter) is LZMA2. Additional filters can be used to improve 10 compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters 11 improve compression ratio of executable data. 12 13 The XZ decompressor in Linux is called XZ Embedded. It supports 14 the LZMA2 filter and optionally also BCJ filters. CRC32 is supported 15 for integrity checking. The home page of XZ Embedded is at 16 <https://tukaani.org/xz/embedded.html>, where you can find the 17 latest version and also information about using the code outside 18 the Linux kernel. 19 20 For userspace, XZ Utils provide a zlib-like compression library 21 and a gzip-like command line tool. XZ Utils can be downloaded from 22 <https://tukaani.org/xz/>. 23 24XZ related components in the kernel 25 26 The xz_dec module provides XZ decompressor with single-call (buffer 27 to buffer) and multi-call (stateful) APIs. The usage of the xz_dec 28 module is documented in include/linux/xz.h. 29 30 The xz_dec_test module is for testing xz_dec. xz_dec_test is not 31 useful unless you are hacking the XZ decompressor. xz_dec_test 32 allocates a char device major dynamically to which one can write 33 .xz files from userspace. The decompressed output is thrown away. 34 Keep an eye on dmesg to see diagnostics printed by xz_dec_test. 35 See the xz_dec_test source code for the details. 36 37 For decompressing the kernel image, initramfs, and initrd, there 38 is a wrapper function in lib/decompress_unxz.c. Its API is the 39 same as in other decompress_*.c files, which is defined in 40 include/linux/decompress/generic.h. 41 42 scripts/xz_wrap.sh is a wrapper for the xz command line tool found 43 from XZ Utils. The wrapper sets compression options to values suitable 44 for compressing the kernel image. 45 46 For kernel makefiles, two commands are provided for use with 47 $(call if_needed). The kernel image should be compressed with 48 $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 49 dictionary. It will also append a four-byte trailer containing the 50 uncompressed size of the file, which is needed by the boot code. 51 Other things should be compressed with $(call if_needed,xzmisc) 52 which will use no BCJ filter and 1 MiB LZMA2 dictionary. 53 54Notes on compression options 55 56 Since the XZ Embedded supports only streams with no integrity check or 57 CRC32, make sure that you don't use some other integrity check type 58 when encoding files that are supposed to be decoded by the kernel. With 59 liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 60 when encoding. With the xz command line tool, use --check=none or 61 --check=crc32. 62 63 Using CRC32 is strongly recommended unless there is some other layer 64 which will verify the integrity of the uncompressed data anyway. 65 Double checking the integrity would probably be waste of CPU cycles. 66 Note that the headers will always have a CRC32 which will be validated 67 by the decoder; you can only change the integrity check type (or 68 disable it) for the actual uncompressed data. 69 70 In userspace, LZMA2 is typically used with dictionary sizes of several 71 megabytes. The decoder needs to have the dictionary in RAM, thus big 72 dictionaries cannot be used for files that are intended to be decoded 73 by the kernel. 1 MiB is probably the maximum reasonable dictionary 74 size for in-kernel use (maybe more is OK for initramfs). The presets 75 in XZ Utils may not be optimal when creating files for the kernel, 76 so don't hesitate to use custom settings. Example: 77 78 xz --check=crc32 --lzma2=dict=512KiB inputfile 79 80 An exception to above dictionary size limitation is when the decoder 81 is used in single-call mode. Decompressing the kernel itself is an 82 example of this situation. In single-call mode, the memory usage 83 doesn't depend on the dictionary size, and it is perfectly fine to 84 use a big dictionary: for maximum compression, the dictionary should 85 be at least as big as the uncompressed data itself. 86 87Future plans 88 89 Creating a limited XZ encoder may be considered if people think it is 90 useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at 91 the fastest settings, so it isn't clear if LZMA2 encoder is wanted 92 into the kernel. 93 94 Support for limited random-access reading is planned for the 95 decompression code. I don't know if it could have any use in the 96 kernel, but I know that it would be useful in some embedded projects 97 outside the Linux kernel. 98 99Conformance to the .xz file format specification 100 101 There are a couple of corner cases where things have been simplified 102 at expense of detecting errors as early as possible. These should not 103 matter in practice all, since they don't cause security issues. But 104 it is good to know this if testing the code e.g. with the test files 105 from XZ Utils. 106 107Reporting bugs 108 109 Before reporting a bug, please check that it's not fixed already 110 at upstream. See <https://tukaani.org/xz/embedded.html> to get the 111 latest code. 112 113 Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on 114 Freenode and talk to Larhzu. I don't actively read LKML or other 115 kernel-related mailing lists, so if there's something I should know, 116 you should email to me personally or use IRC. 117 118 Don't bother Igor Pavlov with questions about the XZ implementation 119 in the kernel or about XZ Utils. While these two implementations 120 include essential code that is directly based on Igor Pavlov's code, 121 these implementations aren't maintained nor supported by him. 122 123