1Command Line Interface for Zstandard library 2============================================ 3 4Command Line Interface (CLI) can be created using the `make` command without any additional parameters. 5There are however other Makefile targets that create different variations of CLI: 6- `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd formats 7- `zstd_nolegacy` : Same as `zstd` but without support for legacy zstd formats 8- `zstd-small` : CLI optimized for minimal size; no dictionary builder, no benchmark, and no support for legacy zstd formats 9- `zstd-compress` : version of CLI which can only compress into zstd format 10- `zstd-decompress` : version of CLI which can only decompress zstd format 11 12 13#### Compilation variables 14`zstd` scope can be altered by modifying the following compilation variables : 15 16- __HAVE_THREAD__ : multithreading is automatically enabled when `pthread` is detected. 17 It's possible to disable multithread support, by setting HAVE_THREAD=0 . 18 Example : make zstd HAVE_THREAD=0 19 It's also possible to force compilation with multithread support, using HAVE_THREAD=1. 20 In which case, linking stage will fail if `pthread` library cannot be found. 21 This might be useful to prevent silent feature disabling. 22 23- __HAVE_ZLIB__ : `zstd` can compress and decompress files in `.gz` format. 24 This is ordered through command `--format=gzip`. 25 Alternatively, symlinks named `gzip` or `gunzip` will mimic intended behavior. 26 `.gz` support is automatically enabled when `zlib` library is detected at build time. 27 It's possible to disable `.gz` support, by setting HAVE_ZLIB=0. 28 Example : make zstd HAVE_ZLIB=0 29 It's also possible to force compilation with zlib support, using HAVE_ZLIB=1. 30 In which case, linking stage will fail if `zlib` library cannot be found. 31 This might be useful to prevent silent feature disabling. 32 33- __HAVE_LZMA__ : `zstd` can compress and decompress files in `.xz` and `.lzma` formats. 34 This is ordered through commands `--format=xz` and `--format=lzma` respectively. 35 Alternatively, symlinks named `xz`, `unxz`, `lzma`, or `unlzma` will mimic intended behavior. 36 `.xz` and `.lzma` support is automatically enabled when `lzma` library is detected at build time. 37 It's possible to disable `.xz` and `.lzma` support, by setting HAVE_LZMA=0 . 38 Example : make zstd HAVE_LZMA=0 39 It's also possible to force compilation with lzma support, using HAVE_LZMA=1. 40 In which case, linking stage will fail if `lzma` library cannot be found. 41 This might be useful to prevent silent feature disabling. 42 43- __HAVE_LZ4__ : `zstd` can compress and decompress files in `.lz4` formats. 44 This is ordered through commands `--format=lz4`. 45 Alternatively, symlinks named `lz4`, or `unlz4` will mimic intended behavior. 46 `.lz4` support is automatically enabled when `lz4` library is detected at build time. 47 It's possible to disable `.lz4` support, by setting HAVE_LZ4=0 . 48 Example : make zstd HAVE_LZ4=0 49 It's also possible to force compilation with lz4 support, using HAVE_LZ4=1. 50 In which case, linking stage will fail if `lz4` library cannot be found. 51 This might be useful to prevent silent feature disabling. 52 53- __ZSTD_LEGACY_SUPPORT__ : `zstd` can decompress files compressed by older versions of `zstd`. 54 Starting v0.8.0, all versions of `zstd` produce frames compliant with the [specification](../doc/zstd_compression_format.md), and are therefore compatible. 55 But older versions (< v0.8.0) produced different, incompatible, frames. 56 By default, `zstd` supports decoding legacy formats >= v0.4.0 (`ZSTD_LEGACY_SUPPORT=4`). 57 This can be altered by modifying this compilation variable. 58 `ZSTD_LEGACY_SUPPORT=1` means "support all formats >= v0.1.0". 59 `ZSTD_LEGACY_SUPPORT=2` means "support all formats >= v0.2.0", and so on. 60 `ZSTD_LEGACY_SUPPORT=0` means _DO NOT_ support any legacy format. 61 if `ZSTD_LEGACY_SUPPORT >= 8`, it's the same as `0`, since there is no legacy format after `7`. 62 Note : `zstd` only supports decoding older formats, and cannot generate any legacy format. 63 64 65#### Aggregation of parameters 66CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`. 67 68 69#### Dictionary builder in Command Line Interface 70Zstd offers a training mode, which can be used to tune the algorithm for a selected 71type of data, by providing it with a few samples. The result of the training is stored 72in a file selected with the `-o` option (default name is `dictionary`), 73which can be loaded before compression and decompression. 74 75Using a dictionary, the compression ratio achievable on small data improves dramatically. 76These compression gains are achieved while simultaneously providing faster compression and decompression speeds. 77Dictionary work if there is some correlation in a family of small data (there is no universal dictionary). 78Hence, deploying one dictionary per type of data will provide the greater benefits. 79Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm 80will rely more and more on previously decoded content to compress the rest of the file. 81 82Usage of the dictionary builder and created dictionaries with CLI: 83 841. Create the dictionary : `zstd --train PathToTrainingSet/* -o dictionaryName` 852. Compress with the dictionary: `zstd FILE -D dictionaryName` 863. Decompress with the dictionary: `zstd --decompress FILE.zst -D dictionaryName` 87 88 89#### Benchmark in Command Line Interface 90CLI includes in-memory compression benchmark module for zstd. 91The benchmark is conducted using given filenames. The files are read into memory and joined together. 92It makes benchmark more precise as it eliminates I/O overhead. 93Multiple filenames can be supplied, as multiple parameters, with wildcards, 94or names of directories can be used as parameters with `-r` option. 95 96The benchmark measures ratio, compressed size, compression and decompression speed. 97One can select compression levels starting from `-b` and ending with `-e`. 98The `-i` parameter selects minimal time used for each of tested levels. 99 100 101#### Usage of Command Line Interface 102The full list of options can be obtained with `-h` or `-H` parameter: 103``` 104Usage : 105 zstd [args] [FILE(s)] [-o file] 106 107FILE : a filename 108 with no FILE, or when FILE is - , read standard input 109Arguments : 110 -# : # compression level (1-19, default:3) 111 -d : decompression 112 -D file: use `file` as Dictionary 113 -o file: result stored into `file` (only if 1 input file) 114 -f : overwrite output without prompting and (de)compress links 115--rm : remove source file(s) after successful de/compression 116 -k : preserve source file(s) (default) 117 -h/-H : display help/long help and exit 118 119Advanced arguments : 120 -V : display Version number and exit 121 -v : verbose mode; specify multiple times to increase verbosity 122 -q : suppress warnings; specify twice to suppress errors too 123 -c : force write to standard output, even if it is the console 124 -l : print information about zstd compressed files 125--ultra : enable levels beyond 19, up to 22 (requires more memory) 126--long : enable long distance matching (requires more memory) 127--no-dictID : don't write dictID into header (dictionary compression) 128--[no-]check : integrity check (default:enabled) 129 -r : operate recursively on directories 130--format=gzip : compress files to the .gz format 131--format=xz : compress files to the .xz format 132--format=lzma : compress files to the .lzma format 133--test : test compressed file integrity 134--[no-]sparse : sparse mode (default:disabled) 135 -M# : Set a memory usage limit for decompression 136-- : All arguments after "--" are treated as files 137 138Dictionary builder : 139--train ## : create a dictionary from a training set of files 140--train-cover[=k=#,d=#,steps=#] : use the cover algorithm with optional args 141--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9) 142 -o file : `file` is dictionary name (default: dictionary) 143--maxdict=# : limit dictionary to specified size (default : 112640) 144--dictID=# : force dictionary ID to specified value (default: random) 145 146Benchmark arguments : 147 -b# : benchmark file(s), using # compression level (default : 1) 148 -e# : test all compression levels from -bX to # (default: 1) 149 -i# : minimum evaluation time in seconds (default : 3s) 150 -B# : cut file into independent blocks of size # (default: no block) 151--priority=rt : set process priority to real-time 152``` 153 154 155#### Long distance matching mode 156The long distance matching mode, enabled with `--long`, is designed to improve 157the compression ratio for files with long matches at a large distance (up to the 158maximum window size, `128 MiB`) while still maintaining compression speed. 159 160Enabling this mode sets the window size to `128 MiB` and thus increases the memory 161usage for both the compressor and decompressor. Performance in terms of speed is 162dependent on long matches being found. Compression speed may degrade if few long 163matches are found. Decompression speed usually improves when there are many long 164distance matches. 165 166Below are graphs comparing the compression speed, compression ratio, and 167decompression speed with and without long distance matching on an ideal use 168case: a tar of four versions of clang (versions `3.4.1`, `3.4.2`, `3.5.0`, 169`3.5.1`) with a total size of `244889600 B`. This is an ideal use case as there 170are many long distance matches within the maximum window size of `128 MiB` (each 171version is less than `128 MiB`). 172 173Compression Speed vs Ratio | Decompression Speed 174---------------------------|--------------------- 175 |  176 177| Method | Compression ratio | Compression speed | Decompression speed | 178|:-------|------------------:|-------------------------:|---------------------------:| 179| `zstd -1` | `5.065` | `284.8 MB/s` | `759.3 MB/s` | 180| `zstd -5` | `5.826` | `124.9 MB/s` | `674.0 MB/s` | 181| `zstd -10` | `6.504` | `29.5 MB/s` | `771.3 MB/s` | 182| `zstd -1 --long` | `17.426` | `220.6 MB/s` | `1638.4 MB/s` | 183| `zstd -5 --long` | `19.661` | `165.5 MB/s` | `1530.6 MB/s`| 184| `zstd -10 --long`| `21.949` | `75.6 MB/s` | `1632.6 MB/s`| 185 186On this file, the compression ratio improves significantly with minimal impact 187on compression speed, and the decompression speed doubles. 188 189On the other extreme, compressing a file with few long distance matches (such as 190the [Silesia compression corpus]) will likely lead to a deterioration in 191compression speed (for lower levels) with minimal change in compression ratio. 192 193The below table illustrates this on the [Silesia compression corpus]. 194 195[Silesia compression corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia 196 197| Method | Compression ratio | Compression speed | Decompression speed | 198|:-------|------------------:|-------------------------:|---------------------------:| 199| `zstd -1` | `2.878` | `231.7 MB/s` | `594.4 MB/s` | 200| `zstd -1 --long` | `2.929` | `106.5 MB/s` | `517.9 MB/s` | 201| `zstd -5` | `3.274` | `77.1 MB/s` | `464.2 MB/s` | 202| `zstd -5 --long` | `3.319` | `51.7 MB/s` | `371.9 MB/s` | 203| `zstd -10` | `3.523` | `16.4 MB/s` | `489.2 MB/s` | 204| `zstd -10 --long`| `3.566` | `16.2 MB/s` | `415.7 MB/s` | 205 206 207 208 209 210
