1Command Line Interface for Zstandard library 2============================================ 3 4Command Line Interface (CLI) can be created using the `make` command without any additional parameters. 5There are however other Makefile targets that create different variations of CLI: 6- `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd formats 7- `zstd_nolegacy` : Same as `zstd` but without support for legacy zstd formats 8- `zstd-small` : CLI optimized for minimal size; no dictionary builder, no benchmark, and no support for legacy zstd formats 9- `zstd-compress` : version of CLI which can only compress into zstd format 10- `zstd-decompress` : version of CLI which can only decompress zstd format 11 12 13#### Compilation variables 14`zstd` scope can be altered by modifying the following `make` variables : 15 16- __HAVE_THREAD__ : multithreading is automatically enabled when `pthread` is detected. 17 It's possible to disable multithread support, by setting `HAVE_THREAD=0`. 18 Example : `make zstd HAVE_THREAD=0` 19 It's also possible to force multithread support, using `HAVE_THREAD=1`. 20 In which case, linking stage will fail if neither `pthread` nor `windows.h` library can be found. 21 This is useful to ensure this feature is not silently disabled. 22 23- __ZSTD_LEGACY_SUPPORT__ : `zstd` can decompress files compressed by older versions of `zstd`. 24 Starting v0.8.0, all versions of `zstd` produce frames compliant with the [specification](../doc/zstd_compression_format.md), and are therefore compatible. 25 But older versions (< v0.8.0) produced different, incompatible, frames. 26 By default, `zstd` supports decoding legacy formats >= v0.4.0 (`ZSTD_LEGACY_SUPPORT=4`). 27 This can be altered by modifying this compilation variable. 28 `ZSTD_LEGACY_SUPPORT=1` means "support all formats >= v0.1.0". 29 `ZSTD_LEGACY_SUPPORT=2` means "support all formats >= v0.2.0", and so on. 30 `ZSTD_LEGACY_SUPPORT=0` means _DO NOT_ support any legacy format. 31 if `ZSTD_LEGACY_SUPPORT >= 8`, it's the same as `0`, since there is no legacy format after `7`. 32 Note : `zstd` only supports decoding older formats, and cannot generate any legacy format. 33 34- __HAVE_ZLIB__ : `zstd` can compress and decompress files in `.gz` format. 35 This is ordered through command `--format=gzip`. 36 Alternatively, symlinks named `gzip` or `gunzip` will mimic intended behavior. 37 `.gz` support is automatically enabled when `zlib` library is detected at build time. 38 It's possible to disable `.gz` support, by setting `HAVE_ZLIB=0`. 39 Example : `make zstd HAVE_ZLIB=0` 40 It's also possible to force compilation with zlib support, `using HAVE_ZLIB=1`. 41 In which case, linking stage will fail if `zlib` library cannot be found. 42 This is useful to prevent silent feature disabling. 43 44- __HAVE_LZMA__ : `zstd` can compress and decompress files in `.xz` and `.lzma` formats. 45 This is ordered through commands `--format=xz` and `--format=lzma` respectively. 46 Alternatively, symlinks named `xz`, `unxz`, `lzma`, or `unlzma` will mimic intended behavior. 47 `.xz` and `.lzma` support is automatically enabled when `lzma` library is detected at build time. 48 It's possible to disable `.xz` and `.lzma` support, by setting `HAVE_LZMA=0` . 49 Example : `make zstd HAVE_LZMA=0` 50 It's also possible to force compilation with lzma support, using `HAVE_LZMA=1`. 51 In which case, linking stage will fail if `lzma` library cannot be found. 52 This is useful to prevent silent feature disabling. 53 54- __HAVE_LZ4__ : `zstd` can compress and decompress files in `.lz4` formats. 55 This is ordered through commands `--format=lz4`. 56 Alternatively, symlinks named `lz4`, or `unlz4` will mimic intended behavior. 57 `.lz4` support is automatically enabled when `lz4` library is detected at build time. 58 It's possible to disable `.lz4` support, by setting `HAVE_LZ4=0` . 59 Example : `make zstd HAVE_LZ4=0` 60 It's also possible to force compilation with lz4 support, using `HAVE_LZ4=1`. 61 In which case, linking stage will fail if `lz4` library cannot be found. 62 This is useful to prevent silent feature disabling. 63 64- __BACKTRACE__ : `zstd` can display a stack backtrace when execution 65 generates a runtime exception. By default, this feature may be 66 degraded/disabled on some platforms unless additional compiler directives are 67 applied. When triaging a runtime issue, enabling this feature can provide 68 more context to determine the location of the fault. 69 Example : `make zstd BACKTRACE=1` 70 71 72#### Aggregation of parameters 73CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`. 74 75 76#### Symlink shortcuts 77It's possible to invoke `zstd` through a symlink. 78When the name of the symlink has a specific value, it triggers an associated behavior. 79- `zstdmt` : compress using all cores available on local system. 80- `zcat` : will decompress and output target file using any of the supported formats. `gzcat` and `zstdcat` are also equivalent. 81- `gzip` : if zlib support is enabled, will mimic `gzip` by compressing file using `.gz` format, removing source file by default (use `--keep` to preserve). If zlib is not supported, triggers an error. 82- `xz` : if lzma support is enabled, will mimic `xz` by compressing file using `.xz` format, removing source file by default (use `--keep` to preserve). If xz is not supported, triggers an error. 83- `lzma` : if lzma support is enabled, will mimic `lzma` by compressing file using `.lzma` format, removing source file by default (use `--keep` to preserve). If lzma is not supported, triggers an error. 84- `lz4` : if lz4 support is enabled, will mimic `lz4` by compressing file using `.lz4` format. If lz4 is not supported, triggers an error. 85- `unzstd` and `unlz4` will decompress any of the supported format. 86- `ungz`, `unxz` and `unlzma` will do the same, and will also remove source file by default (use `--keep` to preserve). 87 88 89#### Dictionary builder in Command Line Interface 90Zstd offers a training mode, which can be used to tune the algorithm for a selected 91type of data, by providing it with a few samples. The result of the training is stored 92in a file selected with the `-o` option (default name is `dictionary`), 93which can be loaded before compression and decompression. 94 95Using a dictionary, the compression ratio achievable on small data improves dramatically. 96These compression gains are achieved while simultaneously providing faster compression and decompression speeds. 97Dictionary work if there is some correlation in a family of small data (there is no universal dictionary). 98Hence, deploying one dictionary per type of data will provide the greater benefits. 99Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm 100will rely more and more on previously decoded content to compress the rest of the file. 101 102Usage of the dictionary builder and created dictionaries with CLI: 103 1041. Create the dictionary : `zstd --train PathToTrainingSet/* -o dictionaryName` 1052. Compress with the dictionary: `zstd FILE -D dictionaryName` 1063. Decompress with the dictionary: `zstd --decompress FILE.zst -D dictionaryName` 107 108 109#### Benchmark in Command Line Interface 110CLI includes in-memory compression benchmark module for zstd. 111The benchmark is conducted using given filenames. The files are read into memory and joined together. 112It makes benchmark more precise as it eliminates I/O overhead. 113Multiple filenames can be supplied, as multiple parameters, with wildcards, 114or names of directories can be used as parameters with `-r` option. 115 116The benchmark measures ratio, compressed size, compression and decompression speed. 117One can select compression levels starting from `-b` and ending with `-e`. 118The `-i` parameter selects minimal time used for each of tested levels. 119 120 121#### Usage of Command Line Interface 122The full list of options can be obtained with `-h` or `-H` parameter: 123``` 124Usage : 125 zstd [args] [FILE(s)] [-o file] 126 127FILE : a filename 128 with no FILE, or when FILE is - , read standard input 129Arguments : 130 -# : # compression level (1-19, default: 3) 131 -d : decompression 132 -D file: use `file` as Dictionary 133 -o file: result stored into `file` (only if 1 input file) 134 -f : overwrite output without prompting and (de)compress links 135--rm : remove source file(s) after successful de/compression 136 -k : preserve source file(s) (default) 137 -h/-H : display help/long help and exit 138 139Advanced arguments : 140 -V : display Version number and exit 141 -v : verbose mode; specify multiple times to increase verbosity 142 -q : suppress warnings; specify twice to suppress errors too 143 -c : force write to standard output, even if it is the console 144 -l : print information about zstd compressed files 145--ultra : enable levels beyond 19, up to 22 (requires more memory) 146--long : enable long distance matching (requires more memory) 147--no-dictID : don't write dictID into header (dictionary compression) 148--[no-]check : integrity check (default: enabled) 149 -r : operate recursively on directories 150--format=gzip : compress files to the .gz format 151--format=xz : compress files to the .xz format 152--format=lzma : compress files to the .lzma format 153--test : test compressed file integrity 154--[no-]sparse : sparse mode (default: disabled) 155 -M# : Set a memory usage limit for decompression 156-- : All arguments after "--" are treated as files 157 158Dictionary builder : 159--train ## : create a dictionary from a training set of files 160--train-cover[=k=#,d=#,steps=#,split=#] : use the cover algorithm with optional args 161--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] : use the fastcover algorithm with optional args 162--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9) 163 -o file : `file` is dictionary name (default: dictionary) 164--maxdict=# : limit dictionary to specified size (default: 112640) 165--dictID=# : force dictionary ID to specified value (default: random) 166 167Benchmark arguments : 168 -b# : benchmark file(s), using # compression level (default: 3) 169 -e# : test all compression levels from -bX to # (default: 1) 170 -i# : minimum evaluation time in seconds (default: 3s) 171 -B# : cut file into independent blocks of size # (default: no block) 172--priority=rt : set process priority to real-time 173``` 174 175#### Restricted usage of Environment Variables 176Using environment variables to set compression/decompression parameters has security implications. Therefore, 177we intentionally restrict its usage. Currently, only `ZSTD_CLEVEL` is supported for setting compression level. 178If the value of `ZSTD_CLEVEL` is not a valid integer, it will be ignored with a warning message. 179Note that command line options will override corresponding environment variable settings. 180 181#### Long distance matching mode 182The long distance matching mode, enabled with `--long`, is designed to improve 183the compression ratio for files with long matches at a large distance (up to the 184maximum window size, `128 MiB`) while still maintaining compression speed. 185 186Enabling this mode sets the window size to `128 MiB` and thus increases the memory 187usage for both the compressor and decompressor. Performance in terms of speed is 188dependent on long matches being found. Compression speed may degrade if few long 189matches are found. Decompression speed usually improves when there are many long 190distance matches. 191 192Below are graphs comparing the compression speed, compression ratio, and 193decompression speed with and without long distance matching on an ideal use 194case: a tar of four versions of clang (versions `3.4.1`, `3.4.2`, `3.5.0`, 195`3.5.1`) with a total size of `244889600 B`. This is an ideal use case as there 196are many long distance matches within the maximum window size of `128 MiB` (each 197version is less than `128 MiB`). 198 199Compression Speed vs Ratio | Decompression Speed 200---------------------------|--------------------- 201 |  202 203| Method | Compression ratio | Compression speed | Decompression speed | 204|:-------|------------------:|-------------------------:|---------------------------:| 205| `zstd -1` | `5.065` | `284.8 MB/s` | `759.3 MB/s` | 206| `zstd -5` | `5.826` | `124.9 MB/s` | `674.0 MB/s` | 207| `zstd -10` | `6.504` | `29.5 MB/s` | `771.3 MB/s` | 208| `zstd -1 --long` | `17.426` | `220.6 MB/s` | `1638.4 MB/s` | 209| `zstd -5 --long` | `19.661` | `165.5 MB/s` | `1530.6 MB/s`| 210| `zstd -10 --long`| `21.949` | `75.6 MB/s` | `1632.6 MB/s`| 211 212On this file, the compression ratio improves significantly with minimal impact 213on compression speed, and the decompression speed doubles. 214 215On the other extreme, compressing a file with few long distance matches (such as 216the [Silesia compression corpus]) will likely lead to a deterioration in 217compression speed (for lower levels) with minimal change in compression ratio. 218 219The below table illustrates this on the [Silesia compression corpus]. 220 221[Silesia compression corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia 222 223| Method | Compression ratio | Compression speed | Decompression speed | 224|:-------|------------------:|------------------:|---------------------:| 225| `zstd -1` | `2.878` | `231.7 MB/s` | `594.4 MB/s` | 226| `zstd -1 --long` | `2.929` | `106.5 MB/s` | `517.9 MB/s` | 227| `zstd -5` | `3.274` | `77.1 MB/s` | `464.2 MB/s` | 228| `zstd -5 --long` | `3.319` | `51.7 MB/s` | `371.9 MB/s` | 229| `zstd -10` | `3.523` | `16.4 MB/s` | `489.2 MB/s` | 230| `zstd -10 --long`| `3.566` | `16.2 MB/s` | `415.7 MB/s` | 231 232 233#### zstdgrep 234 235`zstdgrep` is a utility which makes it possible to `grep` directly a `.zst` compressed file. 236It's used the same way as normal `grep`, for example : 237`zstdgrep pattern file.zst` 238 239`zstdgrep` is _not_ compatible with dictionary compression. 240 241To search into a file compressed with a dictionary, 242it's necessary to decompress it using `zstd` or `zstdcat`, 243and then pipe the result to `grep`. For example : 244`zstdcat -D dictionary -qc -- file.zst | grep pattern` 245
