1zstd(1) -- zstd, zstdmt, unzstd, zstdcat - Compress or decompress .zst files 2============================================================================ 3 4SYNOPSIS 5-------- 6 7`zstd` [*OPTIONS*] [-|_INPUT-FILE_] [-o _OUTPUT-FILE_] 8 9`zstdmt` is equivalent to `zstd -T0` 10 11`unzstd` is equivalent to `zstd -d` 12 13`zstdcat` is equivalent to `zstd -dcf` 14 15 16DESCRIPTION 17----------- 18`zstd` is a fast lossless compression algorithm and data compression tool, 19with command line syntax similar to `gzip (1)` and `xz (1)`. 20It is based on the **LZ77** family, with further FSE & huff0 entropy stages. 21`zstd` offers highly configurable compression speed, 22with fast modes at > 200 MB/s per core, 23and strong modes nearing lzma compression ratios. 24It also features a very fast decoder, with speeds > 500 MB/s per core. 25 26`zstd` command line syntax is generally similar to gzip, 27but features the following differences : 28 29 - Source files are preserved by default. 30 It's possible to remove them automatically by using the `--rm` command. 31 - When compressing a single file, `zstd` displays progress notifications 32 and result summary by default. 33 Use `-q` to turn them off. 34 - `zstd` does not accept input from console, 35 but it properly accepts `stdin` when it's not the console. 36 - `zstd` displays a short help page when command line is an error. 37 Use `-q` to turn it off. 38 39`zstd` compresses or decompresses each _file_ according to the selected 40operation mode. 41If no _files_ are given or _file_ is `-`, `zstd` reads from standard input 42and writes the processed data to standard output. 43`zstd` will refuse to write compressed data to standard output 44if it is a terminal : it will display an error message and skip the _file_. 45Similarly, `zstd` will refuse to read compressed data from standard input 46if it is a terminal. 47 48Unless `--stdout` or `-o` is specified, _files_ are written to a new file 49whose name is derived from the source _file_ name: 50 51* When compressing, the suffix `.zst` is appended to the source filename to 52 get the target filename. 53* When decompressing, the `.zst` suffix is removed from the source filename to 54 get the target filename 55 56### Concatenation with .zst files 57It is possible to concatenate `.zst` files as is. 58`zstd` will decompress such files as if they were a single `.zst` file. 59 60OPTIONS 61------- 62 63### Integer suffixes and special values 64In most places where an integer argument is expected, 65an optional suffix is supported to easily indicate large integers. 66There must be no space between the integer and the suffix. 67 68* `KiB`: 69 Multiply the integer by 1,024 (2\^10). 70 `Ki`, `K`, and `KB` are accepted as synonyms for `KiB`. 71* `MiB`: 72 Multiply the integer by 1,048,576 (2\^20). 73 `Mi`, `M`, and `MB` are accepted as synonyms for `MiB`. 74 75### Operation mode 76If multiple operation mode options are given, 77the last one takes effect. 78 79* `-z`, `--compress`: 80 Compress. 81 This is the default operation mode when no operation mode option is specified 82 and no other operation mode is implied from the command name 83 (for example, `unzstd` implies `--decompress`). 84* `-d`, `--decompress`, `--uncompress`: 85 Decompress. 86* `-t`, `--test`: 87 Test the integrity of compressed _files_. 88 This option is equivalent to `--decompress --stdout` except that the 89 decompressed data is discarded instead of being written to standard output. 90 No files are created or removed. 91* `-b#`: 92 Benchmark file(s) using compression level # 93* `--train FILEs`: 94 Use FILEs as a training set to create a dictionary. 95 The training set should contain a lot of small files (> 100). 96* `-l`, `--list`: 97 Display information related to a zstd compressed file, such as size, ratio, and checksum. 98 Some of these fields may not be available. 99 This command can be augmented with the `-v` modifier. 100 101### Operation modifiers 102 103* `-#`: 104 `#` compression level \[1-19] (default: 3) 105* `--fast[=#]`: 106 switch to ultra-fast compression levels. 107 If `=#` is not present, it defaults to `1`. 108 The higher the value, the faster the compression speed, 109 at the cost of some compression ratio. 110 This setting overwrites compression level if one was set previously. 111 Similarly, if a compression level is set after `--fast`, it overrides it. 112* `--ultra`: 113 unlocks high compression levels 20+ (maximum 22), using a lot more memory. 114 Note that decompression will also require more memory when using these levels. 115* `--long[=#]`: 116 enables long distance matching with `#` `windowLog`, if not `#` is not 117 present it defaults to `27`. 118 This increases the window size (`windowLog`) and memory usage for both the 119 compressor and decompressor. 120 This setting is designed to improve the compression ratio for files with 121 long matches at a large distance. 122 123 Note: If `windowLog` is set to larger than 27, `--long=windowLog` or 124 `--memory=windowSize` needs to be passed to the decompressor. 125* `-T#`, `--threads=#`: 126 Compress using `#` working threads (default: 1). 127 If `#` is 0, attempt to detect and use the number of physical CPU cores. 128 In all cases, the nb of threads is capped to ZSTDMT_NBTHREADS_MAX==200. 129 This modifier does nothing if `zstd` is compiled without multithread support. 130* `--single-thread`: 131 Does not spawn a thread for compression, use a single thread for both I/O and compression. 132 In this mode, compression is serialized with I/O, which is slightly slower. 133 (This is different from `-T1`, which spawns 1 compression thread in parallel of I/O). 134 This mode is the only one available when multithread support is disabled. 135 Single-thread mode features lower memory usage. 136 Final compressed result is slightly different from `-T1`. 137* `--adapt[=min=#,max=#]` : 138 `zstd` will dynamically adapt compression level to perceived I/O conditions. 139 Compression level adaptation can be observed live by using command `-v`. 140 Adaptation can be constrained between supplied `min` and `max` levels. 141 The feature works when combined with multi-threading and `--long` mode. 142 It does not work with `--single-thread`. 143 It sets window size to 8 MB by default (can be changed manually, see `wlog`). 144 Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible. 145 _note_ : at the time of this writing, `--adapt` can remain stuck at low speed 146 when combined with multiple worker threads (>=2). 147* `--rsyncable` : 148 `zstd` will periodically synchronize the compression state to make the 149 compressed file more rsync-friendly. There is a negligible impact to 150 compression ratio, and the faster compression levels will see a small 151 compression speed hit. 152 This feature does not work with `--single-thread`. You probably don't want 153 to use it with long range mode, since it will decrease the effectiveness of 154 the synchronization points, but your milage may vary. 155* `-D file`: 156 use `file` as Dictionary to compress or decompress FILE(s) 157* `--no-dictID`: 158 do not store dictionary ID within frame header (dictionary compression). 159 The decoder will have to rely on implicit knowledge about which dictionary to use, 160 it won't be able to check if it's correct. 161* `-o file`: 162 save result into `file` (only possible with a single _INPUT-FILE_) 163* `-f`, `--force`: 164 overwrite output without prompting, and (de)compress symbolic links 165* `-c`, `--stdout`: 166 force write to standard output, even if it is the console 167* `--[no-]sparse`: 168 enable / disable sparse FS support, 169 to make files with many zeroes smaller on disk. 170 Creating sparse files may save disk space and speed up decompression by 171 reducing the amount of disk I/O. 172 default: enabled when output is into a file, 173 and disabled when output is stdout. 174 This setting overrides default and can force sparse mode over stdout. 175* `--rm`: 176 remove source file(s) after successful compression or decompression 177* `-k`, `--keep`: 178 keep source file(s) after successful compression or decompression. 179 This is the default behavior. 180* `-r`: 181 operate recursively on directories 182* `--format=FORMAT`: 183 compress and decompress in other formats. If compiled with 184 support, zstd can compress to or decompress from other compression algorithm 185 formats. Possibly available options are `zstd`, `gzip`, `xz`, `lzma`, and `lz4`. 186 If no such format is provided, `zstd` is the default. 187* `-h`/`-H`, `--help`: 188 display help/long help and exit 189* `-V`, `--version`: 190 display version number and exit. 191 Advanced : `-vV` also displays supported formats. 192 `-vvV` also displays POSIX support. 193* `-v`: 194 verbose mode 195* `-q`, `--quiet`: 196 suppress warnings, interactivity, and notifications. 197 specify twice to suppress errors too. 198* `--no-progress`: 199 do not display the progress bar, but keep all other messages. 200* `-C`, `--[no-]check`: 201 add integrity check computed from uncompressed data (default: enabled) 202* `--`: 203 All arguments after `--` are treated as files 204 205 206DICTIONARY BUILDER 207------------------ 208`zstd` offers _dictionary_ compression, 209which greatly improves efficiency on small files and messages. 210It's possible to train `zstd` with a set of samples, 211the result of which is saved into a file called a `dictionary`. 212Then during compression and decompression, reference the same dictionary, 213using command `-D dictionaryFileName`. 214Compression of small files similar to the sample set will be greatly improved. 215 216* `--train FILEs`: 217 Use FILEs as training set to create a dictionary. 218 The training set should contain a lot of small files (> 100), 219 and weight typically 100x the target dictionary size 220 (for example, 10 MB for a 100 KB dictionary). 221 222 Supports multithreading if `zstd` is compiled with threading support. 223 Additional parameters can be specified with `--train-fastcover`. 224 The legacy dictionary builder can be accessed with `--train-legacy`. 225 The cover dictionary builder can be accessed with `--train-cover`. 226 Equivalent to `--train-fastcover=d=8,steps=4`. 227* `-o file`: 228 Dictionary saved into `file` (default name: dictionary). 229* `--maxdict=#`: 230 Limit dictionary to specified size (default: 112640). 231* `-#`: 232 Use `#` compression level during training (optional). 233 Will generate statistics more tuned for selected compression level, 234 resulting in a _small_ compression ratio improvement for this level. 235* `-B#`: 236 Split input files in blocks of size # (default: no split) 237* `--dictID=#`: 238 A dictionary ID is a locally unique ID that a decoder can use to verify it is 239 using the right dictionary. 240 By default, zstd will create a 4-bytes random number ID. 241 It's possible to give a precise number instead. 242 Short numbers have an advantage : an ID < 256 will only need 1 byte in the 243 compressed frame header, and an ID < 65536 will only need 2 bytes. 244 This compares favorably to 4 bytes default. 245 However, it's up to the dictionary manager to not assign twice the same ID to 246 2 different dictionaries. 247* `--train-cover[=k#,d=#,steps=#,split=#]`: 248 Select parameters for the default dictionary builder algorithm named cover. 249 If _d_ is not specified, then it tries _d_ = 6 and _d_ = 8. 250 If _k_ is not specified, then it tries _steps_ values in the range [50, 2000]. 251 If _steps_ is not specified, then the default value of 40 is used. 252 If _split_ is not specified or split <= 0, then the default value of 100 is used. 253 Requires that _d_ <= _k_. 254 255 Selects segments of size _k_ with highest score to put in the dictionary. 256 The score of a segment is computed by the sum of the frequencies of all the 257 subsegments of size _d_. 258 Generally _d_ should be in the range [6, 8], occasionally up to 16, but the 259 algorithm will run faster with d <= _8_. 260 Good values for _k_ vary widely based on the input data, but a safe range is 261 [2 * _d_, 2000]. 262 If _split_ is 100, all input samples are used for both training and testing 263 to find optimal _d_ and _k_ to build dictionary. 264 Supports multithreading if `zstd` is compiled with threading support. 265 266 Examples: 267 268 `zstd --train-cover FILEs` 269 270 `zstd --train-cover=k=50,d=8 FILEs` 271 272 `zstd --train-cover=d=8,steps=500 FILEs` 273 274 `zstd --train-cover=k=50 FILEs` 275 276 `zstd --train-cover=k=50,split=60 FILEs` 277 278* `--train-fastcover[=k#,d=#,f=#,steps=#,split=#,accel=#]`: 279 Same as cover but with extra parameters _f_ and _accel_ and different default value of split 280 If _split_ is not specified, then it tries _split_ = 75. 281 If _f_ is not specified, then it tries _f_ = 20. 282 Requires that 0 < _f_ < 32. 283 If _accel_ is not specified, then it tries _accel_ = 1. 284 Requires that 0 < _accel_ <= 10. 285 Requires that _d_ = 6 or _d_ = 8. 286 287 _f_ is log of size of array that keeps track of frequency of subsegments of size _d_. 288 The subsegment is hashed to an index in the range [0,2^_f_ - 1]. 289 It is possible that 2 different subsegments are hashed to the same index, and they are considered as the same subsegment when computing frequency. 290 Using a higher _f_ reduces collision but takes longer. 291 292 Examples: 293 294 `zstd --train-fastcover FILEs` 295 296 `zstd --train-fastcover=d=8,f=15,accel=2 FILEs` 297 298* `--train-legacy[=selectivity=#]`: 299 Use legacy dictionary builder algorithm with the given dictionary 300 _selectivity_ (default: 9). 301 The smaller the _selectivity_ value, the denser the dictionary, 302 improving its efficiency but reducing its possible maximum size. 303 `--train-legacy=s=#` is also accepted. 304 305 Examples: 306 307 `zstd --train-legacy FILEs` 308 309 `zstd --train-legacy=selectivity=8 FILEs` 310 311 312BENCHMARK 313--------- 314 315* `-b#`: 316 benchmark file(s) using compression level # 317* `-e#`: 318 benchmark file(s) using multiple compression levels, from `-b#` to `-e#` (inclusive) 319* `-i#`: 320 minimum evaluation time, in seconds (default: 3s), benchmark mode only 321* `-B#`, `--block-size=#`: 322 cut file(s) into independent blocks of size # (default: no block) 323* `--priority=rt`: 324 set process priority to real-time 325 326**Output Format:** CompressionLevel#Filename : IntputSize -> OutputSize (CompressionRatio), CompressionSpeed, DecompressionSpeed 327 328**Methodology:** For both compression and decompression speed, the entire input is compressed/decompressed in-memory to measure speed. A run lasts at least 1 sec, so when files are small, they are compressed/decompressed several times per run, in order to improve measurement accuracy. 329 330ADVANCED COMPRESSION OPTIONS 331---------------------------- 332### --zstd[=options]: 333`zstd` provides 22 predefined compression levels. 334The selected or default predefined compression level can be changed with 335advanced compression options. 336The _options_ are provided as a comma-separated list. 337You may specify only the options you want to change and the rest will be 338taken from the selected or default compression level. 339The list of available _options_: 340 341- `strategy`=_strat_, `strat`=_strat_: 342 Specify a strategy used by a match finder. 343 344 There are 9 strategies numbered from 1 to 9, from faster to stronger: 345 1=ZSTD\_fast, 2=ZSTD\_dfast, 3=ZSTD\_greedy, 346 4=ZSTD\_lazy, 5=ZSTD\_lazy2, 6=ZSTD\_btlazy2, 347 7=ZSTD\_btopt, 8=ZSTD\_btultra, 9=ZSTD\_btultra2. 348 349- `windowLog`=_wlog_, `wlog`=_wlog_: 350 Specify the maximum number of bits for a match distance. 351 352 The higher number of increases the chance to find a match which usually 353 improves compression ratio. 354 It also increases memory requirements for the compressor and decompressor. 355 The minimum _wlog_ is 10 (1 KiB) and the maximum is 30 (1 GiB) on 32-bit 356 platforms and 31 (2 GiB) on 64-bit platforms. 357 358 Note: If `windowLog` is set to larger than 27, `--long=windowLog` or 359 `--memory=windowSize` needs to be passed to the decompressor. 360 361- `hashLog`=_hlog_, `hlog`=_hlog_: 362 Specify the maximum number of bits for a hash table. 363 364 Bigger hash tables cause less collisions which usually makes compression 365 faster, but requires more memory during compression. 366 367 The minimum _hlog_ is 6 (64 B) and the maximum is 26 (128 MiB). 368 369- `chainLog`=_clog_, `clog`=_clog_: 370 Specify the maximum number of bits for a hash chain or a binary tree. 371 372 Higher numbers of bits increases the chance to find a match which usually 373 improves compression ratio. 374 It also slows down compression speed and increases memory requirements for 375 compression. 376 This option is ignored for the ZSTD_fast strategy. 377 378 The minimum _clog_ is 6 (64 B) and the maximum is 28 (256 MiB). 379 380- `searchLog`=_slog_, `slog`=_slog_: 381 Specify the maximum number of searches in a hash chain or a binary tree 382 using logarithmic scale. 383 384 More searches increases the chance to find a match which usually increases 385 compression ratio but decreases compression speed. 386 387 The minimum _slog_ is 1 and the maximum is 26. 388 389- `minMatch`=_mml_, `mml`=_mml_: 390 Specify the minimum searched length of a match in a hash table. 391 392 Larger search lengths usually decrease compression ratio but improve 393 decompression speed. 394 395 The minimum _mml_ is 3 and the maximum is 7. 396 397- `targetLen`=_tlen_, `tlen`=_tlen_: 398 The impact of this field vary depending on selected strategy. 399 400 For ZSTD\_btopt, ZSTD\_btultra and ZSTD\_btultra2, it specifies 401 the minimum match length that causes match finder to stop searching. 402 A larger `targetLen` usually improves compression ratio 403 but decreases compression speed. 404 405 For ZSTD\_fast, it triggers ultra-fast mode when > 0. 406 The value represents the amount of data skipped between match sampling. 407 Impact is reversed : a larger `targetLen` increases compression speed 408 but decreases compression ratio. 409 410 For all other strategies, this field has no impact. 411 412 The minimum _tlen_ is 0 and the maximum is 999. 413 414- `overlapLog`=_ovlog_, `ovlog`=_ovlog_: 415 Determine `overlapSize`, amount of data reloaded from previous job. 416 This parameter is only available when multithreading is enabled. 417 Reloading more data improves compression ratio, but decreases speed. 418 419 The minimum _ovlog_ is 0, and the maximum is 9. 420 1 means "no overlap", hence completely independent jobs. 421 9 means "full overlap", meaning up to `windowSize` is reloaded from previous job. 422 Reducing _ovlog_ by 1 reduces the reloaded amount by a factor 2. 423 For example, 8 means "windowSize/2", and 6 means "windowSize/8". 424 Value 0 is special and means "default" : _ovlog_ is automatically determined by `zstd`. 425 In which case, _ovlog_ will range from 6 to 9, depending on selected _strat_. 426 427- `ldmHashLog`=_lhlog_, `lhlog`=_lhlog_: 428 Specify the maximum size for a hash table used for long distance matching. 429 430 This option is ignored unless long distance matching is enabled. 431 432 Bigger hash tables usually improve compression ratio at the expense of more 433 memory during compression and a decrease in compression speed. 434 435 The minimum _lhlog_ is 6 and the maximum is 26 (default: 20). 436 437- `ldmMinMatch`=_lmml_, `lmml`=_lmml_: 438 Specify the minimum searched length of a match for long distance matching. 439 440 This option is ignored unless long distance matching is enabled. 441 442 Larger/very small values usually decrease compression ratio. 443 444 The minimum _lmml_ is 4 and the maximum is 4096 (default: 64). 445 446- `ldmBucketSizeLog`=_lblog_, `lblog`=_lblog_: 447 Specify the size of each bucket for the hash table used for long distance 448 matching. 449 450 This option is ignored unless long distance matching is enabled. 451 452 Larger bucket sizes improve collision resolution but decrease compression 453 speed. 454 455 The minimum _lblog_ is 0 and the maximum is 8 (default: 3). 456 457- `ldmHashRateLog`=_lhrlog_, `lhrlog`=_lhrlog_: 458 Specify the frequency of inserting entries into the long distance matching 459 hash table. 460 461 This option is ignored unless long distance matching is enabled. 462 463 Larger values will improve compression speed. Deviating far from the 464 default value will likely result in a decrease in compression ratio. 465 466 The default value is `wlog - lhlog`. 467 468### Example 469The following parameters sets advanced compression options to something 470similar to predefined level 19 for files bigger than 256 KB: 471 472`--zstd`=wlog=23,clog=23,hlog=22,slog=6,mml=3,tlen=48,strat=6 473 474### -B#: 475Select the size of each compression job. 476This parameter is available only when multi-threading is enabled. 477Default value is `4 * windowSize`, which means it varies depending on compression level. 478`-B#` makes it possible to select a custom value. 479Note that job size must respect a minimum value which is enforced transparently. 480This minimum is either 1 MB, or `overlapSize`, whichever is largest. 481 482BUGS 483---- 484Report bugs at: https://github.com/facebook/zstd/issues 485 486AUTHOR 487------ 488Yann Collet 489