10c16b537SWarner Losh<html> 20c16b537SWarner Losh<head> 30c16b537SWarner Losh<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> 4*5ff13fbcSAllan Jude<title>zstd 1.5.2 Manual</title> 50c16b537SWarner Losh</head> 60c16b537SWarner Losh<body> 7*5ff13fbcSAllan Jude<h1>zstd 1.5.2 Manual</h1> 80c16b537SWarner Losh<hr> 90c16b537SWarner Losh<a name="Contents"></a><h2>Contents</h2> 100c16b537SWarner Losh<ol> 110c16b537SWarner Losh<li><a href="#Chapter1">Introduction</a></li> 120c16b537SWarner Losh<li><a href="#Chapter2">Version</a></li> 134d3f1eafSConrad Meyer<li><a href="#Chapter3">Simple API</a></li> 144d3f1eafSConrad Meyer<li><a href="#Chapter4">Explicit context</a></li> 15*5ff13fbcSAllan Jude<li><a href="#Chapter5">Advanced compression API (Requires v1.4.0+)</a></li> 16*5ff13fbcSAllan Jude<li><a href="#Chapter6">Advanced decompression API (Requires v1.4.0+)</a></li> 174d3f1eafSConrad Meyer<li><a href="#Chapter7">Streaming</a></li> 184d3f1eafSConrad Meyer<li><a href="#Chapter8">Streaming compression - HowTo</a></li> 194d3f1eafSConrad Meyer<li><a href="#Chapter9">Streaming decompression - HowTo</a></li> 204d3f1eafSConrad Meyer<li><a href="#Chapter10">Simple dictionary API</a></li> 214d3f1eafSConrad Meyer<li><a href="#Chapter11">Bulk processing dictionary API</a></li> 224d3f1eafSConrad Meyer<li><a href="#Chapter12">Dictionary helper functions</a></li> 23*5ff13fbcSAllan Jude<li><a href="#Chapter13">Advanced dictionary and prefix API (Requires v1.4.0+)</a></li> 244d3f1eafSConrad Meyer<li><a href="#Chapter14">experimental API (static linking only)</a></li> 254d3f1eafSConrad Meyer<li><a href="#Chapter15">Frame size functions</a></li> 264d3f1eafSConrad Meyer<li><a href="#Chapter16">Memory management</a></li> 274d3f1eafSConrad Meyer<li><a href="#Chapter17">Advanced compression functions</a></li> 284d3f1eafSConrad Meyer<li><a href="#Chapter18">Advanced decompression functions</a></li> 294d3f1eafSConrad Meyer<li><a href="#Chapter19">Advanced streaming functions</a></li> 30f7cd7fe5SConrad Meyer<li><a href="#Chapter20">Buffer-less and synchronous inner streaming functions</a></li> 31f7cd7fe5SConrad Meyer<li><a href="#Chapter21">Buffer-less streaming compression (synchronous mode)</a></li> 32f7cd7fe5SConrad Meyer<li><a href="#Chapter22">Buffer-less streaming decompression (synchronous mode)</a></li> 33f7cd7fe5SConrad Meyer<li><a href="#Chapter23">Block level API</a></li> 340c16b537SWarner Losh</ol> 350c16b537SWarner Losh<hr> 360c16b537SWarner Losh<a name="Chapter1"></a><h2>Introduction</h2><pre> 370f743729SConrad Meyer zstd, short for Zstandard, is a fast lossless compression algorithm, targeting 380f743729SConrad Meyer real-time compression scenarios at zlib-level and better compression ratios. 390f743729SConrad Meyer The zstd compression library provides in-memory compression and decompression 400f743729SConrad Meyer functions. 410f743729SConrad Meyer 420f743729SConrad Meyer The library supports regular compression levels from 1 up to ZSTD_maxCLevel(), 430f743729SConrad Meyer which is currently 22. Levels >= 20, labeled `--ultra`, should be used with 440f743729SConrad Meyer caution, as they require more memory. The library also offers negative 450f743729SConrad Meyer compression levels, which extend the range of speed vs. ratio preferences. 460f743729SConrad Meyer The lower the level, the faster the speed (at the cost of compression). 470f743729SConrad Meyer 480c16b537SWarner Losh Compression can be done in: 490c16b537SWarner Losh - a single step (described as Simple API) 5019fcbaf1SConrad Meyer - a single step, reusing a context (described as Explicit context) 510c16b537SWarner Losh - unbounded multiple steps (described as Streaming compression) 520c16b537SWarner Losh 530f743729SConrad Meyer The compression ratio achievable on small data can be highly improved using 540f743729SConrad Meyer a dictionary. Dictionary compression can be performed in: 550f743729SConrad Meyer - a single step (described as Simple dictionary API) 560f743729SConrad Meyer - a single step, reusing a dictionary (described as Bulk-processing 570f743729SConrad Meyer dictionary API) 580f743729SConrad Meyer 590f743729SConrad Meyer Advanced experimental functions can be accessed using 600f743729SConrad Meyer `#define ZSTD_STATIC_LINKING_ONLY` before including zstd.h. 610f743729SConrad Meyer 620f743729SConrad Meyer Advanced experimental APIs should never be used with a dynamically-linked 630f743729SConrad Meyer library. They are not "stable"; their definitions or signatures may change in 640f743729SConrad Meyer the future. Only static linking is allowed. 650c16b537SWarner Losh<BR></pre> 660c16b537SWarner Losh 670c16b537SWarner Losh<a name="Chapter2"></a><h2>Version</h2><pre></pre> 680c16b537SWarner Losh 69f7cd7fe5SConrad Meyer<pre><b>unsigned ZSTD_versionNumber(void); 70f7cd7fe5SConrad Meyer</b><p> Return runtime library version, the value is (MAJOR*100*100 + MINOR*100 + RELEASE). 71f7cd7fe5SConrad Meyer</p></pre><BR> 72f7cd7fe5SConrad Meyer 73f7cd7fe5SConrad Meyer<pre><b>const char* ZSTD_versionString(void); 74f7cd7fe5SConrad Meyer</b><p> Return runtime library version, like "1.4.5". Requires v1.3.0+. 75f7cd7fe5SConrad Meyer</p></pre><BR> 76f7cd7fe5SConrad Meyer 774d3f1eafSConrad Meyer<a name="Chapter3"></a><h2>Simple API</h2><pre></pre> 780c16b537SWarner Losh 790c16b537SWarner Losh<pre><b>size_t ZSTD_compress( void* dst, size_t dstCapacity, 800c16b537SWarner Losh const void* src, size_t srcSize, 810c16b537SWarner Losh int compressionLevel); 820c16b537SWarner Losh</b><p> Compresses `src` content as a single zstd compressed frame into already allocated `dst`. 830c16b537SWarner Losh Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`. 840c16b537SWarner Losh @return : compressed size written into `dst` (<= `dstCapacity), 850c16b537SWarner Losh or an error code if it fails (which can be tested using ZSTD_isError()). 860c16b537SWarner Losh</p></pre><BR> 870c16b537SWarner Losh 880c16b537SWarner Losh<pre><b>size_t ZSTD_decompress( void* dst, size_t dstCapacity, 890c16b537SWarner Losh const void* src, size_t compressedSize); 900c16b537SWarner Losh</b><p> `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames. 910c16b537SWarner Losh `dstCapacity` is an upper bound of originalSize to regenerate. 920c16b537SWarner Losh If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data. 930c16b537SWarner Losh @return : the number of bytes decompressed into `dst` (<= `dstCapacity`), 940c16b537SWarner Losh or an errorCode if it fails (which can be tested using ZSTD_isError()). 950c16b537SWarner Losh</p></pre><BR> 960c16b537SWarner Losh 970c16b537SWarner Losh<pre><b>#define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1) 980c16b537SWarner Losh#define ZSTD_CONTENTSIZE_ERROR (0ULL - 2) 990c16b537SWarner Loshunsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize); 1000c16b537SWarner Losh</b><p> `src` should point to the start of a ZSTD encoded frame. 1010c16b537SWarner Losh `srcSize` must be at least as large as the frame header. 1020c16b537SWarner Losh hint : any size >= `ZSTD_frameHeaderSize_max` is large enough. 1030f743729SConrad Meyer @return : - decompressed size of `src` frame content, if known 1040c16b537SWarner Losh - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined 1050c16b537SWarner Losh - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small) 1060c16b537SWarner Losh note 1 : a 0 return value means the frame is valid but "empty". 1070c16b537SWarner Losh note 2 : decompressed size is an optional field, it may not be present, typically in streaming mode. 1080c16b537SWarner Losh When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size. 1090c16b537SWarner Losh In which case, it's necessary to use streaming mode to decompress data. 1100c16b537SWarner Losh Optionally, application can rely on some implicit limit, 1110c16b537SWarner Losh as ZSTD_decompress() only needs an upper bound of decompressed size. 1120c16b537SWarner Losh (For example, data could be necessarily cut into blocks <= 16 KB). 1130f743729SConrad Meyer note 3 : decompressed size is always present when compression is completed using single-pass functions, 1140f743729SConrad Meyer such as ZSTD_compress(), ZSTD_compressCCtx() ZSTD_compress_usingDict() or ZSTD_compress_usingCDict(). 1150c16b537SWarner Losh note 4 : decompressed size can be very large (64-bits value), 1160c16b537SWarner Losh potentially larger than what local system can handle as a single memory segment. 1170c16b537SWarner Losh In which case, it's necessary to use streaming mode to decompress data. 1180c16b537SWarner Losh note 5 : If source is untrusted, decompressed size could be wrong or intentionally modified. 1190c16b537SWarner Losh Always ensure return value fits within application's authorized limits. 1200c16b537SWarner Losh Each application can set its own limits. 1210c16b537SWarner Losh note 6 : This function replaces ZSTD_getDecompressedSize() 1220c16b537SWarner Losh</p></pre><BR> 1230c16b537SWarner Losh 1240c16b537SWarner Losh<pre><b>unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize); 1250c16b537SWarner Losh</b><p> NOTE: This function is now obsolete, in favor of ZSTD_getFrameContentSize(). 12619fcbaf1SConrad Meyer Both functions work the same way, but ZSTD_getDecompressedSize() blends 12719fcbaf1SConrad Meyer "empty", "unknown" and "error" results to the same return value (0), 12819fcbaf1SConrad Meyer while ZSTD_getFrameContentSize() gives them separate return values. 1290f743729SConrad Meyer @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. 1300c16b537SWarner Losh</p></pre><BR> 1310c16b537SWarner Losh 1322b9c00cbSConrad Meyer<pre><b>size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize); 1332b9c00cbSConrad Meyer</b><p> `src` should point to the start of a ZSTD frame or skippable frame. 1342b9c00cbSConrad Meyer `srcSize` must be >= first frame size 1352b9c00cbSConrad Meyer @return : the compressed size of the first frame starting at `src`, 1362b9c00cbSConrad Meyer suitable to pass as `srcSize` to `ZSTD_decompress` or similar, 1372b9c00cbSConrad Meyer or an error code if input is invalid 1382b9c00cbSConrad Meyer</p></pre><BR> 1392b9c00cbSConrad Meyer 140052d3c12SConrad Meyer<h3>Helper functions</h3><pre></pre><b><pre>#define ZSTD_COMPRESSBOUND(srcSize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) </b>/* margin, from 64 to 0 */ : 0)) /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */<b> 14119fcbaf1SConrad Meyersize_t ZSTD_compressBound(size_t srcSize); </b>/*!< maximum compressed size in worst case single-pass scenario */<b> 1420c16b537SWarner Loshunsigned ZSTD_isError(size_t code); </b>/*!< tells if a `size_t` function result is an error code */<b> 1430c16b537SWarner Loshconst char* ZSTD_getErrorName(size_t code); </b>/*!< provides readable string from an error code */<b> 144*5ff13fbcSAllan Judeint ZSTD_minCLevel(void); </b>/*!< minimum negative compression level allowed, requires v1.4.0+ */<b> 1450c16b537SWarner Loshint ZSTD_maxCLevel(void); </b>/*!< maximum compression level available */<b> 146*5ff13fbcSAllan Judeint ZSTD_defaultCLevel(void); </b>/*!< default compression level, specified by ZSTD_CLEVEL_DEFAULT, requires v1.5.0+ */<b> 1470c16b537SWarner Losh</pre></b><BR> 1484d3f1eafSConrad Meyer<a name="Chapter4"></a><h2>Explicit context</h2><pre></pre> 1490c16b537SWarner Losh 1500c16b537SWarner Losh<h3>Compression context</h3><pre> When compressing many times, 1514d3f1eafSConrad Meyer it is recommended to allocate a context just once, 1524d3f1eafSConrad Meyer and re-use it for each successive compression operation. 1530c16b537SWarner Losh This will make workload friendlier for system's memory. 1544d3f1eafSConrad Meyer Note : re-using context is just a speed / resource optimization. 1554d3f1eafSConrad Meyer It doesn't change the compression ratio, which remains identical. 1564d3f1eafSConrad Meyer Note 2 : In multi-threaded environments, 1574d3f1eafSConrad Meyer use one different context per thread for parallel execution. 1584d3f1eafSConrad Meyer 1590c16b537SWarner Losh</pre><b><pre>typedef struct ZSTD_CCtx_s ZSTD_CCtx; 1600c16b537SWarner LoshZSTD_CCtx* ZSTD_createCCtx(void); 161*5ff13fbcSAllan Judesize_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); </b>/* accept NULL pointer */<b> 1620c16b537SWarner Losh</pre></b><BR> 163a0483764SConrad Meyer<pre><b>size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx, 1640c16b537SWarner Losh void* dst, size_t dstCapacity, 1650c16b537SWarner Losh const void* src, size_t srcSize, 1660c16b537SWarner Losh int compressionLevel); 1679cbefe25SConrad Meyer</b><p> Same as ZSTD_compress(), using an explicit ZSTD_CCtx. 1689cbefe25SConrad Meyer Important : in order to behave similarly to `ZSTD_compress()`, 1699cbefe25SConrad Meyer this function compresses at requested compression level, 1709cbefe25SConrad Meyer __ignoring any other parameter__ . 1719cbefe25SConrad Meyer If any advanced parameter was set using the advanced API, 1729cbefe25SConrad Meyer they will all be reset. Only `compressionLevel` remains. 1739cbefe25SConrad Meyer 1740c16b537SWarner Losh</p></pre><BR> 1750c16b537SWarner Losh 1760c16b537SWarner Losh<h3>Decompression context</h3><pre> When decompressing many times, 1770c16b537SWarner Losh it is recommended to allocate a context only once, 1780c16b537SWarner Losh and re-use it for each successive compression operation. 1790c16b537SWarner Losh This will make workload friendlier for system's memory. 1800c16b537SWarner Losh Use one context per thread for parallel execution. 1810c16b537SWarner Losh</pre><b><pre>typedef struct ZSTD_DCtx_s ZSTD_DCtx; 1820c16b537SWarner LoshZSTD_DCtx* ZSTD_createDCtx(void); 183*5ff13fbcSAllan Judesize_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); </b>/* accept NULL pointer */<b> 1840c16b537SWarner Losh</pre></b><BR> 185a0483764SConrad Meyer<pre><b>size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx, 1860c16b537SWarner Losh void* dst, size_t dstCapacity, 1870c16b537SWarner Losh const void* src, size_t srcSize); 188a0483764SConrad Meyer</b><p> Same as ZSTD_decompress(), 189a0483764SConrad Meyer requires an allocated ZSTD_DCtx. 190a0483764SConrad Meyer Compatible with sticky parameters. 191a0483764SConrad Meyer 1920c16b537SWarner Losh</p></pre><BR> 1930c16b537SWarner Losh 194*5ff13fbcSAllan Jude<a name="Chapter5"></a><h2>Advanced compression API (Requires v1.4.0+)</h2><pre></pre> 195a0483764SConrad Meyer 196a0483764SConrad Meyer<pre><b>typedef enum { ZSTD_fast=1, 197a0483764SConrad Meyer ZSTD_dfast=2, 198a0483764SConrad Meyer ZSTD_greedy=3, 199a0483764SConrad Meyer ZSTD_lazy=4, 200a0483764SConrad Meyer ZSTD_lazy2=5, 201a0483764SConrad Meyer ZSTD_btlazy2=6, 202a0483764SConrad Meyer ZSTD_btopt=7, 203a0483764SConrad Meyer ZSTD_btultra=8, 204a0483764SConrad Meyer ZSTD_btultra2=9 205a0483764SConrad Meyer </b>/* note : new strategies _might_ be added in the future.<b> 206a0483764SConrad Meyer Only the order (from fast to strong) is guaranteed */ 207a0483764SConrad Meyer} ZSTD_strategy; 2080c16b537SWarner Losh</b></pre><BR> 209a0483764SConrad Meyer<pre><b>typedef enum { 210a0483764SConrad Meyer 2112b9c00cbSConrad Meyer </b>/* compression parameters<b> 2122b9c00cbSConrad Meyer * Note: When compressing with a ZSTD_CDict these parameters are superseded 2139cbefe25SConrad Meyer * by the parameters used to construct the ZSTD_CDict. 2149cbefe25SConrad Meyer * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */ 2159cbefe25SConrad Meyer ZSTD_c_compressionLevel=100, </b>/* Set compression parameters according to pre-defined cLevel table.<b> 2169cbefe25SConrad Meyer * Note that exact compression parameters are dynamically determined, 2179cbefe25SConrad Meyer * depending on both compression level and srcSize (when known). 218a0483764SConrad Meyer * Default level is ZSTD_CLEVEL_DEFAULT==3. 219a0483764SConrad Meyer * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT. 220a0483764SConrad Meyer * Note 1 : it's possible to pass a negative compression level. 22137f1f268SConrad Meyer * Note 2 : setting a level does not automatically set all other compression parameters 22237f1f268SConrad Meyer * to default. Setting this will however eventually dynamically impact the compression 22337f1f268SConrad Meyer * parameters which have not been manually set. The manually set 22437f1f268SConrad Meyer * ones will 'stick'. */ 2259cbefe25SConrad Meyer </b>/* Advanced compression parameters :<b> 2269cbefe25SConrad Meyer * It's possible to pin down compression parameters to some specific values. 2279cbefe25SConrad Meyer * In which case, these values are no longer dynamically selected by the compressor */ 228a0483764SConrad Meyer ZSTD_c_windowLog=101, </b>/* Maximum allowed back-reference distance, expressed as power of 2.<b> 2299cbefe25SConrad Meyer * This will set a memory budget for streaming decompression, 2309cbefe25SConrad Meyer * with larger values requiring more memory 2319cbefe25SConrad Meyer * and typically compressing more. 232a0483764SConrad Meyer * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX. 233a0483764SConrad Meyer * Special: value 0 means "use default windowLog". 234a0483764SConrad Meyer * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT 2359cbefe25SConrad Meyer * requires explicitly allowing such size at streaming decompression stage. */ 236a0483764SConrad Meyer ZSTD_c_hashLog=102, </b>/* Size of the initial probe table, as a power of 2.<b> 237a0483764SConrad Meyer * Resulting memory usage is (1 << (hashLog+2)). 238a0483764SConrad Meyer * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX. 239a0483764SConrad Meyer * Larger tables improve compression ratio of strategies <= dFast, 240a0483764SConrad Meyer * and improve speed of strategies > dFast. 241a0483764SConrad Meyer * Special: value 0 means "use default hashLog". */ 242a0483764SConrad Meyer ZSTD_c_chainLog=103, </b>/* Size of the multi-probe search table, as a power of 2.<b> 243a0483764SConrad Meyer * Resulting memory usage is (1 << (chainLog+2)). 244a0483764SConrad Meyer * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX. 245a0483764SConrad Meyer * Larger tables result in better and slower compression. 2469cbefe25SConrad Meyer * This parameter is useless for "fast" strategy. 247a0483764SConrad Meyer * It's still useful when using "dfast" strategy, 248a0483764SConrad Meyer * in which case it defines a secondary probe table. 249a0483764SConrad Meyer * Special: value 0 means "use default chainLog". */ 250a0483764SConrad Meyer ZSTD_c_searchLog=104, </b>/* Number of search attempts, as a power of 2.<b> 251a0483764SConrad Meyer * More attempts result in better and slower compression. 2529cbefe25SConrad Meyer * This parameter is useless for "fast" and "dFast" strategies. 253a0483764SConrad Meyer * Special: value 0 means "use default searchLog". */ 254a0483764SConrad Meyer ZSTD_c_minMatch=105, </b>/* Minimum size of searched matches.<b> 255a0483764SConrad Meyer * Note that Zstandard can still find matches of smaller size, 256a0483764SConrad Meyer * it just tweaks its search algorithm to look for this size and larger. 257a0483764SConrad Meyer * Larger values increase compression and decompression speed, but decrease ratio. 258a0483764SConrad Meyer * Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX. 259a0483764SConrad Meyer * Note that currently, for all strategies < btopt, effective minimum is 4. 260a0483764SConrad Meyer * , for all strategies > fast, effective maximum is 6. 261a0483764SConrad Meyer * Special: value 0 means "use default minMatchLength". */ 262a0483764SConrad Meyer ZSTD_c_targetLength=106, </b>/* Impact of this field depends on strategy.<b> 263a0483764SConrad Meyer * For strategies btopt, btultra & btultra2: 264a0483764SConrad Meyer * Length of Match considered "good enough" to stop search. 265a0483764SConrad Meyer * Larger values make compression stronger, and slower. 266a0483764SConrad Meyer * For strategy fast: 267a0483764SConrad Meyer * Distance between match sampling. 268a0483764SConrad Meyer * Larger values make compression faster, and weaker. 269a0483764SConrad Meyer * Special: value 0 means "use default targetLength". */ 270a0483764SConrad Meyer ZSTD_c_strategy=107, </b>/* See ZSTD_strategy enum definition.<b> 271a0483764SConrad Meyer * The higher the value of selected strategy, the more complex it is, 272a0483764SConrad Meyer * resulting in stronger and slower compression. 273a0483764SConrad Meyer * Special: value 0 means "use default strategy". */ 274a0483764SConrad Meyer </b>/* LDM mode parameters */<b> 275a0483764SConrad Meyer ZSTD_c_enableLongDistanceMatching=160, </b>/* Enable long distance matching.<b> 276a0483764SConrad Meyer * This parameter is designed to improve compression ratio 277a0483764SConrad Meyer * for large inputs, by finding large matches at long distance. 278a0483764SConrad Meyer * It increases memory usage and window size. 279a0483764SConrad Meyer * Note: enabling this parameter increases default ZSTD_c_windowLog to 128 MB 280f7cd7fe5SConrad Meyer * except when expressly set to a different value. 281f7cd7fe5SConrad Meyer * Note: will be enabled by default if ZSTD_c_windowLog >= 128 MB and 282f7cd7fe5SConrad Meyer * compression strategy >= ZSTD_btopt (== compression level 16+) */ 283a0483764SConrad Meyer ZSTD_c_ldmHashLog=161, </b>/* Size of the table for long distance matching, as a power of 2.<b> 284a0483764SConrad Meyer * Larger values increase memory usage and compression ratio, 285a0483764SConrad Meyer * but decrease compression speed. 286a0483764SConrad Meyer * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX 287a0483764SConrad Meyer * default: windowlog - 7. 288a0483764SConrad Meyer * Special: value 0 means "automatically determine hashlog". */ 289a0483764SConrad Meyer ZSTD_c_ldmMinMatch=162, </b>/* Minimum match size for long distance matcher.<b> 290a0483764SConrad Meyer * Larger/too small values usually decrease compression ratio. 291a0483764SConrad Meyer * Must be clamped between ZSTD_LDM_MINMATCH_MIN and ZSTD_LDM_MINMATCH_MAX. 292a0483764SConrad Meyer * Special: value 0 means "use default value" (default: 64). */ 293a0483764SConrad Meyer ZSTD_c_ldmBucketSizeLog=163, </b>/* Log size of each bucket in the LDM hash table for collision resolution.<b> 294a0483764SConrad Meyer * Larger values improve collision resolution but decrease compression speed. 295a0483764SConrad Meyer * The maximum value is ZSTD_LDM_BUCKETSIZELOG_MAX. 296a0483764SConrad Meyer * Special: value 0 means "use default value" (default: 3). */ 297a0483764SConrad Meyer ZSTD_c_ldmHashRateLog=164, </b>/* Frequency of inserting/looking up entries into the LDM hash table.<b> 298a0483764SConrad Meyer * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN). 299a0483764SConrad Meyer * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage. 300a0483764SConrad Meyer * Larger values improve compression speed. 301a0483764SConrad Meyer * Deviating far from default value will likely result in a compression ratio decrease. 302a0483764SConrad Meyer * Special: value 0 means "automatically determine hashRateLog". */ 303a0483764SConrad Meyer 304a0483764SConrad Meyer </b>/* frame parameters */<b> 305a0483764SConrad Meyer ZSTD_c_contentSizeFlag=200, </b>/* Content size will be written into frame header _whenever known_ (default:1)<b> 306a0483764SConrad Meyer * Content size must be known at the beginning of compression. 307a0483764SConrad Meyer * This is automatically the case when using ZSTD_compress2(), 3089cbefe25SConrad Meyer * For streaming scenarios, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */ 309a0483764SConrad Meyer ZSTD_c_checksumFlag=201, </b>/* A 32-bits checksum of content is written at end of frame (default:0) */<b> 310a0483764SConrad Meyer ZSTD_c_dictIDFlag=202, </b>/* When applicable, dictionary's ID is written into frame header (default:1) */<b> 311a0483764SConrad Meyer 312a0483764SConrad Meyer </b>/* multi-threading parameters */<b> 313f7cd7fe5SConrad Meyer </b>/* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).<b> 314f7cd7fe5SConrad Meyer * Otherwise, trying to set any other value than default (0) will be a no-op and return an error. 315f7cd7fe5SConrad Meyer * In a situation where it's unknown if the linked library supports multi-threading or not, 316f7cd7fe5SConrad Meyer * setting ZSTD_c_nbWorkers to any value >= 1 and consulting the return value provides a quick way to check this property. 317f7cd7fe5SConrad Meyer */ 318a0483764SConrad Meyer ZSTD_c_nbWorkers=400, </b>/* Select how many threads will be spawned to compress in parallel.<b> 319f7cd7fe5SConrad Meyer * When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() : 320a0483764SConrad Meyer * ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller, 321f7cd7fe5SConrad Meyer * while compression is performed in parallel, within worker thread(s). 322a0483764SConrad Meyer * (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end : 323a0483764SConrad Meyer * in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call). 324a0483764SConrad Meyer * More workers improve speed, but also increase memory usage. 325f7cd7fe5SConrad Meyer * Default value is `0`, aka "single-threaded mode" : no worker is spawned, 326f7cd7fe5SConrad Meyer * compression is performed inside Caller's thread, and all invocations are blocking */ 327a0483764SConrad Meyer ZSTD_c_jobSize=401, </b>/* Size of a compression job. This value is enforced only when nbWorkers >= 1.<b> 328a0483764SConrad Meyer * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads. 329a0483764SConrad Meyer * 0 means default, which is dynamically determined based on compression parameters. 330*5ff13fbcSAllan Jude * Job size must be a minimum of overlap size, or ZSTDMT_JOBSIZE_MIN (= 512 KB), whichever is largest. 3319cbefe25SConrad Meyer * The minimum size is automatically and transparently enforced. */ 332a0483764SConrad Meyer ZSTD_c_overlapLog=402, </b>/* Control the overlap size, as a fraction of window size.<b> 333a0483764SConrad Meyer * The overlap size is an amount of data reloaded from previous job at the beginning of a new job. 334a0483764SConrad Meyer * It helps preserve compression ratio, while each job is compressed in parallel. 335a0483764SConrad Meyer * This value is enforced only when nbWorkers >= 1. 336a0483764SConrad Meyer * Larger values increase compression ratio, but decrease speed. 337a0483764SConrad Meyer * Possible values range from 0 to 9 : 338a0483764SConrad Meyer * - 0 means "default" : value will be determined by the library, depending on strategy 339a0483764SConrad Meyer * - 1 means "no overlap" 340a0483764SConrad Meyer * - 9 means "full overlap", using a full window size. 341a0483764SConrad Meyer * Each intermediate rank increases/decreases load size by a factor 2 : 342a0483764SConrad Meyer * 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; 0:default 343a0483764SConrad Meyer * default value varies between 6 and 9, depending on strategy */ 344a0483764SConrad Meyer 345a0483764SConrad Meyer </b>/* note : additional experimental parameters are also available<b> 346a0483764SConrad Meyer * within the experimental section of the API. 347a0483764SConrad Meyer * At the time of this writing, they include : 348a0483764SConrad Meyer * ZSTD_c_rsyncable 349a0483764SConrad Meyer * ZSTD_c_format 350a0483764SConrad Meyer * ZSTD_c_forceMaxWindow 351a0483764SConrad Meyer * ZSTD_c_forceAttachDict 3522b9c00cbSConrad Meyer * ZSTD_c_literalCompressionMode 3534d3f1eafSConrad Meyer * ZSTD_c_targetCBlockSize 3549cbefe25SConrad Meyer * ZSTD_c_srcSizeHint 355f7cd7fe5SConrad Meyer * ZSTD_c_enableDedicatedDictSearch 356f7cd7fe5SConrad Meyer * ZSTD_c_stableInBuffer 357f7cd7fe5SConrad Meyer * ZSTD_c_stableOutBuffer 358f7cd7fe5SConrad Meyer * ZSTD_c_blockDelimiters 359f7cd7fe5SConrad Meyer * ZSTD_c_validateSequences 360*5ff13fbcSAllan Jude * ZSTD_c_useBlockSplitter 361*5ff13fbcSAllan Jude * ZSTD_c_useRowMatchFinder 362a0483764SConrad Meyer * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. 363a0483764SConrad Meyer * note : never ever use experimentalParam? names directly; 364a0483764SConrad Meyer * also, the enums values themselves are unstable and can still change. 365a0483764SConrad Meyer */ 366a0483764SConrad Meyer ZSTD_c_experimentalParam1=500, 367a0483764SConrad Meyer ZSTD_c_experimentalParam2=10, 368a0483764SConrad Meyer ZSTD_c_experimentalParam3=1000, 3692b9c00cbSConrad Meyer ZSTD_c_experimentalParam4=1001, 3702b9c00cbSConrad Meyer ZSTD_c_experimentalParam5=1002, 3714d3f1eafSConrad Meyer ZSTD_c_experimentalParam6=1003, 372f7cd7fe5SConrad Meyer ZSTD_c_experimentalParam7=1004, 373f7cd7fe5SConrad Meyer ZSTD_c_experimentalParam8=1005, 374f7cd7fe5SConrad Meyer ZSTD_c_experimentalParam9=1006, 375f7cd7fe5SConrad Meyer ZSTD_c_experimentalParam10=1007, 376f7cd7fe5SConrad Meyer ZSTD_c_experimentalParam11=1008, 377*5ff13fbcSAllan Jude ZSTD_c_experimentalParam12=1009, 378*5ff13fbcSAllan Jude ZSTD_c_experimentalParam13=1010, 379*5ff13fbcSAllan Jude ZSTD_c_experimentalParam14=1011, 380*5ff13fbcSAllan Jude ZSTD_c_experimentalParam15=1012 381a0483764SConrad Meyer} ZSTD_cParameter; 382a0483764SConrad Meyer</b></pre><BR> 383a0483764SConrad Meyer<pre><b>typedef struct { 384a0483764SConrad Meyer size_t error; 385a0483764SConrad Meyer int lowerBound; 386a0483764SConrad Meyer int upperBound; 387a0483764SConrad Meyer} ZSTD_bounds; 388a0483764SConrad Meyer</b></pre><BR> 389a0483764SConrad Meyer<pre><b>ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter cParam); 390a0483764SConrad Meyer</b><p> All parameters must belong to an interval with lower and upper bounds, 391a0483764SConrad Meyer otherwise they will either trigger an error or be automatically clamped. 392a0483764SConrad Meyer @return : a structure, ZSTD_bounds, which contains 393a0483764SConrad Meyer - an error status field, which must be tested using ZSTD_isError() 394a0483764SConrad Meyer - lower and upper bounds, both inclusive 395a0483764SConrad Meyer 396a0483764SConrad Meyer</p></pre><BR> 397a0483764SConrad Meyer 398a0483764SConrad Meyer<pre><b>size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value); 399a0483764SConrad Meyer</b><p> Set one compression parameter, selected by enum ZSTD_cParameter. 400a0483764SConrad Meyer All parameters have valid bounds. Bounds can be queried using ZSTD_cParam_getBounds(). 401a0483764SConrad Meyer Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter). 402a0483764SConrad Meyer Setting a parameter is generally only possible during frame initialization (before starting compression). 403a0483764SConrad Meyer Exception : when using multi-threading mode (nbWorkers >= 1), 404a0483764SConrad Meyer the following parameters can be updated _during_ compression (within same frame): 405a0483764SConrad Meyer => compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy. 406a0483764SConrad Meyer new parameters will be active for next job only (after a flush()). 407a0483764SConrad Meyer @return : an error code (which can be tested using ZSTD_isError()). 408a0483764SConrad Meyer 409a0483764SConrad Meyer</p></pre><BR> 410a0483764SConrad Meyer 411a0483764SConrad Meyer<pre><b>size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize); 412a0483764SConrad Meyer</b><p> Total input data size to be compressed as a single frame. 413a0483764SConrad Meyer Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeFlag. 414a0483764SConrad Meyer This value will also be controlled at end of frame, and trigger an error if not respected. 415a0483764SConrad Meyer @result : 0, or an error code (which can be tested with ZSTD_isError()). 416a0483764SConrad Meyer Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame. 417a0483764SConrad Meyer In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN. 418a0483764SConrad Meyer ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame. 419a0483764SConrad Meyer Note 2 : pledgedSrcSize is only valid once, for the next frame. 420a0483764SConrad Meyer It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN. 421a0483764SConrad Meyer Note 3 : Whenever all input data is provided and consumed in a single round, 422a0483764SConrad Meyer for example with ZSTD_compress2(), 423a0483764SConrad Meyer or invoking immediately ZSTD_compressStream2(,,,ZSTD_e_end), 4242b9c00cbSConrad Meyer this value is automatically overridden by srcSize instead. 425a0483764SConrad Meyer 426a0483764SConrad Meyer</p></pre><BR> 427a0483764SConrad Meyer 428a0483764SConrad Meyer<pre><b>typedef enum { 429a0483764SConrad Meyer ZSTD_reset_session_only = 1, 430a0483764SConrad Meyer ZSTD_reset_parameters = 2, 431a0483764SConrad Meyer ZSTD_reset_session_and_parameters = 3 432a0483764SConrad Meyer} ZSTD_ResetDirective; 433a0483764SConrad Meyer</b></pre><BR> 434a0483764SConrad Meyer<pre><b>size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset); 435a0483764SConrad Meyer</b><p> There are 2 different things that can be reset, independently or jointly : 436a0483764SConrad Meyer - The session : will stop compressing current frame, and make CCtx ready to start a new one. 437a0483764SConrad Meyer Useful after an error, or to interrupt any ongoing compression. 438a0483764SConrad Meyer Any internal data not yet flushed is cancelled. 439a0483764SConrad Meyer Compression parameters and dictionary remain unchanged. 440a0483764SConrad Meyer They will be used to compress next frame. 441a0483764SConrad Meyer Resetting session never fails. 442a0483764SConrad Meyer - The parameters : changes all parameters back to "default". 443a0483764SConrad Meyer This removes any reference to any dictionary too. 444a0483764SConrad Meyer Parameters can only be changed between 2 sessions (i.e. no compression is currently ongoing) 445a0483764SConrad Meyer otherwise the reset fails, and function returns an error value (which can be tested using ZSTD_isError()) 446a0483764SConrad Meyer - Both : similar to resetting the session, followed by resetting parameters. 447a0483764SConrad Meyer 448a0483764SConrad Meyer</p></pre><BR> 449a0483764SConrad Meyer 450a0483764SConrad Meyer<pre><b>size_t ZSTD_compress2( ZSTD_CCtx* cctx, 451a0483764SConrad Meyer void* dst, size_t dstCapacity, 452a0483764SConrad Meyer const void* src, size_t srcSize); 453a0483764SConrad Meyer</b><p> Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API. 454a0483764SConrad Meyer ZSTD_compress2() always starts a new frame. 455a0483764SConrad Meyer Should cctx hold data from a previously unfinished frame, everything about it is forgotten. 456a0483764SConrad Meyer - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*() 457a0483764SConrad Meyer - The function is always blocking, returns when compression is completed. 458a0483764SConrad Meyer Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`. 459a0483764SConrad Meyer @return : compressed size written into `dst` (<= `dstCapacity), 460a0483764SConrad Meyer or an error code if it fails (which can be tested using ZSTD_isError()). 461a0483764SConrad Meyer 462a0483764SConrad Meyer</p></pre><BR> 463a0483764SConrad Meyer 464*5ff13fbcSAllan Jude<a name="Chapter6"></a><h2>Advanced decompression API (Requires v1.4.0+)</h2><pre></pre> 465a0483764SConrad Meyer 466a0483764SConrad Meyer<pre><b>typedef enum { 467a0483764SConrad Meyer 468a0483764SConrad Meyer ZSTD_d_windowLogMax=100, </b>/* Select a size limit (in power of 2) beyond which<b> 469a0483764SConrad Meyer * the streaming API will refuse to allocate memory buffer 470a0483764SConrad Meyer * in order to protect the host from unreasonable memory requirements. 471a0483764SConrad Meyer * This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode. 4722b9c00cbSConrad Meyer * By default, a decompression context accepts window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT). 4732b9c00cbSConrad Meyer * Special: value 0 means "use default maximum windowLog". */ 474a0483764SConrad Meyer 475a0483764SConrad Meyer </b>/* note : additional experimental parameters are also available<b> 476a0483764SConrad Meyer * within the experimental section of the API. 477a0483764SConrad Meyer * At the time of this writing, they include : 47837f1f268SConrad Meyer * ZSTD_d_format 47937f1f268SConrad Meyer * ZSTD_d_stableOutBuffer 480f7cd7fe5SConrad Meyer * ZSTD_d_forceIgnoreChecksum 481*5ff13fbcSAllan Jude * ZSTD_d_refMultipleDDicts 482a0483764SConrad Meyer * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. 483a0483764SConrad Meyer * note : never ever use experimentalParam? names directly 484a0483764SConrad Meyer */ 48537f1f268SConrad Meyer ZSTD_d_experimentalParam1=1000, 486f7cd7fe5SConrad Meyer ZSTD_d_experimentalParam2=1001, 487*5ff13fbcSAllan Jude ZSTD_d_experimentalParam3=1002, 488*5ff13fbcSAllan Jude ZSTD_d_experimentalParam4=1003 489a0483764SConrad Meyer 490a0483764SConrad Meyer} ZSTD_dParameter; 491a0483764SConrad Meyer</b></pre><BR> 492a0483764SConrad Meyer<pre><b>ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam); 493a0483764SConrad Meyer</b><p> All parameters must belong to an interval with lower and upper bounds, 494a0483764SConrad Meyer otherwise they will either trigger an error or be automatically clamped. 495a0483764SConrad Meyer @return : a structure, ZSTD_bounds, which contains 496a0483764SConrad Meyer - an error status field, which must be tested using ZSTD_isError() 497a0483764SConrad Meyer - both lower and upper bounds, inclusive 498a0483764SConrad Meyer 499a0483764SConrad Meyer</p></pre><BR> 500a0483764SConrad Meyer 501a0483764SConrad Meyer<pre><b>size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int value); 502a0483764SConrad Meyer</b><p> Set one compression parameter, selected by enum ZSTD_dParameter. 503a0483764SConrad Meyer All parameters have valid bounds. Bounds can be queried using ZSTD_dParam_getBounds(). 504a0483764SConrad Meyer Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter). 505a0483764SConrad Meyer Setting a parameter is only possible during frame initialization (before starting decompression). 506a0483764SConrad Meyer @return : 0, or an error code (which can be tested using ZSTD_isError()). 507a0483764SConrad Meyer 508a0483764SConrad Meyer</p></pre><BR> 509a0483764SConrad Meyer 5102b9c00cbSConrad Meyer<pre><b>size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset); 5112b9c00cbSConrad Meyer</b><p> Return a DCtx to clean state. 5122b9c00cbSConrad Meyer Session and parameters can be reset jointly or separately. 5132b9c00cbSConrad Meyer Parameters can only be reset when no active frame is being decompressed. 5142b9c00cbSConrad Meyer @return : 0, or an error code, which can be tested with ZSTD_isError() 5152b9c00cbSConrad Meyer 5162b9c00cbSConrad Meyer</p></pre><BR> 5172b9c00cbSConrad Meyer 5184d3f1eafSConrad Meyer<a name="Chapter7"></a><h2>Streaming</h2><pre></pre> 5192b9c00cbSConrad Meyer 5202b9c00cbSConrad Meyer<pre><b>typedef struct ZSTD_inBuffer_s { 5212b9c00cbSConrad Meyer const void* src; </b>/**< start of input buffer */<b> 5222b9c00cbSConrad Meyer size_t size; </b>/**< size of input buffer */<b> 5232b9c00cbSConrad Meyer size_t pos; </b>/**< position where reading stopped. Will be updated. Necessarily 0 <= pos <= size */<b> 5242b9c00cbSConrad Meyer} ZSTD_inBuffer; 5252b9c00cbSConrad Meyer</b></pre><BR> 5262b9c00cbSConrad Meyer<pre><b>typedef struct ZSTD_outBuffer_s { 5272b9c00cbSConrad Meyer void* dst; </b>/**< start of output buffer */<b> 5282b9c00cbSConrad Meyer size_t size; </b>/**< size of output buffer */<b> 5292b9c00cbSConrad Meyer size_t pos; </b>/**< position where writing stopped. Will be updated. Necessarily 0 <= pos <= size */<b> 5302b9c00cbSConrad Meyer} ZSTD_outBuffer; 5312b9c00cbSConrad Meyer</b></pre><BR> 5324d3f1eafSConrad Meyer<a name="Chapter8"></a><h2>Streaming compression - HowTo</h2><pre> 5332b9c00cbSConrad Meyer A ZSTD_CStream object is required to track streaming operation. 5342b9c00cbSConrad Meyer Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources. 5352b9c00cbSConrad Meyer ZSTD_CStream objects can be reused multiple times on consecutive compression operations. 5362b9c00cbSConrad Meyer It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-using already allocated memory. 5372b9c00cbSConrad Meyer 5382b9c00cbSConrad Meyer For parallel execution, use one separate ZSTD_CStream per thread. 5392b9c00cbSConrad Meyer 5402b9c00cbSConrad Meyer note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing. 5412b9c00cbSConrad Meyer 5422b9c00cbSConrad Meyer Parameters are sticky : when starting a new compression on the same context, 5432b9c00cbSConrad Meyer it will re-use the same sticky parameters as previous compression session. 5442b9c00cbSConrad Meyer When in doubt, it's recommended to fully initialize the context before usage. 5452b9c00cbSConrad Meyer Use ZSTD_CCtx_reset() to reset the context and ZSTD_CCtx_setParameter(), 5462b9c00cbSConrad Meyer ZSTD_CCtx_setPledgedSrcSize(), or ZSTD_CCtx_loadDictionary() and friends to 5472b9c00cbSConrad Meyer set more specific parameters, the pledged source size, or load a dictionary. 5482b9c00cbSConrad Meyer 5492b9c00cbSConrad Meyer Use ZSTD_compressStream2() with ZSTD_e_continue as many times as necessary to 5502b9c00cbSConrad Meyer consume input stream. The function will automatically update both `pos` 5512b9c00cbSConrad Meyer fields within `input` and `output`. 5522b9c00cbSConrad Meyer Note that the function may not consume the entire input, for example, because 5532b9c00cbSConrad Meyer the output buffer is already full, in which case `input.pos < input.size`. 5542b9c00cbSConrad Meyer The caller must check if input has been entirely consumed. 5552b9c00cbSConrad Meyer If not, the caller must make some room to receive more compressed data, 5562b9c00cbSConrad Meyer and then present again remaining input data. 5572b9c00cbSConrad Meyer note: ZSTD_e_continue is guaranteed to make some forward progress when called, 5582b9c00cbSConrad Meyer but doesn't guarantee maximal forward progress. This is especially relevant 5592b9c00cbSConrad Meyer when compressing with multiple threads. The call won't block if it can 5602b9c00cbSConrad Meyer consume some input, but if it can't it will wait for some, but not all, 5612b9c00cbSConrad Meyer output to be flushed. 5622b9c00cbSConrad Meyer @return : provides a minimum amount of data remaining to be flushed from internal buffers 5632b9c00cbSConrad Meyer or an error code, which can be tested using ZSTD_isError(). 5642b9c00cbSConrad Meyer 5652b9c00cbSConrad Meyer At any moment, it's possible to flush whatever data might remain stuck within internal buffer, 5662b9c00cbSConrad Meyer using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated. 5672b9c00cbSConrad Meyer Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be enough (return code > 0). 5682b9c00cbSConrad Meyer In which case, make some room to receive more compressed data, and call again ZSTD_compressStream2() with ZSTD_e_flush. 5692b9c00cbSConrad Meyer You must continue calling ZSTD_compressStream2() with ZSTD_e_flush until it returns 0, at which point you can change the 5702b9c00cbSConrad Meyer operation. 5712b9c00cbSConrad Meyer note: ZSTD_e_flush will flush as much output as possible, meaning when compressing with multiple threads, it will 5722b9c00cbSConrad Meyer block until the flush is complete or the output buffer is full. 5732b9c00cbSConrad Meyer @return : 0 if internal buffers are entirely flushed, 5742b9c00cbSConrad Meyer >0 if some data still present within internal buffer (the value is minimal estimation of remaining size), 5752b9c00cbSConrad Meyer or an error code, which can be tested using ZSTD_isError(). 5762b9c00cbSConrad Meyer 5772b9c00cbSConrad Meyer Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame. 5782b9c00cbSConrad Meyer It will perform a flush and write frame epilogue. 5792b9c00cbSConrad Meyer The epilogue is required for decoders to consider a frame completed. 5802b9c00cbSConrad Meyer flush operation is the same, and follows same rules as calling ZSTD_compressStream2() with ZSTD_e_flush. 5812b9c00cbSConrad Meyer You must continue calling ZSTD_compressStream2() with ZSTD_e_end until it returns 0, at which point you are free to 5822b9c00cbSConrad Meyer start a new frame. 5832b9c00cbSConrad Meyer note: ZSTD_e_end will flush as much output as possible, meaning when compressing with multiple threads, it will 5842b9c00cbSConrad Meyer block until the flush is complete or the output buffer is full. 5852b9c00cbSConrad Meyer @return : 0 if frame fully completed and fully flushed, 5862b9c00cbSConrad Meyer >0 if some data still present within internal buffer (the value is minimal estimation of remaining size), 5872b9c00cbSConrad Meyer or an error code, which can be tested using ZSTD_isError(). 5882b9c00cbSConrad Meyer 5892b9c00cbSConrad Meyer 5902b9c00cbSConrad Meyer<BR></pre> 5912b9c00cbSConrad Meyer 5922b9c00cbSConrad Meyer<pre><b>typedef ZSTD_CCtx ZSTD_CStream; </b>/**< CCtx and CStream are now effectively same object (>= v1.3.0) */<b> 5932b9c00cbSConrad Meyer</b></pre><BR> 5942b9c00cbSConrad Meyer<h3>ZSTD_CStream management functions</h3><pre></pre><b><pre>ZSTD_CStream* ZSTD_createCStream(void); 595*5ff13fbcSAllan Judesize_t ZSTD_freeCStream(ZSTD_CStream* zcs); </b>/* accept NULL pointer */<b> 5962b9c00cbSConrad Meyer</pre></b><BR> 5972b9c00cbSConrad Meyer<h3>Streaming compression functions</h3><pre></pre><b><pre>typedef enum { 5982b9c00cbSConrad Meyer ZSTD_e_continue=0, </b>/* collect more data, encoder decides when to output compressed result, for optimal compression ratio */<b> 5992b9c00cbSConrad Meyer ZSTD_e_flush=1, </b>/* flush any data provided so far,<b> 6002b9c00cbSConrad Meyer * it creates (at least) one new block, that can be decoded immediately on reception; 6012b9c00cbSConrad Meyer * frame will continue: any future data can still reference previously compressed data, improving compression. 6022b9c00cbSConrad Meyer * note : multithreaded compression will block to flush as much output as possible. */ 6032b9c00cbSConrad Meyer ZSTD_e_end=2 </b>/* flush any remaining data _and_ close current frame.<b> 6042b9c00cbSConrad Meyer * note that frame is only closed after compressed data is fully flushed (return value == 0). 6052b9c00cbSConrad Meyer * After that point, any additional data starts a new frame. 6062b9c00cbSConrad Meyer * note : each frame is independent (does not reference any content from previous frame). 6072b9c00cbSConrad Meyer : note : multithreaded compression will block to flush as much output as possible. */ 6082b9c00cbSConrad Meyer} ZSTD_EndDirective; 6092b9c00cbSConrad Meyer</pre></b><BR> 6102b9c00cbSConrad Meyer<pre><b>size_t ZSTD_compressStream2( ZSTD_CCtx* cctx, 6112b9c00cbSConrad Meyer ZSTD_outBuffer* output, 6122b9c00cbSConrad Meyer ZSTD_inBuffer* input, 6132b9c00cbSConrad Meyer ZSTD_EndDirective endOp); 6142b9c00cbSConrad Meyer</b><p> Behaves about the same as ZSTD_compressStream, with additional control on end directive. 6152b9c00cbSConrad Meyer - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*() 6162b9c00cbSConrad Meyer - Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode) 6172b9c00cbSConrad Meyer - output->pos must be <= dstCapacity, input->pos must be <= srcSize 6182b9c00cbSConrad Meyer - output->pos and input->pos will be updated. They are guaranteed to remain below their respective limit. 619f7cd7fe5SConrad Meyer - endOp must be a valid directive 6202b9c00cbSConrad Meyer - When nbWorkers==0 (default), function is blocking : it completes its job before returning to caller. 621f7cd7fe5SConrad Meyer - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs to internal worker threads, flush to output whatever is available, 6222b9c00cbSConrad Meyer and then immediately returns, just indicating that there is some data remaining to be flushed. 6232b9c00cbSConrad Meyer The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte. 6242b9c00cbSConrad Meyer - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity, the function delegates to ZSTD_compress2() which is always blocking. 6252b9c00cbSConrad Meyer - @return provides a minimum amount of data remaining to be flushed from internal buffers 6262b9c00cbSConrad Meyer or an error code, which can be tested using ZSTD_isError(). 6272b9c00cbSConrad Meyer if @return != 0, flush is not fully completed, there is still some data left within internal buffers. 6282b9c00cbSConrad Meyer This is useful for ZSTD_e_flush, since in this case more flushes are necessary to empty all buffers. 6292b9c00cbSConrad Meyer For ZSTD_e_end, @return == 0 when internal buffers are fully flushed and frame is completed. 6302b9c00cbSConrad Meyer - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0), 6312b9c00cbSConrad Meyer only ZSTD_e_end or ZSTD_e_flush operations are allowed. 6322b9c00cbSConrad Meyer Before starting a new compression job, or changing compression parameters, 6332b9c00cbSConrad Meyer it is required to fully flush internal buffers. 6342b9c00cbSConrad Meyer 6352b9c00cbSConrad Meyer</p></pre><BR> 6362b9c00cbSConrad Meyer 6372b9c00cbSConrad Meyer<pre><b>size_t ZSTD_CStreamInSize(void); </b>/**< recommended size for input buffer */<b> 6382b9c00cbSConrad Meyer</b></pre><BR> 6394d3f1eafSConrad Meyer<pre><b>size_t ZSTD_CStreamOutSize(void); </b>/**< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block. */<b> 6402b9c00cbSConrad Meyer</b></pre><BR> 6414d3f1eafSConrad Meyer<pre><b>size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel); 6424d3f1eafSConrad Meyer</b>/*!<b> 6434d3f1eafSConrad Meyer * Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue). 6444d3f1eafSConrad Meyer * NOTE: The return value is different. ZSTD_compressStream() returns a hint for 6454d3f1eafSConrad Meyer * the next read size (if non-zero and not an error). ZSTD_compressStream2() 6464d3f1eafSConrad Meyer * returns the minimum nb of bytes left to flush (if non-zero and not an error). 6474d3f1eafSConrad Meyer */ 6484d3f1eafSConrad Meyersize_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input); 6494d3f1eafSConrad Meyer</b>/*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush). */<b> 6504d3f1eafSConrad Meyersize_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output); 6514d3f1eafSConrad Meyer</b>/*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_end). */<b> 6524d3f1eafSConrad Meyersize_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output); 6534d3f1eafSConrad Meyer</b><p> 6542b9c00cbSConrad Meyer ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); 6552b9c00cbSConrad Meyer ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any) 6562b9c00cbSConrad Meyer ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); 6572b9c00cbSConrad Meyer 6584d3f1eafSConrad Meyer</p></pre><BR> 6592b9c00cbSConrad Meyer 6604d3f1eafSConrad Meyer<a name="Chapter9"></a><h2>Streaming decompression - HowTo</h2><pre> 6612b9c00cbSConrad Meyer A ZSTD_DStream object is required to track streaming operations. 6622b9c00cbSConrad Meyer Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources. 6632b9c00cbSConrad Meyer ZSTD_DStream objects can be re-used multiple times. 6642b9c00cbSConrad Meyer 6652b9c00cbSConrad Meyer Use ZSTD_initDStream() to start a new decompression operation. 6662b9c00cbSConrad Meyer @return : recommended first input size 6672b9c00cbSConrad Meyer Alternatively, use advanced API to set specific properties. 6682b9c00cbSConrad Meyer 6692b9c00cbSConrad Meyer Use ZSTD_decompressStream() repetitively to consume your input. 6702b9c00cbSConrad Meyer The function will update both `pos` fields. 6712b9c00cbSConrad Meyer If `input.pos < input.size`, some input has not been consumed. 6722b9c00cbSConrad Meyer It's up to the caller to present again remaining data. 6732b9c00cbSConrad Meyer The function tries to flush all data decoded immediately, respecting output buffer size. 6742b9c00cbSConrad Meyer If `output.pos < output.size`, decoder has flushed everything it could. 6752b9c00cbSConrad Meyer But if `output.pos == output.size`, there might be some data left within internal buffers., 6762b9c00cbSConrad Meyer In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer. 6772b9c00cbSConrad Meyer Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX. 6782b9c00cbSConrad Meyer @return : 0 when a frame is completely decoded and fully flushed, 6792b9c00cbSConrad Meyer or an error code, which can be tested using ZSTD_isError(), 6802b9c00cbSConrad Meyer or any other value > 0, which means there is still some decoding or flushing to do to complete current frame : 6812b9c00cbSConrad Meyer the return value is a suggested next input size (just a hint for better latency) 6822b9c00cbSConrad Meyer that will never request more than the remaining frame size. 6832b9c00cbSConrad Meyer 6842b9c00cbSConrad Meyer<BR></pre> 6852b9c00cbSConrad Meyer 6862b9c00cbSConrad Meyer<pre><b>typedef ZSTD_DCtx ZSTD_DStream; </b>/**< DCtx and DStream are now effectively same object (>= v1.3.0) */<b> 6872b9c00cbSConrad Meyer</b></pre><BR> 6882b9c00cbSConrad Meyer<h3>ZSTD_DStream management functions</h3><pre></pre><b><pre>ZSTD_DStream* ZSTD_createDStream(void); 689*5ff13fbcSAllan Judesize_t ZSTD_freeDStream(ZSTD_DStream* zds); </b>/* accept NULL pointer */<b> 6902b9c00cbSConrad Meyer</pre></b><BR> 6914d3f1eafSConrad Meyer<h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR> 6922b9c00cbSConrad Meyer<pre><b>size_t ZSTD_DStreamInSize(void); </b>/*!< recommended size for input buffer */<b> 6932b9c00cbSConrad Meyer</b></pre><BR> 6942b9c00cbSConrad Meyer<pre><b>size_t ZSTD_DStreamOutSize(void); </b>/*!< recommended size for output buffer. Guarantee to successfully flush at least one complete block in all circumstances. */<b> 6952b9c00cbSConrad Meyer</b></pre><BR> 6964d3f1eafSConrad Meyer<a name="Chapter10"></a><h2>Simple dictionary API</h2><pre></pre> 6972b9c00cbSConrad Meyer 6982b9c00cbSConrad Meyer<pre><b>size_t ZSTD_compress_usingDict(ZSTD_CCtx* ctx, 6992b9c00cbSConrad Meyer void* dst, size_t dstCapacity, 7002b9c00cbSConrad Meyer const void* src, size_t srcSize, 7012b9c00cbSConrad Meyer const void* dict,size_t dictSize, 7022b9c00cbSConrad Meyer int compressionLevel); 7032b9c00cbSConrad Meyer</b><p> Compression at an explicit compression level using a Dictionary. 7042b9c00cbSConrad Meyer A dictionary can be any arbitrary data segment (also called a prefix), 705*5ff13fbcSAllan Jude or a buffer with specified information (see zdict.h). 7062b9c00cbSConrad Meyer Note : This function loads the dictionary, resulting in significant startup delay. 7072b9c00cbSConrad Meyer It's intended for a dictionary used only once. 7082b9c00cbSConrad Meyer Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used. 7092b9c00cbSConrad Meyer</p></pre><BR> 7102b9c00cbSConrad Meyer 7112b9c00cbSConrad Meyer<pre><b>size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx, 7122b9c00cbSConrad Meyer void* dst, size_t dstCapacity, 7132b9c00cbSConrad Meyer const void* src, size_t srcSize, 7142b9c00cbSConrad Meyer const void* dict,size_t dictSize); 7152b9c00cbSConrad Meyer</b><p> Decompression using a known Dictionary. 7162b9c00cbSConrad Meyer Dictionary must be identical to the one used during compression. 7172b9c00cbSConrad Meyer Note : This function loads the dictionary, resulting in significant startup delay. 7182b9c00cbSConrad Meyer It's intended for a dictionary used only once. 7192b9c00cbSConrad Meyer Note : When `dict == NULL || dictSize < 8` no dictionary is used. 7202b9c00cbSConrad Meyer</p></pre><BR> 7212b9c00cbSConrad Meyer 7224d3f1eafSConrad Meyer<a name="Chapter11"></a><h2>Bulk processing dictionary API</h2><pre></pre> 7232b9c00cbSConrad Meyer 7242b9c00cbSConrad Meyer<pre><b>ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize, 7252b9c00cbSConrad Meyer int compressionLevel); 7269cbefe25SConrad Meyer</b><p> When compressing multiple messages or blocks using the same dictionary, 7279cbefe25SConrad Meyer it's recommended to digest the dictionary only once, since it's a costly operation. 7289cbefe25SConrad Meyer ZSTD_createCDict() will create a state from digesting a dictionary. 7299cbefe25SConrad Meyer The resulting state can be used for future compression operations with very limited startup cost. 7302b9c00cbSConrad Meyer ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only. 7319cbefe25SConrad Meyer @dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict. 7329cbefe25SConrad Meyer Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content. 7339cbefe25SConrad Meyer Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer, 7349cbefe25SConrad Meyer in which case the only thing that it transports is the @compressionLevel. 7359cbefe25SConrad Meyer This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively, 7369cbefe25SConrad Meyer expecting a ZSTD_CDict parameter with any data, including those without a known dictionary. 7372b9c00cbSConrad Meyer</p></pre><BR> 7382b9c00cbSConrad Meyer 7392b9c00cbSConrad Meyer<pre><b>size_t ZSTD_freeCDict(ZSTD_CDict* CDict); 7402b9c00cbSConrad Meyer</b><p> Function frees memory allocated by ZSTD_createCDict(). 741*5ff13fbcSAllan Jude If a NULL pointer is passed, no operation is performed. 7422b9c00cbSConrad Meyer</p></pre><BR> 7432b9c00cbSConrad Meyer 7442b9c00cbSConrad Meyer<pre><b>size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx, 7452b9c00cbSConrad Meyer void* dst, size_t dstCapacity, 7462b9c00cbSConrad Meyer const void* src, size_t srcSize, 7472b9c00cbSConrad Meyer const ZSTD_CDict* cdict); 7482b9c00cbSConrad Meyer</b><p> Compression using a digested Dictionary. 7492b9c00cbSConrad Meyer Recommended when same dictionary is used multiple times. 7502b9c00cbSConrad Meyer Note : compression level is _decided at dictionary creation time_, 7512b9c00cbSConrad Meyer and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) 7522b9c00cbSConrad Meyer</p></pre><BR> 7532b9c00cbSConrad Meyer 7542b9c00cbSConrad Meyer<pre><b>ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize); 7552b9c00cbSConrad Meyer</b><p> Create a digested dictionary, ready to start decompression operation without startup delay. 7562b9c00cbSConrad Meyer dictBuffer can be released after DDict creation, as its content is copied inside DDict. 7572b9c00cbSConrad Meyer</p></pre><BR> 7582b9c00cbSConrad Meyer 7592b9c00cbSConrad Meyer<pre><b>size_t ZSTD_freeDDict(ZSTD_DDict* ddict); 7602b9c00cbSConrad Meyer</b><p> Function frees memory allocated with ZSTD_createDDict() 761*5ff13fbcSAllan Jude If a NULL pointer is passed, no operation is performed. 7622b9c00cbSConrad Meyer</p></pre><BR> 7632b9c00cbSConrad Meyer 7642b9c00cbSConrad Meyer<pre><b>size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx, 7652b9c00cbSConrad Meyer void* dst, size_t dstCapacity, 7662b9c00cbSConrad Meyer const void* src, size_t srcSize, 7672b9c00cbSConrad Meyer const ZSTD_DDict* ddict); 7682b9c00cbSConrad Meyer</b><p> Decompression using a digested Dictionary. 7692b9c00cbSConrad Meyer Recommended when same dictionary is used multiple times. 7702b9c00cbSConrad Meyer</p></pre><BR> 7712b9c00cbSConrad Meyer 7724d3f1eafSConrad Meyer<a name="Chapter12"></a><h2>Dictionary helper functions</h2><pre></pre> 7732b9c00cbSConrad Meyer 7742b9c00cbSConrad Meyer<pre><b>unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize); 7752b9c00cbSConrad Meyer</b><p> Provides the dictID stored within dictionary. 7762b9c00cbSConrad Meyer if @return == 0, the dictionary is not conformant with Zstandard specification. 7772b9c00cbSConrad Meyer It can still be loaded, but as a content-only dictionary. 7782b9c00cbSConrad Meyer</p></pre><BR> 7792b9c00cbSConrad Meyer 780*5ff13fbcSAllan Jude<pre><b>unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict); 781*5ff13fbcSAllan Jude</b><p> Provides the dictID of the dictionary loaded into `cdict`. 782*5ff13fbcSAllan Jude If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. 783*5ff13fbcSAllan Jude Non-conformant dictionaries can still be loaded, but as content-only dictionaries. 784*5ff13fbcSAllan Jude</p></pre><BR> 785*5ff13fbcSAllan Jude 7862b9c00cbSConrad Meyer<pre><b>unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict); 7872b9c00cbSConrad Meyer</b><p> Provides the dictID of the dictionary loaded into `ddict`. 7882b9c00cbSConrad Meyer If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. 7892b9c00cbSConrad Meyer Non-conformant dictionaries can still be loaded, but as content-only dictionaries. 7902b9c00cbSConrad Meyer</p></pre><BR> 7912b9c00cbSConrad Meyer 7922b9c00cbSConrad Meyer<pre><b>unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize); 7932b9c00cbSConrad Meyer</b><p> Provides the dictID required to decompressed the frame stored within `src`. 7942b9c00cbSConrad Meyer If @return == 0, the dictID could not be decoded. 7952b9c00cbSConrad Meyer This could for one of the following reasons : 7962b9c00cbSConrad Meyer - The frame does not require a dictionary to be decoded (most common case). 7972b9c00cbSConrad Meyer - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden information. 7982b9c00cbSConrad Meyer Note : this use case also happens when using a non-conformant dictionary. 7992b9c00cbSConrad Meyer - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`). 8002b9c00cbSConrad Meyer - This is not a Zstandard frame. 8012b9c00cbSConrad Meyer When identifying the exact failure cause, it's possible to use ZSTD_getFrameHeader(), which will provide a more precise error code. 8022b9c00cbSConrad Meyer</p></pre><BR> 8032b9c00cbSConrad Meyer 804*5ff13fbcSAllan Jude<a name="Chapter13"></a><h2>Advanced dictionary and prefix API (Requires v1.4.0+)</h2><pre> 8052b9c00cbSConrad Meyer This API allows dictionaries to be used with ZSTD_compress2(), 806*5ff13fbcSAllan Jude ZSTD_compressStream2(), and ZSTD_decompressDCtx(). Dictionaries are sticky, and 8072b9c00cbSConrad Meyer only reset with the context is reset with ZSTD_reset_parameters or 8082b9c00cbSConrad Meyer ZSTD_reset_session_and_parameters. Prefixes are single-use. 8092b9c00cbSConrad Meyer<BR></pre> 8102b9c00cbSConrad Meyer 8112b9c00cbSConrad Meyer<pre><b>size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize); 8122b9c00cbSConrad Meyer</b><p> Create an internal CDict from `dict` buffer. 8132b9c00cbSConrad Meyer Decompression will have to use same dictionary. 8142b9c00cbSConrad Meyer @result : 0, or an error code (which can be tested with ZSTD_isError()). 8152b9c00cbSConrad Meyer Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary, 8162b9c00cbSConrad Meyer meaning "return to no-dictionary mode". 8172b9c00cbSConrad Meyer Note 1 : Dictionary is sticky, it will be used for all future compressed frames. 8182b9c00cbSConrad Meyer To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters). 8192b9c00cbSConrad Meyer Note 2 : Loading a dictionary involves building tables. 8202b9c00cbSConrad Meyer It's also a CPU consuming operation, with non-negligible impact on latency. 8212b9c00cbSConrad Meyer Tables are dependent on compression parameters, and for this reason, 8222b9c00cbSConrad Meyer compression parameters can no longer be changed after loading a dictionary. 8232b9c00cbSConrad Meyer Note 3 :`dict` content will be copied internally. 8242b9c00cbSConrad Meyer Use experimental ZSTD_CCtx_loadDictionary_byReference() to reference content instead. 8252b9c00cbSConrad Meyer In such a case, dictionary buffer must outlive its users. 8262b9c00cbSConrad Meyer Note 4 : Use ZSTD_CCtx_loadDictionary_advanced() 8272b9c00cbSConrad Meyer to precisely select how dictionary content must be interpreted. 8282b9c00cbSConrad Meyer</p></pre><BR> 8292b9c00cbSConrad Meyer 8302b9c00cbSConrad Meyer<pre><b>size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); 8312b9c00cbSConrad Meyer</b><p> Reference a prepared dictionary, to be used for all next compressed frames. 8322b9c00cbSConrad Meyer Note that compression parameters are enforced from within CDict, 8332b9c00cbSConrad Meyer and supersede any compression parameter previously set within CCtx. 834*5ff13fbcSAllan Jude The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs. 8352b9c00cbSConrad Meyer The ignored parameters will be used again if the CCtx is returned to no-dictionary mode. 8362b9c00cbSConrad Meyer The dictionary will remain valid for future compressed frames using same CCtx. 8372b9c00cbSConrad Meyer @result : 0, or an error code (which can be tested with ZSTD_isError()). 8382b9c00cbSConrad Meyer Special : Referencing a NULL CDict means "return to no-dictionary mode". 8392b9c00cbSConrad Meyer Note 1 : Currently, only one dictionary can be managed. 8402b9c00cbSConrad Meyer Referencing a new dictionary effectively "discards" any previous one. 8412b9c00cbSConrad Meyer Note 2 : CDict is just referenced, its lifetime must outlive its usage within CCtx. 8422b9c00cbSConrad Meyer</p></pre><BR> 8432b9c00cbSConrad Meyer 8442b9c00cbSConrad Meyer<pre><b>size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx, 8452b9c00cbSConrad Meyer const void* prefix, size_t prefixSize); 8462b9c00cbSConrad Meyer</b><p> Reference a prefix (single-usage dictionary) for next compressed frame. 8472b9c00cbSConrad Meyer A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end). 8482b9c00cbSConrad Meyer Decompression will need same prefix to properly regenerate data. 8492b9c00cbSConrad Meyer Compressing with a prefix is similar in outcome as performing a diff and compressing it, 8502b9c00cbSConrad Meyer but performs much faster, especially during decompression (compression speed is tunable with compression level). 8512b9c00cbSConrad Meyer @result : 0, or an error code (which can be tested with ZSTD_isError()). 8522b9c00cbSConrad Meyer Special: Adding any prefix (including NULL) invalidates any previous prefix or dictionary 8532b9c00cbSConrad Meyer Note 1 : Prefix buffer is referenced. It **must** outlive compression. 8542b9c00cbSConrad Meyer Its content must remain unmodified during compression. 8552b9c00cbSConrad Meyer Note 2 : If the intention is to diff some large src data blob with some prior version of itself, 8562b9c00cbSConrad Meyer ensure that the window size is large enough to contain the entire source. 8572b9c00cbSConrad Meyer See ZSTD_c_windowLog. 8582b9c00cbSConrad Meyer Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters. 8592b9c00cbSConrad Meyer It's a CPU consuming operation, with non-negligible impact on latency. 8602b9c00cbSConrad Meyer If there is a need to use the same prefix multiple times, consider loadDictionary instead. 8619cbefe25SConrad Meyer Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dct_rawContent). 8622b9c00cbSConrad Meyer Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. 8632b9c00cbSConrad Meyer</p></pre><BR> 8642b9c00cbSConrad Meyer 865a0483764SConrad Meyer<pre><b>size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); 866a0483764SConrad Meyer</b><p> Create an internal DDict from dict buffer, 867a0483764SConrad Meyer to be used to decompress next frames. 868a0483764SConrad Meyer The dictionary remains valid for all future frames, until explicitly invalidated. 869a0483764SConrad Meyer @result : 0, or an error code (which can be tested with ZSTD_isError()). 870a0483764SConrad Meyer Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary, 871a0483764SConrad Meyer meaning "return to no-dictionary mode". 872a0483764SConrad Meyer Note 1 : Loading a dictionary involves building tables, 873a0483764SConrad Meyer which has a non-negligible impact on CPU usage and latency. 874a0483764SConrad Meyer It's recommended to "load once, use many times", to amortize the cost 875a0483764SConrad Meyer Note 2 :`dict` content will be copied internally, so `dict` can be released after loading. 876a0483764SConrad Meyer Use ZSTD_DCtx_loadDictionary_byReference() to reference dictionary content instead. 877a0483764SConrad Meyer Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to take control of 878a0483764SConrad Meyer how dictionary content is loaded and interpreted. 879a0483764SConrad Meyer 880a0483764SConrad Meyer</p></pre><BR> 881a0483764SConrad Meyer 882a0483764SConrad Meyer<pre><b>size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict); 883a0483764SConrad Meyer</b><p> Reference a prepared dictionary, to be used to decompress next frames. 884a0483764SConrad Meyer The dictionary remains active for decompression of future frames using same DCtx. 885*5ff13fbcSAllan Jude 886*5ff13fbcSAllan Jude If called with ZSTD_d_refMultipleDDicts enabled, repeated calls of this function 887*5ff13fbcSAllan Jude will store the DDict references in a table, and the DDict used for decompression 888*5ff13fbcSAllan Jude will be determined at decompression time, as per the dict ID in the frame. 889*5ff13fbcSAllan Jude The memory for the table is allocated on the first call to refDDict, and can be 890*5ff13fbcSAllan Jude freed with ZSTD_freeDCtx(). 891*5ff13fbcSAllan Jude 892a0483764SConrad Meyer @result : 0, or an error code (which can be tested with ZSTD_isError()). 893a0483764SConrad Meyer Note 1 : Currently, only one dictionary can be managed. 894a0483764SConrad Meyer Referencing a new dictionary effectively "discards" any previous one. 895a0483764SConrad Meyer Special: referencing a NULL DDict means "return to no-dictionary mode". 896a0483764SConrad Meyer Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx. 897a0483764SConrad Meyer 898a0483764SConrad Meyer</p></pre><BR> 899a0483764SConrad Meyer 900a0483764SConrad Meyer<pre><b>size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, 901a0483764SConrad Meyer const void* prefix, size_t prefixSize); 902a0483764SConrad Meyer</b><p> Reference a prefix (single-usage dictionary) to decompress next frame. 903a0483764SConrad Meyer This is the reverse operation of ZSTD_CCtx_refPrefix(), 904a0483764SConrad Meyer and must use the same prefix as the one used during compression. 905a0483764SConrad Meyer Prefix is **only used once**. Reference is discarded at end of frame. 906a0483764SConrad Meyer End of frame is reached when ZSTD_decompressStream() returns 0. 907a0483764SConrad Meyer @result : 0, or an error code (which can be tested with ZSTD_isError()). 908a0483764SConrad Meyer Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary 909a0483764SConrad Meyer Note 2 : Prefix buffer is referenced. It **must** outlive decompression. 910a0483764SConrad Meyer Prefix buffer must remain unmodified up to the end of frame, 911a0483764SConrad Meyer reached when ZSTD_decompressStream() returns 0. 9129cbefe25SConrad Meyer Note 3 : By default, the prefix is treated as raw content (ZSTD_dct_rawContent). 913a0483764SConrad Meyer Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section) 914a0483764SConrad Meyer Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost. 915a0483764SConrad Meyer A full dictionary is more costly, as it requires building tables. 916a0483764SConrad Meyer 917a0483764SConrad Meyer</p></pre><BR> 918a0483764SConrad Meyer 9192b9c00cbSConrad Meyer<pre><b>size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx); 9202b9c00cbSConrad Meyersize_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx); 9212b9c00cbSConrad Meyersize_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs); 9222b9c00cbSConrad Meyersize_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds); 9232b9c00cbSConrad Meyersize_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict); 9242b9c00cbSConrad Meyersize_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); 9252b9c00cbSConrad Meyer</b><p> These functions give the _current_ memory usage of selected object. 9262b9c00cbSConrad Meyer Note that object memory usage can evolve (increase or decrease) over time. 927a0483764SConrad Meyer</p></pre><BR> 928a0483764SConrad Meyer 9294d3f1eafSConrad Meyer<a name="Chapter14"></a><h2>experimental API (static linking only)</h2><pre> 930a0483764SConrad Meyer The following symbols and constants 931a0483764SConrad Meyer are not planned to join "stable API" status in the near future. 932a0483764SConrad Meyer They can still change in future versions. 933a0483764SConrad Meyer Some of them are planned to remain in the static_only section indefinitely. 934a0483764SConrad Meyer Some of them might be removed in the future (especially when redundant with existing stable functions) 935a0483764SConrad Meyer 936a0483764SConrad Meyer<BR></pre> 937a0483764SConrad Meyer 9380c16b537SWarner Losh<pre><b>typedef struct { 939f7cd7fe5SConrad Meyer unsigned int offset; </b>/* The offset of the match. (NOT the same as the offset code)<b> 940f7cd7fe5SConrad Meyer * If offset == 0 and matchLength == 0, this sequence represents the last 941f7cd7fe5SConrad Meyer * literals in the block of litLength size. 9429cbefe25SConrad Meyer */ 943f7cd7fe5SConrad Meyer 944f7cd7fe5SConrad Meyer unsigned int litLength; </b>/* Literal length of the sequence. */<b> 945f7cd7fe5SConrad Meyer unsigned int matchLength; </b>/* Match length of the sequence. */<b> 946f7cd7fe5SConrad Meyer 947f7cd7fe5SConrad Meyer </b>/* Note: Users of this API may provide a sequence with matchLength == litLength == offset == 0.<b> 948f7cd7fe5SConrad Meyer * In this case, we will treat the sequence as a marker for a block boundary. 9499cbefe25SConrad Meyer */ 950f7cd7fe5SConrad Meyer 951f7cd7fe5SConrad Meyer unsigned int rep; </b>/* Represents which repeat offset is represented by the field 'offset'.<b> 952f7cd7fe5SConrad Meyer * Ranges from [0, 3]. 953f7cd7fe5SConrad Meyer * 954f7cd7fe5SConrad Meyer * Repeat offsets are essentially previous offsets from previous sequences sorted in 955f7cd7fe5SConrad Meyer * recency order. For more detail, see doc/zstd_compression_format.md 956f7cd7fe5SConrad Meyer * 957f7cd7fe5SConrad Meyer * If rep == 0, then 'offset' does not contain a repeat offset. 958f7cd7fe5SConrad Meyer * If rep > 0: 959f7cd7fe5SConrad Meyer * If litLength != 0: 960f7cd7fe5SConrad Meyer * rep == 1 --> offset == repeat_offset_1 961f7cd7fe5SConrad Meyer * rep == 2 --> offset == repeat_offset_2 962f7cd7fe5SConrad Meyer * rep == 3 --> offset == repeat_offset_3 963f7cd7fe5SConrad Meyer * If litLength == 0: 964f7cd7fe5SConrad Meyer * rep == 1 --> offset == repeat_offset_2 965f7cd7fe5SConrad Meyer * rep == 2 --> offset == repeat_offset_3 966f7cd7fe5SConrad Meyer * rep == 3 --> offset == repeat_offset_1 - 1 967f7cd7fe5SConrad Meyer * 968f7cd7fe5SConrad Meyer * Note: This field is optional. ZSTD_generateSequences() will calculate the value of 969f7cd7fe5SConrad Meyer * 'rep', but repeat offsets do not necessarily need to be calculated from an external 970f7cd7fe5SConrad Meyer * sequence provider's perspective. For example, ZSTD_compressSequences() does not 971f7cd7fe5SConrad Meyer * use this 'rep' field at all (as of now). 972f7cd7fe5SConrad Meyer */ 9739cbefe25SConrad Meyer} ZSTD_Sequence; 9749cbefe25SConrad Meyer</b></pre><BR> 9759cbefe25SConrad Meyer<pre><b>typedef struct { 9760c16b537SWarner Losh unsigned windowLog; </b>/**< largest match distance : larger == more compression, more memory needed during decompression */<b> 9770c16b537SWarner Losh unsigned chainLog; </b>/**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */<b> 9780c16b537SWarner Losh unsigned hashLog; </b>/**< dispatch table : larger == faster, more memory */<b> 9790c16b537SWarner Losh unsigned searchLog; </b>/**< nb of searches : larger == more compression, slower */<b> 980a0483764SConrad Meyer unsigned minMatch; </b>/**< match length searched : larger == faster decompression, sometimes less compression */<b> 9810c16b537SWarner Losh unsigned targetLength; </b>/**< acceptable match size for optimal parser (only) : larger == more compression, slower */<b> 982a0483764SConrad Meyer ZSTD_strategy strategy; </b>/**< see ZSTD_strategy definition above */<b> 9830c16b537SWarner Losh} ZSTD_compressionParameters; 9840c16b537SWarner Losh</b></pre><BR> 9850c16b537SWarner Losh<pre><b>typedef struct { 986a0483764SConrad Meyer int contentSizeFlag; </b>/**< 1: content size will be in frame header (when known) */<b> 987a0483764SConrad Meyer int checksumFlag; </b>/**< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, for error detection */<b> 988a0483764SConrad Meyer int noDictIDFlag; </b>/**< 1: no dictID will be saved into frame header (dictID is only useful for dictionary compression) */<b> 9890c16b537SWarner Losh} ZSTD_frameParameters; 9900c16b537SWarner Losh</b></pre><BR> 9910c16b537SWarner Losh<pre><b>typedef struct { 9920c16b537SWarner Losh ZSTD_compressionParameters cParams; 9930c16b537SWarner Losh ZSTD_frameParameters fParams; 9940c16b537SWarner Losh} ZSTD_parameters; 9950c16b537SWarner Losh</b></pre><BR> 99619fcbaf1SConrad Meyer<pre><b>typedef enum { 99719fcbaf1SConrad Meyer ZSTD_dct_auto = 0, </b>/* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, otherwise it is "rawContent" */<b> 998a0483764SConrad Meyer ZSTD_dct_rawContent = 1, </b>/* ensures dictionary is always loaded as rawContent, even if it starts with ZSTD_MAGIC_DICTIONARY */<b> 999a0483764SConrad Meyer ZSTD_dct_fullDict = 2 </b>/* refuses to load a dictionary if it does not respect Zstandard's specification, starting with ZSTD_MAGIC_DICTIONARY */<b> 100019fcbaf1SConrad Meyer} ZSTD_dictContentType_e; 1001052d3c12SConrad Meyer</b></pre><BR> 100219fcbaf1SConrad Meyer<pre><b>typedef enum { 100319fcbaf1SConrad Meyer ZSTD_dlm_byCopy = 0, </b>/**< Copy dictionary content internally */<b> 10049cbefe25SConrad Meyer ZSTD_dlm_byRef = 1 </b>/**< Reference dictionary content -- the dictionary buffer must outlive its users. */<b> 100519fcbaf1SConrad Meyer} ZSTD_dictLoadMethod_e; 100619fcbaf1SConrad Meyer</b></pre><BR> 1007a0483764SConrad Meyer<pre><b>typedef enum { 1008a0483764SConrad Meyer ZSTD_f_zstd1 = 0, </b>/* zstd frame format, specified in zstd_compression_format.md (default) */<b> 10099cbefe25SConrad Meyer ZSTD_f_zstd1_magicless = 1 </b>/* Variant of zstd frame format, without initial 4-bytes magic number.<b> 1010a0483764SConrad Meyer * Useful to save 4 bytes per generated frame. 1011a0483764SConrad Meyer * Decoder cannot recognise automatically this format, requiring this instruction. */ 1012a0483764SConrad Meyer} ZSTD_format_e; 1013a0483764SConrad Meyer</b></pre><BR> 1014a0483764SConrad Meyer<pre><b>typedef enum { 1015f7cd7fe5SConrad Meyer </b>/* Note: this enum controls ZSTD_d_forceIgnoreChecksum */<b> 1016f7cd7fe5SConrad Meyer ZSTD_d_validateChecksum = 0, 1017f7cd7fe5SConrad Meyer ZSTD_d_ignoreChecksum = 1 1018f7cd7fe5SConrad Meyer} ZSTD_forceIgnoreChecksum_e; 1019f7cd7fe5SConrad Meyer</b></pre><BR> 1020f7cd7fe5SConrad Meyer<pre><b>typedef enum { 1021*5ff13fbcSAllan Jude </b>/* Note: this enum controls ZSTD_d_refMultipleDDicts */<b> 1022*5ff13fbcSAllan Jude ZSTD_rmd_refSingleDDict = 0, 1023*5ff13fbcSAllan Jude ZSTD_rmd_refMultipleDDicts = 1 1024*5ff13fbcSAllan Jude} ZSTD_refMultipleDDicts_e; 1025*5ff13fbcSAllan Jude</b></pre><BR> 1026*5ff13fbcSAllan Jude<pre><b>typedef enum { 1027a0483764SConrad Meyer </b>/* Note: this enum and the behavior it controls are effectively internal<b> 1028a0483764SConrad Meyer * implementation details of the compressor. They are expected to continue 1029a0483764SConrad Meyer * to evolve and should be considered only in the context of extremely 1030a0483764SConrad Meyer * advanced performance tuning. 1031a0483764SConrad Meyer * 10329cbefe25SConrad Meyer * Zstd currently supports the use of a CDict in three ways: 1033a0483764SConrad Meyer * 1034a0483764SConrad Meyer * - The contents of the CDict can be copied into the working context. This 1035a0483764SConrad Meyer * means that the compression can search both the dictionary and input 1036a0483764SConrad Meyer * while operating on a single set of internal tables. This makes 1037a0483764SConrad Meyer * the compression faster per-byte of input. However, the initial copy of 1038a0483764SConrad Meyer * the CDict's tables incurs a fixed cost at the beginning of the 1039a0483764SConrad Meyer * compression. For small compressions (< 8 KB), that copy can dominate 1040a0483764SConrad Meyer * the cost of the compression. 1041a0483764SConrad Meyer * 1042a0483764SConrad Meyer * - The CDict's tables can be used in-place. In this model, compression is 1043a0483764SConrad Meyer * slower per input byte, because the compressor has to search two sets of 1044a0483764SConrad Meyer * tables. However, this model incurs no start-up cost (as long as the 1045a0483764SConrad Meyer * working context's tables can be reused). For small inputs, this can be 1046a0483764SConrad Meyer * faster than copying the CDict's tables. 1047a0483764SConrad Meyer * 10489cbefe25SConrad Meyer * - The CDict's tables are not used at all, and instead we use the working 10499cbefe25SConrad Meyer * context alone to reload the dictionary and use params based on the source 10509cbefe25SConrad Meyer * size. See ZSTD_compress_insertDictionary() and ZSTD_compress_usingDict(). 10519cbefe25SConrad Meyer * This method is effective when the dictionary sizes are very small relative 10529cbefe25SConrad Meyer * to the input size, and the input size is fairly large to begin with. 10539cbefe25SConrad Meyer * 1054a0483764SConrad Meyer * Zstd has a simple internal heuristic that selects which strategy to use 1055a0483764SConrad Meyer * at the beginning of a compression. However, if experimentation shows that 1056a0483764SConrad Meyer * Zstd is making poor choices, it is possible to override that choice with 1057a0483764SConrad Meyer * this enum. 1058a0483764SConrad Meyer */ 1059a0483764SConrad Meyer ZSTD_dictDefaultAttach = 0, </b>/* Use the default heuristic. */<b> 1060a0483764SConrad Meyer ZSTD_dictForceAttach = 1, </b>/* Never copy the dictionary. */<b> 1061a0483764SConrad Meyer ZSTD_dictForceCopy = 2, </b>/* Always copy the dictionary. */<b> 10629cbefe25SConrad Meyer ZSTD_dictForceLoad = 3 </b>/* Always reload the dictionary */<b> 1063a0483764SConrad Meyer} ZSTD_dictAttachPref_e; 1064a0483764SConrad Meyer</b></pre><BR> 10652b9c00cbSConrad Meyer<pre><b>typedef enum { 10662b9c00cbSConrad Meyer ZSTD_lcm_auto = 0, </b>/**< Automatically determine the compression mode based on the compression level.<b> 10672b9c00cbSConrad Meyer * Negative compression levels will be uncompressed, and positive compression 10682b9c00cbSConrad Meyer * levels will be compressed. */ 10692b9c00cbSConrad Meyer ZSTD_lcm_huffman = 1, </b>/**< Always attempt Huffman compression. Uncompressed literals will still be<b> 10702b9c00cbSConrad Meyer * emitted if Huffman compression is not profitable. */ 10719cbefe25SConrad Meyer ZSTD_lcm_uncompressed = 2 </b>/**< Always emit uncompressed literals. */<b> 10722b9c00cbSConrad Meyer} ZSTD_literalCompressionMode_e; 10732b9c00cbSConrad Meyer</b></pre><BR> 1074*5ff13fbcSAllan Jude<pre><b>typedef enum { 1075*5ff13fbcSAllan Jude </b>/* Note: This enum controls features which are conditionally beneficial. Zstd typically will make a final<b> 1076*5ff13fbcSAllan Jude * decision on whether or not to enable the feature (ZSTD_ps_auto), but setting the switch to ZSTD_ps_enable 1077*5ff13fbcSAllan Jude * or ZSTD_ps_disable allow for a force enable/disable the feature. 1078*5ff13fbcSAllan Jude */ 1079*5ff13fbcSAllan Jude ZSTD_ps_auto = 0, </b>/* Let the library automatically determine whether the feature shall be enabled */<b> 1080*5ff13fbcSAllan Jude ZSTD_ps_enable = 1, </b>/* Force-enable the feature */<b> 1081*5ff13fbcSAllan Jude ZSTD_ps_disable = 2 </b>/* Do not use the feature */<b> 1082*5ff13fbcSAllan Jude} ZSTD_paramSwitch_e; 1083*5ff13fbcSAllan Jude</b></pre><BR> 10844d3f1eafSConrad Meyer<a name="Chapter15"></a><h2>Frame size functions</h2><pre></pre> 10850c16b537SWarner Losh 1086*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize); 10872b9c00cbSConrad Meyer</b><p> `src` should point to the start of a series of ZSTD encoded and/or skippable frames 10880c16b537SWarner Losh `srcSize` must be the _exact_ size of this series 10892b9c00cbSConrad Meyer (i.e. there should be a frame boundary at `src + srcSize`) 10900c16b537SWarner Losh @return : - decompressed size of all data in all successive frames 10910c16b537SWarner Losh - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN 10920c16b537SWarner Losh - if an error occurred: ZSTD_CONTENTSIZE_ERROR 10930c16b537SWarner Losh 10940c16b537SWarner Losh note 1 : decompressed size is an optional field, that may not be present, especially in streaming mode. 10950c16b537SWarner Losh When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size. 10960c16b537SWarner Losh In which case, it's necessary to use streaming mode to decompress data. 10970c16b537SWarner Losh note 2 : decompressed size is always present when compression is done with ZSTD_compress() 10980c16b537SWarner Losh note 3 : decompressed size can be very large (64-bits value), 10990c16b537SWarner Losh potentially larger than what local system can handle as a single memory segment. 11000c16b537SWarner Losh In which case, it's necessary to use streaming mode to decompress data. 11010c16b537SWarner Losh note 4 : If source is untrusted, decompressed size could be wrong or intentionally modified. 11020c16b537SWarner Losh Always ensure result fits within application's authorized limits. 11030c16b537SWarner Losh Each application can set its own limits. 11040c16b537SWarner Losh note 5 : ZSTD_findDecompressedSize handles multiple frames, and so it must traverse the input to 11050c16b537SWarner Losh read each contained frame header. This is fast as most of the data is skipped, 11060c16b537SWarner Losh however it does mean that all frame data must be present and valid. 11070c16b537SWarner Losh</p></pre><BR> 11080c16b537SWarner Losh 1109*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize); 11104d3f1eafSConrad Meyer</b><p> `src` should point to the start of a series of ZSTD encoded and/or skippable frames 11112b9c00cbSConrad Meyer `srcSize` must be the _exact_ size of this series 11122b9c00cbSConrad Meyer (i.e. there should be a frame boundary at `src + srcSize`) 11132b9c00cbSConrad Meyer @return : - upper-bound for the decompressed size of all data in all successive frames 1114*5ff13fbcSAllan Jude - if an error occurred: ZSTD_CONTENTSIZE_ERROR 11152b9c00cbSConrad Meyer 11162b9c00cbSConrad Meyer note 1 : an error can occur if `src` contains an invalid or incorrectly formatted frame. 11172b9c00cbSConrad Meyer note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD encoded frame of `src`. 11182b9c00cbSConrad Meyer in this case, `ZSTD_findDecompressedSize` and `ZSTD_decompressBound` return the same value. 11192b9c00cbSConrad Meyer note 3 : when the decompressed size field isn't available, the upper-bound for that frame is calculated by: 11202b9c00cbSConrad Meyer upper-bound = # blocks * min(128 KB, Window_Size) 11212b9c00cbSConrad Meyer 11224d3f1eafSConrad Meyer</p></pre><BR> 11232b9c00cbSConrad Meyer 1124*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize); 1125a0483764SConrad Meyer</b><p> srcSize must be >= ZSTD_FRAMEHEADERSIZE_PREFIX. 11260f743729SConrad Meyer @return : size of the Frame Header, 11270f743729SConrad Meyer or an error code (if srcSize is too small) 11280c16b537SWarner Losh</p></pre><BR> 11290c16b537SWarner Losh 1130f7cd7fe5SConrad Meyer<pre><b>typedef enum { 1131f7cd7fe5SConrad Meyer ZSTD_sf_noBlockDelimiters = 0, </b>/* Representation of ZSTD_Sequence has no block delimiters, sequences only */<b> 1132f7cd7fe5SConrad Meyer ZSTD_sf_explicitBlockDelimiters = 1 </b>/* Representation of ZSTD_Sequence contains explicit block delimiters */<b> 1133f7cd7fe5SConrad Meyer} ZSTD_sequenceFormat_e; 1134f7cd7fe5SConrad Meyer</b></pre><BR> 1135f7cd7fe5SConrad Meyer<pre><b></b><p> Generate sequences using ZSTD_compress2, given a source buffer. 1136f7cd7fe5SConrad Meyer 1137f7cd7fe5SConrad Meyer Each block will end with a dummy sequence 1138f7cd7fe5SConrad Meyer with offset == 0, matchLength == 0, and litLength == length of last literals. 1139f7cd7fe5SConrad Meyer litLength may be == 0, and if so, then the sequence of (of: 0 ml: 0 ll: 0) 1140f7cd7fe5SConrad Meyer simply acts as a block delimiter. 1141f7cd7fe5SConrad Meyer 11429cbefe25SConrad Meyer zc can be used to insert custom compression params. 11439cbefe25SConrad Meyer This function invokes ZSTD_compress2 1144f7cd7fe5SConrad Meyer 1145f7cd7fe5SConrad Meyer The output of this function can be fed into ZSTD_compressSequences() with CCtx 1146f7cd7fe5SConrad Meyer setting of ZSTD_c_blockDelimiters as ZSTD_sf_explicitBlockDelimiters 1147f7cd7fe5SConrad Meyer @return : number of sequences generated 1148f7cd7fe5SConrad Meyer 1149f7cd7fe5SConrad Meyer</p></pre><BR> 1150f7cd7fe5SConrad Meyer 1151*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize); 1152f7cd7fe5SConrad Meyer</b><p> Given an array of ZSTD_Sequence, remove all sequences that represent block delimiters/last literals 1153f7cd7fe5SConrad Meyer by merging them into into the literals of the next sequence. 1154f7cd7fe5SConrad Meyer 1155f7cd7fe5SConrad Meyer As such, the final generated result has no explicit representation of block boundaries, 1156f7cd7fe5SConrad Meyer and the final last literals segment is not represented in the sequences. 1157f7cd7fe5SConrad Meyer 1158f7cd7fe5SConrad Meyer The output of this function can be fed into ZSTD_compressSequences() with CCtx 1159f7cd7fe5SConrad Meyer setting of ZSTD_c_blockDelimiters as ZSTD_sf_noBlockDelimiters 1160f7cd7fe5SConrad Meyer @return : number of sequences left after merging 1161f7cd7fe5SConrad Meyer 1162f7cd7fe5SConrad Meyer</p></pre><BR> 1163f7cd7fe5SConrad Meyer 1164*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size_t dstSize, 1165f7cd7fe5SConrad Meyer const ZSTD_Sequence* inSeqs, size_t inSeqsSize, 1166f7cd7fe5SConrad Meyer const void* src, size_t srcSize); 1167f7cd7fe5SConrad Meyer</b><p> Compress an array of ZSTD_Sequence, generated from the original source buffer, into dst. 1168f7cd7fe5SConrad Meyer If a dictionary is included, then the cctx should reference the dict. (see: ZSTD_CCtx_refCDict(), ZSTD_CCtx_loadDictionary(), etc.) 1169f7cd7fe5SConrad Meyer The entire source is compressed into a single frame. 1170f7cd7fe5SConrad Meyer 1171f7cd7fe5SConrad Meyer The compression behavior changes based on cctx params. In particular: 1172f7cd7fe5SConrad Meyer If ZSTD_c_blockDelimiters == ZSTD_sf_noBlockDelimiters, the array of ZSTD_Sequence is expected to contain 1173f7cd7fe5SConrad Meyer no block delimiters (defined in ZSTD_Sequence). Block boundaries are roughly determined based on 1174f7cd7fe5SConrad Meyer the block size derived from the cctx, and sequences may be split. This is the default setting. 1175f7cd7fe5SConrad Meyer 1176f7cd7fe5SConrad Meyer If ZSTD_c_blockDelimiters == ZSTD_sf_explicitBlockDelimiters, the array of ZSTD_Sequence is expected to contain 1177f7cd7fe5SConrad Meyer block delimiters (defined in ZSTD_Sequence). Behavior is undefined if no block delimiters are provided. 1178f7cd7fe5SConrad Meyer 1179f7cd7fe5SConrad Meyer If ZSTD_c_validateSequences == 0, this function will blindly accept the sequences provided. Invalid sequences cause undefined 1180f7cd7fe5SConrad Meyer behavior. If ZSTD_c_validateSequences == 1, then if sequence is invalid (see doc/zstd_compression_format.md for 1181f7cd7fe5SConrad Meyer specifics regarding offset/matchlength requirements) then the function will bail out and return an error. 1182f7cd7fe5SConrad Meyer 1183f7cd7fe5SConrad Meyer In addition to the two adjustable experimental params, there are other important cctx params. 1184f7cd7fe5SConrad Meyer - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the match finder. It has a minimum value of ZSTD_MINMATCH_MIN. 1185f7cd7fe5SConrad Meyer - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would in typical compression. 1186f7cd7fe5SConrad Meyer - ZSTD_c_windowLog affects offset validation: this function will return an error at higher debug levels if a provided offset 1187f7cd7fe5SConrad Meyer is larger than what the spec allows for a given window log and dictionary (if present). See: doc/zstd_compression_format.md 1188f7cd7fe5SConrad Meyer 1189f7cd7fe5SConrad Meyer Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep is unused. 1190f7cd7fe5SConrad Meyer Note 2: Once we integrate ability to ingest repcodes, the explicit block delims mode must respect those repcodes exactly, 1191f7cd7fe5SConrad Meyer and cannot emit an RLE block that disagrees with the repcode history 1192f7cd7fe5SConrad Meyer @return : final compressed size or a ZSTD error. 11939cbefe25SConrad Meyer 11949cbefe25SConrad Meyer</p></pre><BR> 11959cbefe25SConrad Meyer 1196*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity, 1197*5ff13fbcSAllan Jude const void* src, size_t srcSize, unsigned magicVariant); 1198*5ff13fbcSAllan Jude</b><p> Generates a zstd skippable frame containing data given by src, and writes it to dst buffer. 1199*5ff13fbcSAllan Jude 1200*5ff13fbcSAllan Jude Skippable frames begin with a a 4-byte magic number. There are 16 possible choices of magic number, 1201*5ff13fbcSAllan Jude ranging from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15. 1202*5ff13fbcSAllan Jude As such, the parameter magicVariant controls the exact skippable frame magic number variant used, so 1203*5ff13fbcSAllan Jude the magic number used will be ZSTD_MAGIC_SKIPPABLE_START + magicVariant. 1204*5ff13fbcSAllan Jude 1205*5ff13fbcSAllan Jude Returns an error if destination buffer is not large enough, if the source size is not representable 1206*5ff13fbcSAllan Jude with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore invalid). 1207*5ff13fbcSAllan Jude 1208*5ff13fbcSAllan Jude @return : number of bytes written or a ZSTD error. 1209*5ff13fbcSAllan Jude 1210*5ff13fbcSAllan Jude</p></pre><BR> 1211*5ff13fbcSAllan Jude 1212*5ff13fbcSAllan Jude<pre><b>size_t ZSTD_readSkippableFrame(void* dst, size_t dstCapacity, unsigned* magicVariant, 1213*5ff13fbcSAllan Jude const void* src, size_t srcSize); 1214*5ff13fbcSAllan Jude</b><p> Retrieves a zstd skippable frame containing data given by src, and writes it to dst buffer. 1215*5ff13fbcSAllan Jude 1216*5ff13fbcSAllan Jude The parameter magicVariant will receive the magicVariant that was supplied when the frame was written, 1217*5ff13fbcSAllan Jude i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested 1218*5ff13fbcSAllan Jude in the magicVariant. 1219*5ff13fbcSAllan Jude 1220*5ff13fbcSAllan Jude Returns an error if destination buffer is not large enough, or if the frame is not skippable. 1221*5ff13fbcSAllan Jude 1222*5ff13fbcSAllan Jude @return : number of bytes written or a ZSTD error. 1223*5ff13fbcSAllan Jude 1224*5ff13fbcSAllan Jude</p></pre><BR> 1225*5ff13fbcSAllan Jude 1226*5ff13fbcSAllan Jude<pre><b>unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size); 1227*5ff13fbcSAllan Jude</b><p> Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame. 1228*5ff13fbcSAllan Jude 1229*5ff13fbcSAllan Jude</p></pre><BR> 1230*5ff13fbcSAllan Jude 12314d3f1eafSConrad Meyer<a name="Chapter16"></a><h2>Memory management</h2><pre></pre> 12320c16b537SWarner Losh 1233*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize(int compressionLevel); 1234*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams); 1235*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params); 1236*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateDCtxSize(void); 123737f1f268SConrad Meyer</b><p> These functions make it possible to estimate memory usage 123837f1f268SConrad Meyer of a future {D,C}Ctx, before its creation. 12399cbefe25SConrad Meyer 124037f1f268SConrad Meyer ZSTD_estimateCCtxSize() will provide a memory budget large enough 124137f1f268SConrad Meyer for any compression level up to selected one. 124237f1f268SConrad Meyer Note : Unlike ZSTD_estimateCStreamSize*(), this estimate 124337f1f268SConrad Meyer does not include space for a window buffer. 124437f1f268SConrad Meyer Therefore, the estimation is only guaranteed for single-shot compressions, not streaming. 124537f1f268SConrad Meyer The estimate will assume the input may be arbitrarily large, 124637f1f268SConrad Meyer which is the worst case. 12479cbefe25SConrad Meyer 124837f1f268SConrad Meyer When srcSize can be bound by a known and rather "small" value, 124937f1f268SConrad Meyer this fact can be used to provide a tighter estimation 125037f1f268SConrad Meyer because the CCtx compression context will need less memory. 125137f1f268SConrad Meyer This tighter estimation can be provided by more advanced functions 125237f1f268SConrad Meyer ZSTD_estimateCCtxSize_usingCParams(), which can be used in tandem with ZSTD_getCParams(), 125337f1f268SConrad Meyer and ZSTD_estimateCCtxSize_usingCCtxParams(), which can be used in tandem with ZSTD_CCtxParams_setParameter(). 125437f1f268SConrad Meyer Both can be used to estimate memory using custom compression parameters and arbitrary srcSize limits. 125537f1f268SConrad Meyer 125637f1f268SConrad Meyer Note 2 : only single-threaded compression is supported. 125737f1f268SConrad Meyer ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1. 125837f1f268SConrad Meyer 12590c16b537SWarner Losh</p></pre><BR> 12600c16b537SWarner Losh 1261*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize(int compressionLevel); 1262*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams); 1263*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params); 1264*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateDStreamSize(size_t windowSize); 1265*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize); 12660c16b537SWarner Losh</b><p> ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one. 12670c16b537SWarner Losh It will also consider src size to be arbitrarily "large", which is worst case. 12680c16b537SWarner Losh If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation. 12690c16b537SWarner Losh ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. 12702b9c00cbSConrad Meyer ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1. 127119fcbaf1SConrad Meyer Note : CStream size estimation is only correct for single-threaded compression. 12720c16b537SWarner Losh ZSTD_DStream memory budget depends on window Size. 12730c16b537SWarner Losh This information can be passed manually, using ZSTD_estimateDStreamSize, 12740c16b537SWarner Losh or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame(); 12750c16b537SWarner Losh Note : if streaming is init with function ZSTD_init?Stream_usingDict(), 12760c16b537SWarner Losh an internal ?Dict will be created, which additional size is not estimated here. 12770c16b537SWarner Losh In this case, get total size by adding ZSTD_estimate?DictSize 12780c16b537SWarner Losh</p></pre><BR> 12790c16b537SWarner Losh 1280*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel); 1281*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod); 1282*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod); 12830c16b537SWarner Losh</b><p> ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict(). 128419fcbaf1SConrad Meyer ZSTD_estimateCDictSize_advanced() makes it possible to control compression parameters precisely, like ZSTD_createCDict_advanced(). 128519fcbaf1SConrad Meyer Note : dictionaries created by reference (`ZSTD_dlm_byRef`) are logically smaller. 12860c16b537SWarner Losh 12870c16b537SWarner Losh</p></pre><BR> 12880c16b537SWarner Losh 1289*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize); 1290*5ff13fbcSAllan JudeZSTDLIB_STATIC_API ZSTD_CStream* ZSTD_initStaticCStream(void* workspace, size_t workspaceSize); </b>/**< same as ZSTD_initStaticCCtx() */<b> 129119fcbaf1SConrad Meyer</b><p> Initialize an object using a pre-allocated fixed-size buffer. 129219fcbaf1SConrad Meyer workspace: The memory area to emplace the object into. 129319fcbaf1SConrad Meyer Provided pointer *must be 8-bytes aligned*. 129419fcbaf1SConrad Meyer Buffer must outlive object. 129519fcbaf1SConrad Meyer workspaceSize: Use ZSTD_estimate*Size() to determine 129619fcbaf1SConrad Meyer how large workspace must be to support target scenario. 129719fcbaf1SConrad Meyer @return : pointer to object (same address as workspace, just different type), 129819fcbaf1SConrad Meyer or NULL if error (size too small, incorrect alignment, etc.) 129919fcbaf1SConrad Meyer Note : zstd will never resize nor malloc() when using a static buffer. 130019fcbaf1SConrad Meyer If the object requires more memory than available, 130119fcbaf1SConrad Meyer zstd will just error out (typically ZSTD_error_memory_allocation). 13020c16b537SWarner Losh Note 2 : there is no corresponding "free" function. 130319fcbaf1SConrad Meyer Since workspace is allocated externally, it must be freed externally too. 130419fcbaf1SConrad Meyer Note 3 : cParams : use ZSTD_getCParams() to convert a compression level 130519fcbaf1SConrad Meyer into its associated cParams. 130619fcbaf1SConrad Meyer Limitation 1 : currently not compatible with internal dictionary creation, triggered by 130719fcbaf1SConrad Meyer ZSTD_CCtx_loadDictionary(), ZSTD_initCStream_usingDict() or ZSTD_initDStream_usingDict(). 130819fcbaf1SConrad Meyer Limitation 2 : static cctx currently not compatible with multi-threading. 130919fcbaf1SConrad Meyer Limitation 3 : static dctx is incompatible with legacy support. 13100c16b537SWarner Losh 13110c16b537SWarner Losh</p></pre><BR> 13120c16b537SWarner Losh 1313*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); </b>/**< same as ZSTD_initStaticDCtx() */<b> 131419fcbaf1SConrad Meyer</b></pre><BR> 131519fcbaf1SConrad Meyer<pre><b>typedef void* (*ZSTD_allocFunction) (void* opaque, size_t size); 131619fcbaf1SConrad Meyertypedef void (*ZSTD_freeFunction) (void* opaque, void* address); 131719fcbaf1SConrad Meyertypedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; void* opaque; } ZSTD_customMem; 1318f7cd7fe5SConrad Meyerstatic 1319f7cd7fe5SConrad Meyer#ifdef __GNUC__ 1320f7cd7fe5SConrad Meyer__attribute__((__unused__)) 1321f7cd7fe5SConrad Meyer#endif 1322f7cd7fe5SConrad MeyerZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; </b>/**< this constant defers to stdlib's functions */<b> 132319fcbaf1SConrad Meyer</b><p> These prototypes make it possible to pass your own allocation/free functions. 132419fcbaf1SConrad Meyer ZSTD_customMem is provided at creation time, using ZSTD_create*_advanced() variants listed below. 132519fcbaf1SConrad Meyer All allocation/free operations will be completed using these custom variants instead of regular <stdlib.h> ones. 132619fcbaf1SConrad Meyer 132719fcbaf1SConrad Meyer</p></pre><BR> 132819fcbaf1SConrad Meyer 1329*5ff13fbcSAllan Jude<pre><b>typedef struct POOL_ctx_s ZSTD_threadPool; 1330*5ff13fbcSAllan JudeZSTDLIB_STATIC_API ZSTD_threadPool* ZSTD_createThreadPool(size_t numThreads); 1331*5ff13fbcSAllan JudeZSTDLIB_STATIC_API void ZSTD_freeThreadPool (ZSTD_threadPool* pool); </b>/* accept NULL pointer */<b> 1332*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_CCtx_refThreadPool(ZSTD_CCtx* cctx, ZSTD_threadPool* pool); 1333*5ff13fbcSAllan Jude</b><p> These prototypes make it possible to share a thread pool among multiple compression contexts. 1334*5ff13fbcSAllan Jude This can limit resources for applications with multiple threads where each one uses 1335*5ff13fbcSAllan Jude a threaded compression mode (via ZSTD_c_nbWorkers parameter). 1336*5ff13fbcSAllan Jude ZSTD_createThreadPool creates a new thread pool with a given number of threads. 1337*5ff13fbcSAllan Jude Note that the lifetime of such pool must exist while being used. 1338*5ff13fbcSAllan Jude ZSTD_CCtx_refThreadPool assigns a thread pool to a context (use NULL argument value 1339*5ff13fbcSAllan Jude to use an internal thread pool). 1340*5ff13fbcSAllan Jude ZSTD_freeThreadPool frees a thread pool, accepts NULL pointer. 1341*5ff13fbcSAllan Jude 1342*5ff13fbcSAllan Jude</p></pre><BR> 1343*5ff13fbcSAllan Jude 13444d3f1eafSConrad Meyer<a name="Chapter17"></a><h2>Advanced compression functions</h2><pre></pre> 134519fcbaf1SConrad Meyer 1346*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel); 13470c16b537SWarner Losh</b><p> Create a digested dictionary for compression 1348a0483764SConrad Meyer Dictionary content is just referenced, not duplicated. 1349a0483764SConrad Meyer As a consequence, `dictBuffer` **must** outlive CDict, 1350a0483764SConrad Meyer and its content must remain unmodified throughout the lifetime of CDict. 13519cbefe25SConrad Meyer note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef 13520c16b537SWarner Losh</p></pre><BR> 13530c16b537SWarner Losh 1354*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize); 13550c16b537SWarner Losh</b><p> @return ZSTD_compressionParameters structure for a selected compression level and estimated srcSize. 13560c16b537SWarner Losh `estimatedSrcSize` value is optional, select 0 if not known 13570c16b537SWarner Losh</p></pre><BR> 13580c16b537SWarner Losh 1359*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize); 13600c16b537SWarner Losh</b><p> same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component `ZSTD_compressionParameters`. 1361052d3c12SConrad Meyer All fields of `ZSTD_frameParameters` are set to default : contentSize=1, checksum=0, noDictID=0 13620c16b537SWarner Losh</p></pre><BR> 13630c16b537SWarner Losh 1364*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_checkCParams(ZSTD_compressionParameters params); 13652b9c00cbSConrad Meyer</b><p> Ensure param values remain within authorized range. 13662b9c00cbSConrad Meyer @return 0 on success, or an error code (can be checked with ZSTD_isError()) 13670c16b537SWarner Losh</p></pre><BR> 13680c16b537SWarner Losh 1369*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize); 13700c16b537SWarner Losh</b><p> optimize params for a given `srcSize` and `dictSize`. 13712b9c00cbSConrad Meyer `srcSize` can be unknown, in which case use ZSTD_CONTENTSIZE_UNKNOWN. 13722b9c00cbSConrad Meyer `dictSize` must be `0` when there is no dictionary. 13732b9c00cbSConrad Meyer cPar can be invalid : all parameters will be clamped within valid range in the @return struct. 13742b9c00cbSConrad Meyer This function never fails (wide contract) 13750c16b537SWarner Losh</p></pre><BR> 13760c16b537SWarner Losh 1377*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_compress2") 1378*5ff13fbcSAllan Judesize_t ZSTD_compress_advanced(ZSTD_CCtx* cctx, 13790c16b537SWarner Losh void* dst, size_t dstCapacity, 13800c16b537SWarner Losh const void* src, size_t srcSize, 13810c16b537SWarner Losh const void* dict,size_t dictSize, 13820c16b537SWarner Losh ZSTD_parameters params); 13839cbefe25SConrad Meyer</b><p> Note : this function is now DEPRECATED. 13849cbefe25SConrad Meyer It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters. 1385*5ff13fbcSAllan Jude This prototype will generate compilation warnings. 13860c16b537SWarner Losh</p></pre><BR> 13870c16b537SWarner Losh 1388*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_compress2 with ZSTD_CCtx_loadDictionary") 1389*5ff13fbcSAllan Judesize_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx, 13900c16b537SWarner Losh void* dst, size_t dstCapacity, 13910c16b537SWarner Losh const void* src, size_t srcSize, 1392a0483764SConrad Meyer const ZSTD_CDict* cdict, 1393a0483764SConrad Meyer ZSTD_frameParameters fParams); 1394*5ff13fbcSAllan Jude</b><p> Note : this function is now DEPRECATED. 13959cbefe25SConrad Meyer It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters. 1396*5ff13fbcSAllan Jude This prototype will generate compilation warnings. 13970c16b537SWarner Losh</p></pre><BR> 13980c16b537SWarner Losh 1399*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtx_loadDictionary_byReference(ZSTD_CCtx* cctx, const void* dict, size_t dictSize); 1400a0483764SConrad Meyer</b><p> Same as ZSTD_CCtx_loadDictionary(), but dictionary content is referenced, instead of being copied into CCtx. 1401a0483764SConrad Meyer It saves some memory, but also requires that `dict` outlives its usage within `cctx` 1402a0483764SConrad Meyer</p></pre><BR> 1403a0483764SConrad Meyer 1404*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtx_loadDictionary_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType); 1405a0483764SConrad Meyer</b><p> Same as ZSTD_CCtx_loadDictionary(), but gives finer control over 1406a0483764SConrad Meyer how to load the dictionary (by copy ? by reference ?) 1407a0483764SConrad Meyer and how to interpret it (automatic ? force raw mode ? full mode only ?) 1408a0483764SConrad Meyer</p></pre><BR> 1409a0483764SConrad Meyer 1410*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType); 1411a0483764SConrad Meyer</b><p> Same as ZSTD_CCtx_refPrefix(), but gives finer control over 1412a0483764SConrad Meyer how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) 1413a0483764SConrad Meyer</p></pre><BR> 1414a0483764SConrad Meyer 1415*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtx_getParameter(const ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value); 1416a0483764SConrad Meyer</b><p> Get the requested compression parameter value, selected by enum ZSTD_cParameter, 1417a0483764SConrad Meyer and store it into int* value. 1418a0483764SConrad Meyer @return : 0, or an error code (which can be tested with ZSTD_isError()). 1419a0483764SConrad Meyer 1420a0483764SConrad Meyer</p></pre><BR> 1421a0483764SConrad Meyer 1422*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API ZSTD_CCtx_params* ZSTD_createCCtxParams(void); 1423*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params); </b>/* accept NULL pointer */<b> 1424a0483764SConrad Meyer</b><p> Quick howto : 1425a0483764SConrad Meyer - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure 14262b9c00cbSConrad Meyer - ZSTD_CCtxParams_setParameter() : Push parameters one by one into 1427a0483764SConrad Meyer an existing ZSTD_CCtx_params structure. 1428a0483764SConrad Meyer This is similar to 1429a0483764SConrad Meyer ZSTD_CCtx_setParameter(). 1430a0483764SConrad Meyer - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to 1431a0483764SConrad Meyer an existing CCtx. 1432a0483764SConrad Meyer These parameters will be applied to 1433a0483764SConrad Meyer all subsequent frames. 1434a0483764SConrad Meyer - ZSTD_compressStream2() : Do compression using the CCtx. 1435*5ff13fbcSAllan Jude - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer. 1436a0483764SConrad Meyer 1437a0483764SConrad Meyer This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams() 1438a0483764SConrad Meyer for static allocation of CCtx for single-threaded compression. 1439a0483764SConrad Meyer 1440a0483764SConrad Meyer</p></pre><BR> 1441a0483764SConrad Meyer 1442*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params); 1443a0483764SConrad Meyer</b><p> Reset params to default values. 1444a0483764SConrad Meyer 1445a0483764SConrad Meyer</p></pre><BR> 1446a0483764SConrad Meyer 1447*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel); 1448a0483764SConrad Meyer</b><p> Initializes the compression parameters of cctxParams according to 1449a0483764SConrad Meyer compression level. All other parameters are reset to their default values. 1450a0483764SConrad Meyer 1451a0483764SConrad Meyer</p></pre><BR> 1452a0483764SConrad Meyer 1453*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params); 1454a0483764SConrad Meyer</b><p> Initializes the compression and frame parameters of cctxParams according to 1455a0483764SConrad Meyer params. All other parameters are reset to their default values. 1456a0483764SConrad Meyer 1457a0483764SConrad Meyer</p></pre><BR> 1458a0483764SConrad Meyer 1459*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, int value); 1460a0483764SConrad Meyer</b><p> Similar to ZSTD_CCtx_setParameter. 1461a0483764SConrad Meyer Set one compression parameter, selected by enum ZSTD_cParameter. 1462f7cd7fe5SConrad Meyer Parameters must be applied to a ZSTD_CCtx using 1463f7cd7fe5SConrad Meyer ZSTD_CCtx_setParametersUsingCCtxParams(). 1464f7cd7fe5SConrad Meyer @result : a code representing success or failure (which can be tested with 1465f7cd7fe5SConrad Meyer ZSTD_isError()). 1466a0483764SConrad Meyer 1467a0483764SConrad Meyer</p></pre><BR> 1468a0483764SConrad Meyer 1469*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_getParameter(const ZSTD_CCtx_params* params, ZSTD_cParameter param, int* value); 1470a0483764SConrad Meyer</b><p> Similar to ZSTD_CCtx_getParameter. 1471a0483764SConrad Meyer Get the requested value of one compression parameter, selected by enum ZSTD_cParameter. 1472a0483764SConrad Meyer @result : 0, or an error code (which can be tested with ZSTD_isError()). 1473a0483764SConrad Meyer 1474a0483764SConrad Meyer</p></pre><BR> 1475a0483764SConrad Meyer 1476*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_CCtx_setParametersUsingCCtxParams( 1477a0483764SConrad Meyer ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params); 1478a0483764SConrad Meyer</b><p> Apply a set of ZSTD_CCtx_params to the compression context. 1479a0483764SConrad Meyer This can be done even after compression is started, 1480a0483764SConrad Meyer if nbWorkers==0, this will have no impact until a new compression is started. 1481a0483764SConrad Meyer if nbWorkers>=1, new parameters will be picked up at next job, 1482a0483764SConrad Meyer with a few restrictions (windowLog, pledgedSrcSize, nbWorkers, jobSize, and overlapLog are not updated). 1483a0483764SConrad Meyer 1484a0483764SConrad Meyer</p></pre><BR> 1485a0483764SConrad Meyer 1486*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_compressStream2_simpleArgs ( 1487a0483764SConrad Meyer ZSTD_CCtx* cctx, 1488a0483764SConrad Meyer void* dst, size_t dstCapacity, size_t* dstPos, 1489a0483764SConrad Meyer const void* src, size_t srcSize, size_t* srcPos, 1490a0483764SConrad Meyer ZSTD_EndDirective endOp); 1491a0483764SConrad Meyer</b><p> Same as ZSTD_compressStream2(), 1492a0483764SConrad Meyer but using only integral types as arguments. 1493a0483764SConrad Meyer This variant might be helpful for binders from dynamic languages 1494a0483764SConrad Meyer which have troubles handling structures containing memory pointers. 1495a0483764SConrad Meyer 1496a0483764SConrad Meyer</p></pre><BR> 1497a0483764SConrad Meyer 14984d3f1eafSConrad Meyer<a name="Chapter18"></a><h2>Advanced decompression functions</h2><pre></pre> 14990c16b537SWarner Losh 1500*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API unsigned ZSTD_isFrame(const void* buffer, size_t size); 15010c16b537SWarner Losh</b><p> Tells if the content of `buffer` starts with a valid Frame Identifier. 15020c16b537SWarner Losh Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0. 15030c16b537SWarner Losh Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled. 15040c16b537SWarner Losh Note 3 : Skippable Frame Identifiers are considered valid. 15050c16b537SWarner Losh</p></pre><BR> 15060c16b537SWarner Losh 1507*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize); 15080c16b537SWarner Losh</b><p> Create a digested dictionary, ready to start decompression operation without startup delay. 15090c16b537SWarner Losh Dictionary content is referenced, and therefore stays in dictBuffer. 15100c16b537SWarner Losh It is important that dictBuffer outlives DDict, 15110c16b537SWarner Losh it must remain read accessible throughout the lifetime of DDict 15120c16b537SWarner Losh</p></pre><BR> 15130c16b537SWarner Losh 1514*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); 1515a0483764SConrad Meyer</b><p> Same as ZSTD_DCtx_loadDictionary(), 1516a0483764SConrad Meyer but references `dict` content instead of copying it into `dctx`. 1517a0483764SConrad Meyer This saves memory if `dict` remains around., 1518a0483764SConrad Meyer However, it's imperative that `dict` remains accessible (and unmodified) while being used, so it must outlive decompression. 1519a0483764SConrad Meyer</p></pre><BR> 1520a0483764SConrad Meyer 1521*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType); 1522a0483764SConrad Meyer</b><p> Same as ZSTD_DCtx_loadDictionary(), 1523a0483764SConrad Meyer but gives direct control over 1524a0483764SConrad Meyer how to load the dictionary (by copy ? by reference ?) 1525a0483764SConrad Meyer and how to interpret it (automatic ? force raw mode ? full mode only ?). 1526a0483764SConrad Meyer</p></pre><BR> 1527a0483764SConrad Meyer 1528*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType); 1529a0483764SConrad Meyer</b><p> Same as ZSTD_DCtx_refPrefix(), but gives finer control over 1530a0483764SConrad Meyer how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) 1531a0483764SConrad Meyer</p></pre><BR> 1532a0483764SConrad Meyer 1533*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize); 1534a0483764SConrad Meyer</b><p> Refuses allocating internal buffers for frames requiring a window size larger than provided limit. 1535a0483764SConrad Meyer This protects a decoder context from reserving too much memory for itself (potential attack scenario). 1536a0483764SConrad Meyer This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode. 1537a0483764SConrad Meyer By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) 1538a0483764SConrad Meyer @return : 0, or an error code (which can be tested using ZSTD_isError()). 1539a0483764SConrad Meyer 1540a0483764SConrad Meyer</p></pre><BR> 1541a0483764SConrad Meyer 1542*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int* value); 1543f7cd7fe5SConrad Meyer</b><p> Get the requested decompression parameter value, selected by enum ZSTD_dParameter, 1544f7cd7fe5SConrad Meyer and store it into int* value. 1545f7cd7fe5SConrad Meyer @return : 0, or an error code (which can be tested with ZSTD_isError()). 1546f7cd7fe5SConrad Meyer 1547f7cd7fe5SConrad Meyer</p></pre><BR> 1548f7cd7fe5SConrad Meyer 1549*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_DCtx_setParameter() instead") 1550*5ff13fbcSAllan Judesize_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format); 1551*5ff13fbcSAllan Jude</b><p> This function is REDUNDANT. Prefer ZSTD_DCtx_setParameter(). 1552*5ff13fbcSAllan Jude Instruct the decoder context about what kind of data to decode next. 1553a0483764SConrad Meyer This instruction is mandatory to decode data without a fully-formed header, 1554a0483764SConrad Meyer such ZSTD_f_zstd1_magicless for example. 1555a0483764SConrad Meyer @return : 0, or an error code (which can be tested using ZSTD_isError()). 1556a0483764SConrad Meyer</p></pre><BR> 1557a0483764SConrad Meyer 1558*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_decompressStream_simpleArgs ( 1559a0483764SConrad Meyer ZSTD_DCtx* dctx, 1560a0483764SConrad Meyer void* dst, size_t dstCapacity, size_t* dstPos, 1561a0483764SConrad Meyer const void* src, size_t srcSize, size_t* srcPos); 1562a0483764SConrad Meyer</b><p> Same as ZSTD_decompressStream(), 1563a0483764SConrad Meyer but using only integral types as arguments. 1564a0483764SConrad Meyer This can be helpful for binders from dynamic languages 1565a0483764SConrad Meyer which have troubles handling structures containing memory pointers. 1566a0483764SConrad Meyer 1567a0483764SConrad Meyer</p></pre><BR> 1568a0483764SConrad Meyer 15694d3f1eafSConrad Meyer<a name="Chapter19"></a><h2>Advanced streaming functions</h2><pre> Warning : most of these functions are now redundant with the Advanced API. 1570a0483764SConrad Meyer Once Advanced API reaches "stable" status, 1571a0483764SConrad Meyer redundant functions will be deprecated, and then at some point removed. 1572a0483764SConrad Meyer<BR></pre> 15730c16b537SWarner Losh 1574f7cd7fe5SConrad Meyer<h3>Advanced Streaming compression functions</h3><pre></pre><b><pre></pre></b><BR> 1575*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions") 1576*5ff13fbcSAllan Judesize_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, 15779cbefe25SConrad Meyer int compressionLevel, 15789cbefe25SConrad Meyer unsigned long long pledgedSrcSize); 1579*5ff13fbcSAllan Jude</b><p> This function is DEPRECATED, and equivalent to: 1580f7cd7fe5SConrad Meyer ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); 1581f7cd7fe5SConrad Meyer ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any) 1582f7cd7fe5SConrad Meyer ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); 1583f7cd7fe5SConrad Meyer ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); 1584f7cd7fe5SConrad Meyer 1585f7cd7fe5SConrad Meyer pledgedSrcSize must be correct. If it is not known at init time, use 1586f7cd7fe5SConrad Meyer ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs, 1587f7cd7fe5SConrad Meyer "0" also disables frame content size field. It may be enabled in the future. 1588*5ff13fbcSAllan Jude This prototype will generate compilation warnings. 1589f7cd7fe5SConrad Meyer 1590f7cd7fe5SConrad Meyer</p></pre><BR> 1591f7cd7fe5SConrad Meyer 1592*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions") 1593*5ff13fbcSAllan Judesize_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, 1594f7cd7fe5SConrad Meyer const void* dict, size_t dictSize, 1595f7cd7fe5SConrad Meyer int compressionLevel); 1596*5ff13fbcSAllan Jude</b><p> This function is DEPRECATED, and is equivalent to: 15979cbefe25SConrad Meyer ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); 15989cbefe25SConrad Meyer ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); 15999cbefe25SConrad Meyer ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); 16009cbefe25SConrad Meyer 16019cbefe25SConrad Meyer Creates of an internal CDict (incompatible with static CCtx), except if 16029cbefe25SConrad Meyer dict == NULL or dictSize < 8, in which case no dict is used. 16039cbefe25SConrad Meyer Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if 16049cbefe25SConrad Meyer it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy. 1605*5ff13fbcSAllan Jude This prototype will generate compilation warnings. 16069cbefe25SConrad Meyer 1607f7cd7fe5SConrad Meyer</p></pre><BR> 16089cbefe25SConrad Meyer 1609*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions") 1610*5ff13fbcSAllan Judesize_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, 1611f7cd7fe5SConrad Meyer const void* dict, size_t dictSize, 1612f7cd7fe5SConrad Meyer ZSTD_parameters params, 1613f7cd7fe5SConrad Meyer unsigned long long pledgedSrcSize); 1614*5ff13fbcSAllan Jude</b><p> This function is DEPRECATED, and is approximately equivalent to: 16159cbefe25SConrad Meyer ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); 16169cbefe25SConrad Meyer // Pseudocode: Set each zstd parameter and leave the rest as-is. 16179cbefe25SConrad Meyer for ((param, value) : params) { 16189cbefe25SConrad Meyer ZSTD_CCtx_setParameter(zcs, param, value); 16199cbefe25SConrad Meyer } 16209cbefe25SConrad Meyer ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); 16219cbefe25SConrad Meyer ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); 16229cbefe25SConrad Meyer 16239cbefe25SConrad Meyer dict is loaded with ZSTD_dct_auto and ZSTD_dlm_byCopy. 16249cbefe25SConrad Meyer pledgedSrcSize must be correct. 16259cbefe25SConrad Meyer If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN. 1626*5ff13fbcSAllan Jude This prototype will generate compilation warnings. 16279cbefe25SConrad Meyer 1628f7cd7fe5SConrad Meyer</p></pre><BR> 16299cbefe25SConrad Meyer 1630*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions") 1631*5ff13fbcSAllan Judesize_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict); 1632*5ff13fbcSAllan Jude</b><p> This function is DEPRECATED, and equivalent to: 16339cbefe25SConrad Meyer ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); 16349cbefe25SConrad Meyer ZSTD_CCtx_refCDict(zcs, cdict); 16359cbefe25SConrad Meyer 16369cbefe25SConrad Meyer note : cdict will just be referenced, and must outlive compression session 1637*5ff13fbcSAllan Jude This prototype will generate compilation warnings. 16389cbefe25SConrad Meyer 1639f7cd7fe5SConrad Meyer</p></pre><BR> 16409cbefe25SConrad Meyer 1641*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions") 1642*5ff13fbcSAllan Judesize_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, 1643f7cd7fe5SConrad Meyer const ZSTD_CDict* cdict, 1644f7cd7fe5SConrad Meyer ZSTD_frameParameters fParams, 1645f7cd7fe5SConrad Meyer unsigned long long pledgedSrcSize); 1646f7cd7fe5SConrad Meyer</b><p> This function is DEPRECATED, and is approximately equivalent to: 16479cbefe25SConrad Meyer ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); 16489cbefe25SConrad Meyer // Pseudocode: Set each zstd frame parameter and leave the rest as-is. 16499cbefe25SConrad Meyer for ((fParam, value) : fParams) { 16509cbefe25SConrad Meyer ZSTD_CCtx_setParameter(zcs, fParam, value); 16519cbefe25SConrad Meyer } 16529cbefe25SConrad Meyer ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); 16539cbefe25SConrad Meyer ZSTD_CCtx_refCDict(zcs, cdict); 16549cbefe25SConrad Meyer 16559cbefe25SConrad Meyer same as ZSTD_initCStream_usingCDict(), with control over frame parameters. 16569cbefe25SConrad Meyer pledgedSrcSize must be correct. If srcSize is not known at init time, use 16579cbefe25SConrad Meyer value ZSTD_CONTENTSIZE_UNKNOWN. 1658*5ff13fbcSAllan Jude This prototype will generate compilation warnings. 16599cbefe25SConrad Meyer 1660f7cd7fe5SConrad Meyer</p></pre><BR> 16619cbefe25SConrad Meyer 1662*5ff13fbcSAllan Jude<pre><b>ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions") 1663*5ff13fbcSAllan Judesize_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize); 1664*5ff13fbcSAllan Jude</b><p> This function is DEPRECATED, and is equivalent to: 16652b9c00cbSConrad Meyer ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); 16662b9c00cbSConrad Meyer ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); 1667*5ff13fbcSAllan Jude Note: ZSTD_resetCStream() interprets pledgedSrcSize == 0 as ZSTD_CONTENTSIZE_UNKNOWN, but 1668*5ff13fbcSAllan Jude ZSTD_CCtx_setPledgedSrcSize() does not do the same, so ZSTD_CONTENTSIZE_UNKNOWN must be 1669*5ff13fbcSAllan Jude explicitly specified. 16702b9c00cbSConrad Meyer 16712b9c00cbSConrad Meyer start a new frame, using same parameters from previous frame. 16720f743729SConrad Meyer This is typically useful to skip dictionary loading stage, since it will re-use it in-place. 16730c16b537SWarner Losh Note that zcs must be init at least once before using ZSTD_resetCStream(). 1674052d3c12SConrad Meyer If pledgedSrcSize is not known at reset time, use macro ZSTD_CONTENTSIZE_UNKNOWN. 16750c16b537SWarner Losh If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end. 1676052d3c12SConrad Meyer For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs, 167719fcbaf1SConrad Meyer but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead. 16780c16b537SWarner Losh @return : 0, or an error code (which can be tested using ZSTD_isError()) 1679*5ff13fbcSAllan Jude This prototype will generate compilation warnings. 16800f743729SConrad Meyer 16810c16b537SWarner Losh</p></pre><BR> 16820c16b537SWarner Losh 168319fcbaf1SConrad Meyer<pre><b>typedef struct { 16840f743729SConrad Meyer unsigned long long ingested; </b>/* nb input bytes read and buffered */<b> 16850f743729SConrad Meyer unsigned long long consumed; </b>/* nb input bytes actually compressed */<b> 16860f743729SConrad Meyer unsigned long long produced; </b>/* nb of compressed bytes generated and buffered */<b> 16870f743729SConrad Meyer unsigned long long flushed; </b>/* nb of compressed bytes flushed : not provided; can be tracked from caller side */<b> 16880f743729SConrad Meyer unsigned currentJobID; </b>/* MT only : latest started job nb */<b> 16890f743729SConrad Meyer unsigned nbActiveWorkers; </b>/* MT only : nb of workers actively compressing at probe time */<b> 169019fcbaf1SConrad Meyer} ZSTD_frameProgression; 169119fcbaf1SConrad Meyer</b></pre><BR> 1692*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx); 16930f743729SConrad Meyer</b><p> Tell how many bytes are ready to be flushed immediately. 16940f743729SConrad Meyer Useful for multithreading scenarios (nbWorkers >= 1). 16950f743729SConrad Meyer Probe the oldest active job, defined as oldest job not yet entirely flushed, 16960f743729SConrad Meyer and check its output buffer. 16970f743729SConrad Meyer @return : amount of data stored in oldest job and ready to be flushed immediately. 16980f743729SConrad Meyer if @return == 0, it means either : 16990f743729SConrad Meyer + there is no active job (could be checked with ZSTD_frameProgression()), or 17000f743729SConrad Meyer + oldest job is still actively compressing data, 17010f743729SConrad Meyer but everything it has produced has also been flushed so far, 1702a0483764SConrad Meyer therefore flush speed is limited by production speed of oldest job 1703a0483764SConrad Meyer irrespective of the speed of concurrent (and newer) jobs. 17040f743729SConrad Meyer 17050f743729SConrad Meyer</p></pre><BR> 17060f743729SConrad Meyer 1707f7cd7fe5SConrad Meyer<h3>Advanced Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR> 1708*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); 1709f7cd7fe5SConrad Meyer</b><p> 1710f7cd7fe5SConrad Meyer ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); 1711f7cd7fe5SConrad Meyer ZSTD_DCtx_loadDictionary(zds, dict, dictSize); 1712f7cd7fe5SConrad Meyer 1713f7cd7fe5SConrad Meyer note: no dictionary will be used if dict == NULL or dictSize < 8 1714f7cd7fe5SConrad Meyer Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x 1715f7cd7fe5SConrad Meyer 1716f7cd7fe5SConrad Meyer</p></pre><BR> 1717f7cd7fe5SConrad Meyer 1718*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); 1719f7cd7fe5SConrad Meyer</b><p> 17209cbefe25SConrad Meyer ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); 17219cbefe25SConrad Meyer ZSTD_DCtx_refDDict(zds, ddict); 17229cbefe25SConrad Meyer 17239cbefe25SConrad Meyer note : ddict is referenced, it must outlive decompression session 17249cbefe25SConrad Meyer Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x 17259cbefe25SConrad Meyer 1726f7cd7fe5SConrad Meyer</p></pre><BR> 17279cbefe25SConrad Meyer 1728*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_resetDStream(ZSTD_DStream* zds); 1729f7cd7fe5SConrad Meyer</b><p> 17309cbefe25SConrad Meyer ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); 17319cbefe25SConrad Meyer 17329cbefe25SConrad Meyer re-use decompression parameters from previous init; saves dictionary loading 17339cbefe25SConrad Meyer Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x 17349cbefe25SConrad Meyer 1735f7cd7fe5SConrad Meyer</p></pre><BR> 17369cbefe25SConrad Meyer 1737f7cd7fe5SConrad Meyer<a name="Chapter20"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre> 17380c16b537SWarner Losh This is an advanced API, giving full control over buffer management, for users which need direct control over memory. 17390c16b537SWarner Losh But it's also a complex one, with several restrictions, documented below. 17400c16b537SWarner Losh Prefer normal streaming API for an easier experience. 17410c16b537SWarner Losh 17420c16b537SWarner Losh<BR></pre> 17430c16b537SWarner Losh 1744f7cd7fe5SConrad Meyer<a name="Chapter21"></a><h2>Buffer-less streaming compression (synchronous mode)</h2><pre> 17450c16b537SWarner Losh A ZSTD_CCtx object is required to track streaming operations. 17460c16b537SWarner Losh Use ZSTD_createCCtx() / ZSTD_freeCCtx() to manage resource. 17470c16b537SWarner Losh ZSTD_CCtx object can be re-used multiple times within successive compression operations. 17480c16b537SWarner Losh 17490c16b537SWarner Losh Start by initializing a context. 1750*5ff13fbcSAllan Jude Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression. 17510c16b537SWarner Losh It's also possible to duplicate a reference context which has already been initialized, using ZSTD_copyCCtx() 17520c16b537SWarner Losh 17530c16b537SWarner Losh Then, consume your input using ZSTD_compressContinue(). 17540c16b537SWarner Losh There are some important considerations to keep in mind when using this advanced function : 17550c16b537SWarner Losh - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only. 17560c16b537SWarner Losh - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks. 17570c16b537SWarner Losh - Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario. 17580c16b537SWarner Losh Worst case evaluation is provided by ZSTD_compressBound(). 17590c16b537SWarner Losh ZSTD_compressContinue() doesn't guarantee recover after a failed compression. 17600c16b537SWarner Losh - ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to maximum distance size, see WindowLog). 17610c16b537SWarner Losh It remembers all previous contiguous blocks, plus one separated memory segment (which can itself consists of multiple contiguous blocks) 17620c16b537SWarner Losh - ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps. 17630c16b537SWarner Losh In which case, it will "discard" the relevant memory section from its history. 17640c16b537SWarner Losh 17650c16b537SWarner Losh Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum. 17660c16b537SWarner Losh It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame. 17670c16b537SWarner Losh Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders. 17680c16b537SWarner Losh 17690c16b537SWarner Losh `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again. 17700c16b537SWarner Losh<BR></pre> 17710c16b537SWarner Losh 1772*5ff13fbcSAllan Jude<h3>Buffer-less streaming compression functions</h3><pre></pre><b><pre>ZSTDLIB_STATIC_API size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel); 1773*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel); 1774*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); </b>/**< note: fails if cdict==NULL */<b> 1775*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); </b>/**< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */<b> 17760c16b537SWarner Losh</pre></b><BR> 1777*5ff13fbcSAllan Jude<pre><b>size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */<b> 1778*5ff13fbcSAllan Jude</b></pre><BR> 1779f7cd7fe5SConrad Meyer<a name="Chapter22"></a><h2>Buffer-less streaming decompression (synchronous mode)</h2><pre> 17800c16b537SWarner Losh A ZSTD_DCtx object is required to track streaming operations. 17810c16b537SWarner Losh Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it. 17820c16b537SWarner Losh A ZSTD_DCtx object can be re-used multiple times. 17830c16b537SWarner Losh 17840c16b537SWarner Losh First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader(). 17850c16b537SWarner Losh Frame header is extracted from the beginning of compressed frame, so providing only the frame's beginning is enough. 17860c16b537SWarner Losh Data fragment must be large enough to ensure successful decoding. 17870c16b537SWarner Losh `ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough. 17880c16b537SWarner Losh @result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled. 17890c16b537SWarner Losh >0 : `srcSize` is too small, please provide at least @result bytes on next attempt. 17900c16b537SWarner Losh errorCode, which can be tested using ZSTD_isError(). 17910c16b537SWarner Losh 17920c16b537SWarner Losh It fills a ZSTD_frameHeader structure with important information to correctly decode the frame, 17930c16b537SWarner Losh such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`). 17940c16b537SWarner Losh Note that these values could be wrong, either because of data corruption, or because a 3rd party deliberately spoofs false information. 17950c16b537SWarner Losh As a consequence, check that values remain within valid application range. 17960c16b537SWarner Losh For example, do not allocate memory blindly, check that `windowSize` is within expectation. 17970c16b537SWarner Losh Each application can set its own limits, depending on local restrictions. 17980c16b537SWarner Losh For extended interoperability, it is recommended to support `windowSize` of at least 8 MB. 17990c16b537SWarner Losh 18000c16b537SWarner Losh ZSTD_decompressContinue() needs previous data blocks during decompression, up to `windowSize` bytes. 18010c16b537SWarner Losh ZSTD_decompressContinue() is very sensitive to contiguity, 18020c16b537SWarner Losh if 2 blocks don't follow each other, make sure that either the compressor breaks contiguity at the same place, 18030c16b537SWarner Losh or that previous contiguous segment is large enough to properly handle maximum back-reference distance. 18040c16b537SWarner Losh There are multiple ways to guarantee this condition. 18050c16b537SWarner Losh 18060c16b537SWarner Losh The most memory efficient way is to use a round buffer of sufficient size. 18070c16b537SWarner Losh Sufficient size is determined by invoking ZSTD_decodingBufferSize_min(), 18080c16b537SWarner Losh which can @return an error code if required value is too large for current system (in 32-bits mode). 18090c16b537SWarner Losh In a round buffer methodology, ZSTD_decompressContinue() decompresses each block next to previous one, 18100c16b537SWarner Losh up to the moment there is not enough room left in the buffer to guarantee decoding another full block, 18110c16b537SWarner Losh which maximum size is provided in `ZSTD_frameHeader` structure, field `blockSizeMax`. 18120c16b537SWarner Losh At which point, decoding can resume from the beginning of the buffer. 18130c16b537SWarner Losh Note that already decoded data stored in the buffer should be flushed before being overwritten. 18140c16b537SWarner Losh 18150c16b537SWarner Losh There are alternatives possible, for example using two or more buffers of size `windowSize` each, though they consume more memory. 18160c16b537SWarner Losh 18170c16b537SWarner Losh Finally, if you control the compression process, you can also ignore all buffer size rules, 18180c16b537SWarner Losh as long as the encoder and decoder progress in "lock-step", 18190c16b537SWarner Losh aka use exactly the same buffer sizes, break contiguity at the same place, etc. 18200c16b537SWarner Losh 18210c16b537SWarner Losh Once buffers are setup, start decompression, with ZSTD_decompressBegin(). 18220c16b537SWarner Losh If decompression requires a dictionary, use ZSTD_decompressBegin_usingDict() or ZSTD_decompressBegin_usingDDict(). 18230c16b537SWarner Losh 18240c16b537SWarner Losh Then use ZSTD_nextSrcSizeToDecompress() and ZSTD_decompressContinue() alternatively. 18250c16b537SWarner Losh ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize' to ZSTD_decompressContinue(). 18260c16b537SWarner Losh ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will fail. 18270c16b537SWarner Losh 18280c16b537SWarner Losh @result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity). 18290c16b537SWarner Losh It can be zero : it just means ZSTD_decompressContinue() has decoded some metadata item. 18300c16b537SWarner Losh It can also be an error code, which can be tested with ZSTD_isError(). 18310c16b537SWarner Losh 18320c16b537SWarner Losh A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero. 18330c16b537SWarner Losh Context can then be reset to start a new decompression. 18340c16b537SWarner Losh 18350c16b537SWarner Losh Note : it's possible to know if next input to present is a header or a block, using ZSTD_nextInputType(). 18360c16b537SWarner Losh This information is not required to properly decode a frame. 18370c16b537SWarner Losh 18380c16b537SWarner Losh == Special case : skippable frames 18390c16b537SWarner Losh 18400c16b537SWarner Losh Skippable frames allow integration of user-defined data into a flow of concatenated frames. 18410c16b537SWarner Losh Skippable frames will be ignored (skipped) by decompressor. 18420c16b537SWarner Losh The format of skippable frames is as follows : 18430c16b537SWarner Losh a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F 18440c16b537SWarner Losh b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits 18450c16b537SWarner Losh c) Frame Content - any content (User Data) of length equal to Frame Size 18460c16b537SWarner Losh For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame. 18470c16b537SWarner Losh For skippable frames ZSTD_decompressContinue() always returns 0 : it only skips the content. 18480c16b537SWarner Losh<BR></pre> 18490c16b537SWarner Losh 18500c16b537SWarner Losh<h3>Buffer-less streaming decompression functions</h3><pre></pre><b><pre>typedef enum { ZSTD_frame, ZSTD_skippableFrame } ZSTD_frameType_e; 18510c16b537SWarner Loshtypedef struct { 18520c16b537SWarner Losh unsigned long long frameContentSize; </b>/* if == ZSTD_CONTENTSIZE_UNKNOWN, it means this field is not available. 0 means "empty" */<b> 18530c16b537SWarner Losh unsigned long long windowSize; </b>/* can be very large, up to <= frameContentSize */<b> 18540c16b537SWarner Losh unsigned blockSizeMax; 18550c16b537SWarner Losh ZSTD_frameType_e frameType; </b>/* if == ZSTD_skippableFrame, frameContentSize is the size of skippable content */<b> 18560c16b537SWarner Losh unsigned headerSize; 18570c16b537SWarner Losh unsigned dictID; 18580c16b537SWarner Losh unsigned checksumFlag; 18590c16b537SWarner Losh} ZSTD_frameHeader; 1860a0483764SConrad Meyer</pre></b><BR> 1861*5ff13fbcSAllan Jude<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize); </b>/**< doesn't consume input */<b> 18624d3f1eafSConrad Meyer</b>/*! ZSTD_getFrameHeader_advanced() :<b> 18634d3f1eafSConrad Meyer * same as ZSTD_getFrameHeader(), 18644d3f1eafSConrad Meyer * with added capability to select a format (like ZSTD_f_zstd1_magicless) */ 1865*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format); 1866*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize); </b>/**< when frame content size is not known, pass in frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN */<b> 18674d3f1eafSConrad Meyer</b><p> decode Frame Header, or requires larger `srcSize`. 1868a0483764SConrad Meyer @return : 0, `zfhPtr` is correctly filled, 1869a0483764SConrad Meyer >0, `srcSize` is too small, value is wanted `srcSize` amount, 1870a0483764SConrad Meyer or an error code, which can be tested using ZSTD_isError() 18710f743729SConrad Meyer</p></pre><BR> 18720f743729SConrad Meyer 1873a0483764SConrad Meyer<pre><b>typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e; 1874a0483764SConrad Meyer</b></pre><BR> 1875f7cd7fe5SConrad Meyer<a name="Chapter23"></a><h2>Block level API</h2><pre></pre> 18760c16b537SWarner Losh 18779cbefe25SConrad Meyer<pre><b></b><p> Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 100 bytes). 18789cbefe25SConrad Meyer But users will have to take in charge needed metadata to regenerate data, such as compressed and content sizes. 18790c16b537SWarner Losh 18800c16b537SWarner Losh A few rules to respect : 18810c16b537SWarner Losh - Compressing and decompressing require a context structure 18820c16b537SWarner Losh + Use ZSTD_createCCtx() and ZSTD_createDCtx() 18830c16b537SWarner Losh - It is necessary to init context before starting 18840c16b537SWarner Losh + compression : any ZSTD_compressBegin*() variant, including with dictionary 18850c16b537SWarner Losh + decompression : any ZSTD_decompressBegin*() variant, including with dictionary 18860c16b537SWarner Losh + copyCCtx() and copyDCtx() can be used too 18870c16b537SWarner Losh - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB 18880c16b537SWarner Losh + If input is larger than a block size, it's necessary to split input data into multiple blocks 18899cbefe25SConrad Meyer + For inputs larger than a single block, consider using regular ZSTD_compress() instead. 18909cbefe25SConrad Meyer Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than a block. 18919cbefe25SConrad Meyer - When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero) ! 18929cbefe25SConrad Meyer ===> In which case, nothing is produced into `dst` ! 18939cbefe25SConrad Meyer + User __must__ test for such outcome and deal directly with uncompressed data 18949cbefe25SConrad Meyer + A block cannot be declared incompressible if ZSTD_compressBlock() return value was != 0. 18959cbefe25SConrad Meyer Doing so would mess up with statistics history, leading to potential data corruption. 18969cbefe25SConrad Meyer + ZSTD_decompressBlock() _doesn't accept uncompressed data as input_ !! 18970c16b537SWarner Losh + In case of multiple successive blocks, should some of them be uncompressed, 18980c16b537SWarner Losh decoder must be informed of their existence in order to follow proper history. 18990c16b537SWarner Losh Use ZSTD_insertBlock() for such a case. 19000c16b537SWarner Losh</p></pre><BR> 19010c16b537SWarner Losh 1902*5ff13fbcSAllan Jude<h3>Raw zstd block functions</h3><pre></pre><b><pre>ZSTDLIB_STATIC_API size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx); 1903*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); 1904*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); 1905*5ff13fbcSAllan JudeZSTDLIB_STATIC_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); </b>/**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression. */<b> 19060c16b537SWarner Losh</pre></b><BR> 19070c16b537SWarner Losh</html> 19080c16b537SWarner Losh</body> 1909