Lines Matching +full:full +full:- +full:frame
3 <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
18 <li><a href="#Chapter8">Streaming compression - HowTo</a></li>
19 <li><a href="#Chapter9">Streaming decompression - HowTo</a></li>
25 <li><a href="#Chapter15">Frame size functions</a></li>
30 <li><a href="#Chapter20">Buffer-less and synchronous inner streaming functions</a></li>
31 <li><a href="#Chapter21">Buffer-less streaming compression (synchronous mode)</a></li>
32 <li><a href="#Chapter22">Buffer-less streaming decompression (synchronous mode)</a></li>
38 real-time compression scenarios at zlib-level and better compression ratios.
39 The zstd compression library provides in-memory compression and decompression
43 which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
49 - a single step (described as Simple API)
50 - a single step, reusing a context (described as Explicit context)
51 - unbounded multiple steps (described as Streaming compression)
55 - a single step (described as Simple dictionary API)
56 - a single step, reusing a dictionary (described as Bulk-processing
62 Advanced experimental APIs should never be used with a dynamically-linked
82 </b><p> Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
97 <pre><b>#define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
98 #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
100 </b><p> `src` should point to the start of a ZSTD encoded frame.
101 `srcSize` must be at least as large as the frame header.
103 @return : - decompressed size of `src` frame content, if known
104 - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
105 … - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small)
106 note 1 : a 0 return value means the frame is valid but "empty".
113 …note 3 : decompressed size is always present when compression is completed using single-pass funct…
115 note 4 : decompressed size can be very large (64-bits value),
129 @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise.
133 </b><p> `src` should point to the start of a ZSTD frame or skippable frame.
134 `srcSize` must be >= first frame size
135 @return : the compressed size of the first frame starting at `src`,
140 …ize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) </…
141 …essBound(size_t srcSize); </b>/*!< maximum compressed size in worst case single-pass scenario */<b>
152 and re-use it for each successive compression operation.
154 Note : re-using context is just a speed / resource optimization.
156 Note 2 : In multi-threaded environments,
178 and re-use it for each successive compression operation.
214 * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
215 …ZSTD_c_compressionLevel=100, </b>/* Set compression parameters according to pre-defined cLevel tab…
228 …ZSTD_c_windowLog=101, </b>/* Maximum allowed back-reference distance, expressed as power of 2.<…
242 ZSTD_c_chainLog=103, </b>/* Size of the multi-probe search table, as a power of 2.<b>
287 * default: windowlog - 7.
298 … * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
299 … * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
304 </b>/* frame parameters */<b>
305 …ZSTD_c_contentSizeFlag=200, </b>/* Content size will be written into frame header _whenever known_…
309 …ZSTD_c_checksumFlag=201, </b>/* A 32-bits checksum of content is written at end of frame (default:…
310 …ZSTD_c_dictIDFlag=202, </b>/* When applicable, dictionary's ID is written into frame header (def…
312 </b>/* multi-threading parameters */<b>
313 …</b>/* These parameters are only active if multi-threading is enabled (compiled with build macro Z…
314 … * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
315 * In a situation where it's unknown if the linked library supports multi-threading or not,
325 … * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
338 … * - 0 means "default" : value will be determined by the library, depending on strategy
339 * - 1 means "no overlap"
340 * - 9 means "full overlap", using a full window size.
342 …* 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; …
393 - an error status field, which must be tested using ZSTD_isError()
394 - lower and upper bounds, both inclusive
402 …Setting a parameter is generally only possible during frame initialization (before starting compre…
403 Exception : when using multi-threading mode (nbWorkers >= 1),
404 the following parameters can be updated _during_ compression (within same frame):
412 </b><p> Total input data size to be compressed as a single frame.
413 …Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeFlag.
414 This value will also be controlled at end of frame, and trigger an error if not respected.
416 Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame.
418 ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame.
419 Note 2 : pledgedSrcSize is only valid once, for the next frame.
420 It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN.
436 - The session : will stop compressing current frame, and make CCtx ready to start a new one.
440 They will be used to compress next frame.
442 - The parameters : changes all parameters back to "default".
446 - Both : similar to resetting the session, followed by resetting parameters.
454 ZSTD_compress2() always starts a new frame.
455 Should cctx hold data from a previously unfinished frame, everything about it is forgotten.
456 - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
457 - The function is always blocking, returns when compression is completed.
471 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
496 - an error status field, which must be tested using ZSTD_isError()
497 - both lower and upper bounds, inclusive
505 Setting a parameter is only possible during frame initialization (before starting decompression).
513 Parameters can only be reset when no active frame is being decompressed.
532 <a name="Chapter8"></a><h2>Streaming compression - HowTo</h2><pre>
536 …It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-usin…
543 it will re-use the same sticky parameters as previous compression session.
553 the output buffer is already full, in which case `input.pos < input.size`.
566 using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
567 …Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be enou…
572 block until the flush is complete or the output buffer is full.
577 Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame.
578 It will perform a flush and write frame epilogue.
579 The epilogue is required for decoders to consider a frame completed.
582 start a new frame.
584 block until the flush is complete or the output buffer is full.
585 @return : 0 if frame fully completed and fully flushed,
601 …* frame will continue: any future data can still reference previously compressed data, improving c…
603 ZSTD_e_end=2 </b>/* flush any remaining data _and_ close current frame.<b>
604 … * note that frame is only closed after compressed data is fully flushed (return value == 0).
605 * After that point, any additional data starts a new frame.
606 … * note : each frame is independent (does not reference any content from previous frame).
615 - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
616 …- Compression parameters cannot be changed once compression is started (save a list of exceptions …
617 - output->pos must be <= dstCapacity, input->pos must be <= srcSize
618 …- output->pos and input->pos will be updated. They are guaranteed to remain below their respective…
619 - endOp must be a valid directive
620 …- When nbWorkers==0 (default), function is blocking : it completes its job before returning to cal…
621 …- When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs to …
624 …- Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity, t…
625 - @return provides a minimum amount of data remaining to be flushed from internal buffers
629 … For ZSTD_e_end, @return == 0 when internal buffers are fully flushed and frame is completed.
630 - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
645 * the next read size (if non-zero and not an error). ZSTD_compressStream2()
646 * returns the minimum nb of bytes left to flush (if non-zero and not an error).
660 <a name="Chapter9"></a><h2>Streaming decompression - HowTo</h2><pre>
663 ZSTD_DStream objects can be re-used multiple times.
678 @return : 0 when a frame is completely decoded and fully flushed,
680 …r value > 0, which means there is still some decoding or flushing to do to complete current frame :
682 that will never request more than the remaining frame size.
730 …Dict can be created once and shared by multiple threads concurrently, since its usage is read-only.
751 and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no)
777 It can still be loaded, but as a content-only dictionary.
783 Non-conformant dictionaries can still be loaded, but as content-only dictionaries.
789 Non-conformant dictionaries can still be loaded, but as content-only dictionaries.
793 </b><p> Provides the dictID required to decompressed the frame stored within `src`.
796 - The frame does not require a dictionary to be decoded (most common case).
797 …- The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hid…
798 Note : this use case also happens when using a non-conformant dictionary.
799 …- `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if…
800 - This is not a Zstandard frame.
808 ZSTD_reset_session_and_parameters. Prefixes are single-use.
815 Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
816 meaning "return to no-dictionary mode".
818 To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
820 It's also a CPU consuming operation, with non-negligible impact on latency.
834 The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
835 The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
838 Special : Referencing a NULL CDict means "return to no-dictionary mode".
846 </b><p> Reference a prefix (single-usage dictionary) for next compressed frame.
847 A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end).
859 It's a CPU consuming operation, with non-negligible impact on latency.
870 Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
871 meaning "return to no-dictionary mode".
873 which has a non-negligible impact on CPU usage and latency.
888 will be determined at decompression time, as per the dict ID in the frame.
895 Special: referencing a NULL DDict means "return to no-dictionary mode".
902 </b><p> Reference a prefix (single-usage dictionary) to decompress next frame.
905 Prefix is **only used once**. Reference is discarded at end of frame.
906 End of frame is reached when ZSTD_decompressStream() returns 0.
910 Prefix buffer must remain unmodified up to the end of frame,
915 A full dictionary is more costly, as it requires building tables.
960 * rep == 1 --> offset == repeat_offset_1
961 * rep == 2 --> offset == repeat_offset_2
962 * rep == 3 --> offset == repeat_offset_3
964 * rep == 1 --> offset == repeat_offset_2
965 * rep == 2 --> offset == repeat_offset_3
966 * rep == 3 --> offset == repeat_offset_1 - 1
986 int contentSizeFlag; </b>/**< 1: content size will be in frame header (when known) */<b>
987 …int checksumFlag; </b>/**< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame…
988 …int noDictIDFlag; </b>/**< 1: no dictID will be saved into frame header (dictID is only useful …
997 …ZSTD_dct_auto = 0, </b>/* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, oth…
1004 …ZSTD_dlm_byRef = 1 </b>/**< Reference dictionary content -- the dictionary buffer must outlive …
1008 …ZSTD_f_zstd1 = 0, </b>/* zstd frame format, specified in zstd_compression_format.md (def…
1009 …ZSTD_f_zstd1_magicless = 1 </b>/* Variant of zstd frame format, without initial 4-bytes magic num…
1010 * Useful to save 4 bytes per generated frame.
1034 * - The contents of the CDict can be copied into the working context. This
1037 * the compression faster per-byte of input. However, the initial copy of
1042 * - The CDict's tables can be used in-place. In this model, compression is
1044 * tables. However, this model incurs no start-up cost (as long as the
1048 * - The CDict's tables are not used at all, and instead we use the working
1080 ZSTD_ps_enable = 1, </b>/* Force-enable the feature */<b>
1084 <a name="Chapter15"></a><h2>Frame size functions</h2><pre></pre>
1089 (i.e. there should be a frame boundary at `src + srcSize`)
1090 @return : - decompressed size of all data in all successive frames
1091 - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
1092 - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1098 note 3 : decompressed size can be very large (64-bits value),
1105 read each contained frame header. This is fast as most of the data is skipped,
1106 however it does mean that all frame data must be present and valid.
1112 (i.e. there should be a frame boundary at `src + srcSize`)
1113 @return : - upper-bound for the decompressed size of all data in all successive frames
1114 - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1116 note 1 : an error can occur if `src` contains an invalid or incorrectly formatted frame.
1117 …note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD enc…
1119 …note 3 : when the decompressed size field isn't available, the upper-bound for that frame is calc…
1120 upper-bound = # blocks * min(128 KB, Window_Size)
1126 @return : size of the Frame Header,
1169 The entire source is compressed into a single frame.
1184 …- ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the match f…
1185 …- ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would in ty…
1186 …- ZSTD_c_windowLog affects offset validation: this function will return an error at higher debug l…
1189 …Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep is…
1198 </b><p> Generates a zstd skippable frame containing data given by src, and writes it to dst buffer.
1200 Skippable frames begin with a a 4-byte magic number. There are 16 possible choices of magic number,
1202 …As such, the parameter magicVariant controls the exact skippable frame magic number variant used, …
1206 …with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore inv…
1214 </b><p> Retrieves a zstd skippable frame containing data given by src, and writes it to dst buffer.
1216 …The parameter magicVariant will receive the magicVariant that was supplied when the frame was writ…
1217 i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested
1220 Returns an error if destination buffer is not large enough, or if the frame is not skippable.
1227 …b><p> Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame.
1244 Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
1256 Note 2 : only single-threaded compression is supported.
1270 …ams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression …
1271 Note : CStream size estimation is only correct for single-threaded compression.
1274 or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame();
1291 </b><p> Initialize an object using a pre-allocated fixed-size buffer.
1293 Provided pointer *must be 8-bytes aligned*.
1308 Limitation 2 : static cctx currently not compatible with multi-threading.
1360 </b><p> same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-comp…
1407 and how to interpret it (automatic ? force raw mode ? full mode only ?)
1412 how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?)
1425 - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
1426 - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
1430 - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
1434 - ZSTD_compressStream2() : Do compression using the CCtx.
1435 - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer.
1438 for static allocation of CCtx for single-threaded compression.
1454 </b><p> Initializes the compression and frame parameters of cctxParams according to
1501 </b><p> Tells if the content of `buffer` starts with a valid Frame Identifier.
1502 Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
1503 Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
1504 Note 3 : Skippable Frame Identifiers are considered valid.
1525 and how to interpret it (automatic ? force raw mode ? full mode only ?).
1530 how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?)
1536 …This parameter is only useful in streaming mode, since no internal buffer is allocated in single-p…
1553 This instruction is mandatory to decode data without a fully-formed header,
1587 "0" also disables frame content size field. It may be enabled in the future.
1603 Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if
1616 // Pseudocode: Set each zstd parameter and leave the rest as-is.
1648 // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
1655 same as ZSTD_initCStream_usingCDict(), with control over frame parameters.
1671 start a new frame, using same parameters from previous frame.
1672 This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
1732 re-use decompression parameters from previous init; saves dictionary loading
1737 <a name="Chapter20"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre>
1738 …This is an advanced API, giving full control over buffer management, for users which need direct c…
1744 <a name="Chapter21"></a><h2>Buffer-less streaming compression (synchronous mode)</h2><pre>
1747 ZSTD_CCtx object can be re-used multiple times within successive compression operations.
1755 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
1756 - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
1757 …- Caller must ensure there is enough space in `dst` to store compressed data under worst case scen…
1760 …- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to max…
1762 …- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
1765 Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
1766 …It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
1769 `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
1772 <h3>Buffer-less streaming compression functions</h3><pre></pre><b><pre>ZSTDLIB_STATIC_API size_t ZS…
1779 <a name="Chapter22"></a><h2>Buffer-less streaming decompression (synchronous mode)</h2><pre>
1782 A ZSTD_DCtx object can be re-used multiple times.
1784 First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader().
1785 …Frame header is extracted from the beginning of compressed frame, so providing only the frame's be…
1792 It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
1793 such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
1803 …or that previous contiguous segment is large enough to properly handle maximum back-reference dist…
1808 …which can @return an error code if required value is too large for current system (in 32-bits mode…
1810 …up to the moment there is not enough room left in the buffer to guarantee decoding another full bl…
1818 as long as the encoder and decoder progress in "lock-step",
1832 A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
1836 This information is not required to properly decode a frame.
1840 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
1843 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
1844 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
1845 c) Frame Content - any content (User Data) of length equal to Frame Size
1846 For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
1850 <h3>Buffer-less streaming decompression functions</h3><pre></pre><b><pre>typedef enum { ZSTD_frame,…
1866 …ong long windowSize, unsigned long long frameContentSize); </b>/**< when frame content size is no…
1867 </b><p> decode Frame Header, or requires larger `srcSize`.
1877 <pre><b></b><p> Frame metadata cost is typically ~12 bytes, which can be non-negligible for very…
1881 - Compressing and decompressing require a context structure
1883 - It is necessary to init context before starting
1887 - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
1890 …Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than…
1891 …- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero)…
1905 …); </b>/**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression.…