Lines Matching +full:frame +full:- +full:buffer
1 // SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0-only
3 * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
6 * This source code is licensed under both the BSD-style license (found in the
9 * You may select, at your option, one of the above-listed licenses.
44 real-time compression scenarios at zlib-level and better compression ratios.
45 The zstd compression library provides in-memory compression and decompression
49 which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
55 - a single step (described as Simple API)
56 - a single step, reusing a context (described as Explicit context)
57 - unbounded multiple steps (described as Streaming compression)
61 - a single step (described as Simple dictionary API)
62 - a single step, reusing a dictionary (described as Bulk-processing
68 Advanced experimental APIs should never be used with a dynamically-linked
73 /*------ Version ------*/
98 /* All magic numbers are supposed read/written to/from files/memory using little-endian convention …
101 …50 /* all 16 values, from 0x184D2A50 to 0x184D2A5F, signal the beginning of a skippable frame */
113 * Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
131 * `src` should point to the start of a ZSTD encoded frame.
132 * `srcSize` must be at least as large as the frame header.
134 * @return : - decompressed size of `src` frame content, if known
135 * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
136 …* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too…
137 * note 1 : a 0 return value means the frame is valid but "empty".
144 …* note 3 : decompressed size is always present when compression is completed using single-pass f…
146 * note 4 : decompressed size can be very large (64-bits value),
153 #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
154 #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
162 * @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. */
166 * `src` should point to the start of a ZSTD frame or skippable frame.
167 * `srcSize` must be >= first frame size
168 * @return : the compressed size of the first frame starting at `src`,
175 …ize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /*…
176 …D_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
189 * and re-use it for each successive compression operation.
191 * Note : re-using context is just a speed / resource optimization.
193 * Note 2 : In multi-threaded environments,
216 * and re-use it for each successive compression operation.
240 … Pushed parameters are sticky : they are valid for next compressed frame, and any subsequent frame.
242 * __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .
271 * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
272 … ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
285 ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
299 ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
343 * default: windowlog - 7.
354 … * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
355 … * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
360 /* frame parameters */
361 …ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (de…
365 …ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
366 …ZSTD_c_dictIDFlag=202, /* When applicable, dictionary's ID is written into frame header (default…
368 /* multi-threading parameters */
369 …/* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_…
378 …* Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performe…
390 … * - 0 means "default" : value will be determined by the library, depending on strategy
391 * - 1 means "no overlap"
392 * - 9 means "full overlap", using a full window size.
430 * - an error status field, which must be tested using ZSTD_isError()
431 * - lower and upper bounds, both inclusive
439 …* Setting a parameter is generally only possible during frame initialization (before starting com…
440 * Exception : when using multi-threading mode (nbWorkers >= 1),
441 * the following parameters can be updated _during_ compression (within same frame):
449 * Total input data size to be compressed as a single frame.
450 …* Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeF…
451 * This value will also be controlled at end of frame, and trigger an error if not respected.
453 * Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame.
455 * ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame.
456 * Note 2 : pledgedSrcSize is only valid once, for the next frame.
457 * It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN.
473 * - The session : will stop compressing current frame, and make CCtx ready to start a new one.
477 * They will be used to compress next frame.
479 * - The parameters : changes all parameters back to "default".
483 * - Both : similar to resetting the session, followed by resetting parameters.
489 * ZSTD_compress2() always starts a new frame.
490 * Should cctx hold data from a previously unfinished frame, everything about it is forgotten.
491 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
492 * - The function is always blocking, returns when compression is completed.
517 * the streaming API will refuse to allocate memory buffer
519 …arameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mo…
540 * - an error status field, which must be tested using ZSTD_isError()
541 * - both lower and upper bounds, inclusive
549 …* Setting a parameter is only possible during frame initialization (before starting decompression…
557 * Parameters can only be reset when no active frame is being decompressed.
568 const void* src; /**< start of input buffer */
569 size_t size; /**< size of input buffer */
574 void* dst; /**< start of output buffer */
575 size_t size; /**< size of output buffer */
581 /*-***********************************************************************
582 * Streaming compression - HowTo
587 * It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-us…
594 * it will re-use the same sticky parameters as previous compression session.
604 * the output buffer is already full, in which case `input.pos < input.size`.
616 * At any moment, it's possible to flush whatever data might remain stuck within internal buffer,
617 * using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
618 * Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be en…
623 * block until the flush is complete or the output buffer is full.
625 * >0 if some data still present within internal buffer (the value is minimal estimation …
628 * Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame.
629 * It will perform a flush and write frame epilogue.
630 * The epilogue is required for decoders to consider a frame completed.
633 * start a new frame.
635 * block until the flush is complete or the output buffer is full.
636 * @return : 0 if frame fully completed and fully flushed,
637 * >0 if some data still present within internal buffer (the value is minimal estimation …
653 …* frame will continue: any future data can still reference previously compressed data, improving c…
655 ZSTD_e_end=2 /* flush any remaining data _and_ close current frame.
656 … * note that frame is only closed after compressed data is fully flushed (return value == 0).
657 * After that point, any additional data starts a new frame.
658 … * note : each frame is independent (does not reference any content from previous frame).
664 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
665 …* - Compression parameters cannot be changed once compression is started (save a list of exceptio…
666 * - output->pos must be <= dstCapacity, input->pos must be <= srcSize
667 …* - output->pos and input->pos will be updated. They are guaranteed to remain below their respect…
668 …* - When nbWorkers==0 (default), function is blocking : it completes its job before returning to …
669 …* - When nbWorkers>=1, function is non-blocking : it just acquires a copy of input, and distribut…
672 …* - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity…
673 * - @return provides a minimum amount of data remaining to be flushed from internal buffers
677 …* For ZSTD_e_end, @return == 0 when internal buffers are fully flushed and frame is com…
678 * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
689 /* These buffer sizes are softly recommended.
690 …* They are not required : ZSTD_compressStream*() happily accepts any buffer size, for both input a…
702 ZSTDLIB_API size_t ZSTD_CStreamInSize(void); /**< recommended size for input buffer */
703 ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /**< recommended size for output buffer. Guarantee …
725 * the next read size (if non-zero and not an error). ZSTD_compressStream2()
726 * returns the minimum nb of bytes left to flush (if non-zero and not an error).
735 /*-***************************************************************************
736 * Streaming decompression - HowTo
740 * ZSTD_DStream objects can be re-used multiple times.
750 * The function tries to flush all data decoded immediately, respecting output buffer size.
753 * In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
755 * @return : 0 when a frame is completely decoded and fully flushed,
757 …r value > 0, which means there is still some decoding or flushing to do to complete current frame :
759 * that will never request more than the remaining frame size.
779 ZSTDLIB_API size_t ZSTD_DStreamInSize(void); /*!< recommended size for input buffer */
780 ZSTDLIB_API size_t ZSTD_DStreamOutSize(void); /*!< recommended size for output buffer. Guarantee …
789 * or a buffer with specified information (see dictBuilder/zdict.h).
821 …Dict can be created once and shared by multiple threads concurrently, since its usage is read-only.
839 * and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) */
873 * It can still be loaded, but as a content-only dictionary. */
879 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
883 * Provides the dictID required to decompressed the frame stored within `src`.
886 * - The frame does not require a dictionary to be decoded (most common case).
887 …* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a …
888 * Note : this use case also happens when using a non-conformant dictionary.
889 …* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible…
890 * - This is not a Zstandard frame.
901 * ZSTD_reset_session_and_parameters. Prefixes are single-use.
906 * Create an internal CDict from `dict` buffer.
909 * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
910 * meaning "return to no-dictionary mode".
912 * To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
914 * It's also a CPU consuming operation, with non-negligible impact on latency.
919 * In such a case, dictionary buffer must outlive its users.
928 * The parameters ignored are labled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
929 * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
932 * Special : Referencing a NULL CDict means "return to no-dictionary mode".
939 * Reference a prefix (single-usage dictionary) for next compressed frame.
940 * A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end).
946 * Note 1 : Prefix buffer is referenced. It **must** outlive compression.
952 * It's a CPU consuming operation, with non-negligible impact on latency.
960 * Create an internal DDict from dict buffer,
964 * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
965 * meaning "return to no-dictionary mode".
967 * which has a non-negligible impact on CPU usage and latency.
982 * Special: referencing a NULL DDict means "return to no-dictionary mode".
988 * Reference a prefix (single-usage dictionary) to decompress next frame.
991 * Prefix is **only used once**. Reference is discarded at end of frame.
992 * End of frame is reached when ZSTD_decompressStream() returns 0.
995 * Note 2 : Prefix buffer is referenced. It **must** outlive decompression.
996 * Prefix buffer must remain unmodified up to the end of frame,
1043 …) ((format) == ZSTD_f_zstd1 ? 5 : 1) /* minimum input size required to query frame header size */
1059 #define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
1072 #define ZSTD_WINDOWLOG_LIMIT_DEFAULT 27 /* by default, the streaming decoder will refuse any frame
1076 …* The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional …
1087 #define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
1099 /* --- Advanced types --- */
1105 /* If seqDef.offset > 3, then this is seqDef.offset - 3
1110 * most recent repeat offset - 1
1132 int contentSizeFlag; /**< 1: content size will be in frame header (when known) */
1133 …int checksumFlag; /**< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, fo…
1134 …int noDictIDFlag; /**< 1: no dictID will be saved into frame header (dictID is only useful for …
1150 …ZSTD_dlm_byRef = 1 /**< Reference dictionary content -- the dictionary buffer must outlive its …
1154 …ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default…
1155 … ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
1156 * Useful to save 4 bytes per generated frame.
1168 * - The contents of the CDict can be copied into the working context. This
1171 * the compression faster per-byte of input. However, the initial copy of
1176 * - The CDict's tables can be used in-place. In this model, compression is
1178 * tables. However, this model incurs no start-up cost (as long as the
1182 * - The CDict's tables are not used at all, and instead we use the working
1210 * Frame size functions
1216 * (i.e. there should be a frame boundary at `src + srcSize`)
1217 * @return : - decompressed size of all data in all successive frames
1218 * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
1219 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1225 * note 3 : decompressed size can be very large (64-bits value),
1232 * read each contained frame header. This is fast as most of the data is skipped,
1233 * however it does mean that all frame data must be present and valid. */
1239 * (i.e. there should be a frame boundary at `src + srcSize`)
1240 * @return : - upper-bound for the decompressed size of all data in all successive frames
1241 * - if an error occured: ZSTD_CONTENTSIZE_ERROR
1243 * note 1 : an error can occur if `src` contains an invalid or incorrectly formatted frame.
1244 …* note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD …
1246 …* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is c…
1247 * upper-bound = # blocks * min(128 KB, Window_Size)
1253 * @return : size of the Frame Header,
1278 * does not include space for a window buffer.
1279 * Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
1291 * Note 2 : only single-threaded compression is supported.
1304 …ams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression …
1305 * Note : CStream size estimation is only correct for single-threaded compression.
1308 * or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame();
1328 * Initialize an object using a pre-allocated fixed-size buffer.
1330 * Provided pointer *must be 8-bytes aligned*.
1331 * Buffer must outlive object.
1336 * Note : zstd will never resize nor malloc() when using a static buffer.
1345 * Limitation 2 : static cctx currently not compatible with multi-threading.
1414 …* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component…
1495 /* Force back-reference distances to remain < windowSize,
1532 * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
1533 * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
1537 * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
1541 * - ZSTD_compressStream2() : Do compression using the CCtx.
1542 * - ZSTD_freeCCtxParams() : Free the memory.
1545 * for static allocation of CCtx for single-threaded compression.
1562 * Initializes the compression and frame parameters of cctxParams according to
1610 * Tells if the content of `buffer` starts with a valid Frame Identifier.
1611 * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
1612 * Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
1613 * Note 3 : Skippable Frame Identifiers are considered valid. */
1614 ZSTDLIB_API unsigned ZSTD_isFrame(const void* buffer, size_t size);
1645 …arameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mo…
1664 * MUST be large enough to fit the entire decompressed frame. This will be
1665 * checked when the frame content size is known. The data in the ZSTD_outBuffer
1669 * When this flags is enabled zstd won't allocate an output buffer, because
1671 * an input buffer large enough to fit any compressed block. This will also
1672 * avoid the memcpy() from the internal output buffer to the ZSTD_outBuffer.
1673 * If you need to avoid the input buffer allocation use the buffer-less
1677 * this flag is ALWAYS memory safe, and will never access out-of-bounds
1683 * matches. Normally zstd maintains its own buffer for this purpose, but passing
1684 * this flag tells zstd to use the user provided buffer.
1690 * This instruction is mandatory to decode data without a fully-formed header,
1724 * "0" also disables frame content size field. It may be enabled in the future.
1752 * // Pseudocode: Set each zstd parameter and leave the rest as-is.
1783 * // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
1790 * same as ZSTD_initCStream_usingCDict(), with control over frame parameters.
1806 * start a new frame, using same parameters from previous frame.
1807 * This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
1830 * consumed (input actually compressed) and produced (output) for current frame.
1831 * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
1840 * and check its output buffer.
1880 * re-use decompression parameters from previous init; saves dictionary loading
1887 * Buffer-less and synchronous inner streaming functions
1889 * This is an advanced API, giving full control over buffer management, for users which need direct…
1895 Buffer-less streaming compression (synchronous mode)
1899 ZSTD_CCtx object can be re-used multiple times within successive compression operations.
1908 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
1909 - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
1910 …- Caller must ensure there is enough space in `dst` to store compressed data under worst case scen…
1913 …- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to max…
1915 …- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
1918 Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
1919 …It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
1922 `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
1925 /*===== Buffer-less streaming compression functions =====*/
1937 /*-
1938 Buffer-less streaming decompression (synchronous mode)
1942 A ZSTD_DCtx object can be re-used multiple times.
1944 First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader().
1945 …Frame header is extracted from the beginning of compressed frame, so providing only the frame's be…
1952 It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
1953 such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
1963 …or that previous contiguous segment is large enough to properly handle maximum back-reference dist…
1966 The most memory efficient way is to use a round buffer of sufficient size.
1968 …which can @return an error code if required value is too large for current system (in 32-bits mode…
1969 …In a round buffer methodology, ZSTD_decompressContinue() decompresses each block next to previous …
1970 …up to the moment there is not enough room left in the buffer to guarantee decoding another full bl…
1972 At which point, decoding can resume from the beginning of the buffer.
1973 Note that already decoded data stored in the buffer should be flushed before being overwritten.
1977 Finally, if you control the compression process, you can also ignore all buffer size rules,
1978 as long as the encoder and decoder progress in "lock-step",
1979 aka use exactly the same buffer sizes, break contiguity at the same place, etc.
1992 A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
1996 This information is not required to properly decode a frame.
2000 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
2003 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
2004 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
2005 c) Frame Content - any content (User Data) of length equal to Frame Size
2006 For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
2010 /*===== Buffer-less streaming decompression functions =====*/
2023 * decode Frame Header, or requires larger `srcSize`.
2032 …ed long long windowSize, unsigned long long frameContentSize); /**< when frame content size is no…
2054 Block functions produce and decode raw zstd blocks, without frame metadata.
2055 …Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 1…
2059 - Compressing and decompressing require a context structure
2061 - It is necessary to init context before starting
2065 - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
2068 …Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than…
2069 …- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero)…
2084 …Size); /**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression.…