Lines Matching +full:full +full:- +full:frame
5 * This source code is licensed under both the BSD-style license (found in the
8 * You may select, at your option, one of the above-listed licenses.
45 real-time compression scenarios at zlib-level and better compression ratios.
46 The zstd compression library provides in-memory compression and decompression
50 which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
56 - a single step (described as Simple API)
57 - a single step, reusing a context (described as Explicit context)
58 - unbounded multiple steps (described as Streaming compression)
62 - a single step (described as Simple dictionary API)
63 - a single step, reusing a dictionary (described as Bulk-processing
69 Advanced experimental APIs should never be used with a dynamically-linked
74 /*------ Version ------*/
104 /* All magic numbers are supposed read/written to/from files/memory using little-endian convention …
107 …50 /* all 16 values, from 0x184D2A50 to 0x184D2A5F, signal the beginning of a skippable frame */
118 * Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
136 * `src` should point to the start of a ZSTD encoded frame.
137 * `srcSize` must be at least as large as the frame header.
139 * @return : - decompressed size of `src` frame content, if known
140 * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
141 …* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too…
142 * note 1 : a 0 return value means the frame is valid but "empty".
149 …* note 3 : decompressed size is always present when compression is completed using single-pass f…
151 * note 4 : decompressed size can be very large (64-bits value),
158 #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
159 #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
167 * @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. */
171 * `src` should point to the start of a ZSTD frame or skippable frame.
172 * `srcSize` must be >= first frame size
173 * @return : the compressed size of the first frame starting at `src`,
180 …ize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /*…
181 …D_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
195 * and re-use it for each successive compression operation.
197 * Note : re-using context is just a speed / resource optimization.
199 * Note 2 : In multi-threaded environments,
222 * and re-use it for each successive compression operation.
246 … Pushed parameters are sticky : they are valid for next compressed frame, and any subsequent frame.
248 * __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .
276 * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
277 … ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
290 ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
304 ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
349 * default: windowlog - 7.
360 … * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
361 … * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
366 /* frame parameters */
367 …ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (de…
371 …ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
372 …ZSTD_c_dictIDFlag=202, /* When applicable, dictionary's ID is written into frame header (default…
374 /* multi-threading parameters */
375 …/* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_…
376 … * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
377 * In a situation where it's unknown if the linked library supports multi-threading or not,
387 … * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
400 … * - 0 means "default" : value will be determined by the library, depending on strategy
401 * - 1 means "no overlap"
402 * - 9 means "full overlap", using a full window size.
404 …* 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; …
455 * - an error status field, which must be tested using ZSTD_isError()
456 * - lower and upper bounds, both inclusive
464 …* Setting a parameter is generally only possible during frame initialization (before starting com…
465 * Exception : when using multi-threading mode (nbWorkers >= 1),
466 * the following parameters can be updated _during_ compression (within same frame):
474 * Total input data size to be compressed as a single frame.
475 …* Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeF…
476 * This value will also be controlled at end of frame, and trigger an error if not respected.
478 * Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame.
480 * ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame.
481 * Note 2 : pledgedSrcSize is only valid once, for the next frame.
482 * It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN.
498 * - The session : will stop compressing current frame, and make CCtx ready to start a new one.
502 * They will be used to compress next frame.
504 * - The parameters : changes all parameters back to "default".
508 * - Both : similar to resetting the session, followed by resetting parameters.
514 * ZSTD_compress2() always starts a new frame.
515 * Should cctx hold data from a previously unfinished frame, everything about it is forgotten.
516 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
517 * - The function is always blocking, returns when compression is completed.
544 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
569 * - an error status field, which must be tested using ZSTD_isError()
570 * - both lower and upper bounds, inclusive
578 …* Setting a parameter is only possible during frame initialization (before starting decompression…
586 * Parameters can only be reset when no active frame is being decompressed.
610 /*-***********************************************************************
611 * Streaming compression - HowTo
616 * It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-us…
623 * it will re-use the same sticky parameters as previous compression session.
633 * the output buffer is already full, in which case `input.pos < input.size`.
646 * using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
647 * Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be en…
652 * block until the flush is complete or the output buffer is full.
657 * Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame.
658 * It will perform a flush and write frame epilogue.
659 * The epilogue is required for decoders to consider a frame completed.
662 * start a new frame.
664 * block until the flush is complete or the output buffer is full.
665 * @return : 0 if frame fully completed and fully flushed,
682 …* frame will continue: any future data can still reference previously compressed data, improving c…
684 ZSTD_e_end=2 /* flush any remaining data _and_ close current frame.
685 … * note that frame is only closed after compressed data is fully flushed (return value == 0).
686 * After that point, any additional data starts a new frame.
687 … * note : each frame is independent (does not reference any content from previous frame).
693 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
694 …* - Compression parameters cannot be changed once compression is started (save a list of exceptio…
695 * - output->pos must be <= dstCapacity, input->pos must be <= srcSize
696 …* - output->pos and input->pos will be updated. They are guaranteed to remain below their respect…
697 * - endOp must be a valid directive
698 …* - When nbWorkers==0 (default), function is blocking : it completes its job before returning to …
699 …* - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs …
702 …* - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity…
703 * - @return provides a minimum amount of data remaining to be flushed from internal buffers
707 …* For ZSTD_e_end, @return == 0 when internal buffers are fully flushed and frame is com…
708 * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
755 * the next read size (if non-zero and not an error). ZSTD_compressStream2()
756 * returns the minimum nb of bytes left to flush (if non-zero and not an error).
765 /*-***************************************************************************
766 * Streaming decompression - HowTo
770 * ZSTD_DStream objects can be re-used multiple times.
785 * @return : 0 when a frame is completely decoded and fully flushed,
787 …r value > 0, which means there is still some decoding or flushing to do to complete current frame :
789 * that will never request more than the remaining frame size.
851 …Dict can be created once and shared by multiple threads concurrently, since its usage is read-only.
870 * and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) */
905 * It can still be loaded, but as a content-only dictionary. */
911 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
917 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
921 * Provides the dictID required to decompressed the frame stored within `src`.
924 * - The frame does not require a dictionary to be decoded (most common case).
925 …* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a …
926 * Note : this use case also happens when using a non-conformant dictionary.
927 …* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible…
928 * - This is not a Zstandard frame.
939 * ZSTD_reset_session_and_parameters. Prefixes are single-use.
947 * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
948 * meaning "return to no-dictionary mode".
950 * To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
952 * It's also a CPU consuming operation, with non-negligible impact on latency.
966 * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
967 * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
970 * Special : Referencing a NULL CDict means "return to no-dictionary mode".
977 * Reference a prefix (single-usage dictionary) for next compressed frame.
978 * A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end).
990 * It's a CPU consuming operation, with non-negligible impact on latency.
1002 * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
1003 * meaning "return to no-dictionary mode".
1005 * which has a non-negligible impact on CPU usage and latency.
1020 * will be determined at decompression time, as per the dict ID in the frame.
1027 * Special: referencing a NULL DDict means "return to no-dictionary mode".
1033 * Reference a prefix (single-usage dictionary) to decompress next frame.
1036 * Prefix is **only used once**. Reference is discarded at end of frame.
1037 * End of frame is reached when ZSTD_decompressStream() returns 0.
1041 * Prefix buffer must remain unmodified up to the end of frame,
1046 * A full dictionary is more costly, as it requires building tables.
1091 * typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
1121 …) ((format) == ZSTD_f_zstd1 ? 5 : 1) /* minimum input size required to query frame header size */
1137 #define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
1150 #define ZSTD_WINDOWLOG_LIMIT_DEFAULT 27 /* by default, the streaming decoder will refuse any frame
1154 …* The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional …
1165 #define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
1174 /* --- Advanced types --- */
1200 * rep == 1 --> offset == repeat_offset_1
1201 * rep == 2 --> offset == repeat_offset_2
1202 * rep == 3 --> offset == repeat_offset_3
1204 * rep == 1 --> offset == repeat_offset_2
1205 * rep == 2 --> offset == repeat_offset_3
1206 * rep == 3 --> offset == repeat_offset_1 - 1
1226 int contentSizeFlag; /**< 1: content size will be in frame header (when known) */
1227 …int checksumFlag; /**< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, fo…
1228 …int noDictIDFlag; /**< 1: no dictID will be saved into frame header (dictID is only useful for …
1237 …ZSTD_dct_auto = 0, /* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, otherwi…
1244 …ZSTD_dlm_byRef = 1 /**< Reference dictionary content -- the dictionary buffer must outlive its …
1248 …ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default…
1249 … ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
1250 * Useful to save 4 bytes per generated frame.
1274 * - The contents of the CDict can be copied into the working context. This
1277 * the compression faster per-byte of input. However, the initial copy of
1282 * - The CDict's tables can be used in-place. In this model, compression is
1284 * tables. However, this model incurs no start-up cost (as long as the
1288 * - The CDict's tables are not used at all, and instead we use the working
1320 ZSTD_ps_enable = 1, /* Force-enable the feature */
1325 * Frame size functions
1331 * (i.e. there should be a frame boundary at `src + srcSize`)
1332 * @return : - decompressed size of all data in all successive frames
1333 * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
1334 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1340 * note 3 : decompressed size can be very large (64-bits value),
1347 * read each contained frame header. This is fast as most of the data is skipped,
1348 * however it does mean that all frame data must be present and valid. */
1354 * (i.e. there should be a frame boundary at `src + srcSize`)
1355 * @return : - upper-bound for the decompressed size of all data in all successive frames
1356 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1358 * note 1 : an error can occur if `src` contains an invalid or incorrectly formatted frame.
1359 …* note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD …
1361 …* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is c…
1362 * upper-bound = # blocks * min(128 KB, Window_Size)
1368 * @return : size of the Frame Header,
1412 * The entire source is compressed into a single frame.
1427 …* - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the ma…
1428 …* - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would …
1429 …* - ZSTD_c_windowLog affects offset validation: this function will return an error at higher de…
1432 …* Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep …
1443 * Generates a zstd skippable frame containing data given by src, and writes it to dst buffer.
1445 …* Skippable frames begin with a a 4-byte magic number. There are 16 possible choices of magic numb…
1447 …* As such, the parameter magicVariant controls the exact skippable frame magic number variant used…
1451 …* with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore i…
1459 * Retrieves a zstd skippable frame containing data given by src, and writes it to dst buffer.
1461 …* The parameter magicVariant will receive the magicVariant that was supplied when the frame was wr…
1462 * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested
1465 * Returns an error if destination buffer is not large enough, or if the frame is not skippable.
1473 * Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame.
1491 * Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
1503 * Note 2 : only single-threaded compression is supported.
1516 …ams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression …
1517 * Note : CStream size estimation is only correct for single-threaded compression.
1520 * or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame();
1540 * Initialize an object using a pre-allocated fixed-size buffer.
1542 * Provided pointer *must be 8-bytes aligned*.
1557 * Limitation 2 : static cctx currently not compatible with multi-threading.
1656 …* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component…
1704 * and how to interpret it (automatic ? force raw mode ? full mode only ?) */
1709 * how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */
1739 /* Force back-reference distances to remain < windowSize,
1756 * literals compression based on the compression parameters - specifically,
1802 * structure for the dictionary that is read-optimized.
1822 * In general, you should expect compression to be faster--sometimes very much
1823 * so--and CDict creation to be slightly slower. Eventually, we will probably
1842 * the frame is complete. But, it will still allocate an output buffer
1850 * this flag is ALWAYS memory safe, and will never access out-of-bounds
1866 * calls. Specifically: (out.size - out.pos) will never grow. This gives the
1876 * Zstd will check that (out.size - out.pos) never grows and return an error
1923 * Set to ZSTD_ps_disable to never use row-based matchfinder.
1924 * Set to ZSTD_ps_enable to force usage of row-based matchfinder.
1927 * the row-based matchfinder based on support for SIMD instructions and the window log.
1939 * results than when the two buffers are non-contiguous. This flag forces zstd
1940 * to always load the prefix in non-contiguous mode, even if it happens to be
1962 * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
1963 * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
1967 * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
1971 * - ZSTD_compressStream2() : Do compression using the CCtx.
1972 * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer.
1975 * for static allocation of CCtx for single-threaded compression.
1992 * Initializes the compression and frame parameters of cctxParams according to
2042 * Tells if the content of `buffer` starts with a valid Frame Identifier.
2043 * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
2044 * Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
2045 * Note 3 : Skippable Frame Identifiers are considered valid. */
2066 * and how to interpret it (automatic ? force raw mode ? full mode only ?). */
2071 * how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */
2077 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
2103 * MUST be large enough to fit the entire decompressed frame. This will be
2104 * checked when the frame content size is known. The data in the ZSTD_outBuffer
2112 * If you need to avoid the input buffer allocation use the buffer-less
2116 * this flag is ALWAYS memory safe, and will never access out-of-bounds
2146 * from the set of DDicts based on the dictID in the frame.
2165 * This instruction is mandatory to decode data without a fully-formed header,
2201 * "0" also disables frame content size field. It may be enabled in the future.
2217 * Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if
2229 * // Pseudocode: Set each zstd parameter and leave the rest as-is.
2261 * // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
2268 * same as ZSTD_initCStream_usingCDict(), with control over frame parameters.
2287 * start a new frame, using same parameters from previous frame.
2288 * This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
2312 * consumed (input actually compressed) and produced (output) for current frame.
2313 * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
2363 * re-use decompression parameters from previous init; saves dictionary loading
2370 * Buffer-less and synchronous inner streaming functions
2372 * This is an advanced API, giving full control over buffer management, for users which need direct…
2378 Buffer-less streaming compression (synchronous mode)
2382 ZSTD_CCtx object can be re-used multiple times within successive compression operations.
2390 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
2391 - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
2392 …- Caller must ensure there is enough space in `dst` to store compressed data under worst case scen…
2395 …- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to max…
2397 …- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
2400 Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
2401 …It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
2404 `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
2407 /*===== Buffer-less streaming compression functions =====*/
2422 Buffer-less streaming decompression (synchronous mode)
2426 A ZSTD_DCtx object can be re-used multiple times.
2428 First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader().
2429 …Frame header is extracted from the beginning of compressed frame, so providing only the frame's be…
2436 It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
2437 such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
2447 …or that previous contiguous segment is large enough to properly handle maximum back-reference dist…
2452 …which can @return an error code if required value is too large for current system (in 32-bits mode…
2454 …up to the moment there is not enough room left in the buffer to guarantee decoding another full bl…
2462 as long as the encoder and decoder progress in "lock-step",
2476 A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
2480 This information is not required to properly decode a frame.
2484 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
2487 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
2488 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
2489 c) Frame Content - any content (User Data) of length equal to Frame Size
2490 For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
2494 /*===== Buffer-less streaming decompression functions =====*/
2507 * decode Frame Header, or requires larger `srcSize`.
2516 …ed long long windowSize, unsigned long long frameContentSize); /**< when frame content size is no…
2538 Block functions produce and decode raw zstd blocks, without frame metadata.
2539 …Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 1…
2543 - Compressing and decompressing require a context structure
2545 - It is necessary to init context before starting
2549 - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
2552 …Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than…
2553 …- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero)…
2568 …Size); /**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression.…