1*eda14cbcSMatt Macy# ZSTD-On-ZFS Library Manual 2*eda14cbcSMatt Macy 3*eda14cbcSMatt Macy## Introduction 4*eda14cbcSMatt Macy 5*eda14cbcSMatt MacyThis subtree contains the ZSTD library used in ZFS. It is heavily cut-down by 6*eda14cbcSMatt Macydropping any unneeded files, and combined into a single file, but otherwise is 7*eda14cbcSMatt Macyintentionally unmodified. Please do not alter the file containing the zstd 8*eda14cbcSMatt Macylibrary, besides upgrading to a newer ZSTD release. 9*eda14cbcSMatt Macy 10*eda14cbcSMatt MacyTree structure: 11*eda14cbcSMatt Macy 12*eda14cbcSMatt Macy* `zfs_zstd.c` is the actual `zzstd` kernel module. 13*eda14cbcSMatt Macy* `lib/` contains the the unmodified, [_"amalgamated"_](https://github.com/facebook/zstd/blob/dev/contrib/single_file_libs/README.md) 14*eda14cbcSMatt Macy version of the `Zstandard` library, generated from our template file 15*eda14cbcSMatt Macy* `zstd-in.c` is our template file for generating the library 16*eda14cbcSMatt Macy* `include/`: This directory contains supplemental includes for platform 17*eda14cbcSMatt Macy compatibility, which are not expected to be used by ZFS elsewhere in the 18*eda14cbcSMatt Macy future. Thus we keep them private to ZSTD. 19*eda14cbcSMatt Macy 20*eda14cbcSMatt Macy## Updating ZSTD 21*eda14cbcSMatt Macy 22*eda14cbcSMatt MacyTo update ZSTD the following steps need to be taken: 23*eda14cbcSMatt Macy 24*eda14cbcSMatt Macy1. Grab the latest release of [ZSTD](https://github.com/facebook/zstd/releases). 25*eda14cbcSMatt Macy2. Update `module/zstd/zstd-in.c` if required. (see 26*eda14cbcSMatt Macy `zstd/contrib/single_file_libs/zstd-in.c` in the zstd repository) 27*eda14cbcSMatt Macy3. Generate the "single-file-library" and put it to `module/zstd/lib/`. 28*eda14cbcSMatt Macy4. Copy the following files to `module/zstd/lib/`: 29*eda14cbcSMatt Macy - `zstd/lib/zstd.h` 30*eda14cbcSMatt Macy - `zstd/lib/common/zstd_errors.h` 31*eda14cbcSMatt Macy 32*eda14cbcSMatt MacyThis can be done using a few shell commands from inside the zfs repo: 33*eda14cbcSMatt Macy 34*eda14cbcSMatt Macy~~~sh 35*eda14cbcSMatt Macycd PATH/TO/ZFS 36*eda14cbcSMatt Macy 37*eda14cbcSMatt Macyurl="https://github.com/facebook/zstd" 38*eda14cbcSMatt Macyrelease="$(curl -s "${url}"/releases/latest | grep -oP '(?<=v)[\d\.]+')" 39*eda14cbcSMatt Macyzstd="/tmp/zstd-${release}/" 40*eda14cbcSMatt Macy 41*eda14cbcSMatt Macywget -O /tmp/zstd.tar.gz \ 42*eda14cbcSMatt Macy "${url}/releases/download/v${release}/zstd-${release}.tar.gz" 43*eda14cbcSMatt Macytar -C /tmp -xzf /tmp/zstd.tar.gz 44*eda14cbcSMatt Macy 45*eda14cbcSMatt Macycp ${zstd}/lib/zstd.h module/zstd/lib/ 46*eda14cbcSMatt Macycp ${zstd}/lib/zstd_errors.h module/zstd/lib/ 47*eda14cbcSMatt Macy${zstd}/contrib/single_file_libs/combine.sh \ 48*eda14cbcSMatt Macy -r ${zstd}/lib -o module/zstd/lib/zstd.c module/zstd/zstd-in.c 49*eda14cbcSMatt Macy~~~ 50*eda14cbcSMatt Macy 51*eda14cbcSMatt MacyNote: if the zstd library for zfs is updated to a newer version, 52*eda14cbcSMatt Macythe macro list in include/zstd_compat_wrapper.h usually needs to be updated. 53*eda14cbcSMatt Macythis can be done with some hand crafting of the output of the following 54*eda14cbcSMatt Macyscript: nm zstd.o | awk '{print "#define "$3 " zfs_" $3}' > macrotable 55*eda14cbcSMatt Macy 56*eda14cbcSMatt Macy 57*eda14cbcSMatt Macy## Altering ZSTD and breaking changes 58*eda14cbcSMatt Macy 59*eda14cbcSMatt MacyIf ZSTD made changes that break compatibility or you need to make breaking 60*eda14cbcSMatt Macychanges to the way we handle ZSTD, it is required to maintain backwards 61*eda14cbcSMatt Macycompatibility. 62*eda14cbcSMatt Macy 63*eda14cbcSMatt MacyWe already save the ZSTD version number within the block header to be used 64*eda14cbcSMatt Macyto add future compatibility checks and/or fixes. However, currently it is 65*eda14cbcSMatt Macynot actually used in such a way. 66