1.. index:: encoder 2 3Encoders 4======== 5 6This section gives an overview of encoders, details on the encoders 7that ship with libxo, and documentation for developers of future 8encoders. 9 10Overview 11-------- 12 13The libxo library contains software to generate four "built-in" 14formats: text, XML, JSON, and HTML. These formats are common and 15useful, but there are other common and useful formats that users will 16want, and including them all in the libxo software would be difficult 17and cumbersome. 18 19To allow support for additional encodings, libxo includes a 20"pluggable" extension mechanism for dynamically loading new encoders. 21libxo-based applications can automatically use any installed encoder. 22 23Use the "encoder=XXX" option to access encoders. The following 24example uses the "cbor" encoder, saving the output into a file:: 25 26 df --libxo encoder=cbor > df-output.cbor 27 28Encoders can support specific options that can be accessed by 29following the encoder name with a colon (':') and one of more options, 30separated by a plus sign "+":: 31 32 df --libxo encoder=csv:path=filesystem+leaf=name+no-header 33 34This example instructs libxo to load the "csv" encoder and pass the 35following options:: 36 37 path=filesystem 38 leaf=name 39 no-header 40 41Each of these option is interpreted by the encoder, and all such 42options names and semantics are specific to the particular encoder. 43Refer to the intended encoder for documentation on its options. 44 45.. _csv_encoder: 46 47CSV - Comma Separated Values 48---------------------------- 49 50libxo ships with a custom encoder for "CSV" files, a common format for 51comma separated values. The output of the CSV encoder can be loaded 52directly into spreadsheets or similar applications. 53 54A standard for CSV files is provided in :RFC:`4180`, but since the 55format predates that standard by decades, there are many minor 56differences in CSV file consumers and their expectations. The CSV 57encoder has a number of options to tailor output to those 58expectations. 59 60Consider the following XML:: 61 62 % list-items --libxo xml,pretty 63 <top> 64 <data test="value"> 65 <item test2="value2"> 66 <sku test3="value3" key="key">GRO-000-415</sku> 67 <name key="key">gum</name> 68 <sold>1412</sold> 69 <in-stock>54</in-stock> 70 <on-order>10</on-order> 71 </item> 72 <item> 73 <sku test3="value3" key="key">HRD-000-212</sku> 74 <name key="key">rope</name> 75 <sold>85</sold> 76 <in-stock>4</in-stock> 77 <on-order>2</on-order> 78 </item> 79 <item> 80 <sku test3="value3" key="key">HRD-000-517</sku> 81 <name key="key">ladder</name> 82 <sold>0</sold> 83 <in-stock>2</in-stock> 84 <on-order>1</on-order> 85 </item> 86 </data> 87 </top> 88 89This output is a list of `instances` (named "item"), each containing a 90set of `leafs` ("sku", "name", etc). 91 92The CSV encoder will emit the leaf values in this output as `fields` 93inside a CSV `record`, which is a line containing a set of 94comma-separated values:: 95 96 % list-items --libxo encoder=csv 97 sku,name,sold,in-stock,on-order 98 GRO-000-415,gum,1412,54,10 99 HRD-000-212,rope,85,4,2 100 HRD-000-517,ladder,0,2,1 101 102Be aware that since the CSV encoder looks for data instances, when 103used with :ref:`xo`, the `--instance` option will be needed:: 104 105 % xo --libxo encoder=csv --instance foo 'The {:product} is {:status}\n' stereo "in route" 106 product,status 107 stereo,in route 108 109.. _csv_path: 110 111The `path` Option 112~~~~~~~~~~~~~~~~~ 113 114By default, the CSV encoder will attempt to emit any list instance 115generated by the application. In some cases, this may be 116unacceptable, and a specific list may be desired. 117 118Use the "path" option to limit the processing of output to a specific 119hierarchy. The path should be one or more names of containers or 120lists. 121 122For example, if the "list-items" application generates other lists, 123the user can give "path=top/data/item" as a path:: 124 125 % list-items --libxo encoder=csv:path=top/data/item 126 sku,name,sold,in-stock,on-order 127 GRO-000-415,gum,1412,54,10 128 HRD-000-212,rope,85,4,2 129 HRD-000-517,ladder,0,2,1 130 131Paths are "relative", meaning they need not be a complete set 132of names to the list. This means that "path=item" may be sufficient 133for the above example. 134 135.. _csv_leafs: 136 137The `leafs` Option 138~~~~~~~~~~~~~~~~~~ 139 140The CSV encoding requires that all lines of output have the same 141number of fields with the same order. In contrast, XML and JSON allow 142any order (though libxo forces key leafs to appear before other 143leafs). 144 145To maintain a consistent set of fields inside the CSV file, the same 146set of leafs must be selected from each list item. By default, the 147CSV encoder records the set of leafs that appear in the first list 148instance it processes, and extract only those leafs from future 149instances. If the first instance is missing a leaf that is desired by 150the consumer, the "leaf" option can be used to ensure that an empty 151value is recorded for instances that lack a particular leaf. 152 153The "leafs" option can also be used to exclude leafs, limiting the 154output to only those leafs provided. 155 156In addition, the order of the output fields follows the order in which 157the leafs are listed. "leafs=one.two" and "leafs=two.one" give 158distinct output. 159 160So the "leafs" option can be used to expand, limit, and order the set 161of leafs. 162 163The value of the leafs option should be one or more leaf names, 164separated by a period ("."):: 165 166 % list-items --libxo encoder=csv:leafs=sku.on-order 167 sku,on-order 168 GRO-000-415,10 169 HRD-000-212,2 170 HRD-000-517,1 171 % list-items -libxo encoder=csv:leafs=on-order.sku 172 on-order,sku 173 10,GRO-000-415 174 2,HRD-000-212 175 1,HRD-000-517 176 177Note that since libxo uses terminology from YANG (:RFC:`7950`), the 178data modeling language for NETCONF (:RFC:`6241`), which uses "leafs" 179as the plural form of "leaf". libxo follows that convention. 180 181.. _csv_no_header: 182 183The `no-header` Option 184~~~~~~~~~~~~~~~~~~~~~~ 185 186CSV files typical begin with a line that defines the fields included 187in that file, in an attempt to make the contents self-defining:: 188 189 sku,name,sold,in-stock,on-order 190 GRO-000-415,gum,1412,54,10 191 HRD-000-212,rope,85,4,2 192 HRD-000-517,ladder,0,2,1 193 194There is no reliable mechanism for determining whether this header 195line is included, so the consumer must make an assumption. 196 197The csv encoder defaults to producing the header line, but the 198"no-header" option can be included to avoid the header line. 199 200.. _csv_no_quotes: 201 202The `no-quotes` Option 203~~~~~~~~~~~~~~~~~~~~~~ 204 205:RFC:`4180` specifies that fields containing spaces should be quoted, but 206many CSV consumers do not handle quotes. The "no-quotes" option 207instruct the CSV encoder to avoid the use of quotes. 208 209.. _csv_dos: 210 211The `dos` Option 212~~~~~~~~~~~~~~~~ 213 214:RFC:`4180` defines the end-of-line marker as a carriage return 215followed by a newline. This `CRLF` convention dates from the distant 216past, but its use was anchored in the 1980s by the `DOS` operating 217system. 218 219The CSV encoder defaults to using the standard Unix end-of-line 220marker, a simple newline. Use the "dos" option to use the `CRLF` 221convention. 222 223The Encoder API 224--------------- 225 226The encoder API consists of three distinct phases: 227 228- loading the encoder 229- initializing the encoder 230- feeding operations to the encoder 231 232To load the encoder, libxo will open a shared library named: 233 234 ${prefix}/lib/libxo/encoder/${name}.enc 235 236This file is typically a symbolic link to a dynamic library, suitable 237for `dlopen`(). libxo looks for a symbol called 238`xo_encoder_library_init` inside that library and calls it with the 239arguments defined in the header file "xo_encoder.h". This function 240should look as follows:: 241 242 int 243 xo_encoder_library_init (XO_ENCODER_INIT_ARGS) 244 { 245 arg->xei_version = XO_ENCODER_VERSION; 246 arg->xei_handler = test_handler; 247 248 return 0; 249 } 250 251Several features here allow for future compatibility: the macro 252XO_ENCODER_INIT_ARGS allows the arguments to this function change over 253time, and the XO_ENCODER_VERSION allows the library to tell libxo 254which version of the API it was compiled with. 255 256The function places in xei_handler should be have the signature:: 257 258 static int 259 test_handler (XO_ENCODER_HANDLER_ARGS) 260 { 261 ... 262 263This function will be called with the "op" codes defined in 264"xo_encoder.h". Each op code represents a distinct event in the libxo 265processing model. For example OP_OPEN_CONTAINER tells the encoder 266that a new container has been opened, and the encoder can behave in an 267appropriate manner. 268 269 270