1.\" Copyright (c) 2008 Poul-Henning Kamp 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd February 9, 2008 28.Os 29.Dt FIFOLOG 1 30.Sh NAME 31.Nm fifolog_create , fifolog_writer , fifolog_reader 32.Nd "initialize, write, seek and extract data from a fifolog" 33.Sh SYNOPSIS 34.Nm fifolog_create 35.Op Fl l Ar record-size 36.Op Fl r Ar record-count 37.Op Fl s Ar size 38.Ar file 39.Nm fifolog_reader 40.Op Fl t 41.Op Fl b Ar tstart 42.Op Fl B Ar Tstart 43.Op Fl e Ar tend 44.Op Fl E Ar Tend 45.Op Fl o Ar ofile 46.Op Fl R Ar regexp 47.Op Fl T Ar timefmt 48.Ar file 49.Nm fifolog_writer 50.Op Fl w Ar write-rate 51.Op Fl s Ar sync-rate 52.Op Fl z Ar compression 53.Ar file 54.Sh DESCRIPTION 55Fifologs provide a compact round-robin circular storage for 56recording text and binary information to permanent storage in a bounded 57and predictable fashion, time and space wise. 58.Pp 59A fifolog can be stored either directly on a disk partition or in a 60regular file. 61.Pp 62The input data stream is encoded, compressed and marked up with 63timestamps before it is written to storage, such that it is possible 64to seek out a particular time interval in the stored data, without 65having to decompress the entire logfile. 66.Pp 67The 68.Nm fifolog_create 69utility 70is used to initialize the first sector of a disk device 71or file system file to make it a fifolog and should be called only 72once. 73.Pp 74Running 75.Nm fifolog_create 76on an existing fifolog will reset it so that 77.Nm fifolog_reader 78and 79.Nm fifolog_writer 80will not see the previous contents. 81(The previous contents are not physically erased, and with a bit 82of hand-work all but the first record can be easily recovered.) 83.Pp 84If the 85.Ar file 86does not already exist, 87.Nm fifolog_create 88will attempt to create and 89.Xr ftruncate 2 90it to the specified size, defaulting to 86400 records of 512 bytes 91if the 92.Fl r , l 93or 94.Fl s 95options do not specify otherwise. 96.Pp 97The 98.Nm fifolog_writer 99utility 100will read standard input and write it to the end of the fifolog 101according to the parameters given. 102.Pp 103Writes happen whenever the output buffer is filled with compressed 104data or when either of two timers expire, forcing a partially filled 105buffer to be written. 106.Pp 107The first and faster timer, 108.Fl w Ar write-rate , 109forces available data to be written 110but does not flush and reset the compression dictionary. 111This timer is intended to minimize the amount of logdata lost in RAM 112in case of a crash and by default it fires 10 seconds after 113the previous write. 114.Pp 115The second and slower timer, 116.Fl s Ar sync-rate , 117forces a full flush and reset of the compression 118engine and causes the next record written to be a synchronization 119point with an uncompressed timestamp, making it possible to start 120reading the logfile from that record. 121By default this timer fires a minute after the previous sync. 122.Pp 123The 124.Fl z Ar compression 125option controls the 126.Xr zlib 3 127compression level; legal values are zero to nine which is the default. 128.Pp 129The 130.Nm fifolog_reader 131utility 132will retrieve records from the fifolog according to the specified 133parameters and write them either to standard output or the file specified 134with 135.Fl o . 136.Pp 137It is possible to specify a start and end time to limit the amount 138of data 139.Nm fifolog_reader 140will report. 141The lower-case variants 142.Fl b 143and 144.Fl e 145take a 146.Vt time_t 147value, whereas the upper-case variants 148.Fl B 149and 150.Fl E 151take human-readable specifications such as 152.Dq Li "1 hour ago" . 153.Pp 154The 155.Fl t 156option forces timestamps to be formatted as 157.Dq Li "YYYYMMDDhhmmss" 158instead of as 159.Vt time_t , 160and 161.Fl T 162allows the specification of an 163.Xr strftime 3 164formatting string. 165.Pp 166Finally, records can be filtered such that only records matching the 167.Pq Dv REG_BASIC 168regular expression specified with 169.Fl R 170are output. 171.Sh IMPLEMENTATION NOTES 172The data stored in the fifolog consists of three layers, an outer 173layer that allows searches to synchronization points based on timestamps 174without having to decompress and decode the actual contents, a 175compression layer implemented with 176.Xr zlib 3 , 177and an inner serialization and timestamping layer. 178.Pp 179The exact encoding is described in the 180.Pa fifolog.h 181file. 182.Pp 183Fifolog is particularly well suited for use on Flash based media, where 184it results in much lower write-wear, than a file system with regular 185log files rotated with 186.Xr newsyslog 8 187etc. 188.Sh EXAMPLES 189Create a fifolog with 1024*1024 records of 512 bytes: 190.Pp 191.Dl "fifolog_create -r 10m /tmp/fifolog" 192.Pp 193Write a single record to this file: 194.Pp 195.Dl "date | fifolog_writer /tmp/fifolog" 196.Pp 197Read it back with human readable timestamps: 198.Pp 199.Dl "fifolog_reader -t /tmp/fifolog" 200.Pp 201One particular useful use of 202.Nm fifolog_writer 203is with 204.Xr syslogd 8 205using a line such as this in 206.Xr /etc/syslog.conf 5 : 207.Pp 208.Dl "*.* |fifolog_writer /var/log/syslog_fifolog" 209.Sh HISTORY 210The fifolog tools have been liberated from an open source 211.Tn SCADA 212applications called 213.Dq measured , 214which monitors and controls remote radio navigation 215transmitters for the Danish Air Traffic Control system. 216.Sh AUTHORS 217The fifolog tools were written by 218.An Poul-Henning Kamp . 219