1.\" Copyright (c) 2008 Poul-Henning Kamp 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.Dd February 9, 2008 26.Dt FIFOLOG 1 27.Os 28.Sh NAME 29.Nm fifolog_create , fifolog_writer , fifolog_reader 30.Nd "initialize, write, seek and extract data from a fifolog" 31.Sh SYNOPSIS 32.Nm fifolog_create 33.Op Fl l Ar record-size 34.Op Fl r Ar record-count 35.Op Fl s Ar size 36.Ar file 37.Nm fifolog_reader 38.Op Fl t 39.Op Fl b Ar tstart 40.Op Fl B Ar Tstart 41.Op Fl e Ar tend 42.Op Fl E Ar Tend 43.Op Fl o Ar ofile 44.Op Fl R Ar regexp 45.Op Fl T Ar timefmt 46.Ar file 47.Nm fifolog_writer 48.Op Fl w Ar write-rate 49.Op Fl s Ar sync-rate 50.Op Fl z Ar compression 51.Ar file 52.Sh DESCRIPTION 53Fifologs provide a compact round-robin circular storage for 54recording text and binary information to permanent storage in a bounded 55and predictable fashion, time and space wise. 56.Pp 57A fifolog can be stored either directly on a disk partition or in a 58regular file. 59.Pp 60The input data stream is encoded, compressed and marked up with 61timestamps before it is written to storage, such that it is possible 62to seek out a particular time interval in the stored data, without 63having to decompress the entire logfile. 64.Pp 65The 66.Nm fifolog_create 67utility 68is used to initialize the first sector of a disk device 69or file system file to make it a fifolog and should be called only 70once. 71.Pp 72Running 73.Nm fifolog_create 74on an existing fifolog will reset it so that 75.Nm fifolog_reader 76and 77.Nm fifolog_writer 78will not see the previous contents. 79(The previous contents are not physically erased, and with a bit 80of hand-work all but the first record can be easily recovered.) 81.Pp 82If the 83.Ar file 84does not already exist, 85.Nm fifolog_create 86will attempt to create and 87.Xr ftruncate 2 88it to the specified size, defaulting to 86400 records of 512 bytes 89if the 90.Fl r , l 91or 92.Fl s 93options do not specify otherwise. 94.Pp 95The 96.Nm fifolog_writer 97utility 98will read standard input and write it to the end of the fifolog 99according to the parameters given. 100.Pp 101Writes happen whenever the output buffer is filled with compressed 102data or when either of two timers expire, forcing a partially filled 103buffer to be written. 104.Pp 105The first and faster timer, 106.Fl w Ar write-rate , 107forces available data to be written 108but does not flush and reset the compression dictionary. 109This timer is intended to minimize the amount of logdata lost in RAM 110in case of a crash and by default it fires 10 seconds after 111the previous write. 112.Pp 113The second and slower timer, 114.Fl s Ar sync-rate , 115forces a full flush and reset of the compression 116engine and causes the next record written to be a synchronization 117point with an uncompressed timestamp, making it possible to start 118reading the logfile from that record. 119By default this timer fires a minute after the previous sync. 120.Pp 121The 122.Fl z Ar compression 123option controls the 124.Xr zlib 3 125compression level; legal values are zero to nine which is the default. 126.Pp 127The 128.Nm fifolog_reader 129utility 130will retrieve records from the fifolog according to the specified 131parameters and write them either to standard output or the file specified 132with 133.Fl o . 134.Pp 135It is possible to specify a start and end time to limit the amount 136of data 137.Nm fifolog_reader 138will report. 139The lower-case variants 140.Fl b 141and 142.Fl e 143take a 144.Vt time_t 145value, whereas the upper-case variants 146.Fl B 147and 148.Fl E 149take human-readable specifications such as 150.Dq Li "1 hour ago" . 151.Pp 152The 153.Fl t 154option forces timestamps to be formatted as 155.Dq Li "YYYYMMDDhhmmss" 156instead of as 157.Vt time_t , 158and 159.Fl T 160allows the specification of an 161.Xr strftime 3 162formatting string. 163.Pp 164Finally, records can be filtered such that only records matching the 165.Pq Dv REG_BASIC 166regular expression specified with 167.Fl R 168are output. 169.Sh IMPLEMENTATION NOTES 170The data stored in the fifolog consists of three layers, an outer 171layer that allows searches to synchronization points based on timestamps 172without having to decompress and decode the actual contents, a 173compression layer implemented with 174.Xr zlib 3 , 175and an inner serialization and timestamping layer. 176.Pp 177The exact encoding is described in the 178.Pa fifolog.h 179file. 180.Pp 181Fifolog is particularly well suited for use on Flash based media, where 182it results in much lower write-wear, than a file system with regular 183log files rotated with 184.Xr newsyslog 8 185etc. 186.Sh EXAMPLES 187Create a fifolog with 1024*1024 records of 512 bytes: 188.Pp 189.Dl "fifolog_create -r 10m /tmp/fifolog" 190.Pp 191Write a single record to this file: 192.Pp 193.Dl "date | fifolog_writer /tmp/fifolog" 194.Pp 195Read it back with human readable timestamps: 196.Pp 197.Dl "fifolog_reader -t /tmp/fifolog" 198.Pp 199One particular useful use of 200.Nm fifolog_writer 201is with 202.Xr syslogd 8 203using a line such as this in 204.Xr syslog.conf 5 : 205.Pp 206.Dl "*.* |fifolog_writer /var/log/syslog_fifolog" 207.Sh HISTORY 208The fifolog tools have been liberated from an open source 209.Tn SCADA 210applications called 211.Dq measured , 212which monitors and controls remote radio navigation 213transmitters for the Danish Air Traffic Control system. 214.Sh AUTHORS 215The fifolog tools were written by 216.An Poul-Henning Kamp . 217