1.\" Copyright (c) 1998 Dag-Erling Co�dan Sm�rgrav 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $Id: fetch.3,v 1.6 1998/12/16 15:29:03 des Exp $ 26.\" 27.Dd July 1, 1998 28.Dt FETCH 3 29.Os 30.Sh NAME 31.Nm fetchGetURL , 32.Nm fetchPutURL , 33.Nm fetchStatURL , 34.Nm fetchListURL , 35.Nm fetchParseURL , 36.Nm fetchGet , 37.Nm fetchPut , 38.Nm fetchStat , 39.Nm fetchList , 40.Nm fetchGetFile , 41.Nm fetchPutFile , 42.Nm fetchStatFile , 43.Nm fetchListFile , 44.Nm fetchGetHTTP , 45.Nm fetchPutHTTP , 46.Nm fetchStatHTTP , 47.Nm fetchListHTTP , 48.Nm fetchGetFTP , 49.Nm fetchPutFTP 50.Nm fetchStatFTP 51.Nm fetchListFTP , 52.Nd file transfer library 53.Sh SYNOPSIS 54.Fd #include <sys/param.h> 55.Fd #include <stdio.h> 56.Fd #include <fetch.h> 57.Ft FILE * 58.Fn fetchGetURL "char *URL" "char *flags" 59.Ft FILE * 60.Fn fetchPutURL "char *URL" "char *flags" 61.Ft int 62.Fn fetchStatURL "char *URL" "struct url_stat *us" "char *flags" 63.Ft struct url_ent * 64.Fn fetchListURL "char *URL" "char *flags" 65.Ft struct url * 66.Fn fetchParseURL "char *URL" "char *flags" 67.Ft FILE * 68.Fn fetchGet "struct url *URL" "char *flags" 69.Ft FILE * 70.Fn fetchPut "struct url *URL" "char *flags" 71.Ft int 72.Fn fetchStat "struct url *URL" "struct url_stat *us" "char *flags" 73.Ft struct url_ent * 74.Fn fetchList "struct url *" "char *flags" 75.Ft FILE * 76.Fn fetchGetFile "struct url *u" "char *flags" 77.Ft FILE * 78.Fn fetchPutFile "struct url *u" "char *flags" 79.Ft int 80.Fn fetchStatFile "struct url *URL" "struct url_stat *us" "char *flags" 81.Ft struct url_ent * 82.Fn fetchListFile "struct url *" "char *flags" 83.Ft FILE * 84.Fn fetchGetHTTP "struct url *u" "char *flags" 85.Ft FILE * 86.Fn fetchPutHTTP "struct url *u" "char *flags" 87.Ft int 88.Fn fetchStatHTTP "struct url *URL" "struct url_stat *us" "char *flags" 89.Ft struct url_ent * 90.Fn fetchListHTTP "struct url *" "char *flags" 91.Ft FILE * 92.Fn fetchGetFTP "struct url *u" "char *flags" 93.Ft FILE * 94.Fn fetchPutFTP "struct url *u" "char *flags" 95.Ft int 96.Fn fetchStatFTP "struct url *URL" "struct url_stat *us" "char *flags" 97.Ft struct url_ent * 98.Fn fetchListFTP "struct url *" "char *flags" 99.Sh DESCRIPTION 100.Pp 101These functions implement a high-level library for retrieving and 102uploading files using Uniform Resource Locators (URLs). 103.Pp 104.Fn fetchGetURL 105and 106.Fn fetchPutURL 107constitute the recommended interface to the 108.Nm fetch 109library. They examine the URL passed to them to determine the transfer 110method, and call the appropriate lower-level functions to perform the 111actual transfer. The 112.Fa flags 113argument is a string of characters which specify transfer options. The 114meaning of the individual flags is scheme-dependent, and is detailed 115in the appropriate section below. 116.Pp 117.Fn fetchStatURL 118attempts to obtain the requested document's metadata and fill in the 119structure pointed to by it's second argument. The 120.Fa url_stat 121structure is defined as follows in 122.Aq Pa fetch.h : 123.Bd -literal 124struct url_stat { 125 off_t size; 126 time_t atime; 127 time_t mtime; 128}; 129.Ed 130.Pp 131.Fn fetchListURL 132attempts to list the contents of the directory pointed to by the URL 133provided. If successful, it returns a malloced array of 134.Fa url_ent 135structures. The 136.Fa url_ent 137structure is defined as follows in 138.Aq Pa fetch.h : 139.Bd -literal 140struct url_ent { 141 char name[MAXPATHLEN]; 142 struct url_stat stat; 143}; 144.Ed 145.Pp 146The list is terminated by an entry with an empty name. 147.Pp 148The pointer returned by 149.Fn fetchListURL 150should be freed using 151.Fn free . 152.Pp 153.Fn fetchParseURL 154takes a URL in the form of a null-terminated string and splits it into 155its components function according to the Common Internet Scheme Syntax 156detailed in RFC1738. A regular expression which produces this syntax 157is: 158.Bd -literal 159 <scheme>:(//(<user>(:<pwd>)?@)?<host>(:<port>)?)?/(<document>)? 160.Ed 161.Pp 162Note that some components of the URL are not necessarily relevant to 163all URL schemes. For instance, the file scheme only needs the <scheme> 164and <document> components. 165.Pp 166The pointer returned by 167.Fn fetchParseURL 168should be freed using 169.Fn free . 170.Pp 171.Fn fetchGet , 172.Fn fetchPut 173and 174.Fn fetchStat 175are similar to 176.Fn fetchGetURL , 177.Fn fetchPutURL 178and 179.Fn fetchStatURL , 180except that they expect a pre-parsed URL in the form of a pointer to 181a 182.Fa struct url 183rather than a string. 184.Pp 185All of the 186.Fn fetchGetXXX 187and 188.Fn fetchPutXXX 189functions return a pointer to a stream which can be used to read or 190write data from or to the requested document, respectively. Note that 191although the implementation details of the individual access methods 192vary, it can generally be assumed that a stream returned by one of the 193.Fn fetchGetXXX 194functions is read-only, and that a stream returned by one of the 195.Fn fetchPutXXX 196functions is write-only. 197.Sh FILE SCHEME 198.Fn fetchGetFile 199and 200.Fn fetchPutFile 201provide access to documents which are files in a locally mounted file 202system. Only the <document> component of the URL is used. 203.Pp 204.Fn fetchGetFile 205does not accept any flags. 206.Pp 207.Fn fetchPutFile 208accepts the 209.Fa a 210(append to file) flag. If that flag is specified, the data written to 211the stream returned by 212.Fn fetchPutFile 213will be appended to the previous contents of the file, instead of 214replacing them. 215.Sh FTP SCHEME 216.Fn fetchGetFTP 217and 218.Fn fetchPutFTP 219implement the FTP protocol as described in RFC959. 220.Pp 221If the 222.Fa p 223(passive) flag is specified, a passive (rather than active) connection 224will be attempted. 225.Pp 226If no user name or password is given, the 227.Nm fetch 228library will attempt an anonymous login, with user name "ftp" and 229password "ftp". 230.Sh HTTP SCHEME 231The 232.Fn fetchGetHTTP 233and 234.Fn fetchPutHTTP 235functions implement the HTTP/1.1 protocol. With a little luck, there's 236even a chance that they comply with RFC2068. 237.Pp 238Since there seems to be no good way of implementing the HTTP PUT 239method in a manner consistent with the rest of the 240.Nm fetch 241library, 242.Fn fetchPutHTTP 243is currently unimplemented. 244.Sh RETURN VALUES 245.Fn fetchParseURL 246returns a pointer to a 247.Fa struct url 248containing the individual components of the URL. If it is 249unable to allocate memory, or the URL is syntactically incorrect, 250.Fn fetchParseURL 251returns a NULL pointer. 252.Pp 253The 254.Fn fetchStat 255functions return 0 on success and -1 on failure. 256.Pp 257All other functions return a stream pointer which may be used to 258access the requested document, or NULL if an error occurred. 259.Pp 260.Nm Libfetch 261uses the Common Error Library 262.Nm ( libcom_err ) 263to report errors. The error code passed to 264.Fn com_err 265is one of: 266.Bl -tag -width Er 267.It Bq Er FETCH_ABORT 268Operation aborted 269.It Bq Er FETCH_AUTH 270Authentication failed 271.It Bq Er FETCH_DOWN 272Service unavailable 273.It Bq Er FETCH_EXISTS 274File exists 275.It Bq Er FETCH_FULL 276File system full 277.It Bq Er FETCH_INFO 278Informational response 279.It Bq Er FETCH_MEMORY 280Insufficient memory 281.It Bq Er FETCH_MOVED 282File has moved 283.It Bq Er FETCH_NETWORK 284Network error 285.It Bq Er FETCH_OK 286No error 287.It Bq Er FETCH_PROTO 288Protocol error 289.It Bq Er FETCH_RESOLV 290Resolver error 291.It Bq Er FETCH_SERVER 292Server error 293.It Bq Er FETCH_TEMP 294Temporary error 295.It Bq Er FETCH_TIMEOUT 296Operation timed out 297.It Bq Er FETCH_UNAVAIL 298File is not available 299.It Bq Er FETCH_UNKNOWN 300Unknown error 301.It Bq Er FETCH_URL 302Invalid URL 303.El 304.Pp 305The accompanying error message includes a protocol-specific error code 306and message, e.g. "File is not available (404 Not Found)" 307.Sh ENVIRONMENT 308The FTP and HTTP related functions use the 309.Ev HTTP_PROXY 310and 311.Ev FTP_PROXY 312environment variables, respectively, as the address of a proxy server 313to use for transferring files. 314.Sh SEE ALSO 315.Xr com_err 3 , 316.Xr fetch 1 , 317.Xr ftpio 3 318.Rs 319.%A T. Berners-Lee, L. Masinter & M. McCahill 320.%D December 1994 321.%T Uniform Resource Locators (URL) 322.%O RFC1738 323.Re 324.Rs 325.%A R. Fielding, J. Gettys, J. Mogul, H. Frystyk, T. Berners-Lee 326.%D Januray 1997 327.%B Hypertext Transfer Protocol -- HTTP/1.1 328.%O RFC2068 329.Re 330.Rs 331.%A J. Postel, J. K. Reynolds 332.%D October 1985 333.%B File Transfer Protocol 334.%O RFC959 335.Re 336.Sh NOTES 337The 338.Nm fetch 339library uses the Common Error library, and applications which link 340with 341.Nm libfetch 342must therefore also link with 343.Nm libcom_err . 344.Sh HISTORY 345The 346.Nm fetch 347library first appeared in 348.Fx 3.0 . 349.Sh AUTHORS 350The 351.Nm fetch 352library was mostly written by 353.An Dag-Erling Co�dan Sm�rgrav Aq des@FreeBSD.org 354with numerous suggestions from 355.An Jordan K. Hubbard Aq jkh@FreeBSD.org , 356.An Eugene Skepner Aq eu@qub.com 357and other FreeBSD developers. 358It replaces the older 359.Nm ftpio 360library written by 361.An Poul-Henning Kamp Aq pkh@FreeBSD.org 362and 363.An Jordan K. Hubbard Aq jkh@FreeBSD.org . 364.Pp 365This manual page was written by 366.An Dag-Erling Co�dan Sm�rgrav Aq des@FreeBSD.org 367.Sh BUGS 368Some parts of the library are not yet implemented. The most notable 369examples of this are 370.Fn fetchPutHTTP , 371.Fn fetchStatHTTP , 372.Fn fetchListHTTP , 373.Fn fetchListFTP , 374and FTP proxy support. 375.Pp 376There's no way to select a proxy at run-time other than setting the 377.Ev HTTP_PROXY 378or 379.Ev FTP_PROXY 380environment variables as appropriate. There is also no way to stop the 381FTP and HTTP functions from trying to use a proxy if these variables 382are set. 383.Pp 384HTTP authentication doesn't work. I'm not sure that's a bug in my 385code; as far as I can determine, 386.Nm libfetch 387handles HTTP/1.1 basic authentication correctly as outlined in 388RFC2068, but I haven't been able to find an HTTP server that honors 389the Authentication: header field. Also, 390.Nm libfetch 391does not attempt to interpret and respond to authentication requests 392from the HTTP server. 393.Pp 394No attempt is made to encode spaces etc. within URLs. Spaces in the 395document part of an URLshould be replaced with "%20" in HTTP URLs and 396"\\ " in FTP URLs. 397.Pp 398Error numbers are unique only within a certain context; the error 399codes used for FTP and HTTP overlap, as do those used for resolver and 400system errors. For instance, error code 202 means "Command not 401implemented, superfluous at this site" in an FTP context and 402"Accepted" in an HTTP context. 403.Pp 404The man page is poorly written and produces badly formatted text. 405.Pp 406Tons of other stuff. 407