1.\" Copyright (c) 1998 Dag-Erling Co�dan Sm�rgrav 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd July 1, 1998 28.Dt FETCH 3 29.Os 30.Sh NAME 31.Nm fetchGetURL , 32.Nm fetchPutURL , 33.Nm fetchStatURL , 34.Nm fetchListURL , 35.Nm fetchParseURL , 36.Nm fetchGet , 37.Nm fetchPut , 38.Nm fetchStat , 39.Nm fetchList , 40.Nm fetchGetFile , 41.Nm fetchPutFile , 42.Nm fetchStatFile , 43.Nm fetchListFile , 44.Nm fetchGetHTTP , 45.Nm fetchPutHTTP , 46.Nm fetchStatHTTP , 47.Nm fetchListHTTP , 48.Nm fetchGetFTP , 49.Nm fetchPutFTP 50.Nm fetchStatFTP 51.Nm fetchListFTP , 52.Nd file transfer library 53.Sh SYNOPSIS 54.Fd #include <sys/param.h> 55.Fd #include <stdio.h> 56.Fd #include <fetch.h> 57.Ft FILE * 58.Fn fetchGetURL "char *URL" "char *flags" 59.Ft FILE * 60.Fn fetchPutURL "char *URL" "char *flags" 61.Ft int 62.Fn fetchStatURL "char *URL" "struct url_stat *us" "char *flags" 63.Ft struct url_ent * 64.Fn fetchListURL "char *URL" "char *flags" 65.Ft struct url * 66.Fn fetchParseURL "char *URL" 67.Ft FILE * 68.Fn fetchGet "struct url *URL" "char *flags" 69.Ft FILE * 70.Fn fetchPut "struct url *URL" "char *flags" 71.Ft int 72.Fn fetchStat "struct url *URL" "struct url_stat *us" "char *flags" 73.Ft struct url_ent * 74.Fn fetchList "struct url *" "char *flags" 75.Ft FILE * 76.Fn fetchGetFile "struct url *u" "char *flags" 77.Ft FILE * 78.Fn fetchPutFile "struct url *u" "char *flags" 79.Ft int 80.Fn fetchStatFile "struct url *URL" "struct url_stat *us" "char *flags" 81.Ft struct url_ent * 82.Fn fetchListFile "struct url *" "char *flags" 83.Ft FILE * 84.Fn fetchGetHTTP "struct url *u" "char *flags" 85.Ft FILE * 86.Fn fetchPutHTTP "struct url *u" "char *flags" 87.Ft int 88.Fn fetchStatHTTP "struct url *URL" "struct url_stat *us" "char *flags" 89.Ft struct url_ent * 90.Fn fetchListHTTP "struct url *" "char *flags" 91.Ft FILE * 92.Fn fetchGetFTP "struct url *u" "char *flags" 93.Ft FILE * 94.Fn fetchPutFTP "struct url *u" "char *flags" 95.Ft int 96.Fn fetchStatFTP "struct url *URL" "struct url_stat *us" "char *flags" 97.Ft struct url_ent * 98.Fn fetchListFTP "struct url *" "char *flags" 99.Sh DESCRIPTION 100.Pp 101These functions implement a high-level library for retrieving and 102uploading files using Uniform Resource Locators (URLs). 103.Pp 104.Fn fetchGetURL 105and 106.Fn fetchPutURL 107constitute the recommended interface to the 108.Nm fetch 109library. They examine the URL passed to them to determine the transfer 110method, and call the appropriate lower-level functions to perform the 111actual transfer. The 112.Fa flags 113argument is a string of characters which specify transfer options. The 114meaning of the individual flags is scheme-dependent, and is detailed 115in the appropriate section below. 116.Pp 117.Fn fetchStatURL 118attempts to obtain the requested document's metadata and fill in the 119structure pointed to by it's second argument. The 120.Fa url_stat 121structure is defined as follows in 122.Aq Pa fetch.h : 123.Bd -literal 124struct url_stat { 125 off_t size; 126 time_t atime; 127 time_t mtime; 128}; 129.Ed 130.Pp 131.Fn fetchListURL 132attempts to list the contents of the directory pointed to by the URL 133provided. If successful, it returns a malloced array of 134.Fa url_ent 135structures. The 136.Fa url_ent 137structure is defined as follows in 138.Aq Pa fetch.h : 139.Bd -literal 140struct url_ent { 141 char name[MAXPATHLEN]; 142 struct url_stat stat; 143}; 144.Ed 145.Pp 146The list is terminated by an entry with an empty name. 147.Pp 148The pointer returned by 149.Fn fetchListURL 150should be freed using 151.Fn free . 152.Pp 153.Fn fetchParseURL 154takes a URL in the form of a null-terminated string and splits it into 155its components function according to the Common Internet Scheme Syntax 156detailed in RFC1738. A regular expression which produces this syntax 157is: 158.Bd -literal 159 <scheme>:(//(<user>(:<pwd>)?@)?<host>(:<port>)?)?/(<document>)? 160.Ed 161.Pp 162Note that some components of the URL are not necessarily relevant to 163all URL schemes. For instance, the file scheme only needs the <scheme> 164and <document> components. 165.Pp 166The pointer returned by 167.Fn fetchParseURL 168should be freed using 169.Fn free . 170.Pp 171.Fn fetchGet , 172.Fn fetchPut 173and 174.Fn fetchStat 175are similar to 176.Fn fetchGetURL , 177.Fn fetchPutURL 178and 179.Fn fetchStatURL , 180except that they expect a pre-parsed URL in the form of a pointer to 181a 182.Fa struct url 183rather than a string. 184.Pp 185All of the 186.Fn fetchGetXXX 187and 188.Fn fetchPutXXX 189functions return a pointer to a stream which can be used to read or 190write data from or to the requested document, respectively. Note that 191although the implementation details of the individual access methods 192vary, it can generally be assumed that a stream returned by one of the 193.Fn fetchGetXXX 194functions is read-only, and that a stream returned by one of the 195.Fn fetchPutXXX 196functions is write-only. 197.Sh FILE SCHEME 198.Fn fetchGetFile 199and 200.Fn fetchPutFile 201provide access to documents which are files in a locally mounted file 202system. Only the <document> component of the URL is used. 203.Pp 204.Fn fetchGetFile 205does not accept any flags. 206.Pp 207.Fn fetchPutFile 208accepts the 209.Fa a 210(append to file) flag. If that flag is specified, the data written to 211the stream returned by 212.Fn fetchPutFile 213will be appended to the previous contents of the file, instead of 214replacing them. 215.Sh FTP SCHEME 216.Fn fetchGetFTP 217and 218.Fn fetchPutFTP 219implement the FTP protocol as described in RFC959. 220.Pp 221If the 222.Fa p 223(passive) flag is specified, a passive (rather than active) connection 224will be attempted. 225.Pp 226If the 227.Fa h 228(high) flag is specified, data sockets will be allocated in the high 229port range (see 230.Xr ip 4 ). 231.Pp 232If the 233.Fa d 234(direct) flag is specified, 235.Fn fetchGetFTP 236and 237.Fn fetchPutFTP 238will use a direct connection even if a proxy server is defined. 239.Pp 240If no user name or password is given, the 241.Nm fetch 242library will attempt an anonymous login, with user name "ftp" and 243password "ftp". 244.Sh HTTP SCHEME 245The 246.Fn fetchGetHTTP 247and 248.Fn fetchPutHTTP 249functions implement the HTTP/1.1 protocol. With a little luck, there's 250even a chance that they comply with RFC2068. 251.Pp 252If the 253.Fa d 254(direct) flag is specified, 255.Fn fetchGetHTTP 256and 257.Fn fetchPutHTTP 258will use a direct connection even if a proxy server is defined. 259.Pp 260Since there seems to be no good way of implementing the HTTP PUT 261method in a manner consistent with the rest of the 262.Nm fetch 263library, 264.Fn fetchPutHTTP 265is currently unimplemented. 266.Sh RETURN VALUES 267.Fn fetchParseURL 268returns a pointer to a 269.Fa struct url 270containing the individual components of the URL. If it is 271unable to allocate memory, or the URL is syntactically incorrect, 272.Fn fetchParseURL 273returns a NULL pointer. 274.Pp 275The 276.Fn fetchStat 277functions return 0 on success and -1 on failure. 278.Pp 279All other functions return a stream pointer which may be used to 280access the requested document, or NULL if an error occurred. 281.Pp 282.Nm Libfetch 283uses the Common Error Library 284.Nm ( libcom_err ) 285to report errors. The error code passed to 286.Fn com_err 287is one of: 288.Bl -tag -width Er 289.It Bq Er FETCH_ABORT 290Operation aborted 291.It Bq Er FETCH_AUTH 292Authentication failed 293.It Bq Er FETCH_DOWN 294Service unavailable 295.It Bq Er FETCH_EXISTS 296File exists 297.It Bq Er FETCH_FULL 298File system full 299.It Bq Er FETCH_INFO 300Informational response 301.It Bq Er FETCH_MEMORY 302Insufficient memory 303.It Bq Er FETCH_MOVED 304File has moved 305.It Bq Er FETCH_NETWORK 306Network error 307.It Bq Er FETCH_OK 308No error 309.It Bq Er FETCH_PROTO 310Protocol error 311.It Bq Er FETCH_RESOLV 312Resolver error 313.It Bq Er FETCH_SERVER 314Server error 315.It Bq Er FETCH_TEMP 316Temporary error 317.It Bq Er FETCH_TIMEOUT 318Operation timed out 319.It Bq Er FETCH_UNAVAIL 320File is not available 321.It Bq Er FETCH_UNKNOWN 322Unknown error 323.It Bq Er FETCH_URL 324Invalid URL 325.El 326.Pp 327The accompanying error message includes a protocol-specific error code 328and message, e.g. "File is not available (404 Not Found)" 329.Sh ENVIRONMENT 330The FTP and HTTP related functions use the 331.Ev HTTP_PROXY 332and 333.Ev FTP_PROXY 334environment variables, respectively, as the address of a proxy server 335to use for transferring files. 336.Sh SEE ALSO 337.Xr com_err 3 , 338.Xr fetch 1 , 339.Xr ftpio 3 , 340.Xr ip 4 . 341.Rs 342.%A T. Berners-Lee, L. Masinter & M. McCahill 343.%D December 1994 344.%T Uniform Resource Locators (URL) 345.%O RFC1738 346.Re 347.Rs 348.%A R. Fielding, J. Gettys, J. Mogul, H. Frystyk, T. Berners-Lee 349.%D Januray 1997 350.%B Hypertext Transfer Protocol -- HTTP/1.1 351.%O RFC2068 352.Re 353.Rs 354.%A J. Postel, J. K. Reynolds 355.%D October 1985 356.%B File Transfer Protocol 357.%O RFC959 358.Re 359.Sh NOTES 360The 361.Nm fetch 362library uses the Common Error library, and applications which link 363with 364.Nm libfetch 365must therefore also link with 366.Nm libcom_err . 367.Sh HISTORY 368The 369.Nm fetch 370library first appeared in 371.Fx 3.0 . 372.Sh AUTHORS 373The 374.Nm fetch 375library was mostly written by 376.An Dag-Erling Co�dan Sm�rgrav Aq des@FreeBSD.org 377with numerous suggestions from 378.An Jordan K. Hubbard Aq jkh@FreeBSD.org , 379.An Eugene Skepner Aq eu@qub.com 380and other FreeBSD developers. 381It replaces the older 382.Nm ftpio 383library written by 384.An Poul-Henning Kamp Aq pkh@FreeBSD.org 385and 386.An Jordan K. Hubbard Aq jkh@FreeBSD.org . 387.Pp 388This manual page was written by 389.An Dag-Erling Co�dan Sm�rgrav Aq des@FreeBSD.org 390.Sh BUGS 391Some parts of the library are not yet implemented. The most notable 392examples of this are 393.Fn fetchPutHTTP , 394.Fn fetchStatHTTP , 395.Fn fetchListHTTP , 396.Fn fetchListFTP , 397and FTP proxy support. 398.Pp 399There's no way to select a proxy at run-time other than setting the 400.Ev HTTP_PROXY 401or 402.Ev FTP_PROXY 403environment variables as appropriate. There is also no way to stop the 404FTP and HTTP functions from trying to use a proxy if these variables 405are set. 406.Pp 407HTTP authentication doesn't work. I'm not sure that's a bug in my 408code; as far as I can determine, 409.Nm libfetch 410handles HTTP/1.1 basic authentication correctly as outlined in 411RFC2068, but I haven't been able to find an HTTP server that honors 412the Authentication: header field. Also, 413.Nm libfetch 414does not attempt to interpret and respond to authentication requests 415from the HTTP server. 416.Pp 417No attempt is made to encode spaces etc. within URLs. Spaces in the 418document part of an URLshould be replaced with "%20" in HTTP URLs and 419"\\ " in FTP URLs. 420.Pp 421Error numbers are unique only within a certain context; the error 422codes used for FTP and HTTP overlap, as do those used for resolver and 423system errors. For instance, error code 202 means "Command not 424implemented, superfluous at this site" in an FTP context and 425"Accepted" in an HTTP context. 426.Pp 427.Fn fetchStatFTP 428does not check that the result of an MDTM command is a valid date. 429.Pp 430The HTTP code needs a complete rewrite, or at least a serious cleanup. 431.Pp 432The man page is poorly written and produces badly formatted text. 433.Pp 434Tons of other stuff. 435