xref: /freebsd/usr.bin/fetch/fetch.1 (revision 7f3dea244c40159a41ab22da77a434d7c5b5e85a)
1.\" $Id: fetch.1,v 1.29 1999/02/23 18:51:13 wollman Exp $
2.Dd February 22, 1999
3.Dt FETCH 1
4.Os FreeBSD 4.0
5.Sh NAME
6.Nm fetch
7.Nd retrieve a file by Uniform Resource Locator
8.Sh SYNOPSIS
9.Nm fetch
10.Op Fl AMPablmnpqrtv
11.Op Fl S Ar size
12.Op Fl T Ar timeout
13.Op Fl o Ar file
14.Ar URL
15.Op Ar ...
16.Nm fetch
17.Op Fl MPRlmnpqrv
18.Op Fl S Ar size
19.Op Fl o Ar file
20.Op Fl c Ar dir
21.Fl f Ar file
22.Fl h Ar host
23.Sh DESCRIPTION
24.Nm fetch
25allows a user to transfer files from a remote network site using
26either the
27.Tn FTP
28or the
29.Tn HTTP
30protocol. In the first form of the command, the
31.Ar URL
32may be of the form
33.Li http://site.domain/path/to/the/file
34or
35.Li ftp://site.domain/path/to/the/file .
36To denote a local filename to be copied or linked to (see the
37.Fl l
38flag below), the
39.Em file:/path/to/the/file
40URL form is used.  See URL SYNTAX, below.
41.Pp
42The second form of the command can be used to get a file using the
43.Tn FTP
44protocol, specifying the file name and the remote host with the
45.Fl h
46and the
47.Fl f
48flags.
49.Pp
50The following options are available:
51.Bl -tag -width Fl
52.It Fl A
53Do not automatically follow ``temporary'' (302) redirects.  Some
54broken Web sites will return a redirect instead of a not-found error
55when the requested object does not exist.
56.It Fl a
57Automatically retry the transfer upon soft failures.
58.It Fl b
59Work around a bug in some
60.Tn HTTP
61servers which fail to correctly implement the
62.Tn TCP
63protocol.
64.It Fl c Ar dir
65The file to retrieve is in directory
66.Ar dir
67on the remote host.
68.It Fl f Ar file
69The file to retrieve is named
70.Ar file
71on the remote host.
72.It Fl h Ar host
73The file to retrieve is located on the host
74.Ar host .
75.It Fl l
76If target is a
77.Ar file:/
78style of URL, make a link to the target rather than trying
79to copy it.
80.It Fl M
81.It Fl m
82Mirror mode: Set the modification time of the file so that it is
83identical to the modification time of the file at the remote host.
84If the file already exists on the local host and is identical (as
85gauged by size and modification time), no transfer is done.
86.It Fl n
87Don't preserve the modtime of the transferred file, use the current time.
88.It Fl o Ar file
89Set the output file name to
90.Ar file .
91By default, a ``pathname'' is extracted from the specified URI, and
92its basename is used as the name of the output file.  A
93.Ar file
94argument of
95.Sq Li \&-
96indicates that results are to be directed to the standard output.
97.It Fl P
98.It Fl p
99Use the passive mode of the
100.Tn FTP
101protocol.  This is useful for crossing certain sorts of firewalls.
102.It Fl q
103Quiet mode. Do not report transfer progress on the terminal.
104.It Fl R
105The filenames specified are ``precious'', and should not be deleted
106under any circumstances, even if the transfer failed or was incomplete.
107.It Fl r
108Restart a previously interrupted transfer.
109.It Fl S Ar bytes
110Require the file size reported by
111.Tn FTP
112or
113.Tn HTTP
114server to match the value specified with this option.
115On mismatch, a message is printed and the file will not be fetched.
116If the server does not support reporting of file sizes, the option
117will be ignored and the file will be retrieved anyway.
118This option is useful to prevent
119.Nm fetch
120from downloading a file that is either incomplete or the wrong version,
121given the correct size of the file in advance.
122.It Fl s
123Ask server for size of file in bytes and print it to stdout. Do not
124actually fetch the file.
125.It Fl T Ar seconds
126Set timeout value to
127.Ar seconds.
128Overrides the environment variables
129.Ev FTP_TIMEOUT
130for ftp transfers or
131.Ev HTTP_TIMEOUT
132for http transfers if set.
133.It Fl t
134Work around a different set of buggy
135.Tn TCP
136implementations.
137.It Fl v
138Increase verbosity.  More
139.Fl v Ns \&'s
140result in more information.
141.El
142.Pp
143Many options are also controlled solely by the environment (this is a
144bug).
145.Sh URL SYNTAX
146.Nm
147accepts
148.Tn http
149and
150.Tn ftp
151URL's, as described in RFC1738.  For
152.Tn ftp
153URL's, a username and password may be specified, using the syntax
154.Li ftp://user:password@host/.
155If the path is to be absolute, as opposed to relative to the user's
156home directory, it must start with %2F, as in
157.Li ftp://root:mypass@localhost/%2Fetc/passwd .
158.Nm Fetch
159condenses multiple slashes in an
160.Tn ftp
161URL into a single slash; literal multiple slashes translate to an
162.Tn ftp
163protocol error.
164.Sh PROXY SERVERS
165Many sites use application gateways (``proxy servers'') in their
166firewalls in order to allow communication across the firewall using a
167trusted protocol.  The
168.Nm fetch
169program can use both the
170.Tn FTP
171and the
172.Tn HTTP
173protocol with a proxy server.
174.Tn FTP
175proxy servers can only relay
176.Tn FTP
177requests;
178.Tn HTTP
179proxy servers can relay both
180.Tn FTP
181and
182.Tn HTTP
183requests.
184A proxy server can be configured by defining an environment variable
185named
186.Dq Va PROTO Ns Ev _PROXY ,
187where
188.Va PROTO
189is the name of the protocol in upper case.  The value of the
190environment variable specifies a hostname, optionally followed by a
191colon and a port number.
192.Pp
193The
194.Tn FTP
195proxy client passes the remote username, host and port as the
196.Tn FTP
197session's username, in the form
198.Do Va remoteuser Ns Li \&@ Ns Va remotehost
199.Op Li \&@ Ns Va port
200.Dc .
201The
202.Tn HTTP
203proxy client simply passes the originally-requested URI to the remote
204server in an
205.Tn HTTP
206.Dq Li GET
207request.  HTTP proxy authentication is not yet implemented.
208.Sh HTTP AUTHENTICATION
209The
210.Tn HTTP
211protocol includes support for various methods of authentication.
212Currently, the
213.Dq basic
214method, which provides no security from packet-sniffing or
215man-in-the-middle attacks, is the only method supported in
216.Nm fetch .
217Authentication is enabled by the
218.Ev HTTP_AUTH
219and
220.Ev HTTP_PROXY_AUTH
221environment variables.  Both variables have the same format, which
222consists of space-separated list of parameter settings, where each
223setting consists of a colon-separated list of parameters.  The first
224two parameters are always the (case-insensitive) authentication scheme
225name and the realm in which authentication is to be performed.  If the
226realm is specified as
227.Sq Li \&* ,
228then it will match all realms not specified otherwise.
229.Pp
230The
231.Li basic
232authentication scheme uses two additional optional parameters; the
233first is a user name, and the second is the password associated with
234it.  If either the password or both parameters are not specified in
235the environment, and the standard input of
236.Nm
237is connected to a terminal, then
238.Nm
239will prompt the user to enter the missing parameters.  Thus, if the
240user is known as
241.Dq Li jane
242in the
243.Dq Li WallyWorld
244realm, and has a password of
245.Dq Li QghiLx79
246there, then she might set her
247.Ev HTTP_AUTH
248variable to:
249.Bl -enum -offset indent
250.It
251.Dq Li basic:WallyWorld:jane:QghiLx79
252.It
253.Dq Li basic:WallyWorld:jane ,
254or
255.It
256.Dq Li basic:WallyWorld
257.El
258.Pp
259and
260.Nm
261will prompt for any missing information when it is required.  She might
262also specify a realm of
263.Dq Li \&*
264instead of
265.Dq Li WallyWorld
266to indicate that the parameters can be applied to any realm.  (This is
267most commonly used in a construction such as
268.Dq Li basic:* ,
269which indicates to
270.Nm
271that it may offer to do
272.Li basic
273authentication for any realm.
274.Sh ERRORS
275The
276.Nm
277command returns zero on success, or a non-zero value from
278.Aq Pa sysexits.h
279on failure.  If multiple URIs are given for retrieval,
280.Nm
281will attempt all of them and return zero only if all succeeded
282(otherwise it will return the error from the last failure).
283.Sh ENVIRONMENT
284.Bl -tag -width FTP_PASSIVE_MODE -offset indent
285.It Ev FTP_TIMEOUT
286maximum time, in seconds, to wait before aborting an
287.Tn FTP
288connection.
289.It Ev FTP_LOGIN
290the login name used for
291.Tn FTP
292transfers (default
293.Dq Li anonymous )
294.It Ev FTP_PASSIVE_MODE
295force the use of passive mode FTP
296.It Ev FTP_PASSWORD
297the password used for
298.Tn FTP
299transfers (default
300.Dq Va yourname Ns Li \&@ Ns Va yourhost )
301.It Ev FTP_PROXY
302the address (in the form
303.Do Va hostname Ns
304.Op Li : Ns Va port
305.Dc )
306of a proxy server which understands
307.Tn FTP
308.It Ev HTTP_AUTH
309defines authentication parameters for
310.Tn HTTP
311.It Ev HTTP_PROXY
312the address (in the form
313.Do Va hostname Ns
314.Op Li : Ns Va port
315.Dc )
316of a proxy server which understands
317.Tn HTTP
318.It Ev HTTP_PROXY_AUTH
319defines authentication parameters for
320.Tn HTTP
321proxy servers
322.It Ev HTTP_TIMEOUT
323maximum time, in seconds, to wait before aborting an
324.Tn HTTP
325connection.
326.Sh SEE ALSO
327.Xr ftp 1 ,
328.Xr tftp 1
329.Rs
330.%A R. Fielding
331.%A J. Gettys
332.%A J. Mogul
333.%A H. Frystyk
334.%A T. Berners-Lee
335.%T "Hypertext Transfer Protocol \-\- HTTP/1.1"
336.%O RFC 2068
337.%D January 1997
338.Re
339.Rs
340.%A T. Berners-Lee
341.%A L. Masinter
342.%A M. McCahill
343.%T "Uniform Resource Locators (URL)"
344.%O RFC 1738
345.%D December 1994
346.Re
347.Rs
348.%A J. Postel
349.%A J.K. Reynolds
350.%T "File Transfer Protocol"
351.%O RFC 959 / STD 9
352.%D October 1985
353.Re
354.Rs
355.%A M.R. Horton
356.%T "Standard for interchange of USENET messages."
357.%O RFC 850
358.%D June 1983
359.Re
360.Sh HISTORY
361The
362.Nm fetch
363command appeared in
364.Fx 2.1.5 .
365.Sh AUTHORS
366The original implementation of
367.Nm
368was done by
369.An Jean-Marc Zucconi .
370It was extensively re-worked for
371.Fx 2.2
372by
373.An Garrett Wollman .
374.Sh BUGS
375There are too many environment variables and command-line options.
376.Pp
377The
378.Fl a
379option is only implemented for certain kinds of
380.Tn HTTP
381failures, and no
382.Tn FTP
383failures.
384.Pp
385Only the
386.Dq basic
387authentication mode is implemented for
388.Tn HTTP .
389This should be replaced by digest authentication.
390.Pp
391Some
392.Tn TCP
393implementations (other than
394.Tn FreeBSD )
395fail to correctly implement cases where the
396.Dv SYN
397and/or
398.Dv FIN
399control flags are specified in packets which also contain data.
400The
401.Sq Fl t
402flag works around the latter deficiency and the
403.Sq Fl b
404flag works around the former.  Since these are errors of the server's
405.Tn TCP
406stack, the best we can do is provide these workarounds.  Given a correct
407server, an optimal
408.Tn HTTP
409transfer without
410.Fl t
411and
412.Fl b
413involves a minimum of two round trips (for small replies), one less than
414other implementations.
415.Pp
416The
417.Tn HTTP
418standard requires interpretation of the
419.Tn RFC 850
420date format, which does not provide a century indication.  Versions of
421.Nm fetch
422prior to
423.Fx 3.1
424would interpret all such dates as being in the 1900s.  This version of
425.Nm fetch
426interprets such dates according to the rule given in
427.Tn RFC 2068 :
428.Bd -literal -offset indent
429  o  HTTP/1.1 clients and caches should assume that an RFC-850 date
430     which appears to be more than 50 years in the future is in fact
431     in the past (this helps solve the "year 2000" problem).
432.Ed
433