1.\"- 2.\" Copyright (c) 2000-2014 Dag-Erling Smørgrav 3.\" Copyright (c) 2013-2016 Michael Gmelin <freebsd@grem.de> 4.\" All rights reserved. 5.\" Portions Copyright (c) 1999 Massachusetts Institute of Technology; used 6.\" by permission. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 1. Redistributions of source code must retain the above copyright 12.\" notice, this list of conditions and the following disclaimer 13.\" in this position and unchanged. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in the 16.\" documentation and/or other materials provided with the distribution. 17.\" 3. The name of the author may not be used to endorse or promote products 18.\" derived from this software without specific prior written permission. 19.\" 20.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 21.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 22.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 23.\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, 24.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 25.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 29.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30.\" 31.\" $FreeBSD$ 32.\" 33.Dd October 29, 2020 34.Dt FETCH 1 35.Os 36.Sh NAME 37.Nm fetch 38.Nd retrieve a file by Uniform Resource Locator 39.Sh SYNOPSIS 40.Nm 41.Op Fl 146AadFlMmnPpqRrsUv 42.Op Fl B Ar bytes 43.Op Fl -bind-address= Ns Ar host 44.Op Fl -ca-cert= Ns Ar file 45.Op Fl -ca-path= Ns Ar dir 46.Op Fl -cert= Ns Ar file 47.Op Fl -crl= Ns Ar file 48.Op Fl i Ar file 49.Op Fl -key= Ns Ar file 50.Op Fl N Ar file 51.Op Fl -no-passive 52.Op Fl -no-proxy= Ns Ar list 53.Op Fl -no-sslv3 54.Op Fl -no-tlsv1 55.Op Fl -no-verify-hostname 56.Op Fl -no-verify-peer 57.Op Fl o Ar file 58.Op Fl -referer= Ns Ar URL 59.Op Fl S Ar bytes 60.Op Fl T Ar seconds 61.Op Fl -user-agent= Ns Ar agent-string 62.Op Fl w Ar seconds 63.Ar URL ... 64.Nm 65.Op Fl 146AadFlMmnPpqRrsUv 66.Op Fl B Ar bytes 67.Op Fl -bind-address= Ns Ar host 68.Op Fl -ca-cert= Ns Ar file 69.Op Fl -ca-path= Ns Ar dir 70.Op Fl -cert= Ns Ar file 71.Op Fl -crl= Ns Ar file 72.Op Fl i Ar file 73.Op Fl -key= Ns Ar file 74.Op Fl N Ar file 75.Op Fl -no-passive 76.Op Fl -no-proxy= Ns Ar list 77.Op Fl -no-sslv3 78.Op Fl -no-tlsv1 79.Op Fl -no-verify-hostname 80.Op Fl -no-verify-peer 81.Op Fl o Ar file 82.Op Fl -referer= Ns Ar URL 83.Op Fl S Ar bytes 84.Op Fl T Ar seconds 85.Op Fl -user-agent= Ns Ar agent-string 86.Op Fl w Ar seconds 87.Fl h Ar host Fl f Ar file Oo Fl c Ar dir Oc 88.Sh DESCRIPTION 89The 90.Nm 91utility provides a command-line interface to the 92.Xr fetch 3 93library. 94Its purpose is to retrieve the file(s) pointed to by the URL(s) on the 95command line. 96.Pp 97The following options are available: 98.Bl -tag -width Fl 99.It Fl 1 , -one-file 100Stop and return exit code 0 at the first successfully retrieved file. 101.It Fl 4 , -ipv4-only 102Forces 103.Nm 104to use IPv4 addresses only. 105.It Fl 6 , -ipv6-only 106Forces 107.Nm 108to use IPv6 addresses only. 109.It Fl A , -no-redirect 110Do not automatically follow ``temporary'' (302) redirects. 111Some broken Web sites will return a redirect instead of a not-found 112error when the requested object does not exist. 113.It Fl a , -retry 114Automatically retry the transfer upon soft failures. 115.It Fl B Ar bytes , Fl -buffer-size= Ns Ar bytes 116Specify the read buffer size in bytes. 117The default is 16,384 bytes. 118Attempts to set a buffer size lower than this will be silently 119ignored. 120The number of reads actually performed is reported at verbosity level 121two or higher (see the 122.Fl v 123flag). 124.It Fl -bind-address= Ns Ar host 125Specifies a hostname or IP address to which sockets used for outgoing 126connections will be bound. 127.It Fl c Ar dir 128The file to retrieve is in directory 129.Ar dir 130on the remote host. 131This option is deprecated and is provided for backward compatibility 132only. 133.It Fl -ca-cert= Ns Ar file 134[SSL] 135Path to certificate bundle containing trusted CA certificates. 136If not specified, 137.Pa /usr/local/etc/ssl/cert.pem 138is used. 139If this file does not exist, 140.Pa /etc/ssl/cert.pem 141is used instead. 142If neither file exists and no CA path has been configured, 143OpenSSL's default CA cert and path settings apply. 144The certificate bundle can contain multiple CA certificates. 145The 146.Pa security/ca_root_nss 147port is a common source of a current CA bundle. 148.It Fl -ca-path= Ns Ar dir 149[SSL] 150The directory 151.Ar dir 152contains trusted CA hashes. 153.It Fl -cert= Ns Ar file 154[SSL] 155.Ar file 156is a PEM encoded client certificate/key which will be used in 157client certificate authentication. 158.It Fl -crl= Ns Ar file 159[SSL] 160Points to certificate revocation list 161.Ar file , 162which has to be in PEM format and may contain peer certificates that have 163been revoked. 164.It Fl d , -direct 165Use a direct connection even if a proxy is configured. 166.It Fl F , -force-restart 167In combination with the 168.Fl r 169flag, forces a restart even if the local and remote files have 170different modification times. 171Implies 172.Fl R . 173.It Fl f Ar file 174The file to retrieve is named 175.Ar file 176on the remote host. 177This option is deprecated and is provided for backward compatibility 178only. 179.It Fl h Ar host 180The file to retrieve is located on the host 181.Ar host . 182This option is deprecated and is provided for backward compatibility 183only. 184.It Fl i Ar file , Fl -if-modified-since= Ns Ar file 185If-Modified-Since mode: the remote file will only be retrieved if it 186is newer than 187.Ar file 188on the local host. 189(HTTP only) 190.It Fl -key= Ns Ar file 191[SSL] 192.Ar file 193is a PEM encoded client key that will be used in client certificate 194authentication in case key and client certificate are stored separately. 195.It Fl l , -symlink 196If the target is a file-scheme URL, make a symbolic link to the target 197rather than trying to copy it. 198.It Fl M 199.It Fl m , -mirror 200Mirror mode: if the file already exists locally and has the same size 201and modification time as the remote file, it will not be fetched. 202Note that the 203.Fl m 204and 205.Fl r 206flags are mutually exclusive. 207.It Fl N Ar file , Fl -netrc= Ns Ar file 208Use 209.Ar file 210instead of 211.Pa ~/.netrc 212to look up login names and passwords for FTP sites. 213See 214.Xr ftp 1 215for a description of the file format. 216This feature is experimental. 217.It Fl n , -no-mtime 218Do not preserve the modification time of the transferred file. 219.It Fl -no-passive 220Forces the FTP code to use active mode. 221.It Fl -no-proxy= Ns Ar list 222Either a single asterisk, which disables the use of proxies 223altogether, or a comma- or whitespace-separated list of hosts for 224which proxies should not be used. 225.It Fl -no-sslv3 226[SSL] 227Do not allow SSL version 3 when negotiating the connection. 228This option is deprecated and is provided for backward compatibility 229only. 230SSLv3 is disabled by default. 231Set 232.Ev SSL_ALLOW_SSL3 233to change this behavior. 234.It Fl -no-tlsv1 235[SSL] 236Do not allow TLS version 1 when negotiating the connection. 237.It Fl -no-verify-hostname 238[SSL] 239Do not verify that the hostname matches the subject of the 240certificate presented by the server. 241.It Fl -no-verify-peer 242[SSL] 243Do not verify the peer certificate against trusted CAs. 244.It Fl o Ar file , Fl -output= Ns Ar file 245Set the output file name to 246.Ar file . 247By default, a ``pathname'' is extracted from the specified URI, and 248its basename is used as the name of the output file. 249A 250.Ar file 251argument of 252.Sq Li \&- 253indicates that results are to be directed to the standard output. 254If the 255.Ar file 256argument is a directory, fetched file(s) will be placed within the 257directory, with name(s) selected as in the default behaviour. 258.It Fl P 259.It Fl p , -passive 260Use passive FTP. 261These flags have no effect, since passive FTP is the default, but are 262provided for compatibility with earlier versions where active FTP was 263the default. 264To force active mode, use the 265.Fl -no-passive 266flag or set the 267.Ev FTP_PASSIVE_MODE 268environment variable to 269.Ql NO . 270.It Fl -referer= Ns Ar URL 271Specifies the referrer URL to use for HTTP requests. 272If 273.Ar URL 274is set to 275.Dq auto , 276the document URL will be used as referrer URL. 277.It Fl q , -quiet 278Quiet mode. 279.It Fl R , -keep-output 280The output files are precious, and should not be deleted under any 281circumstances, even if the transfer failed or was incomplete. 282.It Fl r , -restart 283Restart a previously interrupted transfer. 284Note that the 285.Fl m 286and 287.Fl r 288flags are mutually exclusive. 289.It Fl S Ar bytes , Fl -require-size= Ns Ar bytes 290Require the file size reported by the server to match the specified 291value. 292If it does not, a message is printed and the file is not fetched. 293If the server does not support reporting file sizes, this option is 294ignored and the file is fetched unconditionally. 295.It Fl s , -print-size 296Print the size in bytes of each requested file, without fetching it. 297.It Fl T Ar seconds , Fl -timeout= Ns Ar seconds 298Set timeout value to 299.Ar seconds . 300Overrides the environment variables 301.Ev FTP_TIMEOUT 302for FTP transfers or 303.Ev HTTP_TIMEOUT 304for HTTP transfers if set. 305.It Fl U , -passive-portrange-default 306When using passive FTP, allocate the port for the data connection from 307the low (default) port range. 308See 309.Xr ip 4 310for details on how to specify which port range this corresponds to. 311.It Fl -user-agent= Ns Ar agent-string 312Specifies the User-Agent string to use for HTTP requests. 313This can be useful when working with HTTP origin or proxy servers that 314differentiate between user agents. 315.It Fl v , -verbose 316Increase verbosity level. 317.It Fl w Ar seconds , Fl -retry-delay= Ns Ar seconds 318When the 319.Fl a 320flag is specified, wait this many seconds between successive retries. 321.El 322.Pp 323If 324.Nm 325receives a 326.Dv SIGINFO 327signal (see the 328.Cm status 329argument for 330.Xr stty 1 ) , 331the current transfer rate statistics will be written to the 332standard error output, in the same format as the standard completion 333message. 334.Sh ENVIRONMENT 335.Bl -tag -width HTTP_TIMEOUT 336.It Ev FTP_TIMEOUT 337Maximum time, in seconds, to wait before aborting an FTP connection. 338.It Ev HTTP_TIMEOUT 339Maximum time, in seconds, to wait before aborting an HTTP connection. 340.El 341.Pp 342See 343.Xr fetch 3 344for a description of additional environment variables, including 345.Ev FETCH_BIND_ADDRESS , 346.Ev FTP_LOGIN , 347.Ev FTP_PASSIVE_MODE , 348.Ev FTP_PASSWORD , 349.Ev FTP_PROXY , 350.Ev ftp_proxy , 351.Ev HTTP_ACCEPT , 352.Ev HTTP_AUTH , 353.Ev HTTP_PROXY , 354.Ev http_proxy , 355.Ev HTTP_PROXY_AUTH , 356.Ev HTTP_REFERER , 357.Ev HTTP_USER_AGENT , 358.Ev NETRC , 359.Ev NO_PROXY , 360.Ev no_proxy , 361.Ev SSL_CA_CERT_FILE , 362.Ev SSL_CA_CERT_PATH , 363.Ev SSL_CLIENT_CERT_FILE , 364.Ev SSL_CLIENT_KEY_FILE , 365.Ev SSL_CRL_FILE , 366.Ev SSL_ALLOW_SSL3 , 367.Ev SSL_NO_TLS1 , 368.Ev SSL_NO_TLS1_1 , 369.Ev SSL_NO_TLS1_2 , 370.Ev SSL_NO_VERIFY_HOSTNAME 371and 372.Ev SSL_NO_VERIFY_PEER . 373.Sh EXIT STATUS 374The 375.Nm 376command returns zero on success, or one on failure. 377If multiple URLs are listed on the command line, 378.Nm 379will attempt to retrieve each one of them in turn, and will return 380zero only if they were all successfully retrieved. 381.Pp 382If the 383.Fl i 384argument is used and the remote file is not newer than the 385specified file then the command will still return success, 386although no file is transferred. 387.Sh EXAMPLES 388Silently try to fetch the URLs passed as parameters. 389The first one will fail. 390If the second URL succeeds the third one will not be tried: 391.Bd -literal -offset indent 392$ fetch -1 -q https://www.freebsd.org/bad.html \e 393 ftp.freebsd.org/pub/FreeBSD/README.TXT \e 394 https://www.fake.url 395fetch: https://www.freebsd.org/bad.html: Not Found 396.Ed 397.Pp 398Be verbose when retrieving the 399.Pa README.TXT 400file: 401.Bd -literal -offset indent 402$ fetch -v ftp.freebsd.org/pub/FreeBSD/README.TXT 403resolving server address: ftp.freebsd.org:80 404requesting http://ftp.freebsd.org/pub/FreeBSD/README.TXT 405local size / mtime: 4259 / 1431015519 406remote size / mtime: 4259 / 1431015519 407README.TXT 4259 B 44 MBps 00s 408.Ed 409.Pp 410Quietly save the 411.Pa README.TXT 412file as 413.Pa myreadme.txt 414and do not delete the output file under any circumstances: 415.Bd -literal -offset indent 416fetch -o myreadme.txt -q -R ftp.freebsd.org/pub/FreeBSD/README.TXT 417.Ed 418.Pp 419Print the size of the requested file and identify the request with a custom user 420agent string: 421.Bd -literal -offset indent 422$ fetch -s ftp.freebsd.org/pub/FreeBSD/README.TXT 423--user-agent="Mozilla/5.0 (X11; FreeBSD x86_64; rv:78.0) Gecko/20100101" 4243513231 425.Ed 426.Pp 427Restart the transfer of the 428.Pa README.TXT 429file and retry the transfer upon soft failures: 430.Bd -literal -offset indent 431$ fetch -a -r http://ftp.freebsd.org/pub/FreeBSD/README.TXT 432.Ed 433.Sh SEE ALSO 434.Xr fetch 3 , 435.Xr phttpget 8 436.Sh HISTORY 437The 438.Nm 439command appeared in 440.Fx 2.1.5 . 441This implementation first appeared in 442.Fx 4.1 . 443.Sh AUTHORS 444.An -nosplit 445The original implementation of 446.Nm 447was done by 448.An Jean-Marc Zucconi Aq Mt jmz@FreeBSD.org . 449It was extensively re-worked for 450.Fx 2.2 451by 452.An Garrett Wollman Aq Mt wollman@FreeBSD.org , 453and later completely rewritten to use the 454.Xr fetch 3 455library by 456.An Dag-Erling Sm\(/orgrav Aq Mt des@FreeBSD.org 457and 458.An Michael Gmelin Aq Mt freebsd@grem.de . 459.Sh NOTES 460The 461.Fl b 462and 463.Fl t 464options are no longer supported and will generate warnings. 465They were workarounds for bugs in other OSes which this implementation 466does not trigger. 467.Pp 468One cannot both use the 469.Fl h , 470.Fl c 471and 472.Fl f 473options and specify URLs on the command line. 474