1.\"- 2.\" Copyright (c) 2000-2014 Dag-Erling Smørgrav 3.\" Copyright (c) 2013-2016 Michael Gmelin <freebsd@grem.de> 4.\" All rights reserved. 5.\" Portions Copyright (c) 1999 Massachusetts Institute of Technology; used 6.\" by permission. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 1. Redistributions of source code must retain the above copyright 12.\" notice, this list of conditions and the following disclaimer 13.\" in this position and unchanged. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in the 16.\" documentation and/or other materials provided with the distribution. 17.\" 3. The name of the author may not be used to endorse or promote products 18.\" derived from this software without specific prior written permission. 19.\" 20.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 21.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 22.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 23.\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, 24.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 25.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 29.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30.\" 31.Dd October 7, 2023 32.Dt FETCH 1 33.Os 34.Sh NAME 35.Nm fetch 36.Nd retrieve a file by Uniform Resource Locator 37.Sh SYNOPSIS 38.Nm 39.Op Fl 146AadFlMmnPpqRrsUv 40.Op Fl B Ar bytes 41.Op Fl -bind-address= Ns Ar host 42.Op Fl -ca-cert= Ns Ar file 43.Op Fl -ca-path= Ns Ar dir 44.Op Fl -cert= Ns Ar file 45.Op Fl -crl= Ns Ar file 46.Op Fl i Ar file 47.Op Fl -key= Ns Ar file 48.Op Fl N Ar file 49.Op Fl -no-passive 50.Op Fl -no-proxy= Ns Ar list 51.Op Fl -no-sslv3 52.Op Fl -no-tlsv1 53.Op Fl -no-verify-hostname 54.Op Fl -no-verify-peer 55.Op Fl o Ar file 56.Op Fl -referer= Ns Ar URL 57.Op Fl S Ar bytes 58.Op Fl T Ar seconds 59.Op Fl -user-agent= Ns Ar agent-string 60.Op Fl w Ar seconds 61.Ar URL ... 62.Nm 63.Op Fl 146AadFlMmnPpqRrsUv 64.Op Fl B Ar bytes 65.Op Fl -bind-address= Ns Ar host 66.Op Fl -ca-cert= Ns Ar file 67.Op Fl -ca-path= Ns Ar dir 68.Op Fl -cert= Ns Ar file 69.Op Fl -crl= Ns Ar file 70.Op Fl i Ar file 71.Op Fl -key= Ns Ar file 72.Op Fl N Ar file 73.Op Fl -no-passive 74.Op Fl -no-proxy= Ns Ar list 75.Op Fl -no-sslv3 76.Op Fl -no-tlsv1 77.Op Fl -no-verify-hostname 78.Op Fl -no-verify-peer 79.Op Fl o Ar file 80.Op Fl -referer= Ns Ar URL 81.Op Fl S Ar bytes 82.Op Fl T Ar seconds 83.Op Fl -user-agent= Ns Ar agent-string 84.Op Fl w Ar seconds 85.Fl h Ar host Fl f Ar file Oo Fl c Ar dir Oc 86.Sh DESCRIPTION 87The 88.Nm 89utility provides a command-line interface to the 90.Xr fetch 3 91library. 92Its purpose is to retrieve the file(s) pointed to by the URL(s) on the 93command line. 94.Pp 95The following options are available: 96.Bl -tag -width Fl 97.It Fl 1 , -one-file 98Stop and return exit code 0 at the first successfully retrieved file. 99.It Fl 4 , -ipv4-only 100Forces 101.Nm 102to use IPv4 addresses only. 103.It Fl 6 , -ipv6-only 104Forces 105.Nm 106to use IPv6 addresses only. 107.It Fl A , -no-redirect 108Do not automatically follow ``temporary'' (302) redirects. 109Some broken Web sites will return a redirect instead of a not-found 110error when the requested object does not exist. 111.It Fl a , -retry 112Automatically retry the transfer upon soft failures. 113.It Fl B Ar bytes , Fl -buffer-size= Ns Ar bytes 114Specify the read buffer size in bytes. 115The default is 16,384 bytes. 116Attempts to set a buffer size lower than this will be silently 117ignored. 118The number of reads actually performed is reported at verbosity level 119two or higher (see the 120.Fl v 121flag). 122.It Fl -bind-address= Ns Ar host 123Specifies a hostname or IP address to which sockets used for outgoing 124connections will be bound. 125.It Fl c Ar dir 126The file to retrieve is in directory 127.Ar dir 128on the remote host. 129This option is deprecated and is provided for backward compatibility 130only. 131.It Fl -ca-cert= Ns Ar file 132[SSL] 133Path to certificate bundle containing trusted CA certificates. 134Otherwise, 135OpenSSL's default CA cert and path settings apply. 136.It Fl -ca-path= Ns Ar dir 137[SSL] 138The directory 139.Ar dir 140contains trusted CA hashes. 141.It Fl -cert= Ns Ar file 142[SSL] 143.Ar file 144is a PEM encoded client certificate/key which will be used in 145client certificate authentication. 146.It Fl -crl= Ns Ar file 147[SSL] 148Points to certificate revocation list 149.Ar file , 150which has to be in PEM format and may contain peer certificates that have 151been revoked. 152.It Fl d , -direct 153Use a direct connection even if a proxy is configured. 154.It Fl F , -force-restart 155In combination with the 156.Fl r 157flag, forces a restart even if the local and remote files have 158different modification times. 159Implies 160.Fl R . 161.It Fl f Ar file 162The file to retrieve is named 163.Ar file 164on the remote host. 165This option is deprecated and is provided for backward compatibility 166only. 167.It Fl h Ar host 168The file to retrieve is located on the host 169.Ar host . 170This option is deprecated and is provided for backward compatibility 171only. 172.It Fl i Ar file , Fl -if-modified-since= Ns Ar file 173If-Modified-Since mode: the remote file will only be retrieved if it 174is newer than 175.Ar file 176on the local host. 177(HTTP only) 178.It Fl -key= Ns Ar file 179[SSL] 180.Ar file 181is a PEM encoded client key that will be used in client certificate 182authentication in case key and client certificate are stored separately. 183.It Fl l , -symlink 184If the target is a file-scheme URL, make a symbolic link to the target 185rather than trying to copy it. 186.It Fl M 187.It Fl m , -mirror 188Mirror mode: if the file already exists locally and has the same size 189and modification time as the remote file, it will not be fetched. 190Note that the 191.Fl m 192and 193.Fl r 194flags are mutually exclusive. 195.It Fl N Ar file , Fl -netrc= Ns Ar file 196Use 197.Ar file 198instead of 199.Pa ~/.netrc 200to look up login names and passwords for FTP sites. 201See 202.Xr ftp 1 203for a description of the file format. 204This feature is experimental. 205.It Fl n , -no-mtime 206Do not preserve the modification time of the transferred file. 207.It Fl -no-passive 208Forces the FTP code to use active mode. 209.It Fl -no-proxy= Ns Ar list 210Either a single asterisk, which disables the use of proxies 211altogether, or a comma- or whitespace-separated list of hosts for 212which proxies should not be used. 213.It Fl -no-sslv3 214[SSL] 215Do not allow SSL version 3 when negotiating the connection. 216This option is deprecated and is provided for backward compatibility 217only. 218SSLv3 is disabled by default. 219Set 220.Ev SSL_ALLOW_SSL3 221to change this behavior. 222.It Fl -no-tlsv1 223[SSL] 224Do not allow TLS version 1 when negotiating the connection. 225.It Fl -no-verify-hostname 226[SSL] 227Do not verify that the hostname matches the subject of the 228certificate presented by the server. 229.It Fl -no-verify-peer 230[SSL] 231Do not verify the peer certificate against trusted CAs. 232.It Fl o Ar file , Fl -output= Ns Ar file 233Set the output file name to 234.Ar file . 235By default, a ``pathname'' is extracted from the specified URI, and 236its basename is used as the name of the output file. 237A 238.Ar file 239argument of 240.Sq Li \&- 241indicates that results are to be directed to the standard output. 242If the 243.Ar file 244argument is a directory, fetched file(s) will be placed within the 245directory, with name(s) selected as in the default behaviour. 246.It Fl P 247.It Fl p , -passive 248Use passive FTP. 249These flags have no effect, since passive FTP is the default, but are 250provided for compatibility with earlier versions where active FTP was 251the default. 252To force active mode, use the 253.Fl -no-passive 254flag or set the 255.Ev FTP_PASSIVE_MODE 256environment variable to 257.Ql NO . 258.It Fl -referer= Ns Ar URL 259Specifies the referrer URL to use for HTTP requests. 260If 261.Ar URL 262is set to 263.Dq auto , 264the document URL will be used as referrer URL. 265.It Fl q , -quiet 266Quiet mode. 267.It Fl R , -keep-output 268The output files are precious, and should not be deleted under any 269circumstances, even if the transfer failed or was incomplete. 270.It Fl r , -restart 271Restart a previously interrupted transfer. 272Note that the 273.Fl m 274and 275.Fl r 276flags are mutually exclusive. 277.It Fl S Ar bytes , Fl -require-size= Ns Ar bytes 278Require the file size reported by the server to match the specified 279value. 280If it does not, a message is printed and the file is not fetched. 281If the server does not support reporting file sizes, this option is 282ignored and the file is fetched unconditionally. 283.It Fl s , -print-size 284Print the size in bytes of each requested file, without fetching it. 285.It Fl T Ar seconds , Fl -timeout= Ns Ar seconds 286Set timeout value to 287.Ar seconds . 288Overrides the environment variables 289.Ev FTP_TIMEOUT 290for FTP transfers or 291.Ev HTTP_TIMEOUT 292for HTTP transfers if set. 293.It Fl U , -passive-portrange-default 294When using passive FTP, allocate the port for the data connection from 295the low (default) port range. 296See 297.Xr ip 4 298for details on how to specify which port range this corresponds to. 299.It Fl -user-agent= Ns Ar agent-string 300Specifies the User-Agent string to use for HTTP requests. 301This can be useful when working with HTTP origin or proxy servers that 302differentiate between user agents. 303.It Fl v , -verbose 304Increase verbosity level. 305.It Fl w Ar seconds , Fl -retry-delay= Ns Ar seconds 306When the 307.Fl a 308flag is specified, wait this many seconds between successive retries. 309.El 310.Pp 311If 312.Nm 313receives a 314.Dv SIGINFO 315signal (see the 316.Cm status 317argument for 318.Xr stty 1 ) , 319the current transfer rate statistics will be written to the 320standard error output, in the same format as the standard completion 321message. 322.Sh ENVIRONMENT 323.Bl -tag -width HTTP_TIMEOUT 324.It Ev FTP_TIMEOUT 325Maximum time, in seconds, to wait before aborting an FTP connection. 326.It Ev HTTP_TIMEOUT 327Maximum time, in seconds, to wait before aborting an HTTP connection. 328.El 329.Pp 330See 331.Xr fetch 3 332for a description of additional environment variables, including 333.Ev FETCH_BIND_ADDRESS , 334.Ev FTP_LOGIN , 335.Ev FTP_PASSIVE_MODE , 336.Ev FTP_PASSWORD , 337.Ev FTP_PROXY , 338.Ev ftp_proxy , 339.Ev HTTP_ACCEPT , 340.Ev HTTP_AUTH , 341.Ev HTTP_PROXY , 342.Ev http_proxy , 343.Ev HTTP_PROXY_AUTH , 344.Ev HTTP_REFERER , 345.Ev HTTP_USER_AGENT , 346.Ev NETRC , 347.Ev NO_PROXY , 348.Ev no_proxy , 349.Ev SSL_CA_CERT_FILE , 350.Ev SSL_CA_CERT_PATH , 351.Ev SSL_CLIENT_CERT_FILE , 352.Ev SSL_CLIENT_KEY_FILE , 353.Ev SSL_CRL_FILE , 354.Ev SSL_ALLOW_SSL3 , 355.Ev SSL_NO_TLS1 , 356.Ev SSL_NO_TLS1_1 , 357.Ev SSL_NO_TLS1_2 , 358.Ev SSL_NO_VERIFY_HOSTNAME 359and 360.Ev SSL_NO_VERIFY_PEER . 361.Sh EXIT STATUS 362The 363.Nm 364command returns zero on success, or one on failure. 365If multiple URLs are listed on the command line, 366.Nm 367will attempt to retrieve each one of them in turn, and will return 368zero only if they were all successfully retrieved. 369.Pp 370If the 371.Fl i 372argument is used and the remote file is not newer than the 373specified file then the command will still return success, 374although no file is transferred. 375.Sh EXAMPLES 376Silently try to fetch the URLs passed as parameters. 377The first one will fail. 378If the second URL succeeds the third one will not be tried: 379.Bd -literal -offset indent 380$ fetch -1 -q https://www.freebsd.org/bad.html \e 381 ftp.freebsd.org/pub/FreeBSD/README.TXT \e 382 https://www.fake.url 383fetch: https://www.freebsd.org/bad.html: Not Found 384.Ed 385.Pp 386Be verbose when retrieving the 387.Pa README.TXT 388file: 389.Bd -literal -offset indent 390$ fetch -v ftp.freebsd.org/pub/FreeBSD/README.TXT 391resolving server address: ftp.freebsd.org:80 392requesting http://ftp.freebsd.org/pub/FreeBSD/README.TXT 393local size / mtime: 4259 / 1431015519 394remote size / mtime: 4259 / 1431015519 395README.TXT 4259 B 44 MBps 00s 396.Ed 397.Pp 398Quietly save the 399.Pa README.TXT 400file as 401.Pa myreadme.txt 402and do not delete the output file under any circumstances: 403.Bd -literal -offset indent 404fetch -o myreadme.txt -q -R ftp.freebsd.org/pub/FreeBSD/README.TXT 405.Ed 406.Pp 407Print the size of the requested file and identify the request with a custom user 408agent string: 409.Bd -literal -offset indent 410$ fetch -s ftp.freebsd.org/pub/FreeBSD/README.TXT 411--user-agent="Mozilla/5.0 (X11; FreeBSD x86_64; rv:78.0) Gecko/20100101" 4123513231 413.Ed 414.Pp 415Restart the transfer of the 416.Pa README.TXT 417file and retry the transfer upon soft failures: 418.Bd -literal -offset indent 419$ fetch -a -r http://ftp.freebsd.org/pub/FreeBSD/README.TXT 420.Ed 421.Sh SEE ALSO 422.Xr fetch 3 , 423.Xr phttpget 8 424.Sh HISTORY 425The 426.Nm 427command appeared in 428.Fx 2.1.5 . 429This implementation first appeared in 430.Fx 4.1 . 431.Sh AUTHORS 432.An -nosplit 433The original implementation of 434.Nm 435was done by 436.An Jean-Marc Zucconi Aq Mt jmz@FreeBSD.org . 437It was extensively re-worked for 438.Fx 2.2 439by 440.An Garrett Wollman Aq Mt wollman@FreeBSD.org , 441and later completely rewritten to use the 442.Xr fetch 3 443library by 444.An Dag-Erling Sm\(/orgrav Aq Mt des@FreeBSD.org 445and 446.An Michael Gmelin Aq Mt freebsd@grem.de . 447.Sh NOTES 448The 449.Fl b 450and 451.Fl t 452options are no longer supported and will generate warnings. 453They were workarounds for bugs in other OSes which this implementation 454does not trigger. 455.Pp 456One cannot both use the 457.Fl h , 458.Fl c 459and 460.Fl f 461options and specify URLs on the command line. 462