libfetch: don't include fragments in HTTP requests

Summary:
Fragments are reserved for client-side processing, see
https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1

Also, some servers don't l

libfetch: don't include fragments in HTTP requests

Summary:
Fragments are reserved for client-side processing, see
https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1

Also, some servers don't like to receive HTTP requests with fragments.

```
$ fetch 'https://dropbox.com/a/b'
fetch: https://dropbox.com/a/b: Not Found

$ fetch 'https://dropbox.com/a/b#'
fetch: https://dropbox.com/a/b#: Bad Request
```

This is a real-world scenario, where some download link from dropbox
(eventually) redirects to an URL with a fragment:

```
$ fetch -v 'https://www.dropbox.com/sh/<some>/<thing>?dl=1' 2>&1 | grep requesting
requesting https://www.dropbox.com/sh/<some>/<thing>?dl=1
requesting https://www.dropbox.com/scl/fo/<foo>/<bar>?rlkey=<baz>&dl=1
requesting https://<boo>.dl.dropboxusercontent.com/zip_download_get/<some-long-strig>#
```

See how the last redirect ends with a `#`.

Currently, libfetch includes the ending fragment and makes it impossible
to download the file.

Differential Revision: https://reviews.freebsd.org/D46318
MFC after: 2 weeks

show more ...

libfetch: parse IPv4address in IPv6address properly

This fix parsing problems of IPv6 addresses which contains IPv4
addresses. One example is ::192.168.0.1.

Documents are in below:
https://datatrac

libfetch: parse IPv4address in IPv6address properly

This fix parsing problems of IPv6 addresses which contains IPv4
addresses. One example is ::192.168.0.1.

Documents are in below:
https://datatracker.ietf.org/doc/html/rfc2373#page-22

Sponsored by: Juniper Networks, Inc.
MFC after: 1 week
Reviewed by: jhibbits
Differential Revision: https://reviews.freebsd.org/D45896

show more ...

libfetch: parse scheme://domain:/ correctly

This improves URL-parsing compability with cURL, and unbreaks parsing of
similar kinds of URLs after commit 8d9de5b10a24.

Sponsored by: Juniper Networks,

libfetch: parse scheme://domain:/ correctly

This improves URL-parsing compability with cURL, and unbreaks parsing of
similar kinds of URLs after commit 8d9de5b10a24.

Sponsored by: Juniper Networks, Inc.
Reviewed by: des
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D44493

show more ...

lib: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

lib: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Merge ^/head r357408 through r357661.

libfetch: disallow invalid escape sequences

Per RFC1738 escape is "% hex hex"; other sequences do not form a valid URL.

Suggested by: Matthew Dillon
Reviewed by: Matthew Dillon
MFC after: 1 week

Merge ^/head r357179 through r357269.

Fix urldecode buffer overrun.

Reported by: Duncan Overbruck
Security: CVE-2020-7450

Merge ^/head r340918 through r341763.

Improve URL parsing. In particular, convert scheme and host to lowercase.

MFC after: 1 week

Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (nu

Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (null).

While there, clean up handling of netrcfd and add debugging aids.

MFC after: 3 weeks

show more ...

Use __VA_ARGS__ to simplify the DEBUG macro.

MFC after: 3 weeks

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
pr

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

show more ...

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.

While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.

PR: 217723
MFC after: 1 week
Reported by: bapt, bz, cem, ngie

show more ...

Properly initialize netrcfd in fetchParseURL

This fixes ftp with fetch(1) which was broken after r313974

Submitted by: dim
Reported by: olivier
Pointyhat to: bapt

Merge ^/head r313896 through r314128.

Add a file descriptor in struct url for netrc

When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
$

Add a file descriptor in struct url for netrc

When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
${HOME}/.netrc. Now a caller can prepare a fd to the said file and manually
assign it to the structure.

It is also a first step to allow to capsicumize libfetch applications

Reviewed by: allanjude, des
Approved by: des
Differential Revision: https://reviews.freebsd.org/D9678

show more ...

Merge head r233826 through r256722.

Merge from head

IFC @ r254014

IFC @253398

IFC @ r252763

Fix -Wunsequenced warning.

Submitted by: dt71@gmx.com

MFC