Merge ^/head r340427 through r340868.

Make mbstowcs_basic test pass, now that we have more ctype definitions.

Reported by: jenkins
Approved by: kib (mentor, implicit)

Merge ^/head r306412 through r306905.

localedef: Fix ctype dump (fixed wide spread errors)

This commit is from John Marino in dragonfly with the following commit log:

====
This was a CTYPE encoding error involving consecutive points of

localedef: Fix ctype dump (fixed wide spread errors)

This commit is from John Marino in dragonfly with the following commit log:

====
This was a CTYPE encoding error involving consecutive points of the same
ctype. It was reported by myself to Illumos over a year ago but I was
unsure if it was only happening on BSD. Given the cause, the bug is also
present on Illumos.

Basically, if consecutive points were of the exact same ctype, they would
be defined as a range regardless. For example, all of these would be
considered equivalent:

<A> ... <C>, <H> (converts to <A> .. <H>)
<A>, <B>, <H> (converts to <A> .. <H>)
<A>, <J> ... <H> (converts to <A> .. <H>)

So all the points that shouldn't have been defined got "bridged" by the
extreme points.

The effects were recently reported to FreeBSD on PR 213013. There are
countless places were the ctype flags are misdefined, so this is a major
fix that has to be MFC'd.
====

This reveals a bad change I did on the testsuite: while 0x07FF is a valid
unicode it is not used yet (reserved for future use)

PR: 213013
Submitted by: marino@
Reported by: Kurtis Rader <krader@skepticism.us>
Obtained from: Dragonfly
MFC after: 1 month

show more ...

MFH r289384-r293170

Sponsored by: The FreeBSD Foundation

Catch up with head (r291075).

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated fil

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

show more ...

Push the FreeBSD changes under a specific #ifdef to easier tracking what needs
to be upstreamed.

Fix regression test on multibytes

0x07FF and 0x0800 are valid multibyte characters:
'DOUBLE QUESTION MARK' and 'QUESTION EXCLAMATION MARK'

Merge from head

Sponsored by: Gandi.net

Finish merging from head, messed up in previous attempt

Merge from HEAD

Catch up with head, primarily for the 1.14.4.0 firmware.

Merge ^/head r286422 through r286684.

Fix and re-enable UTF-8 tests.

Reintegrate head: r274132-r277384

Sponsored by: The FreeBSD Foundation

Reintegrate head revisions r273096-r277147

Sponsored by: The FreeBSD Foundation

Sync to HEAD@r274766

Merge from head@274682

Mechanically replace #if defined(__FreeBSD__) and #if defined(__NetBSD__) with
their #ifdef equivalents for everything changed in contrib/netbsd-tests. There
are some items from the vendor tree that

Mechanically replace #if defined(__FreeBSD__) and #if defined(__NetBSD__) with
their #ifdef equivalents for everything changed in contrib/netbsd-tests. There
are some items from the vendor tree that use #if defined(__FreeBSD__) or
#if defined(__NetBSD__) which are being left alone

Requested by: bde, rpaulo
Sponsored by: EMC / Isilon Storage Division

show more ...

Sync to HEAD@r274095.

Reintegrate head@r273095

Sponsored by: The FreeBSD Foundation

IFC @r273066

Do initial port of contrib/netbsd-tests/lib/libc/locale

t_io:
- Expect failures potentially related to implementation-specific knowledge of
the zh_TW.Big5 locale [*]

t_mbrtowc:
- Handle unknown loc

Do initial port of contrib/netbsd-tests/lib/libc/locale

t_io:
- Expect failures potentially related to implementation-specific knowledge of
the zh_TW.Big5 locale [*]

t_mbrtowc:
- Handle unknown locales more gracefully (do not test if the locale doesn't
exist)
- Expect failure with mbrtowc_internal dealing with Japanese locales
(potentially related to implementation detail knowledge of the ja_* locales) [*].

t_mbstowcs, t_mbtowc, t_wctomb:
- Handle unknown locales more gracefully (do not test if the locale doesn't
exist)

t_wcstod:
- Treat FreeBSD like NetBSD and Linux in the XXX: FIXME section

[*] More investigation is required to determine the root cause of the failures

Submitted by: pho
Sponsored by: EMC / Isilon Storage Division

show more ...

Merge HEAD@r272944.