MFH r289384-r293170

Sponsored by: The FreeBSD Foundation

Catch up with head (r291075).

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated fil

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

show more ...

collate: Fix expansion substitions (broken upstream too)

Through testing, the user noted that some Cyrillic characters were not
sorting correctly, and this was confirmed.

After extensive testing an

collate: Fix expansion substitions (broken upstream too)

Through testing, the user noted that some Cyrillic characters were not
sorting correctly, and this was confirmed.

After extensive testing and review, the localedef tool was eliminated
as the culprit. The sustitutions were encoded correctly in LC_COLLATE.

The error was mainly in wcscoll where character expansions were
mishandled. The main directive pass routines had to be written to
go back for a new collation value when the "state" variable was set.
Before pointers were being advanced, the second lookup was gettting
applied to the wrong character, etc.

The "eat expansion codes" section on collate.c also had a bug. Later
own, the "state" variable logic was changed to only set if next
code was greater than zero (rather than >= 0).

Some additional cleanups got captured from previous work:
1) The previous commit moved the binary search comment from the
correct location to a wrong location because it's wrong upstream
in Illumos. The comment has little value so I just removed it.
2) Don't check if pointers are null before freeing, this is
redundant as free() handles null pointers.
3) The two binary search trees were standardized wrt initialization
4) On the binary search trees, a negative "high" exits rather than
checking the table count again.

Submitted by: marino
Obtained from: DragonflyBSD

show more ...

libc/collate: minor tweaks / fix

The main "fix" here is properly setting a collate loading error for each
early return. Tweaks include removing unnecessary null checks, adding
assertions (from Illu

libc/collate: minor tweaks / fix

The main "fix" here is properly setting a collate loading error for each
early return. Tweaks include removing unnecessary null checks, adding
assertions (from Illumos) and a couple of variables to reduces code
differences and improve readability. For normal use, there are no
functional changes here.

Obtained from: DragonflyBSD, Illumos

show more ...

Include sys/*.h earlier

Reported by: kib

Fix typo

Use more asprintf
Plug memory leak introduced in previous asprintf addition

Use asprintf/free instead of snprintf

Remove useless variable

Mark __collate_load_tables_l as static
Remove useless addition to Symbols.map

Fix typo
Fix bad location for include

Reported by: jilles

Revamp CTYPE support (from Illumos & Dragonfly)

Obtained from: Dragonfly

The collate functions within libc have been using version 1 and 1.2 of the
packed LC_COLLATE binary formats. These were generated with the colldef
tool, but the new LC_COLLATE files are going to be g

The collate functions within libc have been using version 1 and 1.2 of the
packed LC_COLLATE binary formats. These were generated with the colldef
tool, but the new LC_COLLATE files are going to be generated by the new
localedef tool using CLDR POSIX files as input. The BSD-flavored
version of localedef identifies the format as "BSD 1.0". Any
LC_COLLATE file with a different version will simply not be loaded, and
all LC* categories will get set to "C" (aka "POSIX") locale.

This work is based off of Nexenta's contribution to Illumos.
The integration with xlocale is John Marino's work for Dragonfly.

The following commits will enable localedef tool, disable the colldef
tool, add generated colldef directory, and finally remove colldef from
base.

The only difference with Dragonfly are:
- a few fixes to build with clang
- And identification of the flavor as "BSD 1.0" instead of "Dragonfly 4.4"

Obtained from: Dragonfly

show more ...

Merge fresh head.

IFC @247348.

Sync with HEAD.

IFC @ r244983.

libc: Make various internal file descriptors close-on-exec.

These are obtained via fopen().

Sync from head

Merge head r236710 through r238467.

IFC @238412.

IFC @ r238370

Restore the __collate_load_error global that was accidentally removed in the
xlocale refactoring.

MFC after: 1 week

MFC