1 2Design of xlocale 3================= 4 5The xlocale APIs come from Darwin, although a subset is now part of POSIX 2008. 6They fall into two broad categories: 7 8- Manipulation of per-thread locales (POSIX) 9- Locale-aware functions taking an explicit locale argument (Darwin) 10 11This document describes the implementation of these APIs for FreeBSD. 12 13Goals 14----- 15 16The overall goal of this implementation is to be compatible with the Darwin 17version. Additionally, it should include minimal changes to the existing 18locale code. A lot of the existing locale code originates with 4BSD or earlier 19and has had over a decade of testing. Replacing this code, unless absolutely 20necessary, gives us the potential for more bugs without much benefit. 21 22With this in mind, various libc-private functions have been modified to take a 23locale_t parameter. This causes a compiler error if they are accidentally 24called without a locale. This approach was taken, rather than adding _l 25variants of these functions, to make it harder for accidental uses of the 26global-locale versions to slip in. 27 28Locale Objects 29-------------- 30 31A locale is encapsulated in a `locale_t`, which is an opaque type: a pointer to 32a `struct _xlocale`. The name `_xlocale` is unfortunate, as it does not fit 33well with existing conventions, but is used because this is the name the Darwin 34implementation gives to this structure and so may be used by existing (bad) code. 35 36This structure should include all of the information corresponding to a locale. 37A locale_t is almost immutable after creation. There are no functions that modify it, 38and it can therefore be used without locking. It is the responsibility of the 39caller to ensure that a locale is not deallocated during a call that uses it. 40 41Each locale contains a number of components, one for each of the categories 42supported by `setlocale()`. These are likewise immutable after creation. This 43differs from the Darwin implementation, which includes a deprecated 44`setinvalidrune()` function that can modify the rune locale. 45 46The exception to these mutability rules is a set of `mbstate_t` flags stored 47with each locale. These are used by various functions that previously had a 48static local `mbstate_t` variable. 49 50The components are reference counted, and so can be aliased between locale 51objects. This makes copying locales very cheap. 52 53The Global Locale 54----------------- 55 56All locales and locale components are reference counted. The global locale, 57however, is special. It, and all of its components, are static and so no 58malloc() memory is required when using a single locale. 59 60This means that threads using the global locale are subject to the same 61constraints as with the pre-xlocale libc. Calls to any locale-aware functions 62in threads using the global locale, while modifying the global locale, have 63undefined behaviour. 64 65Because of this, we have to ensure that we always copy the components of the 66global locale, rather than alias them. 67 68It would be cleaner to simply remove the special treatment of the global locale 69and have a locale_t lazily allocated for the global context. This would cost a 70little more `malloc()` memory, so is not done in the initial version. 71 72Caching 73------- 74 75The existing locale implementation included several ad-hoc caching layers. 76None of these were thread safe. Caching is only really of use for supporting 77the pattern where the locale is briefly changed to something and then changed 78back. 79 80The current xlocale implementation removes the caching entirely. This pattern 81is not one that should be encouraged. If you need to make some calls with a 82modified locale, then you should use the _l suffix versions of the calls, 83rather than switch the global locale. If you do need to temporarily switch the 84locale and then switch it back, `uselocale()` provides a way of doing this very 85easily: It returns the old locale, which can then be passed to a subsequent 86call to `uselocale()` to restore it, without the need to load any locale data 87from the disk. 88 89If, in the future, it is determined that caching is beneficial, it can be added 90quite easily in xlocale.c. Given, however, that any locale-aware call is going 91to be a preparation for presenting data to the user, and so is invariably going 92to be part of an I/O operation, this seems like a case of premature 93optimisation. 94 95localeconv 96---------- 97 98The `localeconv()` function is an exception to the immutable-after-creation 99rule. In the classic implementation, this function returns a pointer to some 100global storage, which is initialised with the data from the current locale. 101This is not possible in a multithreaded environment, with multiple locales. 102 103Instead, each locale contains a `struct lconv` that is lazily initialised on 104calls to `localeconv()`. This is not protected by any locking, however this is 105still safe on any machine where word-sized stores are atomic: two concurrent 106calls will write the same data into the structure. 107 108Explicit Locale Calls 109--------------------- 110 111A large number of functions have been modified to take an explicit `locale_t` 112parameter. The old APIs are then reimplemented with a call to `__get_locale()` 113to supply the `locale_t` parameter. This is in line with the Darwin public 114APIs, but also simplifies the modifications to these functions. The 115`__get_locale()` function is now the only way to access the current locale 116within libc. All of the old globals have gone, so there is now a linker error 117if any functions attempt to use them. 118 119The ctype.h functions are a little different. These are not implemented in 120terms of their locale-aware versions, for performance reasons. Each of these 121is implemented as a short inline function. 122 123Differences to Darwin APIs 124-------------------------- 125 126`strtoq_l()` and `strtouq_l() `are not provided. These are extensions to 127deprecated functions - we should not be encouraging people to use deprecated 128interfaces. 129 130Locale Placeholders 131------------------- 132 133The pointer values 0 and -1 have special meanings as `locale_t` values. Any 134public function that accepts a `locale_t` parameter must use the `FIX_LOCALE()` 135macro on it before using it. For efficiency, this can be emitted in functions 136which *only* use their locale parameter as an argument to another public 137function, as the callee will do the `FIX_LOCALE()` itself. 138 139Potential Improvements 140---------------------- 141 142Currently, the current rune set is accessed via a function call. This makes it 143fairly expensive to use any of the ctype.h functions. We could improve this 144quite a lot by storing the rune locale data in a __thread-qualified variable. 145 146Several of the existing FreeBSD locale-aware functions appear to be wrong. For 147example, most of the `strto*()` family should probably use `digittoint_l()`, 148but instead they assume ASCII. These will break if using a character encoding 149that does not put numbers and the letters A-F in the same location as ASCII. 150Some functions, like `strcoll()` only work on single-byte encodings. No 151attempt has been made to fix existing limitations in the libc functions other 152than to add support for xlocale. 153 154Intuitively, setting a thread-local locale should ensure that all locale-aware 155functions can be used safely from that thread. In fact, this is not the case 156in either this implementation or the Darwin one. You must call `duplocale()` 157or `newlocale()` before calling `uselocale()`. This is a bit ugly, and it 158would be better if libc ensure that every thread had its own locale object. 159