workaround against broken duplocale

[Bruno Haible]

Hi,

Running a gnulib testdir on NetBSD, I'm seeing a link error for test-duplocale.
This is due to the fact that the NetBSD people implemented duplocale() without
uselocale().
https://mail-index.netbsd.org/tech-userlevel/2013/04/23/msg007714.html
Which is pretty senseless, since it forces programs to use functions that take
an explicit locale argument (fprintf_l, strfmon_l, and similar). The point
of standardizing duplocale(), newlocale(), uselocale() was to *eliminate*
the need for these non-standard *_l functions.

And on top of that, in NetBSD 7.0,
- strfmon_l is not implemented,
- duplocale(LC_GLOBAL_LOCALE) is broken.

I'm committing these workarounds:
1) Extend the duplocale unit test so that it does meaningful tests even when
uselocale() is not defined. Verified to work on glibc and macOS systems.
2) Extend the duplocale autoconf test to recognize the broken behaviour on
NetBSD.

@ Jörg Sonnenberger: Find attached a program (netbsd7-bug.c) that exhibits
the bug. Its exit code on NetBSD 7.0 is 2. The correct exit code is 0.

Bruno

[Joerg Sonnenberger]

Post by Bruno Haible
Which is pretty senseless, since it forces programs to use functions that take
an explicit locale argument (fprintf_l, strfmon_l, and similar). The point
of standardizing duplocale(), newlocale(), uselocale() was to *eliminate*
the need for these non-standard *_l functions.

uselocale() is fundamentally broken from a design perspective. It can't
be implemented without breaking ABIs. It breaks expectations of existing
applications in funny ways and it is highly undesirable from a
performance spective. It is also by itself useless as building block as
pretty much any higher level runtime like C++ STL wants to have explicit
control over locales.

Post by Bruno Haible
And on top of that, in NetBSD 7.0,
- strfmon_l is not implemented,

This one slipped through. I did a full audit for any locale-sensitive
function, but that one I missed. Thanks for finding it :)

Post by Bruno Haible
- duplocale(LC_GLOBAL_LOCALE) is broken.

I think this is a completely unrelated bug where thousand groupers were
handled incorrectly. Test certainly works on current.

Joerg

[Bruno Haible]

Hi Jörg,

Post by Joerg Sonnenberger
uselocale() is fundamentally broken from a design perspective. It can't
be implemented without breaking ABIs.

Well, glibc did not break ABIs when it introduced uselocale() in 2002.
If you have not exposed too many internals of the locale set by setlocale(),
you shouldn't have any problems. And you have the __RENAME macro, to cope
with these cases.

Post by Joerg Sonnenberger
It breaks expectations of existing applications in funny ways

I would say, it breaks expectations of programmers who have not yet gotten
familiar with the API. But existing applications - that do not invoke
uselocale() - run without modifications and without change in behaviour.

Post by Joerg Sonnenberger
and it is highly undesirable from a performance spective.

Why? If the locale is a single __thread variable (or a set of 6 __thread
variables, if you prefer), it is just as fast as any __thread variable access.

Post by Joerg Sonnenberger
It is also by itself useless as building block as
pretty much any higher level runtime like C++ STL wants to have explicit
control over locales.

Before these functions were standardized, I would have agreed with you:
that a decent object-oriented API for locale_t objects was the way to go,
rather than these odd-to-use newlocale/duplocale APIs with non-extensible
locale_t objects. But now the standard exists, and you do people a favour
if you implement it.

Post by Joerg Sonnenberger

Post by Bruno Haible
And on top of that, in NetBSD 7.0,
- strfmon_l is not implemented,

This one slipped through. I did a full audit for any locale-sensitive
function, but that one I missed. Thanks for finding it :)

FYI, here's the list of symbols from a Mac OS X 10.5 systems that end in '_l':

asprintf_l
atof_l
atoi_l
atol_l
atoll_l
btowc_l
digittoint_l
fgetwc_l
fgetws_l
fprintf_l
fputwc_l
fputws_l
fscanf_l
fwprintf_l
fwscanf_l
getwc_l
getwchar_l
isalnum_l
isalpha_l
isblank_l
iscntrl_l
isdigit_l
isgraph_l
ishexnumber_l
isideogram_l
islower_l
isnumber_l
isphonogram_l
isprint_l
ispunct_l
isrune_l
isspace_l
isspecial_l
isupper_l
iswalnum_l
iswalpha_l
iswblank_l
iswcntrl_l
iswctype_l
iswdigit_l
iswgraph_l
iswhexnumber_l
iswideogram_l
iswlower_l
iswnumber_l
iswphonogram_l
iswprint_l
iswpunct_l
iswrune_l
iswspace_l
iswspecial_l
iswupper_l
iswxdigit_l
isxdigit_l
localeconv_l
mblen_l
mbrlen_l
mbrtowc_l
mbsinit_l
mbsnrtowcs_l
mbsrtowcs_l
mbstowcs_l
mbtowc_l
nextwctype_l
nl_langinfo_l
printf_l
putwc_l
putwchar_l
scanf_l
snprintf_l
sprintf_l
sscanf_l
strcasecmp_l
strcasestr_l
strcoll_l
strfmon_l
strftime_l
strncasecmp_l
strptime_l
strtod_l
strtof_l
strtoimax_l
strtol_l
strtold_l
strtoll_l
strtoq_l
strtoul_l
strtoull_l
strtoumax_l
strtouq_l
strxfrm_l
swprintf_l
swscanf_l
tolower_l
toupper_l
towctrans_l
towlower_l
towupper_l
ungetwc_l
vasprintf_l
vfprintf_l
vfscanf_l
vfwprintf_l
vfwscanf_l
vprintf_l
vscanf_l
vsnprintf_l
vsprintf_l
vsscanf_l
vswprintf_l
vswscanf_l
vwprintf_l
vwscanf_l
wcrtomb_l
wcscoll_l
wcsftime_l
wcsnrtombs_l
wcsrtombs_l
wcstod_l
wcstof_l
wcstoimax_l
wcstol_l
wcstold_l
wcstoll_l
wcstombs_l
wcstoul_l
wcstoull_l
wcstoumax_l
wcswidth_l
wcsxfrm_l
wctob_l
wctomb_l
wctrans_l
wctype_l
wcwidth_l
wprintf_l
wscanf_l

Post by Joerg Sonnenberger

Post by Bruno Haible
- duplocale(LC_GLOBAL_LOCALE) is broken.

I think this is a completely unrelated bug where thousand groupers were
handled incorrectly. Test certainly works on current.

I don't think it's the same as a thousand groupers bug:
- The number the test exercises comes out as "3.5" where "3,5" is expected.
- The month name comes out incorrectly as well.
- In the debugger, when I looked at the memory representation of
duplocale(LC_GLOBAL_LOCALE), I saw a pointer to some C locale data
followed by many NULL pointers.

But I'm pleased to hear that it will be fixed in NetBSD 8.

Bruno

[Joerg Sonnenberger]

Post by Bruno Haible
Hi Jörg,

Post by Joerg Sonnenberger
uselocale() is fundamentally broken from a design perspective. It can't
be implemented without breaking ABIs.

Well, glibc did not break ABIs when it introduced uselocale() in 2002.
If you have not exposed too many internals of the locale set by setlocale(),
you shouldn't have any problems. And you have the __RENAME macro, to cope
with these cases.

I don't know how glibc implemented e.g. ctype.h before. The classic way
to do it is providing a global masking table and that can't be switched
to support uselocale() without breaking the ABI or introducing wonderful
inconsistencies. The alternatives all depend on lying to the compiler or
taking a significant performance hit. The current glibc implementation
falls into the former category.

Post by Bruno Haible

Post by Joerg Sonnenberger
It breaks expectations of existing applications in funny ways

I would say, it breaks expectations of programmers who have not yet gotten
familiar with the API. But existing applications - that do not invoke
uselocale() - run without modifications and without change in behaviour.

This is not true. Before uselocale(), a library could assume that all
threads get consistent locale information if they ask at the same time.
Given that setlocale() can't normally be used in any sane way in a
multi-thread program, the "at the same time" is not really a
restriction. Now take a library that implements a parallel sort for
strings and is collation aware. The library provides the natural
optimization of not using threads on single CPU systems. Does
uselocale() affect the behavior of this library or not? Does it matter
if the library uses a static thread pool or not? This is a very common
pattern and one of the big reasons why C++ doesn't have thread-local
locales in the standard, but explicit locale objects.

The other point is that the far majority of use cases of uselocale()
I have seen so use it for switching to the "C" locale for some parsing
and back afterwards. It is sometimes hidden by a few layers of
indirection, but that's it. The rest is using it as fallback
implementation for lack of explicit locale functions, i.e. in STL.

Post by Bruno Haible

Post by Joerg Sonnenberger
and it is highly undesirable from a performance spective.

Why? If the locale is a single __thread variable (or a set of 6 __thread
variables, if you prefer), it is just as fast as any __thread variable access.

Yes and __thread is slow on many platforms compared to a normal function
call or access to a global variable. Even on platforms where TLS access
is moderately fast, it can easily make common interfaces quite a bit
slower.

Post by Bruno Haible

Post by Joerg Sonnenberger
It is also by itself useless as building block as
pretty much any higher level runtime like C++ STL wants to have explicit
control over locales.

that a decent object-oriented API for locale_t objects was the way to go,
rather than these odd-to-use newlocale/duplocale APIs with non-extensible
locale_t objects. But now the standard exists, and you do people a favour
if you implement it.

I disagree. A lot. As I said above, real world code use supports my
position.

Post by Bruno Haible

Post by Joerg Sonnenberger

Post by Bruno Haible
- duplocale(LC_GLOBAL_LOCALE) is broken.

I think this is a completely unrelated bug where thousand groupers were
handled incorrectly. Test certainly works on current.

- The number the test exercises comes out as "3.5" where "3,5" is expected.
- The month name comes out incorrectly as well.
- In the debugger, when I looked at the memory representation of
duplocale(LC_GLOBAL_LOCALE), I saw a pointer to some C locale data
followed by many NULL pointers.
But I'm pleased to hear that it will be fixed in NetBSD 8.

Hm. Could also be the protected issue. I had forgotten about that one.

Joerg