Subject: Re: [boost] Unicode and codecvt facets
From: Mathias Gaunard (mathias.gaunard_at_[hidden])
Date: 2010-07-05 13:01:40
On 05/07/10 17:27, Artyom wrote:
>>
>> As some may know, I am working on a Unicode library that I plan to submit to
>> Boost fairly soon.
>>
>
> Take a look on Boost.Locale proposal.
I know of it, yes.
But my library purposely *doesn't* use the standard C++ locale subsystem
because it's slow, broken, and inflexible.
Nevertheless I want to provide the ability to bridge my library with
that system.
>
>> The codecs in that library are based around iterators and ranges, but since
>> there was some demand for support
>> for codecvt facets I am working on adapting those into that form as well.
>>
>> Unfortunately, it seems it is only possible to subclass std::codecvt<char,
>> char, mbstate_t> and
>> std::codecvt<wchar_t, char, mbstate_t>.
>
> Yes, these are actually the only specialized classes.
I was hoping I could specialize some more myself.
Some implementations appear to support using arbitrary codecvt facets
just fine, but not GCC's and MSVC's.
> More then that
> std::codecvt<char, char, mbstate_t>
> should be - "noconvert" facet.
I'm talking about types derived from these.
There is no restriction for subclasses of std::codecvt<char, char,
mbstate_t> to be non-convert, only std::codecvt<char, char, mbstate_t> is.
> You can derive from these two classes in re-implement them (like I did in
> Boost.Locale).
That's indeed what I said I can do, but as I said I find that very limiting.
> Also I strongly recommend to take a look on locale and iostreams in standard
> library if you are working with Unicode for C++.
The thing is, I'm not sure it's worth delving into it too much. On top
of being a so-so design, the popular implementations seem to all do
things differently and have different limitations.
>
>>
>> What I wonder is if there is really a point to facets, then.
>> std::codecvt<wchar_t, char, mbstate_t> means that the in-memory charset would
>> be UTF-16 or UTF-32 (depending on the size of wchar_t) while the file would be
>> UTF-8.
>
> Not exactly narrow encoding may be any 8-bit encoding, even something like
> Latin1 or Shift-JIS (and UTF-8 as well).
My library doesn't aim at providing code conversion from/to every
character set ever invented, which is why I just put UTF-8 in there.
Regardless I intend to allow to define a codecvt facet from any pair of
objects modeling the Converter concept; so nothing would prevent someone
from writing one or chaining them to do whatever they want, granted it
converts between char and char or wchar_t and char, since it seems there
is no way around that one.
That way you can also do normalization, case conversion or whatnot with
a codecvt facet.
> C++0x provides char16_t and char32_t to fix this standard's bug.
GCC has those types in C++0x mode, but doesn't support codecvt facets
with them.
>
>>
>> Why do people even use utf8_codecvt_facet anyway? What's wrong with dealing
>> with UTF-8 rather than
>> maybe UTF-16 or UTF-32?
>>
>
> Ask Windows developers, they use wide strings because it is the only way to work
> correctly with their OS.
utf8_codecvt_facet is an utility provided by boost in the detail
namespace, that some libraries not particularly tied to Windows appear
to use.