Subject: Re: [boost] [unicode] Interest Check / Proof of Concept
From: James Porter (porterj_at_[hidden])
Date: 2008-11-19 17:27:07
Zach Laine wrote:
> I would love to see a Unicode support library added to Boost.
> However, I question the usefulness of another string class, or in this
> case another hierarchy of string classes. Interoperability with
> std::string (and QString, and CString, and a thousand other
> API-specific string classes) is always thorny. I'd much rather see an
> iterators- and algorithms-based approach, along the lines of your
> ct_string::iterator.
It might get equally thorny just trying to get the algorithms to
recognize all the strange varieties of strings out there without writing
iterator facades for the lot of them! It's probably possible, but I'm
not I'd want it to be the primary interface for encoding. Most custom
string types (both QString and CString, for instance) are designed to
work with only one encoding (UTF-16 seems popular), so if you had some
reason that you needed to store your strings in UTF-8, or - god forbid -
Shift-JIS, you'd be out of luck.
This is especially important when you're reading in arbitrary data whose
encoding you don't know at compile-time. If someone sends me a message
encoded in Shift-JIS and I want to forward it on, I don't want to have
to decode it into UTF-8 and then re-encode it into Shift-JIS before I
send it; I just want to store it in Shift-JIS.
> Instead of doing this:
>
>> baz.encode(bar,rt::utf8);
>
> I'd rather be able to do something like this:
>
> typedef std::basic_string<some_32bit_char_type> unicode_string;
>
> unicode_string u_string = /*...*/;
> std::string std_string = /*...*/;
>
> typedef boost::recoding_iterator<boost::ucs4, boost::utf8> ucs4_to_utf8_iter;
> std::copy(ucs4_to_utf8_iter(u_string.begin()),
> ucs4_to_utf8_iter(u_string.end()), std::back_inserter(std_string));
std::strings aren't really appropriate for this purpose, at least not
without a lot of changes to their interface, since they're designed for
compile-time-tagged, fixed-width-encoding strings. In your examples, you
have to remember what the source encoding is. This is easy enough if you
know that "all my strings are in UTF-8", but if you start working with
runtime-tagged strings (see my Shift-JIS example above), you'd need to
keep track of every encoding in use.
- Jim