Subject: Re: [boost] [unicode] Interest Check / Proof of Concept
From: Phil Endecott (spam_from_boost_dev_at_[hidden])
Date: 2008-11-20 15:42:41
Eric Niebler wrote:
> Zach Laine wrote:
>>> Over the past few months, I've been tinkering with a Unicode string library.
>>> It's still *far* from finished, but it's far enough along that the overall
>>> structure is visible. I've seen a bunch of Unicode proposals for Boost come
>>> and go, so hopefully this one will address the most common needs people
>>> have.
>>
>> I would love to see a Unicode support library added to Boost.
>> However, I question the usefulness of another string class, or in this
>> case another hierarchy of string classes. Interoperability with
>> std::string (and QString, and CString, and a thousand other
>> API-specific string classes) is always thorny. I'd much rather see an
>> iterators- and algorithms-based approach
> <snip>
>
> Agree. Thanks Zach. I'm discouraged that every time the issue of a
> Unicode library comes up, the discussion immediately descends into a
> debate about how to design yet another string class. Such a high level
> wrapper *might* be useful (strong emphasis on "might"), but the core
> must be the Unicode algorithms, and the design for a Unicode library
> must start there.
I mostly agree. If people want UTF-8 and UTF-16 iterator-adaptors that
will efficiently convert byte-sequence iterators into unicode character
iterators, then I probably already have exactly that. Should I package
it up for review?
There are, however, a few points to consider. Most importantly, if you
operate on a UTF-8 string only using an iterator-adaptor then you'll
miss out on most of the clever features of the encoding. Specifically:
- If you need to search for an ASCII character in a UTF-8 string then
you can do so just by scanning the bytes.
- Similarly, searching for substrings (including substrings with
non-ASCII characters) can be done just by scanning for a bytewise match.
- Sorting can be done using strcmp()-like comparisons on the byte sequences.
An implementation that doesn't somehow exploit these optimisations will
perform sub-optimally, and I don't think that would be acceptable.
I don't really have a complete solution to offer. What I do have is
the beginnings of a character-set traits class with booleans indicating
things like "is an ASCII superset", "is variable-length" etc. The idea
is that algorithms could be specialised based on these traits. I'm not
sure how it all joins together yet though.
Cheers, Phil.