From: Phil Endecott (spam_from_boost_dev_at_[hidden])
Date: 2008-03-02 15:53:12
Sebastian Redl wrote:
> It gets worse. I've tried to implement a very simple "kinda-shift"
> encoding, UTF-16VE. That is, a UTF-16 form that expects a BOM to
> determine endianness. This encoding uses the shift state to remember
> what endian it is in. (No dynamic switching.)
The common case is that you have a BOM at the start, and if there are
any other BOMs they'll be the same. But what I don't know is what the
Unicode specs allow in this respect, and whether it's sensible to
provide explicit support for that limited case as well as the more
general case. (Do the IANA character sets names that I'm using as the
basis for the charset_t enum have any way of distinguishing these
cases, for example? I think the answer is no.)
> Trying to implement this, I've found that it is apparently logically
> impossible to provide bidirectional iterators for shift encodings, like
> ISO 2022-based encodings. These encodings rely on state that can only be
> known by sequentially scanning the string from front to back.
Yes. You may be able to argue in some cases that you can predict the
state during backward traversal IF there are no redundant shifts and if
there are only two states. Again, I don't know whether that's useful
in practice (and I suspect not).
> Any
> attempt to iterate backwards would first have to mark the switch
> positions and what modes they switch from.
>
> This can be worked around for my UTF-16VE, but not for true shift
> encodings. Thus, the charset traits probably need a flag that designates
> the set as a shift encoding and makes the iterator adapter be forward-only.
We could detect the case when skip_forward_char is not implemented.
There are various factors that influence the adapted iterator traversal
tag. For example, I wanted to say that the character iterator has the
same traversal tag as the unit iterator, except that it's not random
access; i.e. min(unit_iter_t,bidirectional). Is there any existing
code anywhere for doing operations like this on iterator traversal
category tags?
> On a side note, Shift-JIS, EUC-JP and ISO-2022-JP are all absurdly
> complex. UTF-8 is so much easier!
Agreed :-(
Phil.