From: Gavin Lambert (boost_at_[hidden])
Date: 2020-01-07 23:16:52
On 7/01/2020 14:58, Yakov Galka wrote:
>> So, while unfortunate, v3 made the correct choice. Paths have to be
>> kept in their original encoding between original source (command line,
>> file, or UI) and file API usage, otherwise you can get weird errors when
>> transcoding produces a different byte sequence that appears identical
>> when actually rendered, but doesn't match the filesystem. Transcoding
>> is only safe when you're going to do something with the string other
>> than using it in a file API.
>
> See above, malformed UTF-16 can be converted to WTF-8 (a UTF-8 superset)
> and back losslessly. The unprecedented introduction of a platform specific
> interface into the standard was, still is, and will always be, a horrible
> mistake.
Given that WTF-8 is not itself supported by the C++ standard library
(and the other formats are), that doesn't seem like a valid argument.
You'd have to campaign for that to be added first.
The main problem though is that once you start allowing transcoding of
any kind, it's a slippery slope to other conversions that can make lossy
changes (such as applying different canonicalisation formats, or
adding/removing layout codepoints such as RTL markers).
Also, if you read the WTF-8 spec, it notes that it is not legal to
directly concatenate two WTF-8 strings (you either have to convert it
back to UCS-16 first, or execute some special handling for the trailing
characters of the first string), which immediately renders it a poor
choice for a path storage format. And indeed a poor choice for any
purpose. (I suspect many people who are using it have conveniently
forgotten that part.)
Although on a related note, I think C++11/17 dropped the ball a bit on
the new encoding-specific character types. It's definitely an
improvement on the prior method, but it would have been better to do
something like:
struct ansi_encoding_t;
struct utf_encoding_t;
typedef encoded_char<ansi_encoding_t, 8> char_t;
typedef encoded_char<utf_encoding_t, 8> char8_t;
typedef encoded_char<utf_encoding_t, 16> char16_t;
Where "encoded_char<E,N>" has storage size equal to N bits (blittable,
and otherwise behaves like a standard integer type) but also carries
around an arbitrary encoding tag type E. This could be used to
distinguish "a string encoded in UTF-8" from "a string encoded in WTF-8"
or "a string encoded in EDBDIC". And supplemental libraries could
define additional encodings and conversion functions, and algorithms
could operate on generic strings of any encoding.