Re: Interpretation of extensions different from Unix/Linux?

[Adam Beneschan <adam@irvine.com>]

On Aug 18, 1:48 pm, "Randy Brukardt" <ra...@rrsoftware.com> wrote:
> "Adam Beneschan" <a...@irvine.com> wrote in message
>
> news:6f80c882-fa03-4ca9-a53e-fae34cea160d@b15g2000yqd.googlegroups.com...
> On Aug 17, 3:28 pm, "Randy Brukardt" <ra...@rrsoftware.com> wrote:
>
> >> The problem here is that String really is not the right type, but since
> >> you
> >> can't have string literals for private types in Ada, you can't make it a
> >> private type. (And if you could have string literals, it still couldn't
> >> be
> >> used with the existing I/O packages, it would be way too incompatible.)
>
> >That wouldn't even be an issue if UTF-8 were strictly a "storage
> >format" as you called it above.  If that were the case, you wouldn't
> >need string literals for it.  I think the problem is that UTF-8 is
> >something of a hybrid.  If all characters in the string are in the
> >32..126 range, the "sequence of octets" stored in the UTF-8 string is
> >identical to the graphic characters stored in a String.  (UTF-8 was
> >designed purposefully so that would happen.)  In cases like that, it
> >makes sense to use a string literal.
>
> Well, the problem here is that it *always* makes sense to use a string
> literal. That's how you specify what you want in storage in Ada.

We might be talking about two different things.  If I had a type whose
purpose was to represent the storage of a string in Huffman code, it
wouldn't make sense to specify its value with a string literal,
because the octets in that storage wouldn't have any direct
relationship with any meaningful graphic characters.  So specifying
the value of such a type with a string literal wouldn't make sense,
except via a conversion routine of some sort.  UTF-8 is different
because in many cases the octets being stored *do* have a direct
relationship with the graphic characters.  That's what I was trying to
get at.  But it's possible that I've lost the plot.


> >Also, I'm afraid that using String can backfire.  If I understand it
> >correctly, the decision was that the Name parameter of Text_IO.Open
> >should be interpreted as a UTF-8 octet sequence even though it's a
> >String.  But the intent is to allow string literals.  At some point,
> >though, some poor innocent programmer in Germany or Spain is going to
> >try to use a string literal (or a Latin-1 string variable) with an
> >umlaut or an accented vowel in it and get totally screwed up since
> >those characters don't represent themselves in UTF-8 encoding, and
> >they'll end up puzzling over how their program created a file with a
> >Chinese character in the middle of the name.  (Yeah, I know, that's
> >very unlikely; most likely the UTF-8 encoding will simply be invalid.)
>
> I've been presuming that UTF-8 encoding started with a BOM or something like
> that, else you couldn't tell it from regular Latin-1 encoding. It would be
> hard to insert a BOM into a string literal by accident!
>
> But I do agree that this issue needs some discussion.

My understanding, which could be wrong, was that the UTF-8 standard
specified how to encode a single "character" (from the entire set of
almost 2**32 possibilities) as a sequence of 8-bit octets.  I wasn't
aware that the standard also said anything about special octet
sequences that precede the encoding of a character *sequence*.  Maybe
it does; I'll have to go back and look.

I don't have a lot of experience working with UTF-8 encoded files (on
Windows, say) to know how they're stored or how applications
distinguish the encodings.  In mail messages, though, I sometimes see
contents and attachments that are encoded in UTF-8, and some that are
encoded in Latin-1.  (My mail reader is pretty primitive so I can get
a good look at exactly how the message is encoded.)  In the UTF-8
messages, I don't see any special characters or anything at the
beginning of the contents; in that case, I think it's information in
the mail message headers that tells the application whether to
interpret the string as encoded in UTF-8 or Latin-1.

... OK, after a bit of research, it looks as though the use of BOM to
distinguish UTF-8 from other encodings isn't part of an official
"standard", but it's widely used by Windows applications.  But I also
note that in the Wikipedia entry for "byte-order mark", they recommend
*against* this practice on Unix-type systems (for example, if a shell
script were stored in this manner, the first character would no longer
be recognized as '!' by utilities that checked for shell scripts).  I
don't know whether this advice is obsolete or not.  Anyway, perhaps
this means that the UTF_Encoding package shouldn't assume the use of
BOM to distinguish UTF-8-encoded strings, but perhaps the routines
there should provide an optional parameter to specify what
"protocol" (if any) is used to distinguish between encodings (a
protocol of None would mean that we assume the calling program knows
by some other means what kind of encoding is being used, and the
UTF_Encoding package just converts single characters according to the
standard without worrying about any kind of header sequence).

                                        -- Adam