Re: strange behaviour of utf-8 files

[Georg Bauhaus <rm.dash-bauhaus@futureapps.de>]

On 18.11.13 10:01, Dmitry A. Kazakov wrote:

>> UTF-8 can actually be so checked (and is checked by typical implementations)
>
> 1. The share of illegal UTF-8 sequences is negligible. The one among Ada
> programs is even less than that.

The share of illegal UTF-8 sequences in source text stays low as
long as policies prevent use of anything but ASCII. But! OTOH,
the difficulty of adapting to use of limited character sets stays
high, unnerving, and costly.

(I know because the source text used here and elsewhere is full of
ASCII-sequences representing ubiquitous Unicode characters. These are
characters that users expect to see. If 1234 codes some common
international character, then having to write

   "abc \x{1234}"
all over the place is a PITA. The need to write

   "abc ["1234"]"
GNAT style does not change that.)

> 2. Latin1 sequences are all legal.

Legality of (only) almost all octets interpreted as Latin-1 characters
does not make the interpretation of string literals correct.
Correctness involves the problem specification, not just Ada.

Which is what matters most: The *user*, the raison d'être of programming,
is not really satisfied when legal programs will actually malfunction
because of legal ambiguity of legal octets. Would anyone be at ease with
similar ambiguity of number literals?

> Now, carefully observe that the program in question was dealt with as if it
> were encoded in Latin1. So much for your theory.

My theory involves programmers, foreign software, and users,
in addition to the mere formalism that you mention.

> ---------------
> P.S. In order to make a point you should take a set of legal [and
> practical] Ada programs encoded in X and then reinterpreted in Y. Then you
> compare how many of them become:

0. useful

> 1. illegal
> 2. remain legal keeping the semantics
> 3. remain legal breaking the semantics

Note that legality can always be established together with 0,
and automatically is, once programmers can easily  specify
character encoding to be something unambiguous.

The stubbornness of 7bit engineering in OSs and in other circumstances
calls for a

   pragma Source_Text_Encoding (...);

With this warning sign in place, both old and new generations of programmers
can do their job.