Re: Why no Ada.Wide_Directories?

[Adam Beneschan <adam@irvine.com>]

On Oct 17, 4:47 pm, ytomino <aghi...@gmail.com> wrote:
> On Oct 18, 6:33 am, "Randy Brukardt" <ra...@rrsoftware.com> wrote:
>
>
>
>
>
>
>
> > Say what?
>
> > Ada.Strings.Encoding (new in Ada 2012) uses a subtype of String to store
> > UTF-8 encoded strings. As such, I'd find it pretty surprising if doing so
> > was "a violation of the standard".
>
> > The intent has always been that Open, Ada.Directories, etc. take UTF-8
> > strings as an option. Presumably the implementation would use a Form to
> > specify that the file names in UTF-8 form rather than Latin-1. (I wasn't
> > able to find a reference for this in a quick search, but I know it has been
> > talked about on several occasions.)
>
> > One of the primary reasons that Ada.Strings.Encoding uses a subtype of
> > String rather than a separate type is so that it can be passed to Open and
> > the like.
>
> > It's probably true that we should standardize on the Form needed to use
> > UTF-8 strings in these contexts, or at least come up with Implementation
> > Advice on that point.
>
> >                                        Randy.
>
> Good news. Thanks for letting know.
> My worry is decreased a little.
>
> However, even if that is right, Form parameters are missing for many
> subprograms.
> Probably, All subprograms in Ada.Directories,
> Ada.Directories.Hierarchical_File_Names, Ada.Command_Line,
> Ada.Environment_Variables and other subprograms having Name parameter
> or returning a file name should have Form parameter.
> (For example, I do Open (X, Form => "UTF-8"). Which does Name (X)
> returns UTF-8 or Latin-1?)
>
> Moreover, in the future, we will always use I/O subprograms as UTF-8
> mode if what you say is realized.
> But other libraries in the standard are explicitly defined as Latin-1.
> It's certain that Ada.Character.Handling.To_Upper breaks UTF-8.

I have a feeling you're fundamentally confused about what UTF-8 is, as
compared to "Latin-1".  Latin-1 is a character mapping.  It defines,
for all integers in the range 0..255, what character that integer
represents (e.g. 77 represents 'M', etc.).  Unicode is a character
mapping that defines characters for a much larger integer range.  For
integers in the range 0..255, the character represented in Unicode is
the same as that in Latin-1; higher integers represent characters in
other alphabets, other symbols, etc.  Those mappings just tell you
what symbols go with what numbers, and they don't say anything about
how the numbers are supposed to be stored.

UTF-8 is an encoding (representation).  It defines, for each non-
negative integer up to a certain point, what bits are used to
represent that integer.  The number of bits is not fixed.  So even if
you're working with characters all in the 0..255 range, some of those
characters will be represented in 8 bits (one byte) and some will take
16 bits (two bytes).

Because of this, it is not feasible to work with strings or characters
in UTF-8 encoding.  Suppose you declare a string

   S : String (1 .. 100);

but you want it to be a UTF-8 string.  How would that work?  If you
want to look at S(50), the computer would have to start at the
beginning of the string and figure out whether each character is
represented as 1 or 2 bytes.  Nobody wants that.

The only sane way to work with strings in memory is to use a format
where every character is the same size (String if all your characters
are in the 0..255 range, Wide_String for 0..65535, Wide_Wide_String
for 0..2**32-1).  Then, if you have a string of bytes in UTF-8 format,
you convert it to a regular (Wide_)(Wide_)String with routines in
Ada.Strings.UTF_Encoding; and it also has routines for converting
regular strings to UTF-8 format.  But you don't want to *keep* strings
in memory and work with them in UTF-8 format.  That's why it doesn't
make sense to have string routines (like
Ada.Strings.Equal_Case_Insensitive or Ada.Character_Handling.To_Upper)
that work with UTF-8.

Hope this solves your problem.

                             -- Adam