Filenames in Ada

Filenames in Ada

[Martin Krischik]

Hello

I was just starting to write a little demo program for Ada.Directories. Aim
was the generation of m3u files for music collection. Allways nice to have
a demo which actually does something.

However, filenames in music collections contain all sort of funny characters 
(especially the my wife's music - allmost entirely in Cyrillic) - no big
deal for modern file systems. But a big deal for Ada as it seams. Not only
are all filenames in Ada.Directories of type String - actually all
filenames anywhere are just String.

How does one deal with modern (utf-8) filenames in Ada?

Is there a chance for an add-on package (to late for Ada 2005) or are we
snookered until Ada 2015.

I also looked at C - they have appropriate functions in <wchar.h> using
Wide_Wide_Character for filenames. 

Martin

Links:
http://en.wikibooks.org/wiki/Ada_Programming/Libraries/Ada.Directories
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: Filenames in Ada

[Martin Dowie]

Martin Krischik wrote:
> Hello
> 
> I was just starting to write a little demo program for Ada.Directories. Aim
> was the generation of m3u files for music collection. Allways nice to have
> a demo which actually does something.
> 
> However, filenames in music collections contain all sort of funny characters 
> (especially the my wife's music - allmost entirely in Cyrillic) - no big
> deal for modern file systems. But a big deal for Ada as it seams. Not only
> are all filenames in Ada.Directories of type String - actually all
> filenames anywhere are just String.
> 
> How does one deal with modern (utf-8) filenames in Ada?
> 
> Is there a chance for an add-on package (to late for Ada 2005) or are we
> snookered until Ada 2015.
> 
> I also looked at C - they have appropriate functions in <wchar.h> using
> Wide_Wide_Character for filenames. 

Aren't there some conversion routines to turn 16/32-bit UTF to/from 
String in AWS or XML/Ada?

Cheers

-- Martin

Re: Filenames in Ada

[Martin Krischik]

Martin Dowie wrote:

> Martin Krischik wrote:
>> Hello
>> 
>> I was just starting to write a little demo program for Ada.Directories.
>> Aim was the generation of m3u files for music collection. Allways nice to
>> have a demo which actually does something.
>> 
>> However, filenames in music collections contain all sort of funny
>> characters (especially the my wife's music - allmost entirely in
>> Cyrillic) - no big deal for modern file systems. But a big deal for Ada
>> as it seams. Not only are all filenames in Ada.Directories of type String
>> - actually all filenames anywhere are just String.
>> 
>> How does one deal with modern (utf-8) filenames in Ada?
>> 
>> Is there a chance for an add-on package (to late for Ada 2005) or are we
>> snookered until Ada 2015.
>> 
>> I also looked at C - they have appropriate functions in <wchar.h> using
>> Wide_Wide_Character for filenames.
> 
> Aren't there some conversion routines to turn 16/32-bit UTF to/from
> String in AWS or XML/Ada?

You missed the point. I am speaking of filenames not file data. 

You can't pass UTF-8 into CreateFileA - you have to use CreateFileW and that
need UTF-16. Actually - really UTF-16 or is it a simple Wide_Character
string. Windows Gurus?

Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: Filenames in Ada

[Dmitry A. Kazakov]

On Wed, 23 Nov 2005 18:36:47 +0100, Martin Krischik wrote:

> However, filenames in music collections contain all sort of funny characters 
> (especially the my wife's music - allmost entirely in Cyrillic) - no big
> deal for modern file systems. But a big deal for Ada as it seams. Not only
> are all filenames in Ada.Directories of type String - actually all
> filenames anywhere are just String.
> 
> How does one deal with modern (utf-8) filenames in Ada?

An interesting question! Indeed the file name parameter is of String type.

Well, some quick check. The following works fine under Fedora / GNAT:

with Ada.Text_IO;                 use Ada.Text_IO;
with Strings_Edit.UTF8.Handling;  use Strings_Edit.UTF8.Handling;

procedure UTF8_Test is
   Name : Wide_String :=
          (  Wide_Character'Val (1092)
          &  Wide_Character'Val (1072)
          &  Wide_Character'Val (1081)
          &  Wide_Character'Val (1083)
          );
   File : File_Type;
begin
   Create (File, Out_File, To_UTF8 (Name));
   Close (File);
end UTF8_Test;

If you have Cyrillic code page installed you will see the file name
correctly spelt in Russian. No wonder, Fedora Linux is natively UTF-8!

Though it cannot work under Windows, because GNAT run-time translates
Create into some CreateFileA Windows API. Moreover Windows is UTF-16. So
there is no chance for Ada.Text_IO.Create. I presume one could directly use
CreateFileW from Windows API with a Wide_String file name, but that's
another story.

> Is there a chance for an add-on package (to late for Ada 2005) or are we
> snookered until Ada 2015.

I don't see any great problem here. I would add UTF-8 variants for all
calls where a file name is mentioned. We could even switch from Latin-1 to
UTF-8. This seems to be independent on Text_IO vs. Wide_Text_IO issue.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Filenames in Ada

[Martin Krischik]

Dmitry A. Kazakov wrote:

> On Wed, 23 Nov 2005 18:36:47 +0100, Martin Krischik wrote:
> 
>> However, filenames in music collections contain all sort of funny
>> characters (especially the my wife's music - allmost entirely in
>> Cyrillic) - no big deal for modern file systems. But a big deal for Ada
>> as it seams. Not only are all filenames in Ada.Directories of type String
>> - actually all filenames anywhere are just String.
 
>> How does one deal with modern (utf-8) filenames in Ada?
> 
> An interesting question! Indeed the file name parameter is of String type.
> 
> Well, some quick check. The following works fine under Fedora / GNAT:
> 
> with Ada.Text_IO;                 use Ada.Text_IO;
> with Strings_Edit.UTF8.Handling;  use Strings_Edit.UTF8.Handling;
> 
> procedure UTF8_Test is
>    Name : Wide_String :=
>           (  Wide_Character'Val (1092)
>           &  Wide_Character'Val (1072)
>           &  Wide_Character'Val (1081)
>           &  Wide_Character'Val (1083)
>           );
>    File : File_Type;
> begin
>    Create (File, Out_File, To_UTF8 (Name));
>    Close (File);
> end UTF8_Test;

Nice and good to know. Only I never did open any files since I was still
reading the directories with Ada.Directories to know what to write into the
files.

Find the code at:
http://cvs.sourceforge.net/viewcvs.py/wikibook-ada/demos/Source/make_m3u.adb?only_with_tag=HEAD&view=markup

at let it loose at a directory with Cyrillic filenames.

> If you have Cyrillic code page installed you will see the file name
> correctly spelt in Russian. No wonder, Fedora Linux is natively UTF-8!

So is SuSE since 9.0.

> Though it cannot work under Windows, because GNAT run-time translates
> Create into some CreateFileA Windows API. Moreover Windows is UTF-16. So
> there is no chance for Ada.Text_IO.Create. I presume one could directly
> use CreateFileW from Windows API with a Wide_String file name, but that's
> another story.

I fear that is the only option available. But how to turn the Windows
filehandle into a Ada one...

>> Is there a chance for an add-on package (to late for Ada 2005) or are we
>> snookered until Ada 2015.
 
> I don't see any great problem here. I would add UTF-8 variants for all
> calls where a file name is mentioned. We could even switch from Latin-1 to
> UTF-8. This seems to be independent on Text_IO vs. Wide_Text_IO issue.

I am not sure. There are quite a few OS dependent functions involved here.

Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: Filenames in Ada

[Dmitry A. Kazakov]

On Thu, 24 Nov 2005 20:13:01 +0100, Martin Krischik wrote:

> Dmitry A. Kazakov wrote:
> 
>> On Wed, 23 Nov 2005 18:36:47 +0100, Martin Krischik wrote:
>> 
> Find the code at:
> http://cvs.sourceforge.net/viewcvs.py/wikibook-ada/demos/Source/make_m3u.adb?only_with_tag=HEAD&view=markup
> 
> at let it loose at a directory with Cyrillic filenames.

It works perfectly well. Ada.Directories returns file names as-is. One
don't need to change anything:

   while More_Entries (Search_Artist) loop
      Get_Next_Entry (Search_Artist, Artist_Entry);
      declare
         Artist : String renames Simple_Name (Artist_Entry);
             -- This is UTF-8 encoded under Linux
      begin
         if (  Artist /= "." -- ASCII-7 is UTF-8!
            and then
               Artist /= ".."
            and then
               Artist /= "Folder Settings"
            )
         then
            Put_Line (Artist); -- Outputs UTF-8
         end if;
      end;
   end loop;

>> Though it cannot work under Windows, because GNAT run-time translates
>> Create into some CreateFileA Windows API. Moreover Windows is UTF-16. So
>> there is no chance for Ada.Text_IO.Create. I presume one could directly
>> use CreateFileW from Windows API with a Wide_String file name, but that's
>> another story.
> 
> I fear that is the only option available. But how to turn the Windows
> filehandle into a Ada one...

I'm afraid there is no other way than to ask ACT 1) to use xxxW stuff for
all file operations and 2) to translate String file names from UTF-8 to
UTF-16 when it goes down to Windows and from UTF-16 to UTF-8 when it goes
back.

>> I don't see any great problem here. I would add UTF-8 variants for all
>> calls where a file name is mentioned. We could even switch from Latin-1 to
>> UTF-8. This seems to be independent on Text_IO vs. Wide_Text_IO issue.
> 
> I am not sure. There are quite a few OS dependent functions involved here.

It cannot be portable.

Well, does VMS still use Radix-50 for file names? (:-)) That's wise!

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Filenames in Ada

[Larry Kilgallen]

In article <hcj87m4mv37e$.1uj3pzelawsm9.dlg@40tude.net>, "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> Well, does VMS still use Radix-50 for file names? (:-))

In terms of APIs for the system services, it never did.

On some of the older VAX-only disks it might be that the internal
storage is RAD50, but the interface exported via RMS uses ASCII,
with only certain characters being legal.

Unless your intention is to write an ODS-1 ACP in Ada, the internal
storage format is immaterial to the Ada programmer.

Re: Filenames in Ada

[krischik]

> Well, does VMS still use Radix-50 for file names? (:-)) That's wise!

Depends on the file system used. Amost all GNU/VMS tools demand an ODS5
file system which is supositly POSIX compliant.

Well I would not know - we use ODS3 here :-( .

Martin

[OT] VMS ODS-5 filesystems, was: Re: Filenames in Ada

[Simon Clubley]

In article <1132902735.252192.317840@g14g2000cwa.googlegroups.com>, "krischik" <krischik@users.sourceforge.net> writes:
>> Well, does VMS still use Radix-50 for file names? (:-)) That's wise!
> 
> Depends on the file system used. Amost all GNU/VMS tools demand an ODS5
> file system which is supositly POSIX compliant.
> 
> Well I would not know - we use ODS3 here :-( .
> 

I think you mean ODS-2. :-)  (ODS-3 is ISO 9660)

If your site policies allow it, you could try one of the freeware container
drivers such as LDDRIVER and create a ODS-5 volume within the container.

Simon.

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP       
Scientific Theory: A testable hypothesis that is supported by a body of evidence

Re: [OT] VMS ODS-5 filesystems, was: Re: Filenames in Ada

[Martin Krischik]

Simon Clubley wrote:

> In article <1132902735.252192.317840@g14g2000cwa.googlegroups.com>,
> "krischik" <krischik@users.sourceforge.net> writes:
>>> Well, does VMS still use Radix-50 for file names? (:-)) That's wise!
>> 
>> Depends on the file system used. Amost all GNU/VMS tools demand an ODS5
>> file system which is supositly POSIX compliant.
>> 
>> Well I would not know - we use ODS3 here :-( .
>> 
> 
> I think you mean ODS-2. :-)  (ODS-3 is ISO 9660)
> 
> If your site policies allow it, you could try one of the freeware
> container drivers such as LDDRIVER and create a ODS-5 volume within the
> container.

Tried that but could not quite get it to work.

I don't mind VMS - just some if the tools drive me insane - like that joke
of a command line shell which can't backspace over a line wrap. So I really
which for I had a GNU/VMS tool chain available.

At least I got a decent editor (vim) now.

Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: [OT] VMS ODS-5 filesystems, was: Re: Filenames in Ada

[Georg Bauhaus]

On Fri, 2005-11-25 at 19:01 +0100, Martin Krischik wrote:


> I don't mind VMS - just some if the tools drive me insane - like that joke
> of a command line shell which can't backspace over a line wrap. So I really
> which for I had a GNU/VMS tool chain available.
> 
> At least I got a decent editor (vim) now.

Can't you use editors to construct more complicated
commands that span a number of lines and need editting?

Re: [OT] VMS ODS-5 filesystems, was: Re: Filenames in Ada

[Martin Krischik]

Georg Bauhaus wrote:

> On Fri, 2005-11-25 at 19:01 +0100, Martin Krischik wrote:
> 
> 
>> I don't mind VMS - just some if the tools drive me insane - like that
>> joke of a command line shell which can't backspace over a line wrap. So I
>> really which for I had a GNU/VMS tool chain available.
>> 
>> At least I got a decent editor (vim) now.
 
> Can't you use editors to construct more complicated
> commands that span a number of lines and need editting?

Actually vim has a "!" command which just passes the rest of the line to the
DCL prompt from which is was started ;-) - very helpfull.

Apart from that I do precicely what you have suggested - only work colleques 
found the resulting files analzed them and where wondering about the
commands used...

Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: [OT] VMS DCL prompt, was: VMS ODS-5 filesystems, was: Re: Filenames in Ada

[Martin Krischik]

Dennis Lee Bieber wrote:

> On Fri, 25 Nov 2005 19:01:06 +0100, Martin Krischik
> <krischik@users.sourceforge.net> declaimed the following in
> comp.lang.ada:
> 
> 
>> I don't mind VMS - just some if the tools drive me insane - like that
>> joke of a command line shell which can't backspace over a line wrap. So I
>> really which for I had a GNU/VMS tool chain available.
>>
> I've not worked VMS for some four years or so, but I could swear
> that I could work with command lines that wrapped over multiple lines...
> Are you referring to a backspace-delete operation, or a cursor-left
> operation? (I vaguely recall that you could delete back over the wrap,
> but you couldn't cursor over it if all you wanted to do was insert a
> character).

Oh, you can enter a commandline over more then one line - as long as you
don't try to edit that line to much. Bun then I have correct myself: it's
more "insert" then "backspace" which makes trouble.

On the other hand: backspace might delete into the wrong direction - or
delete - depends on how you setup your keyboard. Anyway not at all what the
GNU readline library could do for you.

But then - there are areas where VMS is a lot better then Unix. Alone the
ability to have more then one Administrator. So it's not all bad. I think I
would really enjoy working with "GNU/VMS" (that is VMS kernel with an all
GNU toolchain).

Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

[OT] Administrator accounts (was: VMS DCL prompt)

[Björn Persson]

Martin Krischik wrote:
> But then - there are areas where VMS is a lot better then Unix. Alone the
> ability to have more then one Administrator.

Different usernames can have the same user ID in Unix, so you can have 
several usernames with UID 0 but each with its own password. But you 
might not think that counts as multiple administrator accounts.

-- 
Bj�rn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu

Re: [OT] Administrator accounts (was: VMS DCL prompt)

[Larry Kilgallen]

In article <qb6jf.151585$dP1.509419@newsc.telia.net>, =?ISO-8859-1?Q?Bj=F6rn_Persson?= <spam-away@nowhere.nil> writes:
> Martin Krischik wrote:
>> But then - there are areas where VMS is a lot better then Unix. Alone the
>> ability to have more then one Administrator.
> 
> Different usernames can have the same user ID in Unix, so you can have 
> several usernames with UID 0 but each with its own password. But you 
> might not think that counts as multiple administrator accounts.

A critical issue is whether auditing allows one to distinguish the
individuals making use of such privileged access.

Re: Filenames in Ada

[Randy Brukardt]

"Martin Krischik" <krischik@users.sourceforge.net> wrote in message
news:1653090.31FM62oI6I@linux1.krischik.com...
...
> How does one deal with modern (utf-8) filenames in Ada?

We talked about this briefly at a recent meeting. We decided it was too late
to design a proper solution, and virtually all operating systems have a way
to deal with this anyway. (That is that they take UTF-8 names). There's no
conceptual problem with putting UTF-8 into a value of type String, and as
someone else showed, that's typcially supported by Ada implementations.

A bigger problem is that Ada (any flavor) has no built-in support for UTF-8.
I think this is a mistake, but I didn't find much support for adding any
packages. It's insane to use Wide_Wide_String to store any significant
amount of text, as it would waste nearly 3/4s of the space in typical use,
so you'd have to use something else (like UTF-8) for storage anyway.

Since filenames are implementation-defined anyway, there isn't a whole lot
of value to standardizing how they're written. So, it's just up to your
implementer to do something appropriate.

                    Randy Brukardt.

Silly question about strings (was: Filenames in Ada)

[Steve]

This may be a silly question, but why aren't the character dependent 
packages generic?

It seems odd to have separate packages:

  Ada.Strings.Fixed;
  Ada.Strings.Wide_Fixed;
  Ada.Strings.Bounded;
  Ada.Strings.Wide_Bounded;
  Ada.Strings.Unbounded;
  Ada.Strings.Wide_Unbounded;
  Ada.Text_IO;
  Ada.Wide_Text_IO;

Instead of generic packages that use the character type as a parameter, with 
predefined instatiations for character and wide_character similar to what is 
done with numerics.  Perhaps something along the lines of:

  package Ada.Strings.Fixed is new Ada.Strings.String_Base( Character );
  package Ada.Strings.Wide_Fixed is new Ada.Strings.String_Base( 
Wide_Character );

and so on...

It seems like this would eliminate issues with different character sets, and 
avoid the need for a language revision when a 64 bit (or 128 bit) version of 
Unicode arrives.

Steve
(The Duck) 

Re: Silly question about strings (was: Filenames in Ada)

[Larry Kilgallen]

In article <oNmdneZx9ePFrBjenZ2dnUVZ_t2dnZ2d@comcast.com>, "Steve" <nospam_steved94@comcast.net> writes:
> This may be a silly question, but why aren't the character dependent
> packages generic?
>
> It seems odd to have separate packages:
>
>   Ada.Strings.Fixed;
>   Ada.Strings.Wide_Fixed;
>   Ada.Strings.Bounded;
>   Ada.Strings.Wide_Bounded;
>   Ada.Strings.Unbounded;
>   Ada.Strings.Wide_Unbounded;
>   Ada.Text_IO;
>   Ada.Wide_Text_IO;
>
> Instead of generic packages that use the character type as a parameter, with
> predefined instatiations for character and wide_character similar to what is
> done with numerics.  Perhaps something along the lines of:
>
>   package Ada.Strings.Fixed is new Ada.Strings.String_Base( Character );
>   package Ada.Strings.Wide_Fixed is new Ada.Strings.String_Base(
> Wide_Character );
>
> and so on...
>
> It seems like this would eliminate issues with different character sets, and
> avoid the need for a language revision when a 64 bit (or 128 bit) version of
> Unicode arrives.

I would expect those string packages (and especially the string IO
packages) would use operating system facilities in some implementations.
The underlying operating system calls might not be symmetric at all.
The bodies might not even be written in Ada.

Re: Silly question about strings (was: Filenames in Ada)

[Martin Dowie]

Larry Kilgallen wrote:
> I would expect those string packages (and especially the string IO
> packages) would use operating system facilities in some
> implementations.
> The underlying operating system calls might not be symmetric at all.
> The bodies might not even be written in Ada.

They could always be 'defined' as proposed by Steve but the implementation
might use 'magic' to make it happen.

Cheers

-- Martin

Re: Silly question about strings

[Brian May]

>>>>> "Martin" == Martin Dowie <martin.dowie@baesystems.com> writes:


    Martin> They could always be 'defined' as proposed by Steve but
    Martin> the implementation might use 'magic' to make it happen.

... and if this magic meant you could do:

package Ada.Strings.UTF8 is new Ada.Strings.String_Base( UTF8 );

I think everybody would be happy ;-).

Or is that too much magic?

Unfortunately, Steve's proposal didn't take into account Fixed length
vs Unbounded types. Not sure if that could be done with an additional
parameter to String_Base, or if duplicating String_Base would be the
ideal solution.
-- 
Brian May <bam@snoopy.apana.org.au>

Re: Silly question about strings (was: Filenames in Ada)

[Larry Kilgallen]

In article <438582ab_1@glkas0286.greenlnk.net>, "Martin Dowie" <martin.dowie@baesystems.com> writes:
> Larry Kilgallen wrote:
>> I would expect those string packages (and especially the string IO
>> packages) would use operating system facilities in some
>> implementations.
>> The underlying operating system calls might not be symmetric at all.
>> The bodies might not even be written in Ada.
> 
> They could always be 'defined' as proposed by Steve but the implementation
> might use 'magic' to make it happen.

I think magic is typically the least reliable part of an implementation.

Re: Silly question about strings (was: Filenames in Ada)

[Randy Brukardt]

"Steve" <nospam_steved94@comcast.net> wrote in message
news:oNmdneZx9ePFrBjenZ2dnUVZ_t2dnZ2d@comcast.com...
> This may be a silly question, but why aren't the character dependent
> packages generic?
...
> Instead of generic packages that use the character type as a parameter,
with
> predefined instatiations for character and wide_character similar to what
is
> done with numerics.  Perhaps something along the lines of:
>
>   package Ada.Strings.Fixed is new Ada.Strings.String_Base( Character );
>   package Ada.Strings.Wide_Fixed is new
.Strings.String_Base( 
> Wide_Character );
> 
> and so on...
> 
> It seems like this would eliminate issues with different character sets, and 
> avoid the need for a language revision when a 64 bit (or 128 bit) version of 
> Unicode arrives.

Something like this was proposed as an 11th hour suggestion (this year). But it was thought to be just too late to design it properly, thus nothing was done with that.

For Janus/Ada, such a generic would have terrible performance. Since we share all generic bodies, the code would have to deal with arrays of unknown size elements. Yuck. The predefined instantiations would help that some (as they wouldn't necessarily need to be implemented as generic instantiations), but user-defined string types would use real instantiations and probably would be unusable.

                            Randy Brukardt.

Re: Silly question about strings (was: Filenames in Ada)

[tmoran]

>>   package Ada.Strings.Fixed is new Ada.Strings.String_Base( Character );
>...
>Something like this was proposed as an 11th hour suggestion (this year).
Since the generic type parameter is merely any discrete type, presumably
you would support (with appropriate OS calls where needed):
  package Ada.Strings.Colors is new Ada.Strings.String_Base(Colors);
  package Ada.Strings.Unbounded_Integers
    is new Ada.Strings.Unbounded_String_Base(Integer):

Re: Silly question about strings (was: Filenames in Ada)

[Randy Brukardt]

<tmoran@acm.org> wrote in message
news:1MadnQhFHPh0cRbenZ2dnUVZ_smdnZ2d@comcast.com...
> >>   package Ada.Strings.Fixed is new
Ada.Strings.String_Base( Character );
> >...
> >Something like this was proposed as an 11th hour suggestion (this year).
> Since the generic type parameter is merely any discrete type, presumably
> you would support (with appropriate OS calls where needed):
>   package Ada.Strings.Colors is new Ada.Strings.String_Base(Colors);
>   package Ada.Strings.Unbounded_Integers
>     is new Ada.Strings.Unbounded_String_Base(Integer):

Right. And there was some objection to that, given the overlap with the
containers library. (How would you decide to use an unbounded array of
integers or an vector of integers? They have similar sets of operations.)

                             Randy.

Re: Silly question about strings (was: Filenames in Ada)

[Björn Persson]

tmoran@acm.org wrote:
> Since the generic type parameter is merely any discrete type, presumably
> you would support (with appropriate OS calls where needed):
>   package Ada.Strings.Colors is new Ada.Strings.String_Base(Colors);
>   package Ada.Strings.Unbounded_Integers
>     is new Ada.Strings.Unbounded_String_Base(Integer):

In that case they should be called arrays or vectors rather than strings.

-- 
Bj�rn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu

Re: Filenames in Ada

[Martin Krischik]

Randy Brukardt wrote:

> "Martin Krischik" <krischik@users.sourceforge.net> wrote in message
> news:1653090.31FM62oI6I@linux1.krischik.com...
> ...
>> How does one deal with modern (utf-8) filenames in Ada?
> 
> We talked about this briefly at a recent meeting. We decided it was too
> late to design a proper solution, and virtually all operating systems have
> a way to deal with this anyway. (That is that they take UTF-8 names).
> There's no conceptual problem with putting UTF-8 into a value of type
> String, and as someone else showed, that's typcially supported by Ada
> implementations.

Only Windows wants UTF-16 and extra APIs instead . And that is indeed the
problem I have.

Just a question. I get an NAME_ERROR exception in
Ada.Directories.More_Entries. Is that actually correct? Should the
exception not be in Ada.Directories.Simple_Name?

Only when I convert the name to a string it becomes relevant that it can't
be represented into Latin 1.

See source in

http://cvs.sourceforge.net/viewcvs.py/wikibook-ada/demos/Source/make_m3u.adb?only_with_tag=HEAD&view=markup

> A bigger problem is that Ada (any flavor) has no built-in support for
> UTF-8. I think this is a mistake, but I didn't find much support for
> adding any packages.

Well, that can be rectified with external packages. Text_IO is not as easy
to replace

> It's insane to use Wide_Wide_String to store any 
> significant amount of text, as it would waste nearly 3/4s of the space in
> typical use, so you'd have to use something else (like UTF-8) for storage
> anyway.

Tell the C99 designers. They though that wchar_t should be large enough to
hold all possible values - without defining what "all possible values"
actually is. As it is GNU C defines it as 32 bit.

> Since filenames are implementation-defined anyway, there isn't a whole lot
> of value to standardizing how they're written. So, it's just up to your
> implementer to do something appropriate.

As long as you don't "push them with the nose on it" as we say in Germany:

"An implementation may support Wide_String or Wide_Wide_String variants for
passing filenames if supported by the platform"

they won't do nothing.

Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: Filenames in Ada

[Martin Dowie]

Martin Krischik wrote:
> Just a question. I get an NAME_ERROR exception in
> Ada.Directories.More_Entries. Is that actually correct? Should the
> exception not be in Ada.Directories.Simple_Name?

Sounds like a bug to me...

Assuming you are using GNAT, there is also note there is a bug in the 
AdaCore "Compose" function. You should change:

       if not Is_Valid_Path_Name (Containing_Directory) then
          raise Name_Error;

to:

       if Containing_Directory /= "" and
          not Is_Valid_Path_Name (Containing_Directory) then
          raise Name_Error;

I've reported this in Bugzilla yonks ago but no changes in GCC yet, see 
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21346

Cheers

-- Martin

Re: Filenames in Ada

[Björn Persson]

Let's see if I understand the problem. Windows has two functions for 
each file operation, one -A version that expects or returns a file name 
in some 8-bit encoding like Windows-1252, and one -W version that 
expects or returns a file name in UTF-16 or maybe UCS-2? And all the 
file operations in the Ada library take and return file names as String, 
that is, Latin-1? And Gnat's implementation pretends that Latin-1 is 
identical to whatever 8-bit encoding Windows is using, and passes these 
Strings to Windows' -A functions, leaving you with no way to handle 
filenames that can't be expressed in said 8-bit encoding? Is that right?

It is my intention to add an encoding-aware interface to Ada.Directories 
under EAstrings.OS. For that to work reasonably on Windows, this problem 
needs to be solved. I suppose I also need to fix this in EAstrings.IO. I 
will need help from a Windows programmer to do this. (Of course I also 
need to get transcoding implemented on Windows before EAstrings will be 
of any use there.)

It seems that the right thing to do would be to tap into the Gnat 
library and make UTF-16 (or UCS-2) versions of the file operations. It 
could be as easy as changing the parameter type and replacing calls to 
the Windows functions with their -W equivalents, or it could be very hairy.

We'll need to determine whether it is UTF-16 or UCS-2. This page lists 
code page numbers for a whole lot of encodings, but UTF-16 is missing:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/intl/unicode_81rn.asp

I take that as a hint that UTF-16 is Windows' idea of wide strings, and 
that all the others are considered "multi-byte character sets" or 
whatever the term is.

-- 
Bj�rn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu

Re: Filenames in Ada

[Martin Krischik]

Bjï¿œrn Persson wrote:

> Let's see if I understand the problem. Windows has two functions for
> each file operation, one -A version that expects or returns a file name
> in some 8-bit encoding like Windows-1252, and one -W version that
> expects or returns a file name in UTF-16 or maybe UCS-2?

Well the Windows API in question where designed at a time when UTF-16 and
UCS-2 where still the same - that is Unicode had no codes defined above the
65535 border. At that time programmers did not care - or understood - the
difference between the two.

VFAT-32 is most likely a UCS-2 filesystem (anyone from china to confirm
that?). I remember an article about the "new" VFAT technology wasting
"enormous" amount of storrage using UCS-2 for character encoding.

Obviously the article came from an Latin-1 based country ;-) .

> And all the 
> file operations in the Ada library take and return file names as String,
> that is, Latin-1? And Gnat's implementation pretends that Latin-1 is
> identical to whatever 8-bit encoding Windows is using, and passes these
> Strings to Windows' -A functions, leaving you with no way to handle
> filenames that can't be expressed in said 8-bit encoding? Is that right?

Yes indeed. But I take it that on a Russian system the Windows-1251 code
page is active and all filenames are expressed using that and not Latin 1.

> It is my intention to add an encoding-aware interface to Ada.Directories
> under EAstrings.OS. For that to work reasonably on Windows, this problem
> needs to be solved. I suppose I also need to fix this in EAstrings.IO. I
> will need help from a Windows programmer to do this. (Of course I also
> need to get transcoding implemented on Windows before EAstrings will be
> of any use there.)

It is sad that XML/Ada has no UCS-2 and UCS-4 convertion available - but
AdaCL allready has that - so not problem for you really.

> It seems that the right thing to do would be to tap into the Gnat
> library and make UTF-16 (or UCS-2) versions of the file operations. It
> could be as easy as changing the parameter type and replacing calls to
> the Windows functions with their -W equivalents, or it could be very
> hairy.

I had that idea as well and did take a look. Lots of "pragma Import" there.

> We'll need to determine whether it is UTF-16 or UCS-2. This page lists
> code page numbers for a whole lot of encodings, but UTF-16 is missing:
> 
>
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/intl/unicode_81rn.asp
> 
> I take that as a hint that UTF-16 is Windows' idea of wide strings, and
> that all the others are considered "multi-byte character sets" or
> whatever the term is.

Well there seems an better article:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/naming_a_file.asp

I wonder about that \\?\ stuff and what it really means
 
Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: Filenames in Ada

[Björn Persson]

Martin Krischik wrote:
> But I take it that on a Russian system the Windows-1251 code
> page is active and all filenames are expressed using that and not Latin 1.

Speaking of that, do you know how to find out which code page is active? 
Can I get it from C's nl_langinfo like in Unix?

> It is sad that XML/Ada has no UCS-2 and UCS-4 convertion available - but
> AdaCL allready has that - so not problem for you really.

My main problem is lack of time. I found a job a year ago. Darn! ;-)

>>It seems that the right thing to do would be to tap into the Gnat
>>library and make UTF-16 (or UCS-2) versions of the file operations. It
>>could be as easy as changing the parameter type and replacing calls to
>>the Windows functions with their -W equivalents, or it could be very
>>hairy.
> 
> I had that idea as well and did take a look. Lots of "pragma Import" there.

I take it you mean that's a complication. You can't just import other 
functions?

> Well there seems an better article:
> 
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/naming_a_file.asp

Well, it says "Windows stores the long file names on disk in Unicode", 
so now we have to guess which encoding it is they call "Unicode". I'm 
guessing UTF-16, because UTF-16 was defined by Unicode while I think 
UCS-2 was defined by ISO.

Maybe you can do an experiment? Create a file with a surrogate pair in 
the name and see how it's shown in the file manager. You may get boxes 
or something if Windows doesn't have the glyphs, but if you see only one 
box it's obviusly been interpreted as UTF-16. If you see two boxes or 
you get some error then Windows seems to expect UCS-2.

> I wonder about that \\?\ stuff and what it really means

It looks like a crude hack to allow longer paths by bypassing parts of 
the library. Anyway it's not relevant to the question of character 
encodings.

-- 
Bjï¿œrn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu

Re: Filenames in Ada

[Martin Krischik]

Bjï¿œrn Persson wrote:

> Martin Krischik wrote:
>> But I take it that on a Russian system the Windows-1251 code
>> page is active and all filenames are expressed using that and not Latin
>> 1.
> 
> Speaking of that, do you know how to find out which code page is active?
> Can I get it from C's nl_langinfo like in Unix?

Depends. cygwin and mingw do indeed support those functions - but MS-C does
not. MS is a real pain when it comes to standart functionality. I guess
they don't want to support full ANSI/ISO C: It cost money and my lead to
less custom as porting away becomes easier.
 
>> It is sad that XML/Ada has no UCS-2 and UCS-4 convertion available - but
>> AdaCL allready has that - so not problem for you really.
> 
> My main problem is lack of time. I found a job a year ago. Darn! ;-)

Damm - I got the bloody same problem ;-) .
 
>>>It seems that the right thing to do would be to tap into the Gnat
>>>library and make UTF-16 (or UCS-2) versions of the file operations. It
>>>could be as easy as changing the parameter type and replacing calls to
>>>the Windows functions with their -W equivalents, or it could be very
>>>hairy.
>> 
>> I had that idea as well and did take a look. Lots of "pragma Import"
>> there.
> 
> I take it you mean that's a complication. You can't just import other
> functions?

Well it usually is some 

pragma Import (C, ....., "_gnat_.....");

So we have Ada -> C -> libC  and we don't only have to replace the Ada
functions but the C functions as well.

>> Well there seems an better article:

>>http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/naming_a_file.asp

> Well, it says "Windows stores the long file names on disk in Unicode",
> so now we have to guess which encoding it is they call "Unicode". I'm
> guessing UTF-16, because UTF-16 was defined by Unicode while I think
> UCS-2 was defined by ISO.

I am more pessimistic here. If MS likes to implement an encoding with
variable length why would they have used an 16 encoding for long filenames
in VFAT? At that time MP3 libraries where uncommon and almost all filenames
where plain ASCII.

> Maybe you can do an experiment? Create a file with a surrogate pair in
> the name and see how it's shown in the file manager. You may get boxes
> or something if Windows doesn't have the glyphs, but if you see only one
> box it's obviusly been interpreted as UTF-16. If you see two boxes or
> you get some error then Windows seems to expect UCS-2.

Anybody got an example? If not I get one from Wikibooks. There is a good
book an "Mandarin Chinese" there which should have enough examples ;-).

>> I wonder about that \\?\ stuff and what it really means
> 
> It looks like a crude hack to allow longer paths by bypassing parts of
> the library.

Well, Windows is full of Hacks as well. An so is Unix. Just yesterday I read
that the "Uni" in Unix stands for "single user". Originally Unix was a
scaled down single user alternative to "Multrix". Well that explains a lot.
Indeed there is only one real user on Unix - the user with the ID 0.

> Anyway it's not relevant to the question of character 
> encodings.

The question is: Which encoding is used for \\?\ Filenames. If it is UTF-8
it would solve some of our problems.

Martin

-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: Filenames in Ada

[Björn Persson]

Martin Krischik wrote:
> Bjï¿œrn Persson wrote:
>>Speaking of that, do you know how to find out which code page is active?
>>Can I get it from C's nl_langinfo like in Unix?
> 
> Depends. cygwin and mingw do indeed support those functions - but MS-C does
> not.

I seem to recall that Gnat for Windows uses MinGW, so maybe MinGW's 
functions are available without installing extra libraries?

> Well it usually is some 
> 
> pragma Import (C, ....., "_gnat_.....");
> 
> So we have Ada -> C -> libC  and we don't only have to replace the Ada
> functions but the C functions as well.

Yes, that's more work of course.

>>Maybe you can do an experiment? Create a file with a surrogate pair in
>>the name and see how it's shown in the file manager. You may get boxes
>>or something if Windows doesn't have the glyphs, but if you see only one
>>box it's obviusly been interpreted as UTF-16. If you see two boxes or
>>you get some error then Windows seems to expect UCS-2.
> 
> Anybody got an example? If not I get one from Wikibooks. There is a good
> book an "Mandarin Chinese" there which should have enough examples ;-).

It doesn't necessarily need to mean anything you know. ;-) Just feed 
0061 D800 DC00 0062 to CreateFileW or whatever the function is called, 
and see if it's shown as "a?b" or "a??b". (The surrogate pair D800 DC00 
forms the character U+10000, "Linear B syllable B008 A".)

How new is your Windows by the way?

> The question is: Which encoding is used for \\?\ Filenames. If it is UTF-8
> it would solve some of our problems.

I see. Well it says only the "Unicode versions" of the functions allow 
that kind of path. That would be the -W functions, which take wide 
strings, so I would think they expect UTF-16. Either way we still have 
to come up with a way to call them.

-- 
Bjï¿œrn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu

Re: Filenames in Ada

[Dave Thompson]

On Thu, 01 Dec 2005 07:59:45 +0100, Martin Krischik
<krischik@users.sourceforge.net> wrote:
<snip>
> Well, Windows is full of Hacks as well. An so is Unix. Just yesterday I read
> that the "Uni" in Unix stands for "single user". Originally Unix was a
> scaled down single user alternative to "Multrix". Well that explains a lot.

Multics, the Multiplexed Information and Computing Service. See
www.multicians.org for (much!) more. Not single user -- even the very
first, internal Bell Labs systems had multiple concurrent users -- but
small numbers of users: then perhaps 2-10 on a minicomputer, today
typically 10s to 100s (real users e.g. shell not say HTTP requesters,
which could be hugely more), compared to the big-iron mainframe
timesharing systems of the day which were 100s to many 1000s.

> Indeed there is only one real user on Unix - the user with the ID 0.
> 
What do you mean "real"? There is only one superuser id, or
equivalently all system/manager privileges/roles are combined into one
userid, which is certainly a limitation that is sometimes a problem.
But Unix does (or perhaps Unices do, since there are now a range of
them) support multiple "ordinary" users pretty well. If competently
administered, which many personal (Linux etc.) systems aren't.

- David.Thompson1 at worldnet.att.net

Re: Filenames in Ada

[Martin Krischik]

Dave Thompson wrote:

> On Thu, 01 Dec 2005 07:59:45 +0100, Martin Krischik
> <krischik@users.sourceforge.net> wrote:
> <snip>
>> Well, Windows is full of Hacks as well. An so is Unix. Just yesterday I
>> read that the "Uni" in Unix stands for "single user". Originally Unix was
>> a scaled down single user alternative to "Multrix". Well that explains a
>> lot.
> 
> Multics, the Multiplexed Information and Computing Service. See
> www.multicians.org for (much!) more. Not single user -- even the very
> first, internal Bell Labs systems had multiple concurrent users -- but
> small numbers of users: then perhaps 2-10 on a minicomputer, today
> typically 10s to 100s (real users e.g. shell not say HTTP requesters,
> which could be hugely more), compared to the big-iron mainframe
> timesharing systems of the day which were 100s to many 1000s.
> 
>> Indeed there is only one real user on Unix - the user with the ID 0.
>> 
> What do you mean "real"? There is only one superuser id, or
> equivalently all system/manager privileges/roles are combined into one
> userid, which is certainly a limitation that is sometimes a problem.
> But Unix does (or perhaps Unices do, since there are now a range of
> them) support multiple "ordinary" users pretty well. If competently
> administered, which many personal (Linux etc.) systems aren't.

big-iron machines normally need more then one superuser - sometimes some of
them might want a holiday. May be not every superuser need all the
superpowers.

I permanently have a root console open on my system and when I use a Linux
system where I don't have the root password I feel like a cripple.

I don't feel that way on VMS - there is a SET PRIVILEGE command there.

Martin 
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: program privileges (was: Filenames in Ada)

[Georg Bauhaus]

On Thu, 2005-12-15 at 21:23 +0100, Martin Krischik wrote:
> Dave Thompson wrote:


> > What do you mean "real"? There is only one superuser id, or
> > equivalently all system/manager privileges/roles are combined into one
> > userid, which is certainly a limitation that is sometimes a problem.
> > But Unix does (or perhaps Unices do, since there are now a range of
> > them) support multiple "ordinary" users pretty well. If competently
> > administered, which many personal (Linux etc.) systems aren't.
> 


> I permanently have a root console open on my system and when I use a Linux
> system where I don't have the root password I feel like a cripple.

I'd suggest setting up a sudo environment and a file system
convention like the one used in HP-Unix,
/opt/program/{typical unix hierarchy}

This way you can use plain old groups and have kind of superuser
privileges in these areas. (Unless some programs needs system
resources requiring root privileges.)

I'm happy when I don't have to do systems administration all the
time and execute every silly little program configuration as root.
I can otherwise concentrate on the applications.

Re: Filenames in Ada

[Simon Wright]

Martin Krischik <krischik@users.sourceforge.net> writes:

> I permanently have a root console open on my system and when I use a
> Linux system where I don't have the root password I feel like a
> cripple.
>
> I don't feel that way on VMS - there is a SET PRIVILEGE command
> there.

I don't see the difference between SET PRIV and sudo, really (of
course SET PRIV has all that feature set re: AUTHPRIV etc!)

Re: Filenames in Ada

[Larry Kilgallen]

In article <m2bqzf4q3q.fsf@grendel.local>, Simon Wright <simon@pushface.org> writes:
> Martin Krischik <krischik@users.sourceforge.net> writes:
> 
>> I permanently have a root console open on my system and when I use a
>> Linux system where I don't have the root password I feel like a
>> cripple.
>>
>> I don't feel that way on VMS - there is a SET PRIVILEGE command
>> there.
> 
> I don't see the difference between SET PRIV and sudo, really (of
> course SET PRIV has all that feature set re: AUTHPRIV etc!)

I do not know what "sudo" is, but VMS people who have seen Unix systems
say that a considerable difference is that VMS users can be restricted
in the set of privileges to which they have access.  The apparent set of
43 privileges or so should not be taken at face value, but instead one
should (gasp) actually read the documentation and see how those 43 are
divided into 7 "categories" (actually levels) according to how much
damage they could do to the system.  But even within a single level,
making distinctions regarding which privileges are enabled does protect
against some inadvertent cockpit errors.

Of course a major strength of SET PRIVILEGE, as distinguished from
logging into the SYSTEM username, is that auditing is according to
individual username so that actions can be tied back to a real person.
I would hope the same is provided on Unix.

privileges in Unix and VMS (was: Filenames in Ada)

[Björn Persson]

Larry Kilgallen wrote:
> I do not know what "sudo" is,

This is how Fedora's Sudo package is described:

Sudo (superuser do) allows a system administrator to give certain
users (or groups of users) the ability to run some (or all) commands
as root while logging all commands and arguments. Sudo operates on a
per-command basis.  It is not a replacement for the shell.  Features
include: the ability to restrict what commands a user may run on a
per-host basis, copious logging of each command (providing a clear
audit trail of who did what), a configurable timeout of the sudo
command, and the ability to use the same configuration file (sudoers)
on many different machines.

-- 
Bj�rn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu