Latin_1 and portability

Latin_1 and portability

[Xavier Serrand]

Hello,

For return (or enter)

when using Text_Io.Get_Immediate (E);

on UNIX or Linux Suse Character'Pos (E) = 10
on Windows 2000 Character'Pos (E) = 13

How can I manage to work properly ...
Do i have to parametrize my application reading characters from a file
... and looking at the values recognized by the functions ... Is there
anothers means ?

Thanks for Ada's Gurus who can help me a very little bit .....

xavier SERRAND

RE: Latin_1 and portability

[amado.alves]

<<...on UNIX or Linux Suse Character'Pos (E) = 10
on Windows 2000 Character'Pos (E) = 13

How can I manage to work properly ...>>

Depends on the application. For text processing applications that span several operating systems, I often use Latin_1.NEL for newline in the main application and provide 1-N conversors.

Re: Latin_1 and portability

[Georg Bauhaus]

Xavier Serrand <xavier.serrand@free.fr> wrote:
: Hello,
: 
: For return (or enter)
: 
: when using Text_Io.Get_Immediate (E);
: 
: on UNIX or Linux Suse Character'Pos (E) = 10
: on Windows 2000 Character'Pos (E) = 13
: 
: How can I manage to work properly ...
: Do i have to parametrize my application reading characters from a file
: ... and looking at the values recognized by the functions ... Is there
: anothers means ?

Do you read the characters from the keyboard?
And had you pressed RETURN in the above example?

If so, in order to be prepared for a variety of keyboards and
systems and character handling done by systems or run time
libraries, you can hide the details of what character you
will get for RETURN.
If you know that you will either get Latin_1.CR or Latin_1.LF,
and both Characters have the same meaning in your application, they
both could be mapped to the same character by a wrapper function,
for example.

Re: Latin_1 and portability

[Jeffrey Carter]

Xavier Serrand wrote:

> For return (or enter)
> 
> when using Text_Io.Get_Immediate (E);
> 
> on UNIX or Linux Suse Character'Pos (E) = 10
> on Windows 2000 Character'Pos (E) = 13

You could write

if Text_IO.End_Of_Line then
    Text_IO.Skip_Line;
    -- End of line processing
else
    Text_IO.Get_Immediate (E);
    -- Normal character processing
end if;

or

Text_IO.Get_Immediate (E);

case E is
when Latin_1.CR | Latin_1.LF =>
    -- End of line processing
when ... -- other special cases
when others =>
    -- Normal character processing
end case;

-- 
Jeff Carter
"Perfidious English mouse-dropping hoarders."
Monty Python & the Holy Grail
10

Re: Latin_1 and portability

[Pascal Obry]

xavier.serrand@free.fr (Xavier Serrand) writes:

> For return (or enter)
> 
> when using Text_Io.Get_Immediate (E);
> 
> on UNIX or Linux Suse Character'Pos (E) = 10
> on Windows 2000 Character'Pos (E) = 13

On Windows you should get 13+10 (CRLF) on UNIX only 10 (LF). You could try to
skip the 13 and call Get_Immediate a second time in this case.

Pascal.

-- 

--|------------------------------------------------------
--| Pascal Obry                           Team-Ada Member
--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE
--|------------------------------------------------------
--|         http://perso.wanadoo.fr/pascal.obry
--| "The best way to travel is by means of imagination"
--|
--| gpg --keyserver wwwkeys.pgp.net --recv-key C1082595

Re: Latin_1 and portability

[Ludovic Brenta]

Pascal Obry <p.obry@wanadoo.fr> writes:

> xavier.serrand@free.fr (Xavier Serrand) writes:
> 
> > For return (or enter)
> > 
> > when using Text_Io.Get_Immediate (E);
> > 
> > on UNIX or Linux Suse Character'Pos (E) = 10
> > on Windows 2000 Character'Pos (E) = 13
> 
> On Windows you should get 13+10 (CRLF) on UNIX only 10 (LF). You
> could try to skip the 13 and call Get_Immediate a second time in
> this case.

Just in case you're new to computers: CR stands for Carriage Return
and LF stands for Line Feed.  These are instructions for the cursor of
the terminal or the printing head of a teletype or printer, and are
encoded in ASCII as 13 and 10 respectively (in decimal; in hex they
are 0D and 0A respectively).  CR causes the cursor to go the the left
of the page, LF causes it to go down one line.

By convention, text files (and the terminal) on Windows use a sequence
of CR and LF to signal the end of a line.  Unices and GNU/Linux use a
different convention with only LF.  That is why Pascal's advice of
just ignoring the CR should get you going.  However, others have
pointed at function Text_IO.End_Of_Line, which does this for you in a
more portable way not relying on particular ASCII codes.

-- 
Ludovic Brenta.

Re: Latin_1 and portability

[Xavier Serrand]

Georg Bauhaus <sb463ba@l1-hrz.uni-duisburg.de> wrote in message news:<br7814$3ia$1@a1-hrz.uni-duisburg.de>...
> Xavier Serrand <xavier.serrand@free.fr> wrote:
> : Hello,
> : 
> : For return (or enter)
> : 
> : when using Text_Io.Get_Immediate (E);
> : 
> : on UNIX or Linux Suse Character'Pos (E) = 10
> : on Windows 2000 Character'Pos (E) = 13
> : 
> : How can I manage to work properly ...
> : Do i have to parametrize my application reading characters from a file
> : ... and looking at the values recognized by the functions ... Is there
> : anothers means ?
> 
> Do you read the characters from the keyboard?
> And had you pressed RETURN in the above example?
> 
> If so, in order to be prepared for a variety of keyboards and
> systems and character handling done by systems or run time
> libraries, you can hide the details of what character you
> will get for RETURN.
> If you know that you will either get Latin_1.CR or Latin_1.LF,
> and both Characters have the same meaning in your application, they
> both could be mapped to the same character by a wrapper function,
> for example.

Hello Mister Georg Bauhaus,

First, I read characters from the keyboard, and i've pressed RETURN...
Thanks for this advise... i actually don't need to make differences
between  CR and LF.

Xavier SERRAND

Re: Latin_1 and portability

[Robert I. Eachus]

Xavier Serrand wrote:

> Hello,
> 
> For return (or enter)
> 
> when using Text_Io.Get_Immediate (E);
> 
> on UNIX or Linux Suse Character'Pos (E) = 10
> on Windows 2000 Character'Pos (E) = 13
> 
> How can I manage to work properly ...
> Do i have to parametrize my application reading characters from a file
> ... and looking at the values recognized by the functions ... Is there
> anothers means ?
> 
> Thanks for Ada's Gurus who can help me a very little bit .....

This turns out not to be an Ada issue, but you still have to deal with 
it.  The Ada Reference Manual doesn't specify what indicates the end of 
a line.  This is because the choice of how lines are delimited is a 
property of the file type.  On Windows, .txt files use CR (13) as a line 
delimiter and on Unix LF (10) is used.  There are also formats that use 
CR followed by LF, or VT(11), and many formats use FF(12) as an end of 
line AND page mark.

If you are reading from a file, then you can let Text_IO handle the 
issue, even if the file format is one where the end of a line is 
represented implicitly with a character count.  (Not too common 
anymore.)  But you are trying to read from a user input device.  What do 
you get?  Whatever the user gives you, as mapped by the OS. ;-)  I would 
tend to test for all four:  E in Latin1.LF..Latin.CR.  That should deal 
with most existing operating systems.  The operating system maps the 
keys on the keyboard to ASCII or other characters.  In most operating 
systems, you can choose some other keyboard mapping that the standard. 
I actually have about six set up, but the two I use most are United 
States and International.  This allows me to type characters like 
o-diaresis, or e-acute.  It also lets me type Japanese, Chinese, and 
Korean, but you do not want to deal with that.

-- 
                                           Robert I. Eachus

100% Ada, no bugs--the only way to create software.

Re: Latin_1 and portability

[Larry Kilgallen]

In article <gcednamCzLqiekqi4p2dnA@comcast.com>, "Robert I. Eachus" <rieachus@comcast.net> writes:
> Xavier Serrand wrote:
> 
>> Hello,
>> 
>> For return (or enter)
>> 
>> when using Text_Io.Get_Immediate (E);

> If you are reading from a file, then you can let Text_IO handle the 
> issue, even if the file format is one where the end of a line is 
> represented implicitly with a character count.  (Not too common 
> anymore.)  But you are trying to read from a user input device.  What do 
> you get?  Whatever the user gives you, as mapped by the OS. ;-)

Are you saying that some implementations of TEXT_IO.GET_LINE do not
handle OS terminals properly ?

I thought GET_LINE was the way around OS-dependencies in this area.

RE: Latin_1 and portability

[amado.alves]

<<
Are you saying that some implementations of TEXT_IO.GET_LINE do not
handle OS terminals properly ?

I thought GET_LINE was the way around OS-dependencies in this area.
>>
 
I find this fascinating. You can have an X-conventioned text file in a Y system, where X and Y are operating systems (with distinct end of line conventions). If you compile your Ada program to the Y target, it will not read the X file properly. Amongs other unwarranted effects. Thit is why I adopt an 1-N conversion approach. Say 8-bit characters. I use Latin_1.NEL for end of line in the main application (the "1" side), and provide (simple) converters to/from the N conventions.

RE: Latin_1 and portability

[Larry Kilgallen]

In article <mailman.110.1071282456.31149.comp.lang.ada@ada-france.org>, "amado.alves" <amado.alves@netcabo.pt> writes:
> <<
> Are you saying that some implementations of TEXT_IO.GET_LINE do not
> handle OS terminals properly ?
> 
> I thought GET_LINE was the way around OS-dependencies in this area.
>>>
> =20
> I find this fascinating. You can have an X-conventioned text file in a Y =
> system, where X and Y are operating systems (with distinct end of line =
> conventions). If you compile your Ada program to the Y target, it will =
> not read the X file properly. Amongs other unwarranted effects. Thit is =
> why I adopt an 1-N conversion approach. Say 8-bit characters. I use =
> Latin_1.NEL for end of line in the main application (the "1" side), and =
> provide (simple) converters to/from the N conventions.

That depends on your transfer mechanism.  ISO-9660 includes an indicator
regarding which of the 4 major EOL indicators is used in a file, and the
OS can readily honor that.  I have been told that some TCP/IP implementations
do conversion on the fly.

Taking random bits from one operating system to another is bound to fail
when the operating systems expects a particular format (text file). This
is no an Ada problem.

RE: Latin_1 and portability

[amado.alves]

<<...ISO-9660 includes an indicator
regarding which of the 4 major EOL indicators>>
 
I bet all except NEL: as I said, fascinating stuff :-)
 
<<...I have been told that some TCP/IP implementations
do conversion on the fly.>>
 
I don't think I have ever met one implementation of TCP/IP or magnetic media types (that's ISO-9660 right?) that actually convert. Maybe the standards have places for it as you said (I believe you for ISO). But it is not effective in the systems I'm used to (Windows, Linux). Not to mention the remaining characters outside ASCII. It's a mess. Of course AdaOS will do the right thing :-)
 

Re: Latin_1 and portability

[Robert I. Eachus]

Larry Kilgallen wrote:

> Are you saying that some implementations of TEXT_IO.GET_LINE do not
> handle OS terminals properly ?
> 
> I thought GET_LINE was the way around OS-dependencies in this area.

Get_Line is fine, if you want to read lines.  But if you want to read 
characters with Get_Immediate, you get to see the control characters.

-- 
                                           Robert I. Eachus

100% Ada, no bugs--the only way to create software.

Re: Latin_1 and portability

[Björn Persson]

On Sat, 13 Dec 2003 13:35:17 -0000
"amado.alves" <amado.alves@netcabo.pt> wrote:

> I don't think I have ever met one implementation of TCP/IP or magnetic
> media types (that's ISO-9660 right?) that actually convert.

ISO 9660 is for optical media - CDs and DVDs. Not that it matters.

TCP is on a far too low level to convert line breaks, and IP is even lower.
FTP tries to do this, fails, and causes endless problems with corruption of
non-text files. Even FTP is on a too low level.

> Of course AdaOS will do the right thing :-)

Now I'm curious. What is the right thing to do?

Björn Persson

RE: Latin_1 and portability

[amado.alves]

/*
> ISO 9660 is for optical media - CDs and DVDs.
I was thinking CDs and said magnetic. Old brain. At least I didn't say vinyl :-)
*/

>> Of course AdaOS will do the right thing :-)

>Now I'm curious. What is the right thing to do?

Hmm... good point :-) 24-bit Unicode everywhere inside the system, and absolutely flawless converters to/from all formats?

Re: Latin_1 and portability

[Larry Kilgallen]

In article <20031213155408.356b7ec3.bjorn_persson.spam-is-evil@sverige.nu>, =?ISO-8859-1?Q?Bj=F6rn?= Persson <bjorn_persson.spam-is-evil@sverige.nu> writes:
> On Sat, 13 Dec 2003 13:35:17 -0000
> "amado.alves" <amado.alves@netcabo.pt> wrote:
> 
>> I don't think I have ever met one implementation of TCP/IP or magnetic
>> media types (that's ISO-9660 right?) that actually convert.
> 
> ISO 9660 is for optical media - CDs and DVDs. Not that it matters.

Actually it can be used for magnetic disks as well, but the major reason
to choose it over OS-native formats is portability, and that is typically
done with optical media.

> TCP is on a far too low level to convert line breaks, and IP is even lower.
> FTP tries to do this, fails, and causes endless problems with corruption of
> non-text files. Even FTP is on a too low level.

Sorry, I was the one who said TCP/IP and I should have said FTP.
I have not heard of any troubles on such conversion, but the FTP
conversion would obviously only be used when the source system
indicated whether a file is text or not.

>> Of course AdaOS will do the right thing :-)
> 
> Now I'm curious. What is the right thing to do?

	1. Keep track of whether a local file is text or not.
	2. Convert on the fly for FTP with less capable operating systems
	3. Support ISO-9660 format in full, for both read and write.

RE: Latin_1 and portability

[Larry Kilgallen]

In article <mailman.111.1071322524.31149.comp.lang.ada@ada-france.org>, "amado.alves" <amado.alves@netcabo.pt> writes:
> <<...ISO-9660 includes an indicator
> regarding which of the 4 major EOL indicators>>
> =20
> I bet all except NEL: as I said, fascinating stuff :-)

	It handles:
		CR separators
		LF separators
		CR/LF separators
		counted lines with no separator characters

> <<...I have been told that some TCP/IP implementations
> do conversion on the fly.>>
> =20
> I don't think I have ever met one implementation of TCP/IP or magnetic =
> media types (that's ISO-9660 right?) that actually convert. Maybe the =
> standards have places for it as you said (I believe you for ISO). But it =
> is not effective in the systems I'm used to (Windows, Linux).

You can lead an operating system to standards but you cannot make it drink :-)
Perhaps the originators of Linux decided to only partially implement the
standard because the standards group charges money for a copy :-)

Re: Latin_1 and portability

[Björn Persson]

On 13 Dec 2003 16:17:06 -0600
Kilgallen@SpamCop.net (Larry Kilgallen) wrote:

> I have not heard of any troubles on such conversion, but the FTP
> conversion would obviously only be used when the source system
> indicated whether a file is text or not.

FTP clients typically let the user select ASCII mode or image mode
manually. If you want conversion you have to remember to switch to ASCII
mode before you transfer a text file, and then switch back to image mode
before you transfer a non-text file. People will inevitably forget this
now and then, and get either an unconverted text file or a corrupt non-text
file. There is a reason why the Portable Network Graphics format's "magic
signature" incudes both a CR/LF pair and a single LF. It was designed to
provide for easy detection of this kind of file corruption. It's very
clever - and should never have been necessary.

RFC 959 actually specifies that ASCII mode is the default, but luckily
clients that automatically set image mode seem to be common.

Björn Persson

Re: Latin_1 and portability

[Björn Persson]

On Sat, 13 Dec 2003 15:24:56 -0000
"amado.alves" <amado.alves@netcabo.pt> wrote:

> >> Of course AdaOS will do the right thing :-)
> 
> >Now I'm curious. What is the right thing to do?
> 
> Hmm... good point :-) 24-bit Unicode everywhere inside the system, and
> absolutely flawless converters to/from all formats?

Imagine a file server running on AdaOS in a heterogenous network. Let's say
files are served with SMB (Microsoft's protocol), so that both Windows, Mac
and Unix clients can connect. I don't think SMB lets the server ask the
clients what operating system they are running, and I'd be very surprised
if it has a method of specifying whether a file is text or not. Even if SMB
can do that, the client operating systems usually won't know what kind of
data is in the files that various programs produce, so they can't tell the
server.

As a result, the file server can't know which files should be converted, it
can't know which format to expect from a certain client, and it can't know
which format a certain client needs; and there's not much AdaOS can do to
help.

Björn Persson

Re: Latin_1 and portability

[Jeff C,]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 571 bytes --]


"Bj�rn Persson" <bjorn_persson.spam-is-evil@sverige.nu> wrote in message
news:20031214034629.3960d5a8.bjorn_persson.spam-is-evil@sverige.nu...
As a result, the file server can't know which files should be converted, it
can't know which format to expect from a certain client, and it can't know
which format a certain client needs; and there's not much AdaOS can do to
help.

Bj�rn Persson



Ah..But you miss the ultimate power of AdaOS. Since it does not exist and is
never likely to exist, it is capable of claiming to perform all sorts of
endless wonders...

Re: Latin_1 and portability

[Nick Roberts]

Jeff C, wrote:

> "Bj�rn Persson" <bjorn_persson.spam-is-evil@sverige.nu> wrote in message
> news:20031214034629.3960d5a8.bjorn_persson.spam-is-evil@sverige.nu...

> > As a result, the file server can't know which files should be converted, it
> > can't know which format to expect from a certain client, and it can't know
> > which format a certain client needs; and there's not much AdaOS can do to
> > help.
> 
> Ah..But you miss the ultimate power of AdaOS. Since it does not exist and is
> never likely to exist, it is capable of claiming to perform all sorts of
> endless wonders...

It's true that AdaOS does not yet exist, and it surely will not for several 
years yet, but I remain as determined as ever to see it through, however 
long that takes.

Anyway, it's not really of importance to Bj�rn's point. I think all he 
meant to imply is that there's not much /some future operating system/ can 
do to help. In essence, that's quite correct. However, for the curious, 
AdaOS will have a way of coping with the problem (as it will have to, of 
course).

AdaOS will be object oriented, and files will be objects, and so strictly 
typed. Of course one type will be 'opaque binary', but all text files will 
have suitable types, parametised by encoding and character set.

I anticipate developing (and evolving) a piece of software in AdaOS which 
will use a strategy to guess the correct type of imported files. When 
obtaining a file by HTTP or FTP, it may be able to obtain a MIME type 
string, for example. For files coming from some kinds of systems, it may 
use a part of the name (the 'file type extension'). It may examine the 
file's contents. It might ask the user to supply or confirm the type. 
(Generally, the user will have to be asked for other information about the 
file anyway.)

Normally, every external file imported into a system will be permanently 
represented by an internal object. In this object will be stored a lot of 
information about the external file, including the (confirmed) type, and 
this type would generally be re-used whenever the file was re-accessed. (In 
fact, the file contents may be temporarily cached in the object anyway, 
according to parametised criteria.)

-- 
Nick Roberts
  __________________________________________________________
|  Fight Spam! Join EuroCAUCE: http://www.euro.cauce.org/  |
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~