Howto read line from a stream

Howto read line from a stream

[Tomek Walkuski]

Hi group,

I have a stream, which is defined as:
Channel : Stream_Access;

This stream is associated with a socket. If I knew how long is input
message I could do:
Msg : String (1 .. Known_Length);
String'Read (Channel, Msg);

Is there any smart solution? Or should I read character by character
from a stream until I get CR & LF ?

Re: Howto read line from a stream

[Tomek Wałkuski]

I have done so far:

   function Read_Line (Channel : in Stream_Access) return String is
      Buffer : String (1 .. 1);
      Result : Unbounded_String;
   begin
      loop
         String'Read (Channel, Buffer);
         Append (Result, Buffer);
         exit when Buffer (1) = ASCII.LF;
      end loop;
      return To_String(Result);
   end Read_Line;

I know, this is completely NOT smart solution.

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Sun, 31 May 2009 03:41:46 -0700 (PDT), Tomek Walkuski wrote:

> I have a stream, which is defined as:
> Channel : Stream_Access;
>
> This stream is associated with a socket. If I knew how long is input
> message I could do:
> Msg : String (1 .. Known_Length);
> String'Read (Channel, Msg);

One potential problem with using Streams as an interface to sockets is that
there is nothing in the Ada standard that mandates Stream_Element to be an
octet of bits, as socket I/O demands.

> Is there any smart solution?

If you know the size of packets you can read it as a whole. Typically
protocols deploy some headers determining the length of a packet body. The
header is usually read octet by octet, and the rest in bigger chunks.

However, String'Read is not necessarily mapped into one single socket's
read. Quite possible is that it will be split into separate Character'Read
implemented by independent reads.

(I assume that you are using TCP/IP rather than UDP)

> Or should I read character by character
> from a stream until I get CR & LF ?

No. There is no any CR's or LF's in the stream unless the protocol defines
them as a part of the content.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Sun, 31 May 2009 04:29:33 -0700 (PDT), Tomek Wa�kuski wrote:

> I have done so far:
> 
>    function Read_Line (Channel : in Stream_Access) return String is
>       Buffer : String (1 .. 1);
>       Result : Unbounded_String;
>    begin
>       loop
>          String'Read (Channel, Buffer);
>          Append (Result, Buffer);
>          exit when Buffer (1) = ASCII.LF;
>       end loop;
>       return To_String(Result);
>    end Read_Line;
> 
> I know, this is completely NOT smart solution.

It is OK except for Unbounded_String, especially Append per each character.
That should be very slow.

I would do something like:

with Ada.Finalization;
with Ada.Streams; use Ada.Streams;
package Buffers is
      -- Buffer to accumulate read lines
   type Line_Buffer is tagged limited private;
      -- Get the line in the buffer (longer than the line read)
   function Get_Line (Buffer : Line_Buffer) return not null access String;
      -- Get the length of the line in the buffer
   function Get_Length (Buffer : Line_Buffer) return Natural;
      -- Read new line into the buffer
   procedure Read
             (  Buffer : in out Line_Buffer;
                Stream : in out Root_Stream_Type'Class
             );
private
   type String_Ptr is access String;
   type Line_Buffer is new Ada.Finalization.Limited_Controlled with record
      Length : Natural := 0;
      Line   : String_Ptr; -- The line body, dynamically allocated
   end record;
   overriding procedure Finalize (Buffer : in out Line_Buffer);
end Buffers;

You would use it like:

   loop
      Read (Buffer, Stream);  -- Read line
      declare
          Line : String renames Buffer.Get_Line (1..Buffer.Get_Length));
      begin
         ... -- Process Line
      end;
   end loop;

The implementation could be like:

with Ada.Unchecked_Deallocation;
package body Buffers is
   Increment : constant := 1024;
   procedure Free is new Ada.Unchecked_Deallocation (String, String_Ptr);

   function Get_Line (Buffer : Line_Buffer)
      return not null access String is
   begin
      return Buffer.Line;
   end Get_Line;

   function Get_Length (Buffer : Line_Buffer) return Natural is
   begin
      return Buffer.Length;
   end Get_Length;

   procedure Read
             (  Buffer : in out Line_Buffer;
                Stream : in out Root_Stream_Type'Class
             )  is
      Data : Character;
   begin
      Buffer.Length := 0;
      loop
         Character'Read (Stream'Access, Data);
         exit when Data = Character'Val (10);
         if Buffer.Line = null then
            Buffer.Line := new String (1..Increment);
         end if;
         if Buffer.Line'Length = Buffer.Length then
            declare
               Old : String_Ptr := Buffer.Line;
            begin
               Buffer.Line := new String (1..Old'Length + Increment);
               Buffer.Line (1..Old'Length) := Old.all;
               Free (Old);
            end;
         end if;
         Buffer.Length := Buffer.Length + 1;
         Buffer.Line (Buffer.Length) := Data;
      end loop;
   end Read;

   procedure Finalize (Buffer : in out Line_Buffer) is
   begin
      Free (Buffer.Line);
   end Finalize;

end Buffers;

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Tomek Wałkuski]

Thank you Dimitry. I have implemented something like this:

---------- CODE -----------
   function Get_Line (Channel : in Stream_Access) return String is
      type String_Access is access String;
      Buffer      : Character;
      Result      : String_Access;
      Quit        : Boolean := False;
      Line_Length : Natural := 0;
      procedure Free is new Ada.Unchecked_Deallocation (String,
String_Access);
   begin
      loop
         Character'Read (Channel, Buffer);
         if Result = null then
            Result := new String (1 .. 1);
         end if;
         if Result'Length = Line_Length then
            declare
               Old : String_Access := Result;
            begin
               Result := new String (1 .. Old'Length +1);
               Result (1 .. Old'Length) := Old.all;
               Free (Old);
            end;
         end if;
         Line_Length := Line_Length + 1;
         Result (Line_Length) := Buffer;
         if Quit = True and then Buffer = ASCII.LF then
            exit;
         elsif Buffer = ASCII.CR then
            Quit := True;
         else
            Quit := False;
         end if;
      end loop;
      return Result.all;
   end Get_Line;
---------- CODE -----------

My protocol says that every line ends with CR & LF, only length can
vary.

Re: Howto read line from a stream

[Tomek Wałkuski]

On 31 Maj, 14:56, Tomek Wałkuski <tomek.walku...@gmail.com> wrote:
> Thank you Dimitry. I have implemented something like this:

I decided to return String_Access, instead of String, because of
memory leak in my previous solution. I can't declare as:

   function Get_Line (Channel : in Stream_Access) return not null
access String;

when I want to use it as:

   declare
      Msg : String_Access := Get_Line (Channel);
   begin
      --  Process line here
      Free (Msg);
   end;

so I did this:

   function Get_Line (Channel : in Stream_Access) return
String_Access;

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Sun, 31 May 2009 05:56:41 -0700 (PDT), Tomek Wa�kuski wrote:

> Thank you Dimitry. I have implemented something like this:
> 
> ---------- CODE -----------
>    function Get_Line (Channel : in Stream_Access) return String is
>       type String_Access is access String;
>       Buffer      : Character;
>       Result      : String_Access;
>       Quit        : Boolean := False;
>       Line_Length : Natural := 0;
>       procedure Free is new Ada.Unchecked_Deallocation (String,
> String_Access);
>    begin
>       loop
>          Character'Read (Channel, Buffer);
>          if Result = null then
>             Result := new String (1 .. 1);
>          end if;
>          if Result'Length = Line_Length then
>             declare
>                Old : String_Access := Result;
>             begin
>                Result := new String (1 .. Old'Length +1);
>                Result (1 .. Old'Length) := Old.all;
>                Free (Old);
>             end;
>          end if;
>          Line_Length := Line_Length + 1;
>          Result (Line_Length) := Buffer;
>          if Quit = True and then Buffer = ASCII.LF then
>             exit;
>          elsif Buffer = ASCII.CR then
>             Quit := True;
>          else
>             Quit := False;
>          end if;
>       end loop;
>       return Result.all;
>    end Get_Line;
> ---------- CODE -----------

The problem with above is that it is at least as inefficient as 
Unbounded_String:

1. You allocate a new string per each input byte;
2. You do not keep the allocated buffer between calls to Get_Line;
3. You copy the buffer content once read;
4. It leaks memory.

The idea I tried to convey by my solution was to address the issues 1-3:

1. The buffer is allocated in big pieces, so that it would adjust to the 
maximal line length in few iterations.
2. The buffer is kept between calls. 1 & 2 mean that there would be only 
one or two allocations per session.
3. The buffer content is not copied. Get_Line returns of an access to 
String, of which slice is then renamed.

> My protocol says that every line ends with CR & LF, only length can
> vary.

The best practice of dealing with texts (streams or not) is to use LF as a 
line terminator and remove training (or any) CR's from the buffer. This 
makes it working under both Linux and Windows.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[sjw]

On May 31, 12:34 pm, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
wrote:

> One potential problem with using Streams as an interface to sockets is that
> there is nothing in the Ada standard that mandates Stream_Element to be an
> octet of bits, as socket I/O demands.

Can't imagine what reason an implementer would have to make a stream
element anything other than an octet! (for any general-purpose
processor you'd actually be likely to come across).

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Sun, 31 May 2009 08:38:57 -0700 (PDT), sjw wrote:

> On May 31, 12:34�pm, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
> wrote:
> 
>> One potential problem with using Streams as an interface to sockets is that
>> there is nothing in the Ada standard that mandates Stream_Element to be an
>> octet of bits, as socket I/O demands.
> 
> Can't imagine what reason an implementer would have to make a stream
> element anything other than an octet! (for any general-purpose
> processor you'd actually be likely to come across).

This is a question you should better address to the designers of the Ada
standard. For some reasons they didn't take the responsibility to mandate
that Stream_Element'Size = 8.

I am not a language lawyer, but I remember a DSP processor with
sizeof(char)=4.

I suggest that it should be difficult quite there to have:

   type Stream_Element_Array is
        array(Stream_Element_Offset range <>) of aliased Stream_Element;

(note *aliased*), with 8 bit Stream_Element and packed
Stream_Element_Array.

Honestly, I have no idea how socket library should function there (I don;t
remember if it had any). Probably it extended and truncated each octet.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Niklas Holsti]

Dmitry A. Kazakov wrote:

> I am not a language lawyer, but I remember a DSP processor with
> sizeof(char)=4.

Apologies for picking a nit, but that can't be the case for a 
standard-conforming C implementation -- the C standard mandates 
that sizeof(char) = 1 because "sizeof" uses the size of "char" as 
its unit of measurement.

No doubt you mean that "char" objects on this DSP were 32 bits = 4 
octets. I know that at least one GCC port for the Analog Devices 
ADSP-21020 had such "char" objects. Since that processor only has 
32-bit word addressing a smaller size for "char" would have 
required the C compiler to invent its own "octet address" type, 
with much complication and overhead. I believe that an Ada compiler 
could do that with less overhead, because the internal structure of 
access values is better hidden than in C. Perhaps an Ada compiler 
could choose 8-bit stream elements for such processors, with 
tolerable overhead.

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .

Re: Howto read line from a stream

[sjw]

On May 31, 5:07 pm, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
wrote:
> On Sun, 31 May 2009 08:38:57 -0700 (PDT), sjw wrote:
> > On May 31, 12:34 pm, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
> > wrote:
>
> >> One potential problem with using Streams as an interface to sockets is that
> >> there is nothing in the Ada standard that mandates Stream_Element to be an
> >> octet of bits, as socket I/O demands.
>
> > Can't imagine what reason an implementer would have to make a stream
> > element anything other than an octet! (for any general-purpose
> > processor you'd actually be likely to come across).
>
> This is a question you should better address to the designers of the Ada
> standard. For some reasons they didn't take the responsibility to mandate
> that Stream_Element'Size = 8.

Fairly strong hint, I think, in http://www.adaic.com/standards/05aarm/html/AA-13-13.html
(1).

An implementation with Stream_Element'Size /= 8 would be in a world of
its own, probably better off not to bother with a socket library!

> I am not a language lawyer, but I remember a DSP processor with
> sizeof(char)=4.

I spent a lot of time with a processor that (if it had had a C
compiler) would have had 24-bit chars.

> I suggest that it should be difficult quite there to have:
>
>    type Stream_Element_Array is
>         array(Stream_Element_Offset range <>) of aliased Stream_Element;
>
> (note *aliased*), with 8 bit Stream_Element and packed
> Stream_Element_Array.

See http://www.adaic.com/standards/05aarm/html/AA-13-13-1.html (11.c/2)

Re: Howto read line from a stream

[Jeffrey R. Carter]

Tomek Wałkuski wrote:
> I have done so far:
> 
>    function Read_Line (Channel : in Stream_Access) return String is
>       Buffer : String (1 .. 1);
>       Result : Unbounded_String;
>    begin
>       loop
>          String'Read (Channel, Buffer);
>          Append (Result, Buffer);
>          exit when Buffer (1) = ASCII.LF;
>       end loop;
>       return To_String(Result);
>    end Read_Line;

I might do something like

function Get_Line (Channel : in Stream_Access) return String is
    Char : Character;
begin -- Get_Line
    Character'Read (Channel, Char);

    if Char = Ada.Characters.Latin_1.LF then
       return String'(1 => Char);
    else
       return Char & Get_Line (Channel);
    end if;
end Get_Line;

This is easily extended if needed to ensure that the terminating LF follows a CR.

-- 
Jeff Carter
"Have you gone berserk? Can't you see that that man is a ni?"
Blazing Saddles
38

Re: Howto read line from a stream

[Pascal Obry]

Dmitry A. Kazakov a �crit :
> It is OK except for Unbounded_String, especially Append per each character.
> That should be very slow.

This depends on the implementation. GNAT makes this efficiently as there
is some internal buffer so the string is not reallocated for each and
every character.

The RM does not mandate slowness :)

Pascal.

-- 

--|------------------------------------------------------
--| Pascal Obry                           Team-Ada Member
--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE
--|------------------------------------------------------
--|    http://www.obry.net  -  http://v2p.fr.eu.org
--| "The best way to travel is by means of imagination"
--|
--| gpg --keyserver keys.gnupg.net --recv-key F949BD3B

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Sun, 31 May 2009 15:00:24 -0700 (PDT), sjw wrote:

> An implementation with Stream_Element'Size /= 8 would be in a world of
> its own, probably better off not to bother with a socket library!

Well, streams are already in that worlds because stream I/O is not
portable. Therefore using Ada stream as an interface to the socket stream
is questionable.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[björn lundin]

On 31 Maj, 13:29, Tomek Wałkuski <tomek.walku...@gmail.com> wrote:
> I have done so far:
>
>    function Read_Line (Channel : in Stream_Access) return String is
>       Buffer : String (1 .. 1);
>       Result : Unbounded_String;
>    begin
>       loop
>          String'Read (Channel, Buffer);
>          Append (Result, Buffer);
>          exit when Buffer (1) = ASCII.LF;
>       end loop;
>       return To_String(Result);
>    end Read_Line;
>
> I know, this is completely NOT smart solution.

I would look into the solution in Ada Sockets,
procedure Get_Line, by Sam Tardieu, I think
<http://www.rfc1149.net/devel/adasockets>
--
Björn Lundin

Re: Howto read line from a stream

[Randy Brukardt]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> wrote in message 
news:18nn8atp3gdq1$.6g5cqv5h6u1c.dlg@40tude.net...
...
>> My protocol says that every line ends with CR & LF, only length can
>> vary.
>
> The best practice of dealing with texts (streams or not) is to use LF as a
> line terminator and remove training (or any) CR's from the buffer. This
> makes it working under both Linux and Windows.

That's lousy advice if he is working with a protocol that is defined outside 
of his system. For instance, most Internet protocols (like HTTP, SMTP, and 
FTP) require lines to be ended with CR LF. Lines that don't include the CR 
are simply wrong and do not conform to the protocol. The originating system 
is irrelevant, Windows, Linux, Mac, whatever. Indeed, the lack of those 
characters is usually a good sign that spammers, malware, and the like is 
trying to access your system. For instance, my spam filter blocks e-mail 
messages that are missing the CRs required by the protocol, because only 
clueless or sloppy systems miss them (in addition, some older e-mail clients 
would crash on such messages, and there still is a danger to allowing them 
through).

                                                       Randy.

Re: Howto read line from a stream

[Randy Brukardt]

"sjw" <simon.j.wright@mac.com> wrote in message 
news:03bab021-0df6-445b-b2f7-7a3ab770448c@b1g2000vbc.googlegroups.com...
On May 31, 12:34 pm, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
wrote:
>> One potential problem with using Streams as an interface to sockets is 
>> that
>> there is nothing in the Ada standard that mandates Stream_Element to be 
>> an
>> octet of bits, as socket I/O demands.
>
>Can't imagine what reason an implementer would have to make a stream
>element anything other than an octet! (for any general-purpose
>processor you'd actually be likely to come across).

Our U2200 compiler used Stream_Element'Size = 9, as that was the native 
character size on that machine (36-bit words). Using 'Size = 8 would have 
required horrific code to access elements.

I don't think there are many modern machines that would be other than 8, 
however.

(And socket I/O probably used 9-bit bytes on that machine as well, although 
I don't remember the details anymore.)

                                       Randy.

Re: Howto read line from a stream

[anon]

Try the "PingPong" program that has been commented in the GNAT.SOCKETS 
specs ( g-socket.ads ). Some editiing is required. Like setting the IP to 
127.0.0.1 local loopback and removing some if not all "Set_Socket_Option" 
statements, (some OSs have problems with these statements). 


In <83317a97-dae5-4c84-a1ac-88a87833cf3f@q14g2000vbn.googlegroups.com>, Tomek Walkuski <tomek.walkuski@gmail.com> writes:
>Hi group,
>
>I have a stream, which is defined as:
>Channel : Stream_Access;
>
>This stream is associated with a socket. If I knew how long is input
>message I could do:
>Msg : String (1 .. Known_Length);
>String'Read (Channel, Msg);
>
>Is there any smart solution? Or should I read character by character
>from a stream until I get CR & LF ?

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Mon, 1 Jun 2009 18:30:31 -0500, Randy Brukardt wrote:

> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> wrote in message 
> news:18nn8atp3gdq1$.6g5cqv5h6u1c.dlg@40tude.net...
> ...
>>> My protocol says that every line ends with CR & LF, only length can
>>> vary.
>>
>> The best practice of dealing with texts (streams or not) is to use LF as a
>> line terminator and remove training (or any) CR's from the buffer. This
>> makes it working under both Linux and Windows.
> 
> That's lousy advice if he is working with a protocol that is defined outside 
> of his system. For instance, most Internet protocols (like HTTP, SMTP, and 
> FTP) require lines to be ended with CR LF. Lines that don't include the CR 
> are simply wrong and do not conform to the protocol. The originating system 
> is irrelevant, Windows, Linux, Mac, whatever. Indeed, the lack of those 
> characters is usually a good sign that spammers, malware, and the like is 
> trying to access your system. For instance, my spam filter blocks e-mail 
> messages that are missing the CRs required by the protocol, because only 
> clueless or sloppy systems miss them (in addition, some older e-mail clients 
> would crash on such messages, and there still is a danger to allowing them 
> through).

I don't think it is a good idea, how to handle poorly designed and badly
implemented protocols. We are living in a real world. It is silly to imply
that an incoherent implementation should indicate a security problem. The
problem is elsewhere, in the protocol itself. Anyways, I doubt that
somebody could statistically confirm that the percentage of malformed line
terminators is higher among unwanted postings.

P.S. Do you remember early GNAT versions that refused to compile Ada source
files with MS-DOS conventions when under Linux? Thank goodness, they are
gone.

P.P.S. There are lots of text-oriented protocols used in automotive. The
device vendors just do not care to handle LFs and CRs as specified (if
specified! (:-)). You buy a $10**5+ device and have to communicate with it.
Guess which side has to be fixed?

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Georg Bauhaus]

Dmitry A. Kazakov schrieb:

> P.P.S. There are lots of text-oriented protocols used in automotive. The
> device vendors just do not care to handle LFs and CRs as specified (if
> specified! (:-)). You buy a $10**5+ device and have to communicate with it.
> Guess which side has to be fixed?

The trouble is that '\n' in C source does not necessarily
mean '\n' in the data, same for New_Line.
How early in a programmer's life does she learn to assume
that '\n' means '\n', 'A' means 65 and CHAR_BIT is 8 ?
The earlier this damage is done, the more difficult it
is to mend.

I understand they still use ASCII tables at universities
when they explain character data.  They do not explain the
concept of a data type with possible representations of
values, as should be.  Let alone representations
of ends of lines.  This bit of teaching, eternalized as
a tradition, undermines the fundamental (and early)
understanding of types.

So having protocols depend on an understanding of line
termination is at odds with common knowledge;  when
common knowledge is used to implement protocols, some
effects become predictable consequences.

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Tue, 02 Jun 2009 11:36:43 +0200, Georg Bauhaus wrote:

> Dmitry A. Kazakov schrieb:
> 
>> P.P.S. There are lots of text-oriented protocols used in automotive. The
>> device vendors just do not care to handle LFs and CRs as specified (if
>> specified! (:-)). You buy a $10**5+ device and have to communicate with it.
>> Guess which side has to be fixed?
> 
> The trouble is that '\n' in C source does not necessarily
> mean '\n' in the data, same for New_Line.
> How early in a programmer's life does she learn to assume
> that '\n' means '\n', 'A' means 65 and CHAR_BIT is 8 ?
> The earlier this damage is done, the more difficult it
> is to mend.
> 
> I understand they still use ASCII tables at universities
> when they explain character data.  They do not explain the
> concept of a data type with possible representations of
> values, as should be.  Let alone representations
> of ends of lines.  This bit of teaching, eternalized as
> a tradition, undermines the fundamental (and early)
> understanding of types.

Right. It is utmost damaging. Students simply do not understand the
difference between value and a machine representation of. I have frequent
brain-dead discussions about "binary strings", "adding color pixels" (as
integers of course) and similar rubbish. The problem is that C++ language
family when taught as the first language inflicts a persistent damaging
effect on young programmers, because it does not promote the notion of
abstract data type. On the contrary it makes them believe in existence of
some magic chars and ints (existing in the real world), and at the same
time in some "classes" which are totally different (non-existent, composed
of ints, additive because, composed of ints, etc).

(Even some Ada programmers believe that tagged types are somewhat less or
more types than others.)

> So having protocols depend on an understanding of line
> termination is at odds with common knowledge;  when
> common knowledge is used to implement protocols, some
> effects become predictable consequences.

I would rather say common ignorance than knowledge.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Randy Brukardt]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> wrote in message 
news:1dlvlys7r0tyh$.zyytlf2vt5zm.dlg@40tude.net...
> On Mon, 1 Jun 2009 18:30:31 -0500, Randy Brukardt wrote:
...
> I don't think it is a good idea, how to handle poorly designed and badly
> implemented protocols. We are living in a real world. It is silly to imply
> that an incoherent implementation should indicate a security problem. The
> problem is elsewhere, in the protocol itself. Anyways, I doubt that
> somebody could statistically confirm that the percentage of malformed line
> terminators is higher among unwanted postings.

Maybe in general, but in the case of spam, there is a very strong 
correlation, simply because the majority of all legitimate mail is sent by 
only a few mail clients (Outlook, Thunderbird, GMail, etc.) and these do in 
fact follow the protocol standards. So anything that doesn't follow it is 
much more likely to be junk. Indeed, spam filtering is 100% about apply 
statistical models, since there is really no way to predict what good mail 
will look like.

It's possible that there is more of this problem with other mail these days; 
I would think that spammers would be more careful these days if for no 
reason than such malformed mail never gets delivered. But there still is an 
amazing amount of malformed mail.

Note that I created and manage the spam filter (in Ada, of course) on our 
mail servers, and we average 100,000 spams per week (about 99% spam). So I 
see a lot of spam in a week, even now when 97% or so gets automatically 
blocked or deleted.

> P.P.S. There are lots of text-oriented protocols used in automotive. The
> device vendors just do not care to handle LFs and CRs as specified (if
> specified! (:-)). You buy a $10**5+ device and have to communicate with 
> it.
> Guess which side has to be fixed?

I would argue that is mainly a problem with the specifications, rather than 
the device. But I feel your pain; hardly anything in the real world quite 
follows the specifications. Generally, you have handle screwed up line 
endings, but remember that fact so that you can then quarantine the mail 
(the reason being that you still need to scan the mail to see if it would be 
automatically deleted for other reasons). So it's pretty much the worst of 
both worlds. Browsers and HTML are even worse; if they refused to display a 
lot of improper junk, filtering would not be so hard.

                                                         Randy.

Re: Howto read line from a stream

[Tomek Wałkuski]

On 1 Cze, 02:05, "Jeffrey R. Carter"
<spam.jrcarter....@nospam.acm.org> wrote:
> I might do something like
>
> function Get_Line (Channel : in Stream_Access) return String is
>     Char : Character;
> begin -- Get_Line
>     Character'Read (Channel, Char);
>
>     if Char = Ada.Characters.Latin_1.LF then
>        return String'(1 => Char);
>     else
>        return Char & Get_Line (Channel);
>     end if;
> end Get_Line;
>
How to measure which solution (mine or yours) is faster?

Re: Howto read line from a stream

[Jeffrey R. Carter]

Tomek Wałkuski wrote:
>>
> How to measure which solution (mine or yours) is faster?

I don't really care which one is faster. The question is whether they meet the 
timing requirements for the application. If that's true for both, then it 
becomes a question of which is easier to understand and modify.

-- 
Jeff Carter
"He didn't get that nose from playing ping-pong."
Never Give a Sucker an Even Break
110

Re: Howto read line from a stream

[sjw]

On Jun 3, 4:49 pm, Tomek Wałkuski <tomek.walku...@gmail.com> wrote:
> On 1 Cze, 02:05, "Jeffrey R. Carter"<spam.jrcarter....@nospam.acm.org> wrote:
> > I might do something like
>
> > function Get_Line (Channel : in Stream_Access) return String is
> >     Char : Character;
> > begin -- Get_Line
> >     Character'Read (Channel, Char);
>
> >     if Char = Ada.Characters.Latin_1.LF then
> >        return String'(1 => Char);
> >     else
> >        return Char & Get_Line (Channel);
> >     end if;
> > end Get_Line;
>
> How to measure which solution (mine or yours) is faster?

Under many modern OSs Ada.Calendar.Clock will give you microsecond
accuracy.
But if you are dealing with character i/o I hardly think it matters!
Jeffrey's solution may take quite a bit of stack, if that's an issue.

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Wed, 3 Jun 2009 12:07:16 -0700 (PDT), sjw wrote:

> But if you are dealing with character i/o I hardly think it matters!
> Jeffrey's solution may take quite a bit of stack, if that's an issue.

You terribly underestimate the huge "progress" of idiotic protocols,
beginning with XML. The hardware protocol I am presently dealing with has
configuration files of 50K source lines long. The files are *text*,
"human-readable". Such a file merely describes how to communicate a device
with a dozen of physical variables. Speed does matter!

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Georg Bauhaus]

Dmitry A. Kazakov wrote:
> On Wed, 3 Jun 2009 12:07:16 -0700 (PDT), sjw wrote:
> 
>> But if you are dealing with character i/o I hardly think it matters!
>> Jeffrey's solution may take quite a bit of stack, if that's an issue.
> 
> You terribly underestimate the huge "progress" of idiotic protocols,
> beginning with XML.

Let's say, idiotic uses of XML. If the job is downloading
ready made configuration data to some hardware device, then
I'd find building an XML parser into the device as questionable
as doing the same for some Ada-like syntax. JSON is only slightly
better.

But FWIW, validated XML is easily and realiably transformed
into a packed, hardware-friendly, lossles, valid, efficient
configuration stream, tailored to the hardware's needs.

XML is for MOM.  Let there be middleware.

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Wed, 03 Jun 2009 21:43:05 +0200, Georg Bauhaus wrote:

> Dmitry A. Kazakov wrote:
>> On Wed, 3 Jun 2009 12:07:16 -0700 (PDT), sjw wrote:
>> 
>>> But if you are dealing with character i/o I hardly think it matters!
>>> Jeffrey's solution may take quite a bit of stack, if that's an issue.
>> 
>> You terribly underestimate the huge "progress" of idiotic protocols,
>> beginning with XML.
> 
> Let's say, idiotic uses of XML.

Which are all uses of, since XML is totally useless, unsuitable both for
humans and for computers.

> If the job is downloading
> ready made configuration data to some hardware device, then
> I'd find building an XML parser into the device as questionable
> as doing the same for some Ada-like syntax. JSON is only slightly
> better.

Because the concept of "downloading ready made configuration data to some
hardware device" is rubbish. Remember the last time you downloaded
configuration data  into your ballpoint pen?

XML is used by people incapable to design a communication protocol. Last
time it was guys who described the device internal memory layout in an XML
file and suggested to directly read/write into the device memory according
to this "configuration"! The device is now widely used commercially.

> But FWIW, validated XML is easily and realiably transformed
> into a packed, hardware-friendly, lossles, valid, efficient
> configuration stream, tailored to the hardware's needs.

Why should I convert anything? Who does write it? Who does read it? It is a
cart before the horse. They use XML just because they want to. This is the
only reason, which is irrational.

> XML is for MOM.  Let there be middleware.

Middleware needs no XML. (XML was thought as a middleware by people who do
not really understand the role and the requirements put on a middleware.
Otherwise, they would immediately see that XML fulfills nether of.)

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Maciej Sobczak]

On 3 Cze, 20:04, "Jeffrey R. Carter"
<spam.jrcarter....@nospam.acm.org> wrote:

> I don't really care which one is faster. The question is whether they meet the
> timing requirements for the application.

The typical requirement is that your application must be faster then
the competition, otherwise you go out of business.
Do not assume that everybody is coding against the specification which
is given up-front. This might be true when doing contract work (ie.
responding to demand), but not when pushing a new product on the
market (ie. creating a demand).

In any case (performance aside), I find the idea of linear recursion
on each single character in the string a bit... bizarre, especially in
the context of the *externally* provided content.
The memory requirements for your version are also not easy to
establish in any intuitive way - the concatenation of return values is
probably performed on the heap and at the same time control flow
pushes down the stack. Yuck! :-)

--
Maciej Sobczak * www.msobczak.com * www.inspirel.com

Database Access Library for Ada: www.inspirel.com/soci-ada

Re: Howto read line from a stream

[Georg Bauhaus]

Dmitry A. Kazakov wrote:

> Why should I convert anything?

To get data from outside input, reshaped as needed.
(STEP 3)

> Who does write it?

Admins, operators, ...  They use their preferred, but customized
to your grammar, XML editor that (a) lets them write consistent
configurations and no others and (b) uses XML behind the
scenes only.   (Much better than information nicely
scattered in some Excel sheet.  Better than some
proprietary system dependent GUI for config making
as these seem to have a limited life time built in.)
(STEP 1)

> Who does read it?

The middleware.  Your middleware program will process
the XML data produced by outsiders during STEP 1 above.
Wouldn't this processing step resemble an integration
build with regard to the state of compilation
units (XML documents) where a compiler (validating XML parser)
has seen the units?
(STEP 2)

STEP 4:
You have valid data ready for consumption as is...

Otherwise, there is a problem with input sent to you,
and caused by something outside your responsibility.

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Thu, 04 Jun 2009 00:09:40 +0200, Georg Bauhaus wrote:

> Dmitry A. Kazakov wrote:
> 
>> Why should I convert anything?
> 
> To get data from outside input, reshaped as needed.
> (STEP 3)

I don't see how getting data is related to conversion of. (Unless you would
claim that XML does some physical measurements...) 

>> Who does write it?
> 
> Admins, operators, ...  They use their preferred, but customized
> to your grammar, XML editor that (a) lets them write consistent
> configurations and no others and (b) uses XML behind the
> scenes only.   (Much better than information nicely
> scattered in some Excel sheet.  Better than some
> proprietary system dependent GUI for config making
> as these seem to have a limited life time built in.)
> (STEP 1)

Do you plan an admin position for your ball-point pen? You must be a very
wealthy man...

The point is, either the configuration is trivial and you don't need any,
or else it is non-trivial and then a representation of it as a tree does
not work.

XML is rooted in dark ages of computing, from the times when people didn't
really understand that there is no data. Because data without behavior is
noise. You cannot handle complexity merely by providing tree, or for that
matter relational views (another end of the "axis of evil" is RDBs). It
does not add any value.

>> Who does read it?
> 
> The middleware.  Your middleware program will process
> the XML data produced by outsiders during STEP 1 above.
> Wouldn't this processing step resemble an integration
> build with regard to the state of compilation
> units (XML documents) where a compiler (validating XML parser)
> has seen the units?
> (STEP 2)

My middleware binds publishers and subscribers without XML.

> STEP 4:
> You have valid data ready for consumption as is...
> 
> Otherwise, there is a problem with input sent to you,
> and caused by something outside your responsibility.

If input is connected, it cannot be outside my responsibility. Remember the
discussion about preconditions and contracts? In a running program there is
no such case as "outside someone's responsibility", otherwise it is a bug.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Jean-Pierre Rosen]

Maciej Sobczak a �crit :

> The typical requirement is that your application must be faster then
> the competition, otherwise you go out of business.
If this were true, Linux would have wiped out Windows...

(Sorry, couldn't resist)
-- 
---------------------------------------------------------
           J-P. Rosen (rosen@adalog.fr)
Visit Adalog's web site at http://www.adalog.fr

Re: Howto read line from a stream

[Ludovic Brenta]

Jean-Pierre Rosen wrote on comp.lang.ada:
> Maciej Sobczak a écrit :
> > The typical requirement is that your application must be faster then
> > the competition, otherwise you go out of business.
>
> If this were true, Linux would have wiped out Windows...
>
> (Sorry, couldn't resist)

No, I think MS-DOS was faster still.

(couldn't resist either).

--
Ludovic Brenta.

Re: Howto read line from a stream

[Georg Bauhaus]

Dmitry A. Kazakov schrieb:

> Do you plan an admin position for your ball-point pen? You must be a very
> wealthy man...

Reminds me that I need to get one or two ballpen refills.

> The point is, either the configuration is trivial and you don't need any,
> or else it is non-trivial and then a representation of it as a tree does
> not work.

What would be the reason that trees don't work?
I notice that alternative formats have tree structure, such as

Name.Space.This = 123
Name.Space.That = foo
Another.Thing = bar

Dot, almighty...
(Java properties files).


> XML is rooted in dark ages of computing,

The dark ages of configuration are not over, and they will
not be over as long as we are here;  remember the
protocols being written in quality C?  ;-)

> from the times when people didn't
> really understand that there is no data. Because data without behavior is
> noise.

Data and behavior can have a *separate* representation
when moving between parts of a technical process.


> It
> does not add any value.

Config data does not need to *add* value.
Just *provide* values.  Valid values.  XML validation
is a start, and can be performed anywhere.
Here in the chain XML can add value to some technical
production process.


> My middleware binds publishers and subscribers without XML.

Since you are the one who has written the
middleware (a one producer connected system?
long term maintenance contracts? cooperative
atmosphere?),
I suppose it is unlikely processing some cleverly
made outside octet format such as videotext.


> If input is connected, it cannot be outside my responsibility.

Producing data can be an outside your responsibility.
I'd expect input validation to be more likely successful
when data has been preprocessed.  This is somewhat
like a journalist would be responsible, in part, for
what he writes before the editor does his work.

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Thu, 04 Jun 2009 11:41:38 +0200, Georg Bauhaus wrote:

> Dmitry A. Kazakov schrieb:
> 
>> The point is, either the configuration is trivial and you don't need any,
>> or else it is non-trivial and then a representation of it as a tree does
>> not work.
> 
> What would be the reason that trees don't work?

Because non-trivial configuration cannot be reasonably represented by a
tree. BTW, Ada program as a configuration for some processor. Why do we use
Ada instead of XML? After all any FSA can be represented by a tree. Go to
Intel tell them that they are outdated with that "Itanic", they should go
for a native-XML processor!

> I notice that alternative formats have tree structure, such as
> 
> Name.Space.This = 123
> Name.Space.That = foo
> Another.Thing = bar
> 
> Dot, almighty...
> (Java properties files).

What about a constraint that Name.Space.This cannot be 123 when
Another.Thing is not bar?

How about:

Personal_Foto = ?
Joint_Distribution_of_Possibilities_of_baz_by_Normal_Pressure  = ?

>> XML is rooted in dark ages of computing,
> 
> The dark ages of configuration are not over, and they will
> not be over as long as we are here;  remember the
> protocols being written in quality C?  ;-)

At least they were properly documented and didn't contain 2K keywords, 2
different languages and required three compilers to deal with, because one
of the languages is a meta language describing another one. (This is a
configuration protocol I have to deal with this protocol right now. The
configuration of a device that yields 8 values is 993 lines long.)

>> from the times when people didn't
>> really understand that there is no data. Because data without behavior is
>> noise.
> 
> Data and behavior can have a *separate* representation
> when moving between parts of a technical process.

Data and behavior don't exist without each other. They cannot have separate
representations.

>> It does not add any value.
> 
> Config data does not need to *add* value.

Right, that is why I don't need config data.

> Just *provide* values.  Valid values.  XML validation
> is a start, and can be performed anywhere.

XML validation does not validate the values. It validates *itself*. A
totally useless thing, as I said.

> Here in the chain XML can add value to some technical
> production process.

I don't see how XML can add a value to the process without knowing how the
process functions and how to influence the process. XML is a "Ding an
sich", it has no technical purpose. 

>> If input is connected, it cannot be outside my responsibility.
> 
> Producing data can be an outside your responsibility.

So? It is connected. Look, producing 220V for the wall outlet is not your
responsibility. But when you put your fingers into it, you as a subscriber
will have to deal with that unfortunate situation...

> I'd expect input validation to be more likely successful
> when data has been preprocessed.

I expect that VGA pug does not fit to the wall outlet. You propose that all
plugs were same, supplied with instructions for "preprocessing" of 220V AC
into VGA signal. I am curios how to preprocess off-line something that
exists on-line.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Georg Bauhaus]

Dmitry A. Kazakov schrieb:
> On Thu, 04 Jun 2009 11:41:38 +0200, Georg Bauhaus wrote:
> 
>> Dmitry A. Kazakov schrieb:
>>
>>> The point is, either the configuration is trivial and you don't need any,
>>> or else it is non-trivial and then a representation of it as a tree does
>>> not work.
>> What would be the reason that trees don't work?
> 
> Because non-trivial configuration cannot be reasonably represented by a
> tree. BTW, Ada program as a configuration for some processor. Why do we use
> Ada instead of XML?

*We* use Ada instead of XML, but *we* are the front end in program
production, wheras an Ada compiler's middleware does *not* use Ada
as a data representation format.  To the best of my knowledge,
it uses a format that works well with compilers: lists or trees!
XML uses graphs BTW.

>> I notice that alternative formats have tree structure, such as
>>
>> Name.Space.This = 123
>> Name.Space.That = foo
>> Another.Thing = bar

> What about a constraint that Name.Space.This cannot be 123 when
> Another.Thing is not bar?

I was referring to XML editting customized to leading to consistent
and only to consistent configurations.  I think the last time
they tried to capture much more than syntax with the help of a
grammar was Algol 68.



>>> XML is rooted in dark ages of computing,
>> The dark ages of configuration are not over, and they will
>> not be over as long as we are here;  remember the
>> protocols being written in quality C?  ;-)
> 
> At least they were properly documented

Every proper XML grammar is documented.  In addition, there
are standard ways of documenting the grammar.

On the other hand, programmer behavior cannot be caused by XML.
From what I learn, a negative attitude towards XML is frequently
articulated by programmers who know little about XML, about grammars,
or about a diligent selection of names and whose impression
of XML is largely based on XML spit. (That's internal data
structures, as offered by some persistence/serialization
frameworks that, by default, just frame every component
with its name... Again, not the fault of XML, the cause
is elsewhere.)

> and didn't contain 2K keywords,

As does a good XML grammar.


> (The
> configuration of a device that yields 8 values is 993 lines long.)

I don't know how a ratio of 993:8 is representative of the
technical properties of XML's grammars.  It might be
representative of get-the-job-out-the-door management
strategies.


>>> from the times when people didn't
>>> really understand that there is no data. Because data without behavior is
>>> noise.
>> Data and behavior can have a *separate* representation
>> when moving between parts of a technical process.
> 
> Data and behavior don't exist without each other. They cannot have separate
> representations.

They can be represented as (data, behavior) or, without
wanting to imply too much,
> 
>>> It does not add any value.
>> Config data does not need to *add* value.
> 
> Right, that is why I don't need config data.
> 
>> Just *provide* values.  Valid values.  XML validation
>> is a start, and can be performed anywhere.
> 
> XML validation does not validate the values. It validates *itself*.

Self-validation is not sensible.  How could it.  XML validation
performs language defined checks.  Nothing more, nothing less.
XML  can be used in integration.  When some input in whatever
format is lifted over some wall (middleware), you need programs
that deal with the delivery.  Proper packaging helps with
the transport.


>> Here in the chain XML can add value to some technical
>> production process.
> 
> I don't see how XML can add a value to the process without knowing how the
> process functions and how to influence the process. XML is a "Ding an
> sich", it has no technical purpose.

XML has rules.  It does not magically provide the rules of
a specific technical process. Ad hoc example:

<!ELEMENT  Flow  (Pipes | Wires) >

Either pipes for water, or wires for electric current,
but not both!  Water through wires? An error that can
be prevented grammatically.

<!ELEMENT  Wires
   (content model of electrical config) >

<!ELEMENT  Pipes
   (content model of water config) >

Naturally, the logic of an entire application cannot
be the same as some XML document.
Your remaining responsibility here is to mention syntax errors
to the other end.  Your problem becomes their problem.

Re: Howto read line from a stream

[Maciej Sobczak]

On 4 Cze, 10:56, Jean-Pierre Rosen <ro...@adalog.fr> wrote:

> > The typical requirement is that your application must be faster then
> > the competition, otherwise you go out of business.
>
> If this were true, Linux would have wiped out Windows...
>
> (Sorry, couldn't resist)

I understand the joke, but unfortunately it does not work for this
case. One of these two products gained the market first, so the rest
is really more about social engineering that genuine technical
competition.
In other words, being faster is not enough for Linux to wipe out
Windows, although certainly if it was slower than Windows, then you
might forget about it altogether.

And, to make it a bit more interesting, I have recently run a small
interprocess communication (based of local TCP/IP) benchmark in two
environments:
- Mac OS
- Windows XP in the virtual machine running on the same Mac OS

You might think that the second environment was obviously in a
disadvantage.

To my big surprise, the second test was repeatably and consistently
faster. Not by a large margin, but measurably (several percent). Of
course Mac OS is not Linux, so I do not want to conclude anything
about Linux this way, but certainly I would not be terribly shocked to
see something similar happening on Linux as well.
Interesting, isn't it?

--
Maciej Sobczak * www.msobczak.com * www.inspirel.com

Database Access Library for Ada: www.inspirel.com/soci-ada

Re: Howto read line from a stream

[Jean-Pierre Rosen]

Maciej Sobczak a �crit :

> And, to make it a bit more interesting, I have recently run a small
> interprocess communication (based of local TCP/IP) benchmark in two
> environments:
> - Mac OS
> - Windows XP in the virtual machine running on the same Mac OS
> 
> You might think that the second environment was obviously in a
> disadvantage.
> 
> To my big surprise, the second test was repeatably and consistently
> faster. Not by a large margin, but measurably (several percent). Of
> course Mac OS is not Linux, so I do not want to conclude anything
> about Linux this way, but certainly I would not be terribly shocked to
> see something similar happening on Linux as well.
> Interesting, isn't it?
> 
The only constant I have seen about efficiency is that when you do the
measurement, you are always surprised by the result...
-- 
---------------------------------------------------------
           J-P. Rosen (rosen@adalog.fr)
Visit Adalog's web site at http://www.adalog.fr

Re: Howto read line from a stream

[andrew]

Just a random thought here:
If the read a Character, increase size of buffer, append the
Character, free old buffer cycle is "a problem" then why not read a
byte and write (append) it to a file?  Then, when there is no more
data to read you could read the entire line from the file in one call.

Again, just a thought.

Andrew

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Thu, 4 Jun 2009 06:05:25 -0700 (PDT), Maciej Sobczak wrote:

> And, to make it a bit more interesting, I have recently run a small
> interprocess communication (based of local TCP/IP) benchmark in two
> environments:
> - Mac OS
> - Windows XP in the virtual machine running on the same Mac OS
> 
> You might think that the second environment was obviously in a
> disadvantage.
> 
> To my big surprise, the second test was repeatably and consistently
> faster. Not by a large margin, but measurably (several percent).

Our measurements shown that Windows sockets (TCP/IP and PGM) function quite
good. Who could expect that from MS? (:-))

> Of course Mac OS is not Linux, so I do not want to conclude anything
> about Linux this way, but certainly I would not be terribly shocked to
> see something similar happening on Linux as well.
> Interesting, isn't it?

Rather sad.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Thu, 04 Jun 2009 14:14:33 +0200, Georg Bauhaus wrote:

> Dmitry A. Kazakov schrieb:
>> On Thu, 04 Jun 2009 11:41:38 +0200, Georg Bauhaus wrote:
>> 
>>> Dmitry A. Kazakov schrieb:
>>>
>>>> The point is, either the configuration is trivial and you don't need any,
>>>> or else it is non-trivial and then a representation of it as a tree does
>>>> not work.
>>> What would be the reason that trees don't work?
>> 
>> Because non-trivial configuration cannot be reasonably represented by a
>> tree. BTW, Ada program as a configuration for some processor. Why do we use
>> Ada instead of XML?
> 
> *We* use Ada instead of XML, but *we* are the front end in program
> production, wheras an Ada compiler's middleware does *not* use Ada
> as a data representation format.

You mean object, library and executable files?

> To the best of my knowledge,
> it uses a format that works well with compilers: lists or trees!
> XML uses graphs BTW.

So, down with ELF let it be XML!?

>>> I notice that alternative formats have tree structure, such as
>>>
>>> Name.Space.This = 123
>>> Name.Space.That = foo
>>> Another.Thing = bar
> 
>> What about a constraint that Name.Space.This cannot be 123 when
>> Another.Thing is not bar?
> 
> I was referring to XML editting customized to leading to consistent
> and only to consistent configurations.

I don't know what "customized XML editing" is. But since it is obviously
not XML, then I don't see how XML helps the purpose of "customization" and
"editing" of itself. A trivial constraint like above cannot be described in
XML. What else need to be said?

>>>> XML is rooted in dark ages of computing,
>>> The dark ages of configuration are not over, and they will
>>> not be over as long as we are here;  remember the
>>> protocols being written in quality C?  ;-)
>> 
>> At least they were properly documented
> 
> Every proper XML grammar is documented.  In addition, there
> are standard ways of documenting the grammar.

Sorry, but XML grammar is an equivalent to, say, in a byte-oriented stream
protocol, to the sentence "the stream consists of bytes." What a lavish
piece of documentation!

Look, I don't need any grammars in a reasonable designed protocol.

>> (The
>> configuration of a device that yields 8 values is 993 lines long.)
> 
> I don't know how a ratio of 993:8 is representative of the
> technical properties of XML's grammars.  It might be
> representative of get-the-job-out-the-door management
> strategies.

It shows where it goes.

>>> Just *provide* values.  Valid values.  XML validation
>>> is a start, and can be performed anywhere.
>> 
>> XML validation does not validate the values. It validates *itself*.
> 
> Self-validation is not sensible.  How could it.  XML validation
> performs language defined checks.  Nothing more, nothing less.

Great. This is the reason why I should take this language? Because there
exist a validator? I am impressed!

> XML  can be used in integration.  When some input in whatever
> format is lifted over some wall (middleware), you need programs
> that deal with the delivery.  Proper packaging helps with
> the transport.

I don't see how XML helps me to deal with "whatever format" of an input.
Say I have a temperature sensor connected via a CAN bus (... encoding
description follows...). What exactly can XML do with that?

>>> Here in the chain XML can add value to some technical
>>> production process.
>> 
>> I don't see how XML can add a value to the process without knowing how the
>> process functions and how to influence the process. XML is a "Ding an
>> sich", it has no technical purpose.
> 
> XML has rules.  It does not magically provide the rules of
> a specific technical process.

What is the purpose of XML as a language here. Can you point out what does
XML describe except than its own gadgets?

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Howto read line from a stream

[Georg Bauhaus]

Dmitry A. Kazakov schrieb:

>>> Because non-trivial configuration cannot be reasonably represented by a
>>> tree. BTW, Ada program as a configuration for some processor. Why do we use
>>> Ada instead of XML?
>> *We* use Ada instead of XML, but *we* are the front end in program
>> production, wheras an Ada compiler's middleware does *not* use Ada
>> as a data representation format.
> 
> You mean object, library and executable files?

No, I mean ASTs, parse lists, parse trees, possibly annotated,
that are handed from stage to stage.

>> To the best of my knowledge,
>> it uses a format that works well with compilers: lists or trees!
>> XML uses graphs BTW.
> 
> So, down with ELF let it be XML!?

ELF refers to output of a compiler, not its middle end...

>>>> I notice that alternative formats have tree structure, such as
>>>>
>>>> Name.Space.This = 123
>>>> Name.Space.That = foo
>>>> Another.Thing = bar
>>> What about a constraint that Name.Space.This cannot be 123 when
>>> Another.Thing is not bar?
>> I was referring to XML editting customized to leading to consistent
>> and only to consistent configurations.
> 
> I don't know what "customized XML editing" is.

First, the basic structure of a configuration data item follows
from the productions of your XML grammar.  XML editing
equipment (including: ubuquituous office programs) can be
programmed to include additional type checks, constraint
checks, etc.  They do not stop human checks in addition
to the XML syntax checks and validity checks that are for freed.


> A trivial constraint like above cannot be described in
> XML. What else need to be said?

XML is for MOM etc, not a programmming language, not a
logic language.  The additional logic goes into (a) the
customization (b) the heads of the config makers and
(c) the input checking routines of your middleware.


>>>>> XML is rooted in dark ages of computing,
>>>> The dark ages of configuration are not over, and they will
>>>> not be over as long as we are here;  remember the
>>>> protocols being written in quality C?  ;-)
>>> At least they were properly documented
>> Every proper XML grammar is documented.  In addition, there
>> are standard ways of documenting the grammar.
> 
> Sorry, but XML grammar is an equivalent to, say, in a byte-oriented stream
> protocol, to the sentence "the stream consists of bytes." What a lavish
> piece of documentation!

I don't understand this comment, it sure does not address
Relax NG documentation, for example, or content models.
NOTE: I should again emphasize that the final configuration
data is the result of a programmatical transformation
of XML documents. XML was not intended for fully controlled,
maximmaly efficient internal data streams.

> Look, I don't need any grammars in a reasonable designed protocol.

Your protocols do not even place any order on items?
There is not even a regular structure in a reasonably designed
protocol?


>>>> Just *provide* values.  Valid values.  XML validation
>>>> is a start, and can be performed anywhere.
>>> XML validation does not validate the values. It validates *itself*.
>> Self-validation is not sensible.  How could it.  XML validation
>> performs language defined checks.  Nothing more, nothing less.
> 
> Great. This is the reason why I should take this language? Because there
> exist a validator? I am impressed!

XML validation is a well tested wheel.  Available everywhere.
If you need additional logic when reading input from the
uncontrolled outside world, you can build on top of XML
infrastructure.  Consider this:

CUSTOMER:  Here is a Word 2007 document describing the
system configuration in nice, if somewhat ambigous prose,
we are open to discussion.
We cannot predict the order of items, BTW.

versus

CUSTOMER:  Here is an XML document that follows the grammar
you have sent us.  The root element is ... and we will leave
it up to you to fill in element ... with what you see fit.

For example, it might be "this end"s job to set up a system
such that "the other end" can supply a great-product-archive,
to be run.
All required software is installed then, because that is what
was requested in the XML description of the system, and the
description was programmatically connected to the systems
software configuration machinery. (parsed, type checked,
...)


>> XML  can be used in integration.  When some input in whatever
>> format is lifted over some wall (middleware), you need programs
>> that deal with the delivery.  Proper packaging helps with
>> the transport.
> 
> I don't see how XML helps me to deal with "whatever format" of an input.
> Say I have a temperature sensor connected via a CAN bus (... encoding
> description follows...). What exactly can XML do with that?

Uhm, is sensor data what you consider either message oriented middleware
or else configuration data?  I had thought of this:

   --> XML description of system setup
     --> input checking
       --> transformed description of system setup
         --> download system settings

Another:

   --> XML description of stock exchange items (excl. RT prices)
     --> input checking
       --> transformed description of stock exchange items
         --> stored in internal data base

The idea is that someone from the other side of the middleware
border never needs to look at the transformed description.
The latter is internal.  They also need not expose their internal
data structures. Instead, they produce a data stream. The
stream is a valid XML instance of some agreed upon XML
grammar.


> Can you point out what does
> XML describe except than its own gadgets?

XML documents use the vocabulary that you define in a grammar.

Re: Howto read line from a stream

[Ludovic Brenta]

Jean-Pierre Rosen wrote on comp.lang.ada:
> The only constant I have seen about efficiency is that when you do the
> measurement, you are always surprised by the result...

That's a good constant.  If we weren't at least occasionally surprised
by the results, we might as well save ourselves the trouble of
measuring :)

--
Ludovic Brenta.

Re: Howto read line from a stream

[Dmitry A. Kazakov]

On Thu, 04 Jun 2009 18:33:22 +0200, Georg Bauhaus wrote:

> Dmitry A. Kazakov schrieb:
> 
>>>> Because non-trivial configuration cannot be reasonably represented by a
>>>> tree. BTW, Ada program as a configuration for some processor. Why do we use
>>>> Ada instead of XML?
>>> *We* use Ada instead of XML, but *we* are the front end in program
>>> production, wheras an Ada compiler's middleware does *not* use Ada
>>> as a data representation format.
>> 
>> You mean object, library and executable files?
> 
> No, I mean ASTs, parse lists, parse trees, possibly annotated,
> that are handed from stage to stage.

I see, that was a joke... (:-))

>> A trivial constraint like above cannot be described in
>> XML. What else need to be said?
> 
> XML is for MOM etc, not a programmming language, not a
> logic language.

Yes, I know what XML is not. (:-))

> The additional logic goes into (a) the
> customization (b) the heads of the config makers and
> (c) the input checking routines of your middleware.

So XML does not solve the problem you proposed it as a solution for.

>>>>>> XML is rooted in dark ages of computing,
>>>>> The dark ages of configuration are not over, and they will
>>>>> not be over as long as we are here;  remember the
>>>>> protocols being written in quality C?  ;-)
>>>> At least they were properly documented
>>> Every proper XML grammar is documented.  In addition, there
>>> are standard ways of documenting the grammar.
>> 
>> Sorry, but XML grammar is an equivalent to, say, in a byte-oriented stream
>> protocol, to the sentence "the stream consists of bytes." What a lavish
>> piece of documentation!
> 
> I don't understand this comment, it sure does not address
> Relax NG documentation, for example, or content models.

[...]
>> Look, I don't need any grammars in a reasonable designed protocol.
> 
> Your protocols do not even place any order on items?

Do you call it grammar? OK, in some sense it is. But there is no need to
document it. For example, there certainly is a grammar of the text files
used for Ada (C, C++ etc) programs. It describes that a character #2
follows a character #1. It is a very impressive grammar...

>>>>> Just *provide* values.  Valid values.  XML validation
>>>>> is a start, and can be performed anywhere.
>>>> XML validation does not validate the values. It validates *itself*.
>>> Self-validation is not sensible.  How could it.  XML validation
>>> performs language defined checks.  Nothing more, nothing less.
>> 
>> Great. This is the reason why I should take this language? Because there
>> exist a validator? I am impressed!
> 
> XML validation is a well tested wheel. Available everywhere.
> If you need additional logic when reading input from the
> uncontrolled outside world, you can build on top of XML
> infrastructure.

If everything I need must be built on top of XML, I prefer to build it
rather on the [solid] ground.

>>> XML  can be used in integration.  When some input in whatever
>>> format is lifted over some wall (middleware), you need programs
>>> that deal with the delivery.  Proper packaging helps with
>>> the transport.
>> 
>> I don't see how XML helps me to deal with "whatever format" of an input.
>> Say I have a temperature sensor connected via a CAN bus (... encoding
>> description follows...). What exactly can XML do with that?
> 
> Uhm, is sensor data what you consider either message oriented middleware
> or else configuration data?

To put it clear I don't see how XML can describe this so, that a COTS XML
compiler (parser) would produce an executable/interpretable code.

If I describe sensor in Ada language, I can take a compiler, translate the
description (an Ada program) into an executable/interpretable code. Where
to buy such XML compiler? What is the target of XML.

I know that XML provides an infrastructure. The point is that I don't need
it, since it does not solve any concrete problem. The only problems it
solves (like validation) are ones inflicted by this infrastructure itself.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de