Problems with Ada.Streams.Read (blocking)

Problems with Ada.Streams.Read (blocking)

[Dennis Hoppe]

Hi,

I've written a minimal example to access an ftp server (like FileZilla). 
First, let's have a look at the code snippet:

-- START
with Ada.Text_IO;
with Ada.Streams;
with GNAT.Sockets; use GNAT.Sockets;
use type Ada.Streams.Stream_Element_Count;

procedure Test is
   Client: Socket_Type;
   Address: Sock_Addr_Type;
   Channel: Stream_Access;
   Data   : Ada.Streams.Stream_Element_Array (1 .. 1);
   Offset : Ada.Streams.Stream_Element_Count;

begin
   Initialize;
   Create_Socket(Client);
   Address.Addr := Inet_Addr("127.0.0.1");
   Address.Port := 21;
   Connect_Socket (Client,Address);
   Channel := Stream(Client);

   loop  -- reads in the welcome message
     Ada.Streams.Read (Channel.all, Data(1..1), Offset);
     exit when Offset = 0;
     -- alternative: exit when Offset /= Data'Last
     for I in 1 .. Offset loop
        Ada.Text_IO.Put (Character'Val (Data (I)));
     end loop;
   end loop;
end Test;
-- END

The problem is, that Ada.Streams.Read is blocking, if the end of the
stream is reached. I found many examples, that outline, that the
variable Offset will be 0, if no further elements are on the stream.
But this seems not to be the case, unfortunately.

For a ftp server/client situation, each command is terminate by <CRLF>,
so I enhanced the exit condition to:

loop
   Ada.Streams.Read (Channel.all, Data(1..2), Offset);
   exit when (Character'Val (Data(1)) = ASCII.CR and Character'Val 
(Data(2)) = ASCII.LF);
   -- code omitted
end loop;

Of course, the Stream_Element_Array is enhanced to (1..2).

This approach works very well, but some ftp commands send a messages
over several lines. I do not know in advance, how many lines I should
read in. Subsequently, Ada.Streams.Read has to be called in a loop, 
which will eventually block, again.

How can I query the stream, if new elements are ready to read?

Many thanks in advance,
   Dennis

Re: Problems with Ada.Streams.Read (blocking)

[Dmitry A. Kazakov]

On Fri, 08 Aug 2008 15:24:43 +0200, Dennis Hoppe wrote:

> This approach works very well, but some ftp commands send a messages
> over several lines. I do not know in advance, how many lines I should
> read in. Subsequently, Ada.Streams.Read has to be called in a loop, 
> which will eventually block, again.
> 
> How can I query the stream, if new elements are ready to read?

You have to implement the application layer protocol above the stream.
Stream is a flow of stream elements, nothing more. If you have a
record-oriented layer above it, that segments the stream into records, then
you define a procedure to receive one record according to the definition
of. For instance, a sequence of stream elements until CR/LF:

procedure Read_Record
   (  Channel : in out Socket_Type;
       Buffer : in out Stream_Element_Array;
      Last : out Stream_Element_Offset
   )  is
   -- Reads the stream until CR/LF filling Buffer. Last is the index of
   -- of the last element of the packet. It is Buffer'First - 1 when
   -- the record body is empty.
   Index : Stream_Element_Offset;
   Has_CR : Boolean := False;
begin
   while Index in Buffer'Range loop
      Read (Channel, Buffer (Index));
      if Has_CR and then
         Storage_Element'Pos (Buffer (Index) = Character'Pos (LF)
      then
         Last := Index - 2;
         return;
      else
         Has_CR :=
            Storage_Element'Pos (Buffer (Index) = Character'Pos (CR);
      end if;
   end loop;
   raise Data_Overrun_Error; -- Too large packet
end Read_Record;

> How can I query the stream, if new elements are ready to read?

You need not, because record-oriented protocols let you know either the
record size in advance or else provide a way to determine the record end.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Problems with Ada.Streams.Read (blocking)

[Dennis Hoppe]

Hi Dmitri,

thank you for your reply, but it does not solve my problem. You just 
encapsulate the challenge in a nicer way. The application stops
at runtime while invoking Ada.Streams.Read to often, again.

I corrected your snippet to be compatible with my test-framework.


with Ada.Text_IO;
with Ada.Streams; use Ada.Streams;
with GNAT.Sockets; use GNAT.Sockets;
use type Ada.Streams.Stream_Element_Count;
with System.Storage_Elements; use System.Storage_Elements;

procedure Test is
   Client: Socket_Type;
   Address: Sock_Addr_Type;
   Channel: Stream_Access;
   Data   : Ada.Streams.Stream_Element_Array (1 .. 2);
   Offset : Ada.Streams.Stream_Element_Count := 1;

   procedure Read_Record (
       Channel : in out Stream_Access;
       Buffer  : in out Stream_Element_Array;
       Last    : out Stream_Element_Offset)  is
     Index : Stream_Element_Offset := 1;
     Has_CR : Boolean := False;
   begin
     while Index in Buffer'Range loop
       Ada.Streams.Read (Channel.all, Buffer, Index);
       if Has_CR and then
	Stream_Element'Pos (Buffer (Index)) = Character'Pos (ASCII.LF)
       then
	Last := Index - 2;
         return;
       else
	Has_CR :=
          Stream_Element'Pos (Buffer (Index)) = Character'Pos (ASCII.CR);
       end if;
     end loop;
     raise Constraint_Error; -- Too large packet
   end Read_Record;

begin
    Initialize;
    Create_Socket(Client);
    Address.Addr := Inet_Addr("127.0.0.1");
    Address.Port := 21;
    Connect_Socket (Client, Address);
    Channel := Stream (Client);

    Read_Record (Channel, Data, Offset);

    for I in 1 .. Offset loop
      Ada.Text_IO.Put (Character'Val (Data(I)));
    end loop;
end Test;


The Ada specification mentions the following:

"The Read operation transfers stream elements from the specified stream 
to fill the array Item. Elements are transferred until Item'Length 
elements have been transferred, or until the end of the stream is 
reached. If any elements are transferred, the index of the last stream 
element transferred is returned in Last. Otherwise, Item'First - 1 is 
returned in Last. Last is less than Item'Last only if the end of the 
stream is reached."

(<http://www.adaic.org/standards/05aarm/html/AA-13-13-1.html>)


Actually, some invocation of Ada.Streams.Read (Channel.all, Buffer, 
Index) should lead to

   Index < Buffer'Last

if the end of the stream is reached. This is never the case, because 
"read" is blocking, if not all elements in Buffer could be filled up.
If Buffer is Buffer(1..1), Index went never to 0, for example.

I am now able to read line per line sent by the ftp server, but if
I only once invoke my read method to often, the whole program hangs up
due to the blocking behavior of Ada.Stream.Read.


Best regards,
   Dennis

Re: Problems with Ada.Streams.Read (blocking)

[Dmitry A. Kazakov]

On Fri, 08 Aug 2008 20:00:44 +0200, Dennis Hoppe wrote:

> Actually, some invocation of Ada.Streams.Read (Channel.all, Buffer, 
> Index) should lead to
> 
>    Index < Buffer'Last
> 
> if the end of the stream is reached.

You never reach the end of a TCP/IP stream unless the client closes the
socket, which is usually indicates an error.

The point is that you should never care about the stream end.

> I am now able to read line per line sent by the ftp server, but if
> I only once invoke my read method to often, the whole program hangs up
> due to the blocking behavior of Ada.Stream.Read.

No, it does because you incorrectly implement the protocol. Typically, you
send a request to the server and it answers with a chain of records. The
number and the content of the records is determined by the protocol. So you
can always know what to expect. Pseudo-code looks like

   -- Processing a request
   send request
   loop
      read one record
      interpret the content of
      exit when no records should follow
   end loop
   -- Ready to issue another request
   ...

You have mentioned FTP. There is a clear definition of how a FTP server
response ends:

"The client can identify the last line of the response as follows: it
begins with three ASCII digits and a space; previous lines do not. The
three digits form a code. Codes between 100 and 199 indicate marks; codes
between 200 and 399 indicate acceptance; codes between 400 and 599 indicate
rejection."
  --  http://cr.yp.to/ftp/request.html

If you correctly implement the protocol your application will never block
unless the server is not busy. And you have to read out all response
otherwise the server will be block. So both parties know the states of each
other.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Problems with Ada.Streams.Read (blocking)

[Dennis Hoppe]

Dmitry A. Kazakov wrote:
> You never reach the end of a TCP/IP stream unless the client closes the
> socket, which is usually indicates an error.
> 
> The point is that you should never care about the stream end.

Seems, that I am not as familiar with streams as I should ;)


> You have mentioned FTP. There is a clear definition of how a FTP server
> response ends:
> 
> "The client can identify the last line of the response as follows: it
> begins with three ASCII digits and a space; previous lines do not. The
> three digits form a code. Codes between 100 and 199 indicate marks; codes
> between 200 and 399 indicate acceptance; codes between 400 and 599 indicate
> rejection."
>   --  http://cr.yp.to/ftp/request.html
> 
> If you correctly implement the protocol your application will never block
> unless the server is not busy. And you have to read out all response
> otherwise the server will be block. So both parties know the states of each
> other.

Yeah, I am used to the status codes, but I am struggeling with the 
welcome message from ftp servers :-)  FileZilla send three or four lines 
of information, all with status code 220. By means of your weblink 
(cr.yp.to - hehe), I saw the following:

      150-This is the first line of a mark
      123-This line does not end the mark; note the hyphen
      150 This line ends the mark
      226-This is the first line of the second response
       226 This line does not end the response; note the leading space
      226 This is the last line of the response, using code 226

This solves already the problem, because indeed the series of 
consecutive commands is finished, if three digits were followed by a 
blank. Each line of the welcome message begins with 220- and only the
last one with 220_ (blank).

Ok, I will never have a blocking read call in the future ;-)

Thank you very much, you made my day!

Best regards,
   Dennis

Re: Problems with Ada.Streams.Read (blocking)

[Maciej Sobczak]

On 8 Sie, 20:51, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
wrote:

> You never reach the end of a TCP/IP stream unless the client closes the
> socket, which is usually indicates an error.

This is not true. There are lots of protocols that use the end-of-
stream as an indicator of the end of request group. In such protocols
the client is free to issue new requests whenever he likes and the
server will dutifully *wait* for each of them.

Telnet is one such protocol. HTTP is another (client can ask for the
page and its dependent files like images via a single connection).
CORBA (and virtually every other RPC-like middleware) uses it for
handling many RPC requests from a single client. And so on - the list
is endless.

> The point is that you should never care about the stream end.

The point is that end of stream indicates that the client is not
interested in interaction anymore. This is a valid protocol state.

This is different from the client's perspective, but I can imagine
protocols where the server can respond with potentially endless
answer, not necessarily continuous in time (online stock price
updates?), and finish after a timeout or some other even that is
outside of the client's control. Again, end of stream is a valid
protocol state.

> If you correctly implement the protocol your application will never block

Your application can also block if the other party blocks in the
middle of responding. You cannot fully control it.

--
Maciej Sobczak * www.msobczak.com * www.inspirel.com

Re: Problems with Ada.Streams.Read (blocking)

[Dmitry A. Kazakov]

On Fri, 8 Aug 2008 14:25:33 -0700 (PDT), Maciej Sobczak wrote:

> On 8 Sie, 20:51, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
> wrote:
> 
>> You never reach the end of a TCP/IP stream unless the client closes the
>> socket, which is usually indicates an error.
> 
> This is not true. There are lots of protocols that use the end-of-
> stream as an indicator of the end of request group.

I meant Ada stream associated with the socket. You cannot end it otherwise
than by closing the socket, if at all. I didn't look into GNAT sockets, but
it is well thinkable that it implementation never ever ends the stream and
instead of that raises Socket_Error upon an I/O on a closed socket.

> In such protocols
> the client is free to issue new requests whenever he likes and the
> server will dutifully *wait* for each of them.

No, that is full-duplex communication, which technically is not
client-server. But in this case too, end of stream is never used to bound a
request or response. It would be wasting resources. Worse than that, it
would make the protocol unusable for the cases where the connection cannot
be dropped, like serial communications, CAN bus, etc.

>> The point is that you should never care about the stream end.
> 
> The point is that end of stream indicates that the client is not
> interested in interaction anymore. This is a valid protocol state.

(OP designed a client)

> This is different from the client's perspective, but I can imagine
> protocols where the server can respond with potentially endless
> answer, not necessarily continuous in time (online stock price
> updates?), and finish after a timeout or some other even that is
> outside of the client's control. Again, end of stream is a valid
> protocol state.

Surely, the network protocol for data exchange I designed for a middleware
has exactly this design. Many other protocols used in automotive industry
are of full-duplex asynchronous nature. But end of stream is never used
there, otherwise than to indicate broken connection.

>> If you correctly implement the protocol your application will never block
> 
> Your application can also block if the other party blocks in the
> middle of responding. You cannot fully control it.

Yes, I wrote "unless server is busy." This is not blocking. Compare it with
the way Ada defines a potentially blocking operation. A non-blocking
operation is not meant instant, it is time-bounded. Delays caused by
network latencies, server load etc semantically are not lookups.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Problems with Ada.Streams.Read (blocking)

[Adam Beneschan]

On Aug 8, 6:24 am, Dennis Hoppe <dennis.ho...@hoppinet.de> wrote:
> Hi,
>
> I've written a minimal example to access an ftp server (like FileZilla).
> First, let's have a look at the code snippet:
>
> -- START
> with Ada.Text_IO;
> with Ada.Streams;
> with GNAT.Sockets; use GNAT.Sockets;
> use type Ada.Streams.Stream_Element_Count;
>
> procedure Test is
>    Client: Socket_Type;
>    Address: Sock_Addr_Type;
>    Channel: Stream_Access;
>    Data   : Ada.Streams.Stream_Element_Array (1 .. 1);
>    Offset : Ada.Streams.Stream_Element_Count;
>
> begin
>    Initialize;
>    Create_Socket(Client);
>    Address.Addr := Inet_Addr("127.0.0.1");
>    Address.Port := 21;
>    Connect_Socket (Client,Address);
>    Channel := Stream(Client);
>
>    loop  -- reads in the welcome message
>      Ada.Streams.Read (Channel.all, Data(1..1), Offset);
>      exit when Offset = 0;
>      -- alternative: exit when Offset /= Data'Last
>      for I in 1 .. Offset loop
>         Ada.Text_IO.Put (Character'Val (Data (I)));
>      end loop;
>    end loop;
> end Test;
> -- END
>
> The problem is, that Ada.Streams.Read is blocking, if the end of the
> stream is reached. I found many examples, that outline, that the
> variable Offset will be 0, if no further elements are on the stream.
> But this seems not to be the case, unfortunately.
>
> For a ftp server/client situation, each command is terminate by <CRLF>,
> so I enhanced the exit condition to:
>
> loop
>    Ada.Streams.Read (Channel.all, Data(1..2), Offset);
>    exit when (Character'Val (Data(1)) = ASCII.CR and Character'Val
> (Data(2)) = ASCII.LF);
>    -- code omitted
> end loop;
>
> Of course, the Stream_Element_Array is enhanced to (1..2).
>
> This approach works very well, but some ftp commands send a messages
> over several lines. I do not know in advance, how many lines I should
> read in. Subsequently, Ada.Streams.Read has to be called in a loop,
> which will eventually block, again.
>
> How can I query the stream, if new elements are ready to read?

I'm not familiar with GNAT.Sockets, but looking at the spec it appears
that there are a couple routines that might help: Control_Socket which
lets you specify non-blocking I/O, and Check_Selector which I think
can be used to query whether data is available, if you give it a
Timeout of zero.  Anyway, I haven't tried anything and I have no idea
whether it's appropriate for your problem, but it seems like it might
help.  If not, my apologies.

                                  -- Adam

Re: Problems with Ada.Streams.Read (blocking)

[Dennis Hoppe]

Hi Adam,


Adam Beneschan wrote:
> I'm not familiar with GNAT.Sockets, but looking at the spec it appears
> that there are a couple routines that might help: Control_Socket which
> lets you specify non-blocking I/O, and Check_Selector which I think
> can be used to query whether data is available, if you give it a
> Timeout of zero.  Anyway, I haven't tried anything and I have no idea
> whether it's appropriate for your problem, but it seems like it might
> help.  If not, my apologies.
> 
>                                   -- Adam


I tried to implement your idea:

with Ada.Text_IO;
with Ada.Streams; use Ada.Streams;
with GNAT.Sockets; use GNAT.Sockets;
use type Ada.Streams.Stream_Element_Count;

procedure Test is
    Client: Socket_Type;
    Address: Sock_Addr_Type;
    Channel: Stream_Access;
    Data   : Ada.Streams.Stream_Element_Array (1 .. 256);
    Offset : Ada.Streams.Stream_Element_Count := 1;
    Request : Request_Type;

begin
    Initialize;
    Create_Socket(Client);
    Address.Addr := Inet_Addr("127.0.0.1");
    Address.Port := 21;
    Connect_Socket (Client, Address);

    Request := (Name => Non_Blocking_IO, Enabled => True);
    Control_Socket (Client, Request);  -- make socket I/O non-blocking
    Channel := Stream (Client);

    declare
       Message : String (1 .. 256);
    begin
       String'Read (Channel, Message);
       Ada.Text_IO.Put_Line (Message);
    end;
end Test;


The code blocks now in line

   Control_Socket (Client, Request);  -- make socket I/O non-blocking

The application crashes with: Segmentation fault. *shrugging*

Maybe Create_Selector is the way to go...


Best regards,
   Dennis

Re: Problems with Ada.Streams.Read (blocking)

[anon]

Check the source files for the "Ada Terminal Emulator - version 2.3"

http://members.optusnet.com.au/~rosshigson/terminal.htm#_Source_Distribution_1

Binaries and Source can be found there.  Since, both TELNET and FTP protocols 
are related you should be able to find out how they goyt around the blocking 
concept. Or they may even created a query routine that you might be able to 
adopt for your program.

Note: In the emulator the package uses it own Sockets packages which is just 
a renamed version of the GNAT.Sockets packages.

In <g7hher$mtt$1@aioe.org>, Dennis Hoppe <dennis.hoppe@hoppinet.de> writes:
>Hi,
>
>I've written a minimal example to access an ftp server (like FileZilla). 
>First, let's have a look at the code snippet:
>
>-- START
>with Ada.Text_IO;
>with Ada.Streams;
>with GNAT.Sockets; use GNAT.Sockets;
>use type Ada.Streams.Stream_Element_Count;
>
>procedure Test is
>   Client: Socket_Type;
>   Address: Sock_Addr_Type;
>   Channel: Stream_Access;
>   Data   : Ada.Streams.Stream_Element_Array (1 .. 1);
>   Offset : Ada.Streams.Stream_Element_Count;
>
>begin
>   Initialize;
>   Create_Socket(Client);
>   Address.Addr := Inet_Addr("127.0.0.1");
>   Address.Port := 21;
>   Connect_Socket (Client,Address);
>   Channel := Stream(Client);
>
>   loop  -- reads in the welcome message
>     Ada.Streams.Read (Channel.all, Data(1..1), Offset);
>     exit when Offset = 0;
>     -- alternative: exit when Offset /= Data'Last
>     for I in 1 .. Offset loop
>        Ada.Text_IO.Put (Character'Val (Data (I)));
>     end loop;
>   end loop;
>end Test;
>-- END
>
>The problem is, that Ada.Streams.Read is blocking, if the end of the
>stream is reached. I found many examples, that outline, that the
>variable Offset will be 0, if no further elements are on the stream.
>But this seems not to be the case, unfortunately.
>
>For a ftp server/client situation, each command is terminate by <CRLF>,
>so I enhanced the exit condition to:
>
>loop
>   Ada.Streams.Read (Channel.all, Data(1..2), Offset);
>   exit when (Character'Val (Data(1)) = ASCII.CR and Character'Val 
>(Data(2)) = ASCII.LF);
>   -- code omitted
>end loop;
>
>Of course, the Stream_Element_Array is enhanced to (1..2).
>
>This approach works very well, but some ftp commands send a messages
>over several lines. I do not know in advance, how many lines I should
>read in. Subsequently, Ada.Streams.Read has to be called in a loop, 
>which will eventually block, again.
>
>How can I query the stream, if new elements are ready to read?
>
>Many thanks in advance,
>   Dennis

Re: Problems with Ada.Streams.Read (blocking)

[Paweł 'Nivertius' Płazieński]

Dennis Hoppe wrote:

> This approach works very well, but some ftp commands send a messages
> over several lines. I do not know in advance, how many lines I should
> read in. Subsequently, Ada.Streams.Read has to be called in a loop,
> which will eventually block, again.
> 
> How can I query the stream, if new elements are ready to read?

Altrought you tried using a Control_Socket to read N_Bytes_To_Read. If
there's something in the local buffer, you can read without blocking.
Probably you can wait some time and test if there's something, if not, it
won't probably be sent. This can be good or a totally bad way to do it
[random latencies on the line for example]. Just passing around.

BTW It never work for me, I tried to use it several times. ;-))
    Maybe you'll get more luck with it.

-- 
Pawe� P�azie�ski aka Nivertius
"In the end, there will be Ada, XML and gzip"