Sending Variable Length Messages with GNAT.Sockets

Sending Variable Length Messages with GNAT.Sockets

[markp]

I am currently using GNAT.Sockets to send data via streams. A snippet
of my code is is as follows:

.
.
My_Stream_Access := GNAT.Sockets.Stream(Socket => Socket);
.
.
Msg_Types.Msg_Header_Type'Write(
   My_Stream_Access,
   Msg_Type);

This code works fine. It appears GNAT.Sockets works off a message type
in this manner.

My question is: What if the data type "Msg_Type" was a variable length
message? Is there a way to send only "x" bytes for Msg_Type, and not
the entire thing?

Thanks,

Mark

Re: Sending Variable Length Messages with GNAT.Sockets

[Adrien Plisson]

markp wrote:
> My question is: What if the data type "Msg_Type" was a variable length
> message? Is there a way to send only "x" bytes for Msg_Type, and not
> the entire thing?

it depends on your requirements.

if your receiver is an Ada software, use 'Output instead of 'Write, 
and there will be no problems.

in any case, you can use 'Write, but you will have no way to tell the 
size of the message from the stream, unless the message itself 
contains its length (a pretty common case). reading variable length 
messages with 'Read can become really difficult.

now choose the way you implement your variable length message. there 
are three kind of variable length types:
- variant records with a discriminant
- arrays
- tagged types
each have their pros and their cons...

also, i can think of no way to send "x" bytes of a type and not the 
entire thing, unless writing your own stream input and output procedures.

-- 
rien

Re: Sending Variable Length Messages with GNAT.Sockets

[markp]

Thank you for your quick reply. Our target platforms are not Ada, so we
chose to use the write. Could you tell me some place I could look to
research how to write my own stream input/output routines? I could pass
a byte count in with the message to be sent, but the stream calls seem
to only take a type, not a length. For example, in our application
code, we have a data type that can hold up an array of 512 entries, but
we might only want to send the first 32, for example. We have built
into the message header how many bytes there are, but Ada would know
nothing about that. We are used to the traditional "address, byte
count" wat of dealing with sockets

Any help you could provide would be greatly appreciated.

Thank you very much,

Mark.

Re: Sending Variable Length Messages with GNAT.Sockets

[Jeff C]

Adrien Plisson wrote:
> markp wrote:
> 
>> My question is: What if the data type "Msg_Type" was a variable length
>> message? Is there a way to send only "x" bytes for Msg_Type, and not
>> the entire thing?
> 
> 
> it depends on your requirements.
> 
> if your receiver is an Ada software, use 'Output instead of 'Write, and 
> there will be no problems.
> 
> in any case, you can use 'Write, but you will have no way to tell the 
> size of the message from the stream, unless the message itself contains 
> its length (a pretty common case). reading variable length messages with 
> 'Read can become really difficult.
> 
> now choose the way you implement your variable length message. there are 
> three kind of variable length types:
> - variant records with a discriminant
> - arrays
> - tagged types
> each have their pros and their cons...
> 
> also, i can think of no way to send "x" bytes of a type and not the 
> entire thing, unless writing your own stream input and output procedures.
> 

Note that the last statement is almost written as if this is a bad 
thing. (And I suspect someone that tried it for the first time might 
feel that way)...But, there is really nothing wrong with writing your 
own input and output procedures. It is mildly painful once and then the 
rest of the code ends up pretty clean.

Re: Sending Variable Length Messages with GNAT.Sockets

[markp]

Ok,

You'll have to excuse my ignorance, but I have very little experience
with Streams. Could you point me in the right direction?

Thanks

Re: Sending Variable Length Messages with GNAT.Sockets

[Adrien Plisson]

markp wrote:
> Thank you for your quick reply. Our target platforms are not Ada, so we
> chose to use the write. Could you tell me some place I could look to
> research how to write my own stream input/output routines? 

i don't know of any place where you can find examples of custom 
read/write. you may try http://www.adapower.com/ or other ada related 
example enabled website. i think the first place is understanding 
ARM95 13.13 and all its subsections. we also discussed it a bit in 
"TCP/IP Sockets with GNAT.Sockets" on comp.lang.ada last week.

basically, you declare one/both of the following procedure:

procedure Msg_Write(
     Stream : access Ada.Streams.Root_Stream_Type'Class;
     Message : in Msg_Type);
for Msg_Type'Write use Msg_Write;

procedure Msg_Read(
     Stream : access Ada.Streams.Root_Stream_Type'Class;
     Message : out Msg_Type);
for Msg_Type'Read use Msg_Read;

(beware of the freezing rule which can lead to the unclear messages:
no more representation items for type "..." defined at line ...
representation item appears too late)


> I could pass
> a byte count in with the message to be sent, but the stream calls seem
> to only take a type, not a length. For example, in our application
> code, we have a data type that can hold up an array of 512 entries, but
> we might only want to send the first 32, for example. We have built
> into the message header how many bytes there are, but Ada would know
> nothing about that. We are used to the traditional "address, byte
> count" wat of dealing with sockets

when writing your own procedure to write to/read from streams, you can 
use any other stream attributes. as such, you can first read/write the 
header using standard 'Read or 'Write attribute. from this you will 
have the size in bytes. by carefully using the 'Size attribute, you 
can easily find the number of entries in your message. then, you are 
back to the fixed size case and can read or write just what is needed.

for more complicated stuffs, you can use the Read and Write procedures 
of type Ada.Streams.Root_Stream_Type'Class which allows you to 
read/write at the "Stream_Elements" level (basically the byte level).

-- 
rien

Re: Sending Variable Length Messages with GNAT.Sockets

[Adrien Plisson]

Jeff C wrote:
>> also, i can think of no way to send "x" bytes of a type and not the 
>> entire thing, unless writing your own stream input and output procedures.
>>
> 
> Note that the last statement is almost written as if this is a bad 
> thing. (And I suspect someone that tried it for the first time might 
> feel that way)...But, there is really nothing wrong with writing your 
> own input and output procedures. It is mildly painful once and then the 
> rest of the code ends up pretty clean.

sorry for not being very friendly in this statement, but it was not my 
intention to let people feel that way.

personnaly i think using well-defined custom stream attributes allows 
a more language independant use of streams, and as such a better 
interoperability between softwares. so it is clearly a good thing.

-- 
rien

Re: Sending Variable Length Messages with GNAT.Sockets

[John B. Matthews]

In article <1115637556.074009.113830@o13g2000cwo.googlegroups.com>,
 "markp" <markwork66@yahoo.com> wrote:
[...]
> Could you tell me some place I could look to
> research how to write my own stream input/output routines?
[...]
> Mark.

There's an excellent introduction in the _Ada_95_Rationale_ in section A.4.1, 
"Stream Input and Output".

<http://www.adahome.com/LRM/95/Rationale/rat95html/rat95-p3-a.html#4>

-- 
John
jmatthews at wright dot edu
www dot wright dot edu/~john.matthews/

Re: Sending Variable Length Messages with GNAT.Sockets

[markp]

Thanks John.

I still have a fundamental problem. I am working on legacy code in
which data is defined in records (or arrays of records) which are sent
in variable lengths. We use address, byte count to send the exact
number of bytes. Using Streams, it seems they type has to be of
Stream_element_Array. The only way I can see to send/receive these
"variable" messages is to first read our message header which contains
the number of bytes in the message. Then, create a Stream_Element_Array
of the exact size for the next read. Then, do an unchecked conversion
or a copy bytes routine to get the data into my data type for
processing. Does this sound correct?

Lastly, using the 'Read function on streams, am I guaranteed to block
until I receive all bytes?

Thanks,

Mark

Re: Sending Variable Length Messages with GNAT.Sockets

[Adrien Plisson]

markp wrote:
> Thanks John.
> 
> I still have a fundamental problem. I am working on legacy code in
> which data is defined in records (or arrays of records) which are sent
> in variable lengths. We use address, byte count to send the exact
> number of bytes. Using Streams, it seems they type has to be of
> Stream_element_Array. The only way I can see to send/receive these
> "variable" messages is to first read our message header which contains
> the number of bytes in the message. Then, create a Stream_Element_Array
> of the exact size for the next read. Then, do an unchecked conversion
> or a copy bytes routine to get the data into my data type for
> processing. Does this sound correct?

this sounds bof bof... a bit complicated...

since you seem still struggling on this, here is some help, based on 
wild guesses i made given your previous posts:

with Ada.Text_IO,
      Ada.Streams,
      Ada.Streams.Stream_IO;
procedure Main is
     -- let's assume you have this kind of records
     type Data_Record is record
         Some : Integer;
         Dummy : String (1..8);
         Fields : Float;
     end record;

     -- according to the description you gave in a previous post,
     -- your message type is implemented (simplified) this way
     type Data_Record_Array is array (1..512) of Data_Record;
     type Message is record
         -- the message header (very simplified)
         -- holds the number of actual records in the data
         Number_Of_Records : Natural;
         -- the message data
         Data : Data_Record_Array;
     end record;

     -- declaration of the custom stream attribute procedures
     procedure Message_Read( Stream : access 
Ada.Streams.Root_Stream_Type'Class; Msg : out Message );
     for Message'Read use Message_Read;
     procedure Message_Write( Stream : access 
Ada.Streams.Root_Stream_Type'Class; Msg : in Message );
     for Message'Write use Message_Write;

     -- implementation of Message'Read
     procedure Message_Read( Stream : access 
Ada.Streams.Root_Stream_Type'Class; Msg : out Message ) is
         Size : Natural;
     begin
         -- first read the header
         -- WARNING: on the network, the size is in bytes,
         -- not in number of records
         Natural'Read( Stream, Size );
         Msg.Number_Of_Records := Size / (Data_Record'Size / 
Ada.Streams.Stream_Element'Size);

         -- then read the data
         for I in Msg.Data'First..Msg.Data'First + Msg.Number_Of_Records 
- 1 loop
             Data_Record'Read( Stream, Msg.Data(I) );
         end loop;
     end Message_Read;

     -- implementation of Message'Write
     procedure Message_Write( Stream : access 
Ada.Streams.Root_Stream_Type'Class; Msg : in Message ) is
     begin
         -- first write the header
         -- WARNING: on the network, the size is in bytes,
         -- not in number of records
         Natural'Write( Stream, Msg.Number_Of_Records * Data_Record'Size 
/ Ada.Streams.Stream_Element'Size );

         -- then write the data
         for I in Msg.Data'First..Msg.Data'First + Msg.Number_Of_Records 
- 1 loop
             Data_Record'Write( Stream, Msg.Data(I) );
         end loop;
     end Message_Write;


     -- this is for to simplify test writing
     use Ada.Streams.Stream_IO;

begin
     -- since socket streams and file streams are identical (really, all 
streams work the same)
     -- we will use files for testing

     -- first write some stuffs to a stream
     declare
         F : File_Type;
         S : Stream_Access;
         M : Message;

     begin
         -- create a new file
         Create( F, Out_File, "test.out" );
         -- get the stream corresponding to the file
         S := Stream( F );

         -- create 5 messages
         for I in 1..5 loop
             -- fill each message with dummy data
             M.Number_Of_Records := I * 2;
             for J in 1..M.Number_Of_Records loop
                 M.Data(J) := Data_Record'(Some => J, Dummy => 
"20050509", Fields => 42.0 / Float( I ) + Float( J ));
             end loop;

             -- write the data to the stream
             Message'Write( S, M );
         end loop;

         -- close the file
         Close( F );
     end;

     -- here you can look at "test.out" and see what it contains.

     -- now read some stuffs from the stream
     declare
         F : File_Type;
         S : Stream_Access;
         M : Message;

     begin
         -- open the file
         Open( F, In_File, "test.out" );
         -- get the stream corresponding to the file
         S := Stream( F );

         -- read all messages on the stream
         loop
             Message'Read( S, M );

             -- display the content of the messages
             Ada.Text_IO.Put_Line( "Message found : " & Natural'Image( 
M.Number_Of_Records ) & " records =>" );
             for I in M.Data'First..M.Data'First + M.Number_Of_Records - 
1 loop
                 Ada.Text_IO.Put_Line( "    " & Integer'Image( 
M.Data(I).Some ) & " " & M.Data(I).Dummy & " " & Float'Image( 
M.Data(I).Fields ) );
             end loop;
         end loop;
     exception
         when End_Error =>
             -- when there are no more messages in the stream file,
             -- close it
             Close( F );
     end;
end Main;

-- 
rien

Re: Sending Variable Length Messages with GNAT.Sockets

[Simon Wright]

"markp" <markwork66@yahoo.com> writes:

> I still have a fundamental problem. I am working on legacy code in
> which data is defined in records (or arrays of records) which are sent
> in variable lengths. We use address, byte count to send the exact
> number of bytes. Using Streams, it seems they type has to be of
> Stream_element_Array. The only way I can see to send/receive these
> "variable" messages is to first read our message header which contains
> the number of bytes in the message. Then, create a Stream_Element_Array
> of the exact size for the next read. Then, do an unchecked conversion
> or a copy bytes routine to get the data into my data type for
> processing. Does this sound correct?

Sounds pretty good to me; though the compiler is going to complain
about the unchecked conversion if your data type isn't the size
specified in the header?

Assuming you have type my_array_type is array (Natural range <>) of
your record type, try something like

   read the number of bytes;
   compute the number of array elements;
   handle the error if it doesn't match exactly;
   declare
      Readable : my_array_type (1 .. number_of_array_elements);   
   begin
      my_array_type'Read (str, Readable);
      return Readable;  --  or something related
   end;

> Lastly, using the 'Read function on streams, am I guaranteed to block
> until I receive all bytes?

I believe so (unless the other end closes, or you have a non-blocking
socket, in which case I'd expect End_Error).

Re: Sending Variable Length Messages with GNAT.Sockets

[markp]

Thanks for your reply.

As we both expected, it worked fine.

Re: Sending Variable Length Messages with GNAT.Sockets

[Simon Wright]

Simon Wright <simon@pushface.org> writes:

>    declare
>       Readable : my_array_type (1 .. number_of_array_elements);   
>    begin
>       my_array_type'Read (str, Readable);

I forgot to say, you need to be careful about the components of
records if you're going to let the compiler use the default
'Read/'Write.

If the C side says

  struct rec {
    char flag;
    int count;
  };

there will typically be 3 bytes of padding between flag & count, for 8
bytes on the wire.

If the Ada side says

  type Rec is record
     Flag : Interfaces.Unsigned_8;
     Count : Interfaces.Integer_32;
  end record;
  pragma Convention (C, Rec);

the 'Write will output *5* bytes; but an unchecked conversion to
Stream_Element_Array (1 .. 8) will succeed & do The Right Thing
(endianness permitting).

Re: Sending Variable Length Messages with GNAT.Sockets

[Marc A. Criley]

Simon Wright wrote:

> If the C side says
> 
>   struct rec {
>     char flag;
>     int count;
>   };
> 
> there will typically be 3 bytes of padding between flag & count, for 8
> bytes on the wire.
> 
> If the Ada side says
> 
>   type Rec is record
>      Flag : Interfaces.Unsigned_8;
>      Count : Interfaces.Integer_32;
>   end record;
>   pragma Convention (C, Rec);
> 
> the 'Write will output *5* bytes; but an unchecked conversion to
> Stream_Element_Array (1 .. 8) will succeed & do The Right Thing
> (endianness permitting).

This is why trying to use Ada.Streams "out-of-the-box" to communicate 
between Ada applications and those written in other languages is 
problematic at best.  You pretty much have to write your own stream I/O 
routines for each type to be transferred to get the data into an 
interoperable layout, especially for structured types like those in 
Simon's example.

I've pretty much given up on doing that, only using streams for 
intra-system Ada-to-Ada communication, and doing Unchecked_Conversions 
to Stream_Element_Array and using GNAT.Sockets for all external comms.

Marc A. Criley
www.mckae.com

Re: Sending Variable Length Messages with GNAT.Sockets

[tmoran]

> This is why trying to use Ada.Streams "out-of-the-box" to communicate
> between Ada applications and those written in other languages is
> problematic at best.  You pretty much have to write your own stream I/O
> routines for each type to be transferred to get the data into an
  You wouldn't expect to use default Streams to read/write disk files to
be exchanged with arbitrary other programs.  For just that reason we
included a generic capability (like Ada.Sequential_IO's) in Claw.Sockets.
Also, a lot of internet stuff is LF-ended text, which needs to be handled
more like Text_IO than Streams.  It's rather optimistic to expect socket
IO to be *simpler* than disk IO.

Re: Sending Variable Length Messages with GNAT.Sockets

[Simon Wright]

"Marc A. Criley" <mcNOSPAM@mckae.com> writes:

> Simon Wright wrote:
>
>> If the C side says
>>   struct rec {
>>     char flag;
>>     int count;
>>   };
>> there will typically be 3 bytes of padding between flag & count, for
>> 8
>> bytes on the wire.
>> If the Ada side says
>>   type Rec is record
>>      Flag : Interfaces.Unsigned_8;
>>      Count : Interfaces.Integer_32;
>>   end record;
>>   pragma Convention (C, Rec);
>> the 'Write will output *5* bytes; but an unchecked conversion to
>> Stream_Element_Array (1 .. 8) will succeed & do The Right Thing
>> (endianness permitting).
>
> This is why trying to use Ada.Streams "out-of-the-box" to communicate
> between Ada applications and those written in other languages is
> problematic at best.  You pretty much have to write your own stream
> I/O routines for each type to be transferred to get the data into an
> interoperable layout, especially for structured types like those in
> Simon's example.
>
> I've pretty much given up on doing that, only using streams for
> intra-system Ada-to-Ada communication, and doing Unchecked_Conversions
> to Stream_Element_Array and using GNAT.Sockets for all external comms.

Our project has lots of Ada (our bit) and lots of C (the other
bits). The interfaces are defined using only 32-bit quantities (after
a packed 20-byte header which initially caused us a pile of grief,
especially since we wanted to be interoperable between PC and
PowerPC).

On our side there's quite a lot of generated code to cope with mapping
Booleans to packed arrays of bits, driven by UML tags in the models,
but once the data is in the 32-bit form we do indeed use Streams.

I suppose we could have achieved the same end by writing our own
'Read, 'Write, but that might have been a bridge too far; and this way
at least the main code generator isn't intertwingled with the network
i/o part. I can't remember whether that argument occurred to me at the
time!

Re: Sending Variable Length Messages with GNAT.Sockets

[Jacob Sparre Andersen]

Simon Wright wrote:

> On our side there's quite a lot of generated code to cope with
> mapping Booleans to packed arrays of bits, driven by UML tags in the
> models, but once the data is in the 32-bit form we do indeed use
> Streams.

Why not use representation clauses for doing those mappings?  (or is
the generated code actually mostly generated representation clauses?)

Jacob
-- 
"It ain't rocket science!"

Re: Sending Variable Length Messages with GNAT.Sockets

[Simon Wright]

Jacob Sparre Andersen <sparre@nbi.dk> writes:

> Simon Wright wrote:
>
>> On our side there's quite a lot of generated code to cope with
>> mapping Booleans to packed arrays of bits, driven by UML tags in the
>> models, but once the data is in the 32-bit form we do indeed use
>> Streams.
>
> Why not use representation clauses for doing those mappings?  (or is
> the generated code actually mostly generated representation clauses?)

Because rep clauses are endian-dependent. I know that there are
techniques for writing bit offsets in an endian-independent way, but
if humans are involved it's often easier (for both writing and
reviewing) to have an Unsigned_32 and do shifts and masks in the good
old C way (IME).

I have used the

  in the spec
     declare un-represented record

  in the body
     read raw bytes
     if big-endian then
        declare
           big-endian represented derived record
           unchecked conversion from raw bytes to represented record
        begin
           convert raw bytes to represented record
           return conversion from represented record to un-represented record
        end
     else
        the same for the little-endian representation

technique, but it's a lot of work!