records containing variable length arrays [long]

records containing variable length arrays [long]

[Mats Karlssohn]

I'm currently wrestling with an interesting problem. I feel that this
have probably been discussed previously, but I couldn't find anything
on it neither on AdaPower nor on the cla. archives on google.

I would like to declare a record type containing two (possibly more)
arrays of variable lengths, the length of each array is another element
of the record. I also need to specify an exact representation for the
record since it is a message that I cant change the format of.

A piece of code to illustrate (please not that this is NOT compilable)

package Communication is
   -- Basic types
   type Byte is mod 2**8;
   for Byte'size use 8;

   type Word is mod 2**16;
   for Word'size use 16;


   -- Variable length buffer
   type Buffer is array(Byte range <>) of Byte;
   pragma Pack(Buffer);

   -- Declaring the variable length elements is one major headache
   -- note that at least Output_Length may legally be 0
   type Message is
      record
         Magic           : Word;
         Operation       : Word;
         Status          : Word;
         Response_Length : Byte;
         Output_Length   : Byte;
         Response_Data   : Buffer(0..(Response_Length - 1));
         Output_Data     : Buffer(0..(Output_Length   - 1));
         CRC             : Word;
      end record;

   -- the other big problem is to get the correct representation
   for Message use
      record
          Magic           at 0 range 0..15;
          Operation       at 2 range 0..15;
          Status          at 4 range 0..15;
          Response_Length at 4 range 0.. 7;
          Output_Length   at 5 range 0.. 7;
          Response_Data   at 6 range 0..(Response_Length * Byte'size - 1);
          Response_Data   at (6 + Response_Length) range 0..(Output_Length * Byte'size - 1);
          CRC             at (6 + Response_Length + Output_Length) range 0..15;
      end record;

end Communication;

The compiler (Gnat 3.13p on Linux) says:
$ gnatgcc -c -g -gnata -gnatf communication.ads
communication.ads:23:39: component "Response_Length" cannot be used before end of record declaration
communication.ads:24:39: component "Output_Length" cannot be used before end of record declaration
communication.ads:34:11: component "Response_Length" overlaps "Status" at line 33
communication.ads:35:11: component "Output_Length" overlaps "Status" at line 33
communication.ads:36:42: "Response_Length" is undefined
communication.ads:37:35: "Response_Length" is undefined
communication.ads:37:62: "Output_Length" is undefined
communication.ads:38:35: "Response_Length" is undefined
communication.ads:38:53: "Output_Length" is undefined
$

I have currently solved this by overlaying the record with a large
enough array of Byte, and then calculating what portions of that I
must use Unchecked_Conversion on to find the data I need.
This works BUT:
* It is ugly and I would like to do this as as much of a schoolexample
  that I can possibly do.
* Performance may be a bottleneck (probably not in the particular case
  above) so I would really like to avoid copying any of the elements.


Any suggestions ?

Thanks for any help!

-- 
Mats Karlssohn, developer                         mailto:mats@mida.se  
Mida Systemutveckling AB                          http://www.mida.se
Box 64, S-732 22 ARBOGA, SWEDEN
Phone: +46-(0)589-89808   Fax: +46-(0)589-89809

Re: records containing variable length arrays [long]

[Jeffrey Carter]

Mats Karlssohn wrote:
> 
> I would like to declare a record type containing two (possibly more)
> arrays of variable lengths, the length of each array is another element
> of the record. I also need to specify an exact representation for the
> record since it is a message that I cant change the format of.
> 
> A piece of code to illustrate (please not that this is NOT compilable)
> 
> package Communication is
>    -- Basic types
>    type Byte is mod 2**8;
>    for Byte'size use 8;
> 
>    type Word is mod 2**16;
>    for Word'size use 16;
> 
>    -- Variable length buffer
>    type Buffer is array(Byte range <>) of Byte;
>    pragma Pack(Buffer);
> 
>    -- Declaring the variable length elements is one major headache
>    -- note that at least Output_Length may legally be 0
>    type Message is
>       record
>          Magic           : Word;
>          Operation       : Word;
>          Status          : Word;
>          Response_Length : Byte;
>          Output_Length   : Byte;
>          Response_Data   : Buffer(0..(Response_Length - 1));
>          Output_Data     : Buffer(0..(Output_Length   - 1));
>          CRC             : Word;
>       end record;

The basic concept for what you're trying to do is a discriminated record
type:

type Bounded_String (Max_Length : Positive) is record
   Current_Length : Natural := 0;
   Value          : String (1 .. Max_Length) := (others => '*');
end record;

> 
>    -- the other big problem is to get the correct representation
>    for Message use
>       record
>           Magic           at 0 range 0..15;
>           Operation       at 2 range 0..15;
>           Status          at 4 range 0..15;
>           Response_Length at 4 range 0.. 7;
>           Output_Length   at 5 range 0.. 7;
>           Response_Data   at 6 range 0..(Response_Length * Byte'size - 1);
>           Response_Data   at (6 + Response_Length) range 0..(Output_Length * Byte'size - 1);
>           CRC             at (6 + Response_Length + Output_Length) range 0..15;
>       end record;
> 
> end Communication;

However, you have additional problems, such as overlaying your lengths,
which would be your discriminants, with Status. I presume that you
receive your message as a simple Buffer. In that case you might be able
to work something along these lines

type Message (Response_Length : Byte; Output_Length : Byte) is record
   Magic         : Word;
   Operation     : Word;
   Response_Data : Buffer (1 .. Response_Length);
   Output_Data   : Buffer (1 .. Output_Length);
   CRC           : Word;
end record;

I have specified the Buffers starting at one because Ada does not allow
an expression containing a discriminant.

for Message use record
   Magic           at 0 range 0..15;
   Operation       at 2 range 0..15;
   Response_Length at 4 range 0.. 7;
   Output_Length   at 5 range 0.. 7;
end record;
pragma Pack (Message);

We will have to ensure that the compiler will lay out the Buffers and
CRC properly, but that seems the most likely result.

Assume we have

Raw_Message : Buffer (0 .. Some_Value);

containing the raw received message. Then we can translate it by doing

Translate_Message : declare
   subtype Raw_Buffer is Buffer (Raw_Message'range);

   subtype This_Message is
      Message (Response_Length => Raw_Message (4),
               Output_Length   => Raw_Message (5) );

   function Translate is new Ada.Unchecked_Conversion (Source =>
Raw_Buffer,
                                                       Target =>
This_Message);

   Translated : [constant] This_Message := Translate (Raw_Message);
begin -- Translate_Message
   -- Reference Translated here
end Translate_Message;

Note that if Translated is constant, a good compiler will not copy the
contents of Raw_Message to create Translated.

Your only issue now is to reconstruct Status from the 2 discriminants.
This is reasonably simple and left as an exercise for the reader.

-- 
Jeffrey Carter

Re: records containing variable length arrays [long]

[Mats Karlssohn]

Jeffrey Carter wrote:
> 
> Mats Karlssohn wrote:
%<
> The basic concept for what you're trying to do is a discriminated record
> type:
> 
> type Bounded_String (Max_Length : Positive) is record
>    Current_Length : Natural := 0;
>    Value          : String (1 .. Max_Length) := (others => '*');
> end record;

Yes, that's right.

> >    -- the other big problem is to get the correct representation
> >    for Message use
> >       record
%<
> >           Status          at 4 range 0..15;
> >           Response_Length at 4 range 0.. 7;

This is a typo... "Response_Length at 5 range 0..7;" is the correct
representation statement, this also means that all the following
components move up one byte.

%<
> However, you have additional problems, such as overlaying your lengths,
> which would be your discriminants, with Status. I presume that you
> receive your message as a simple Buffer. In that case you might be able
> to work something along these lines

Yes, I receive a C pointer (converted to System.Address) and an integer
Length from a buffer manager for the shared memory where thit beast
lives. I forgot to mention that I really only neet to read this data
structure, I will NOT need to assign to it.

> type Message (Response_Length : Byte; Output_Length : Byte) is record
>    Magic         : Word;
>    Operation     : Word;
>    Response_Data : Buffer (1 .. Response_Length);
>    Output_Data   : Buffer (1 .. Output_Length);
>    CRC           : Word;
> end record;
> 
> I have specified the Buffers starting at one because Ada does not allow
> an expression containing a discriminant.

That's OK but not ideal... I can handle that in the access functions.
The spec. says that it is indexed from 0 but I'll just have to take care
of it.

The problem comes when Output_Length becomes 0, a value which it can
legally have. In that case the Output_Data buffer is 0 bytes long and
hence disapears from the message. This is kind of an almost error, not
the normal use, but still, allowed.
Any suggestions on that one ?


> for Message use record
>    Magic           at 0 range 0..15;
>    Operation       at 2 range 0..15;
>    Response_Length at 4 range 0.. 7;
>    Output_Length   at 5 range 0.. 7;
> end record;
> pragma Pack (Message);

Hmmm... yes, I think I see. I think that I havn't done my homework, I
really didn't think of trying to include the discriminants in the
representation.

> We will have to ensure that the compiler will lay out the Buffers and
> CRC properly, but that seems the most likely result.

Ahh... this seems doable, I'll look into it some more this evening.

%<
> Your only issue now is to reconstruct Status from the 2 discriminants.
> This is reasonably simple and left as an exercise for the reader.

And not an issue since I mistyped....


TNX 1e6

-- 
Mats Karlssohn, developer                         mailto:mats@mida.se  
Mida Systemutveckling AB                          http://www.mida.se
Box 64, S-732 22 ARBOGA, SWEDEN
Phone: +46-(0)589-89808   Fax: +46-(0)589-89809

Re: records containing variable length arrays [long]

[Stephen Leake]

Mats Karlssohn <mats@mida.se> writes:

> I'm currently wrestling with an interesting problem. I feel that this
> have probably been discussed previously, but I couldn't find anything
> on it neither on AdaPower nor on the cla. archives on google.
> 
> I would like to declare a record type containing two (possibly more)
> arrays of variable lengths, the length of each array is another element
> of the record. I also need to specify an exact representation for the
> record since it is a message that I cant change the format of.
> 
> <snip>

Do you really need to declare the message as a single type? That is,
do you really need to pass objects of that type around? Or can you get
by with reading the message from a stream, and building a set of
internal objects that other packages access in a controlled way?

The point being that there are many reasonable communications
protocols that simply cannot be expressed as an Ada record, and this
seems to be one of them. The correct approach is to encode the
protocol as a procedure that reads from a stream, not as an Ada record
type. 

-- 
-- Stephe

Re: records containing variable length arrays [long]

[Jeffrey Carter]

Mats Karlssohn wrote:
> 
> Jeffrey Carter wrote:
> >
> > Mats Karlssohn wrote:
> 
> > >    -- the other big problem is to get the correct representation
> > >    for Message use
> > >       record
> %<
> > >           Status          at 4 range 0..15;
> > >           Response_Length at 4 range 0.. 7;
> 
> This is a typo... "Response_Length at 5 range 0..7;" is the correct
> representation statement, this also means that all the following
> components move up one byte.

Good. That makes life easier for you.

> 
> %<
> > However, you have additional problems, such as overlaying your lengths,
> > which would be your discriminants, with Status. I presume that you
> > receive your message as a simple Buffer. In that case you might be able
> > to work something along these lines
> 
> Yes, I receive a C pointer (converted to System.Address) and an integer
> Length from a buffer manager for the shared memory where thit beast
> lives. I forgot to mention that I really only neet to read this data
> structure, I will NOT need to assign to it.

In that case you can access it simply. We'll call what you get from C
Data_Address and Data_Length. Then you can say

Raw_Data : constant Buffer (1 .. Data_Length);
-- or (7 .. Data_Length + 6), if you prefer
pragma Import (Ada, Raw_Data); -- Prevent any initialization of Raw_Data
for Raw_Data'Address use Data_Address;

You can then Translate this to an appropriately constrained record type.
Since you do not need to modify it, you can make it constant, which
should avoid copying the data.

If you want to live dangerously, you can avoid the buffer representation
and simply apply the address clause to the record object.

-- 
Jeff Carter
"Perfidious English mouse-dropping hoarders."
Monty Python & the Holy Grail

Re: records containing variable length arrays [long]

[Mats Karlssohn]

I will try to address two posts with the same reply here...

Jeffrey Carter wrote:
%<
> In that case you can access it simply. We'll call what you get from C
> Data_Address and Data_Length. Then you can say
> 
> Raw_Data : constant Buffer (1 .. Data_Length);
> -- or (7 .. Data_Length + 6), if you prefer
> pragma Import (Ada, Raw_Data); -- Prevent any initialization of Raw_Data
> for Raw_Data'Address use Data_Address;
> 
> You can then Translate this to an appropriately constrained record type.
> Since you do not need to modify it, you can make it constant, which
> should avoid copying the data.
>
> If you want to live dangerously, you can avoid the buffer representation
> and simply apply the address clause to the record object.

Stephen Leake wrote:
%<
> Do you really need to declare the message as a single type? That is,
> do you really need to pass objects of that type around? Or can you get
> by with reading the message from a stream, and building a set of
> internal objects that other packages access in a controlled way?

Last night i came up with a partitial solution that seems promising.
I realised that I will need to pass pointers to the messages around
to different routines.

I ended up declaring the record in the way that was discussed earlier.
Then I declare an access type to the record and use an instance of
System.Address_To_Access_Conversions to convert the C pointer to an
Ada pointer. I'll probably wrap another layer around the buffer
manager and let that glue perform the address to access conversion.

Right now this seems to work but I have a layout problem in another
similar (but not identical) record type. See below for a new question.

Stephen Leake wrote:
> The point being that there are many reasonable communications
> protocols that simply cannot be expressed as an Ada record, and this
> seems to be one of them. The correct approach is to encode the
> protocol as a procedure that reads from a stream, not as an Ada record
> type.

Well, I guess that could be done, but I really do not want to copy
the data around. I gets dumped into shared (actually reflected)
memory by another CPU and in some cases the records can be quite
big and arriving fast.


Now, for the new (but related) question.

Given this declaration and representation:

   type Message(Response_Length : Byte; Output_Length : Byte) is
      record
         Magic           : Word;
         Operation       : Word;
         Status          : Word;
         Response_Data   : Buffer(1..Response_Length);
         Output_Data     : Buffer(1..Output_Length);
         CRC             : Word;
      end record;

   for Message use
      record
          Magic           at 0 range 0..15;
          Operation       at 2 range 0..15;
          Status          at 4 range 0..15;
          Response_Length at 6 range 0.. 7;
          Output_Length   at 7 range 0.. 7;
      end record;

   pragma Pack(Message);

Is there a way to explicitly put at least the CRC element into the
representation clause ? My experiments haven't gotten me anywhere
on this part.

-- 
Mats Karlssohn, developer                         mailto:mats@mida.se  
Mida Systemutveckling AB                          http://www.mida.se
Box 64, S-732 22 ARBOGA, SWEDEN
Phone: +46-(0)589-89808   Fax: +46-(0)589-89809

Re: records containing variable length arrays [long]

[Jeffrey Carter]

Mats Karlssohn wrote:
> 
> I ended up declaring the record in the way that was discussed earlier.
> Then I declare an access type to the record and use an instance of
> System.Address_To_Access_Conversions to convert the C pointer to an
> Ada pointer. I'll probably wrap another layer around the buffer
> manager and let that glue perform the address to access conversion.

You can probably avoid the conversion of the pointer. Given

type Message (...) is record ...;
for Message use record ...;

and something like

void get (buffer* p, int* len);

you can define

type Message_Ptr is access all Message;
pragma Convention (C, Message_Ptr);

procedure Get (Ptr : out Message_Ptr; Length : out Interfaces.C.Int);
pragma Import (C, Get, "get");

and use the access value you get from Get directly.

> Given this declaration and representation:
> 
>    type Message(Response_Length : Byte; Output_Length : Byte) is
>       record
>          Magic           : Word;
>          Operation       : Word;
>          Status          : Word;
>          Response_Data   : Buffer(1..Response_Length);
>          Output_Data     : Buffer(1..Output_Length);
>          CRC             : Word;
>       end record;
> 
>    for Message use
>       record
>           Magic           at 0 range 0..15;
>           Operation       at 2 range 0..15;
>           Status          at 4 range 0..15;
>           Response_Length at 6 range 0.. 7;
>           Output_Length   at 7 range 0.. 7;
>       end record;
> 
>    pragma Pack(Message);
> 
> Is there a way to explicitly put at least the CRC element into the
> representation clause ? My experiments haven't gotten me anywhere
> on this part.

Only if it comes before the variable-length part of the record. The
values used in a record representation clause have to be static, which
means known at compile time.

-- 
Jeffrey Carter

Re: records containing variable length arrays [long]

[Jeffrey Carter]

Jeffrey Carter wrote:
> 
> Mats Karlssohn wrote:
> >
> > I ended up declaring the record in the way that was discussed earlier.
> > Then I declare an access type to the record and use an instance of
> > System.Address_To_Access_Conversions to convert the C pointer to an
> > Ada pointer. I'll probably wrap another layer around the buffer
> > manager and let that glue perform the address to access conversion.
> 
> You can probably avoid the conversion of the pointer. Given
> 
> type Message (...) is record ...;
> for Message use record ...;
> 
> and something like
> 
> void get (buffer* p, int* len);

This is probably wrong, but my C is bad, so I tend to get things wrong.
What is needed here is the equivalent of Ada's "P : out Ptr_To_Byte",
which on second look is probably

byte** p

> 
> you can define
> 
> type Message_Ptr is access all Message;
> pragma Convention (C, Message_Ptr);
> 
> procedure Get (Ptr : out Message_Ptr; Length : out Interfaces.C.Int);
> pragma Import (C, Get, "get");
> 
> and use the access value you get from Get directly.
> 
> > Given this declaration and representation:
> >
> >    type Message(Response_Length : Byte; Output_Length : Byte) is
> >       record
> >          Magic           : Word;
> >          Operation       : Word;
> >          Status          : Word;
> >          Response_Data   : Buffer(1..Response_Length);
> >          Output_Data     : Buffer(1..Output_Length);
> >          CRC             : Word;
> >       end record;
> >
> >    for Message use
> >       record
> >           Magic           at 0 range 0..15;
> >           Operation       at 2 range 0..15;
> >           Status          at 4 range 0..15;
> >           Response_Length at 6 range 0.. 7;
> >           Output_Length   at 7 range 0.. 7;
> >       end record;
> >
> >    pragma Pack(Message);
> >
> > Is there a way to explicitly put at least the CRC element into the
> > representation clause ? My experiments haven't gotten me anywhere
> > on this part.
> 
> Only if it comes before the variable-length part of the record. The
> values used in a record representation clause have to be static, which
> means known at compile time.
> 
> --
> Jeffrey Carter


-- 
Jeffrey Carter

Re: records containing variable length arrays [long]

[Mats Karlssohn]

Jeffrey Carter wrote:
%<
> You can probably avoid the conversion of the pointer. Given
%<
> type Message_Ptr is access all Message;
> pragma Convention (C, Message_Ptr);
> 
> procedure Get (Ptr : out Message_Ptr; Length : out Interfaces.C.Int);
> pragma Import (C, Get, "get");

Hmmm, I feel I really need to do some reading on and experimenting with
the import/export and convention pragmas. If it's possible to convert
the C pointer in this way the solution of the problem will become wery
nice and uncluttered...

%<
> > Is there a way to explicitly put at least the CRC element into the
> > representation clause ? My experiments haven't gotten me anywhere
> > on this part.
> 
> Only if it comes before the variable-length part of the record. The
> values used in a record representation clause have to be static, which
> means known at compile time.

Oh... that's too bad. Well, I'll have to see what I might do about it.
I guess that I could split the record declaration into three or four
pieces and also declaring a few constants along with them. But it I
worry about the execution times for this piece so I'll probably just
omit the representation for the last part and add a few testcases that
will show if the assumptions is correct.


Thanks anyway, my vacation was nice (and well earned).

-- 
Mats Karlssohn, developer                         mailto:mats@mida.se  
Mida Systemutveckling AB                          http://www.mida.se
Box 64, S-732 22 ARBOGA, SWEDEN
Phone: +46-(0)589-89808   Fax: +46-(0)589-89809