Variable length raw-byte data

Variable length raw-byte data

[Julian Morrison]

I'm trying to shim to a C library that returns some data as a length in
bytes and a pointer to the first byte. The length is not known at compile
time. Is there any way I can morph this into an array of mod 2**8? I'm
using the latest Linux GNAT.

RE: Variable length raw-byte data

[Beard, Frank]

Try looking at Interfaces.C and Interfaces.C.Pointers.
If it's string data, look at Interfaces.C.Strings.
I've used this successfully to interface to C.

Or

In your interface routine, receive the pointer as an
address, and then unchecked convert it.

  type Byte is mod 2**8;

  type Byte_Array is array (positive range <>) of Byte;

  type Byte_Array_Pointer is access Byte_Array;

  buffer_Address : System.Address;
  buffer_Pointer : Byte_Array_Pointer;

begin
...
  status := Get_C_Stuff(length         => length,
                        buffer_Address => buffer_Address);

  if (length > 0) then
    declare
      subtype Constrained_Byte_Array is Byte_Array(1..length);
      function To_Buffer_Pointer is
        new Ada.Unchecked_Conversion(System.Address,Constrained_Byte_Array);
    begin
      buffer_Pointer := To_Buffer_Pointer(buffer_Address);
    end;
    ...
  end if;
...

Something like that.  (Constructive corrections only please!)
I haven't done it this way in a while.

Frank

-----Original Message-----
From: Julian Morrison [mailto:julian.morrison@virgin.net]
Sent: Monday, December 11, 2000 2:38 PM
To: comp.lang.ada@ada.eu.org
Subject: Variable length raw-byte data


I'm trying to shim to a C library that returns some data as a length in
bytes and a pointer to the first byte. The length is not known at compile
time. Is there any way I can morph this into an array of mod 2**8? I'm
using the latest Linux GNAT.
_______________________________________________
comp.lang.ada mailing list
comp.lang.ada@ada.eu.org
http://ada.eu.org/mailman/listinfo/comp.lang.ada

Re: Variable length raw-byte data

[Jeff Carter]

Often, you can receive the pointer as an address and use it for an
object.

type Byte_List is array (Positive range <>) of Unsigned_8;

C_Thing (Length, Pointer);

Use_Thing : declare
   Thing : Byte_List (1 .. Length);
   for Thing'Address use Pointer;
begin -- Use_Thing
   ...
end Use_Thing;

-- 
Jeff Carter
"I blow my nose on you."
Monty Python & the Holy Grail

RE: Variable length raw-byte data

[tmoran]

>Or
>In your interface routine, receive the pointer as an
>address, and then unchecked convert it.
> ...
>  type Byte_Array is array (positive range <>) of Byte;
>  buffer_Address : System.Address;
> ...
>      subtype Constrained_Byte_Array is Byte_Array(1..length);
>      function To_Buffer_Pointer is
>        new Ada.Unchecked_Conversion(System.Address,Constrained_Byte_Array);
> ...
>Something like that.  (Constructive corrections only please!)
  Instead of trying to use Unchecked_Conversion to change an address
into an access type (I presume that was meant here), use
System.Address_To_Access_Conversions.  That should work.  But a
System.Address need not be the same size as an access type (consider
various Intel memory models), and even an access to an unconstrained
array (which will involve an actual 'range being stored somewhere) and
an access to a constrained array (where the compiler might keep the
range somewhere else) may be different, and different from a
System.Address.

>I'm trying to shim to a C library that returns some data as a length in
>bytes and a pointer to the first byte. The length is not known at compile
 type Data_Array is array(1 .. 10_000) of Byte; -- big enough constrained array
 type Data_Array_Ptr is access all Data_Array;
 P : aliased Data_Array_Ptr;
 N : aliased Natural;
 ...
 if C_Func(N'access, P'access) /= Failure then ...
 -- P.all(0 .. N) contains result

If Data_Array is not constrained, then a pointer to one, ie a
Data_Array_Ptr, must handle the range somewhere.  If it's a fat pointer,
it's unlikely C_Func will be so kind as to supply it correctly.  If it
points to a descriptor of some kind for the actual array, it's again
unlikely C_Func will return such a beast.

RE: Variable length raw-byte data

[Beard, Frank]

Yes, I forgot about this.  I've used this several times
in the past with great success.

Frank

-----Original Message-----
From: Jeff Carter [mailto:jrcarter@acm.org]

Often, you can receive the pointer as an address and use it for an
object.

type Byte_List is array (Positive range <>) of Unsigned_8;

C_Thing (Length, Pointer);

Use_Thing : declare
   Thing : Byte_List (1 .. Length);
   for Thing'Address use Pointer;
begin -- Use_Thing
   ...
end Use_Thing;

-- 
Jeff Carter
"I blow my nose on you."
Monty Python & the Holy Grail
_______________________________________________
comp.lang.ada mailing list
comp.lang.ada@ada.eu.org
http://ada.eu.org/mailman/listinfo/comp.lang.ada

RE: Variable length raw-byte data

[Beard, Frank]

Your right.  When I wrote the code that used Unchecked_Conversion,
it was on an HP-UX box with the Alsys Ada83 compiler that didn't
have System.Address_To_Access_Conversions.  The addresses there
did translate nicely to access types.

With Ada95, the only interfaces I've written to C have used the
Interfaces.C... packages.

Frank

-----Original Message-----
From: tmoran@acm.org [mailto:tmoran@acm.org]

  Instead of trying to use Unchecked_Conversion to change an address
into an access type (I presume that was meant here), use
System.Address_To_Access_Conversions.  That should work.  But a
System.Address need not be the same size as an access type (consider
various Intel memory models), and even an access to an unconstrained
array (which will involve an actual 'range being stored somewhere) and
an access to a constrained array (where the compiler might keep the
range somewhere else) may be different, and different from a
System.Address.

Re: Variable length raw-byte data

[Robert Dewar]

In article
<976563484.10868.0.nnrp-01.c1ed21d4@news.demon.co.uk>,
  "Julian Morrison" <julian.morrison@virgin.net> wrote:
> I'm trying to shim to a C library that returns some data as a
length in
> bytes and a pointer to the first byte. The length is not known
at compile
> time. Is there any way I can morph this into an array of mod
2**8? I'm
> using the latest Linux GNAT.



The important thing is to NEVER use pointers to unconstrained
arrays in such a case. this is asking for non-portable, peculiar
behavior.

A good type to use in this case is what we in GNAT-land call
big arrays:

   type byte is mod 2 ** 8;

   type memory is array (natural) of byte;
   type memptr is access memory;

Now it is almost certainly safe to use address_to_access
conversions to go between address and memptr.

Of course you have to be careful not to access the array
out of bounds, but that's going to be up to the calling
program anyway if the bounds don't come built in :-)


Sent via Deja.com
http://www.deja.com/

RE: Variable length raw-byte data

[Beard, Frank]

>The important thing is to NEVER use pointers to unconstrained
>arrays in such a case. this is asking for non-portable, peculiar
>behavior.

That's exactly why my example contained a pointer to the constrained
subtype.

> A good type to use in this case is what we in GNAT-land call
> big arrays:
>
>   type byte is mod 2 ** 8;
>
>   type memory is array (natural) of byte;
>   type memptr is access memory;

So you're going to allocate 2GB to handle what might
be only 10 bytes (or even 1K or 10K).  That should
work well on a dedicated processor with 64 K of memory.

And again if you use a pointer to the constrained subtype,
you only allocate what you need. 

Or are you saying, you will be kind and make sure you
don't go beyond the length value returned, instead of
letting Ada constraint checking make sure you don't do it.

>Now it is almost certainly safe to use address_to_access
>conversions to go between address and memptr.
>
>Of course you have to be careful not to access the array
>out of bounds, but that's going to be up to the calling
>program anyway if the bounds don't come built in :-)

And you're going to depend on C to do this?

Re: Variable length raw-byte data

[Tristan Gingold]

In article <916h4j$egt$1@nnrp1.deja.com>, Robert Dewar wrote:
>In article
><976563484.10868.0.nnrp-01.c1ed21d4@news.demon.co.uk>,
>  "Julian Morrison" <julian.morrison@virgin.net> wrote:
>> I'm trying to shim to a C library that returns some data as a
>length in
>> bytes and a pointer to the first byte. The length is not known
>at compile
>> time. Is there any way I can morph this into an array of mod
>2**8? I'm
>> using the latest Linux GNAT.
>
>
>
>The important thing is to NEVER use pointers to unconstrained
>arrays in such a case. this is asking for non-portable, peculiar
>behavior.
>
>A good type to use in this case is what we in GNAT-land call
>big arrays:
>
>   type byte is mod 2 ** 8;
>
>   type memory is array (natural) of byte;
>   type memptr is access memory;
>
>Now it is almost certainly safe to use address_to_access
>conversions to go between address and memptr.
AFAIR, address_to_access_conversions has only one generic parameter: the
Object.
So, you need to use the access type defined by the package.
Right or wrong ?

Related question: why it is so ? (why AAC doesn't have the same paramaters
as unchecked_conversion ?)

Tristan.

Re: Variable length raw-byte data

[David Botton]

Since Byte_List is unconstrained is there not a potential for problem here
with additional data associated with the type for bounds?

> type Byte_List is array (Positive range <>) of Unsigned_8;

Would this be a better solution:

C_Thing (Length, Pointer)

declare
    type Byte_List is array (1 .. Length) of Unsigned_8;
    Thing : Byte_List;
    for Thing'Address use Pointer;

begin
   ...
end;


I frequently code the above as:

C_Thing (Length, Pointer)

declare
    type Byte_List is array (1 .. Length) of Unsigned_8;
    type Pointer_To_Byte_List is access all Byte_List;

    function To_Pointer_To_Byte_List is
       new Ada.Unchecked_Conversion (System.Address, Pointer_To_Byte_List);

    Thing : Pointer_To_Byte_List := To_Pointer_To_Byte_List
(Pointer'Address);
begin
 ...
end;

Is there a reason that using the for Thing'Address would be a better
solution then the second method?

David Botton

Re: Variable length raw-byte data

[Lutz Donnerhacke]

* David Botton wrote:
>C_Thing (Length, Pointer)
>
>declare
>    type Byte_List is array (1 .. Length) of Unsigned_8;
>    Thing : Byte_List;
>    for Thing'Address use Pointer;

Usually I add:
     pragma Convention(Byte_List, Thing);

>Is there a reason that using the for Thing'Address would be a better
>solution then the second method?

Personally I suspect an array to be a complex data structure and am not sure
if the Unchecked_Conversion or the 'Address points really to the included
data with out the pragam above.

-- 
	      http://www.tm.oneiros.de/calendar/2001/index.html

RE: Variable length raw-byte data

[Robert Dewar]

In article
<B6A1A9B09E52D31183ED00A0C9E0888C469942@nctswashxchg.
nctswash.navy.mil>,

> So you're going to allocate 2GB to handle what
> might be only 10 bytes (or even 1K or 10K).  That
> should work well on a dedicated processor with 64 K
> of memory.

Well you sure missed the point here :-)

Of *course* you never allocate an instance of a
big array. in fact it is not a bad idea to add

  for big_string_ptr'storage_Size use 0;

to emphasize this. The ONLY way you create instances
of the pointer is by unchecked conversion.
>
> And again if you use a pointer to the constrained
> subtype, you only allocate what you need.

Again, you never do any allocations.
>
> Or are you saying, you will be kind and make sure
> you don't go beyond the length value returned,
> instead of letting Ada constraint checking make
> sure you don't do it.

Well remember that the case being discussed here
is one in which the bounds are not part of the value
in any case. If you know the bound BEFORE you look
at the value, then of course you can make a subtype
that is the right length, but this is not always
the case, the obvious example being a C-style
null-terminated string, and here of course, yes,
the program must be "kind" and make sure it does
not reference past the terminating zero byte.

> >Now it is almost certainly safe to use
> >address_to_access conversions to go between
> >address and memptr.
> >
> >Of course you have to be careful not to access the
> >array out of bounds, but that's going to be up to
> >the calling program anyway if the bounds don't
> >come built in :-)

> And you're going to depend on C to do this?

That's the point, you *can't* depend on C to do this.
And if you are importing the data from the C world,
you cannot depend on Ada to magically protect you
from inherently dangerous C data structures!


Sent via Deja.com
http://www.deja.com/

RE: Variable length raw-byte data

[Larry Kilgallen]

In article <9185vs$nh8$1@nnrp1.deja.com>, Robert Dewar <robert_dewar@my-deja.com> writes:
> In article
> <B6A1A9B09E52D31183ED00A0C9E0888C469942@nctswashxchg.
> nctswash.navy.mil>,
> 
>> So you're going to allocate 2GB to handle what
>> might be only 10 bytes (or even 1K or 10K).  That
>> should work well on a dedicated processor with 64 K
>> of memory.
> 
> Well you sure missed the point here :-)
> 
> Of *course* you never allocate an instance of a
> big array. in fact it is not a bad idea to add
> 
>   for big_string_ptr'storage_Size use 0;
> 
> to emphasize this. The ONLY way you create instances
> of the pointer is by unchecked conversion.

That is an interesting trick.  Is it new for Ada 95 ?

Looking at the nearest (old) manual for VAX/DEC/Compaq Ada (83),
it says that if the value specified is zero the default value of
"no limit" will be used.  Of course that is in the blue type,
meaning a description specific to that compiler. Is their approach
now outlawed for Ada 95, or was it outlawed by an AI for Ada 83 ?

RE: Variable length raw-byte data

[Robert Dewar]

In article <8Mavy1o6gTqY@eisner.decus.org>,
  Kilgallen@eisner.decus.org.nospam (Larry Kilgallen)
wrote:
> Is their approach
> now outlawed for Ada 95, or was it outlawed by an
> AI for Ada 83 ?

It was considered to be a ramification in an Ada 83
AI. It is clearly wrong to treat 0 as unlimited in
this context (and silly, because the default if no
such clause is given should be unlimited, the whole
point of a storage size clause is to set a limit).



Sent via Deja.com
http://www.deja.com/

Re: Variable length raw-byte data

[Robert Dewar]

In article <slrn93f6iu.vo.lutz@taranis.iks-jena.de>,
  lutz@iks-jena.de (Lutz Donnerhacke) wrote:

    pragma Convention(Byte_List, Thing);

What on earth is that supposed to mean. I don't know
of any Ada compiler that supports the convention
Byte_List -- most certainly this is wrong or highly
non-portable, most likely the former, and I cannot
guess what you meant to be saying here!


Sent via Deja.com
http://www.deja.com/

RE: Variable length raw-byte data

[Beard, Frank]

-----Original Message-----
From: Robert Dewar [mailto:robert_dewar@my-deja.com]

> Of *course* you never allocate an instance of a
> big array. in fact it is not a bad idea to add

Allocate was a bad choice of words.  But you missed
my real point.  My biggest concern is that by doing
this:

   type byte is mod 2 ** 8;

   type memory is array (natural) of byte;
   type memptr is access memory;

   myMemory : memptr;

You are telling the compiler that you're pointing
at a 2GB array and depending on the programmer to be
"kind" and use the length returned (smacks way too
much of C/C++), instead of letting the language tell
you when you've over-reached your bounds.  What if
the programmer inadvertently uses "myMemory.all'last"?  
If you use the constrained sub-type approach, you
can still use 'first and 'last, and it properly sets
the state data of the access type.

Of course the counter-danger is that the programmer
will inadvertently uncheck convert something using
the unconstrained type instead of the constrained
subtype.  Which will likely lead to peculiar behavior.

Another approach is to do the following:

  type Byte is mod 2**8;

  buffer_Address : System.Address;

begin
...
  status := Get_C_Stuff(length         => length,
                        buffer_Address => buffer_Address);

  if (length > 0) then
    declare
      type Byte_Array is Byte_Array(1..length) of Byte;
      type Byte_Array_Pointer is access Byte_Array;
      buffer_Pointer : Byte_Array_Pointer;
      function To_Buffer_Pointer is
        new Ada.Unchecked_Conversion(System.Address,Byte_Array);
    begin
      buffer_Pointer := To_Buffer_Pointer(buffer_Address);
    end;
    ...
  end if;
...

This is fine so long as everything you need to do is
within the declare block.

Every system I've worked on, both the unconstrained
subtype and the declare type have worked fine.

The DEC VAX Ada compiler would let you do unchecked
conversion with unconstrained types, other compilers
require constrained types.  What's the rule in Ada 95?
Does the language require both types to be constrained?

Frank

Re: Variable length raw-byte data

[Jeff Carter]

David Botton wrote:
> 
> Since Byte_List is unconstrained is there not a potential for problem here
> with additional data associated with the type for bounds?
> 
> > type Byte_List is array (Positive range <>) of Unsigned_8;

This shouldn't cause a problem, because in

Thing : Byte_List (1 .. Length);
for Thing'Address use Pointer;

Thing'Address is defined to be the address of Thing (Thing'First). If
the compiler needs to store any additional information, it has to put
that information elsewhere.

> 
> Would this be a better solution:
> 
> C_Thing (Length, Pointer)
> 
> declare
>     type Byte_List is array (1 .. Length) of Unsigned_8;
>     Thing : Byte_List;
>     for Thing'Address use Pointer;
> 
> begin
>    ...
> end;

This is certainly OK, too.

> 
> I frequently code the above as:
> 
> C_Thing (Length, Pointer)
> 
> declare
>     type Byte_List is array (1 .. Length) of Unsigned_8;
>     type Pointer_To_Byte_List is access all Byte_List;
> 
>     function To_Pointer_To_Byte_List is
>        new Ada.Unchecked_Conversion (System.Address, Pointer_To_Byte_List);
> 
>     Thing : Pointer_To_Byte_List := To_Pointer_To_Byte_List
> (Pointer'Address);
> begin
>  ...
> end;
> 
> Is there a reason that using the for Thing'Address would be a better
> solution then the second method?

Yes, the second method is not portable. While Unchecked_Conversion
between System.Address and an access type may work with some compilers,
it definitely does NOT work with all compilers.

Since you have a constrained type here, you could use
System.Address_To_Access_Conversions to obtain a valid access value, but
why add the added baggage of access-type semantics when you can make
Thing a Byte_List?

-- 
Jeff Carter
"You tiny-brained wipers of other people's bottoms!"
Monty Python & the Holy Grail

Re: Variable length raw-byte data

[Lutz Donnerhacke]

* Robert Dewar wrote:
>In article <slrn93f6iu.vo.lutz@taranis.iks-jena.de>,
>  lutz@iks-jena.de (Lutz Donnerhacke) wrote:
>
>    pragma Convention(Byte_List, Thing);
>
>What on earth is that supposed to mean.

Copy & Waste. Sorry. I'd like to change:
    pragma Convention(C, Thing);
to
    pragma Convention(C, Byte_List);
and got the wrong field.

-- 
	      http://www.tm.oneiros.de/calendar/2001/index.html

RE: Variable length raw-byte data

[Robert Dewar]

In article
<B6A1A9B09E52D31183ED00A0C9E0888C469948@nctswashxchg.
nctswash.navy.mil>,
  comp.lang.ada@ada.eu.org wrote:
> -----Original Message-----
> From: Robert Dewar
[mailto:robert_dewar@my-deja.com]
>
> > Of *course* you never allocate an instance of a
> > big array. in fact it is not a bad idea to add
>
> Allocate was a bad choice of words.

It wasn't just a matter of words, you specifically
expressed concern about *allocating* 2 gigs on a
64K machine. Not much ambiguity there!

> But you missed my real point.  My biggest concern
> is that by doing this:
>
>    type byte is mod 2 ** 8;
>
>    type memory is array (natural) of byte;
>    type memptr is access memory;
>
>    myMemory : memptr;
>
> You are telling the compiler that you're pointing
> at a 2GB array and depending on the programmer to
> be "kind" and use the length returned (smacks way
> too much of C/C++),

Please carefully read my response, which in fact
responds to this exact point, there is nothing that
I missed here.

Look at the subject line of this thread "variable
length raw-byte data".

When you are dealing with raw data streams from
a foreign environment, then as I described in my
previous post, there are two cases.

1. You know the length in advance, in this case, of
course you use a constrained array of the right
length. No one argues with that, since it is standard
practice.

2. You do not know the length (my example in my
previous post was a null terminated C string, yes
you can find the length but only AFTER you have
imported the data into the Ada environment, and
the point of the big array approach is to allow
you do to do that importation).

> instead of letting the language tell
> you when you've over-reached your bounds.

The language cannot do magic, there is no way for
you to deal with the null terminated string case in
some automatic language controlled way.

> If you use the constrained sub-type approach, you
> can still use 'first and 'last, and it properly
> sets the state data of the access type.

Yes, of course, that's elementary, and I cannot
imagine ANY Ada programmer not understanding that
of course you use a constrained array if the length
is known in advance.

> Of course the counter-danger is that the programmer
> will inadvertently uncheck convert something using
> the unconstrained type instead of the constrained
> subtype.  Which will likely lead to peculiar
> behavior.

You should avoid using the unconstrained type in
all cases with foreign data, that's my point. Using
unconstrained types leads you into nasty
implementation dependent areas.

> Another approach is to do the following:
>
>   type Byte is mod 2**8;
>
>   buffer_Address : System.Address;
>
> begin
> ...
>   status := Get_C_Stuff(length         => length,
>                         buffer_Address =>
>                                  buffer_Address);
>
>   if (length > 0) then
>     declare
>       type Byte_Array is Byte_Array(1..length) of
>                                              Byte;

----> Well you sure managed to crowd a lot
----> of errors into one line here :-)

1. You cannot have Byte_Array on both sides of a
declaration like this

2. You mean either a new array declaration, or a
derived type declaration, or a subtype declaration.
What you have written is a confused mixture of the
three :-)

Let's assume you meant one of the following

type Byte_Array is new Raw_Storage (1 .. length);
subtype Byte_Array is Raw_Storage (1 ., length);
type Byte_Array is array (1 .. length) of Byte;

It does not matter which, all three are reasonable
choices in context. (I always suggest that people
compile code before they post :-)

>       type Byte_Array_Pointer is access Byte_Array;
>       buffer_Pointer : Byte_Array_Pointer;
>       function To_Buffer_Pointer is
>         new
>Ada.Unchecked_Conversion(System.Address,Byte_Array);

------> ooops, you mean to convert to
        Byte_Array_Ptr here, the above UC is totally
        meaningless. Furthermore, in Ada 95 we should
        always use Address_To_Access conversions.

>     begin
>       buffer_Pointer :=
To_Buffer_Pointer(buffer_Address);
>     end;
>     ...
>   end if;
> ...
>
> This is fine so long as everything you need to do
is
> within the declare block.
>
> Every system I've worked on, both the unconstrained
> subtype and the declare type have worked fine.

Your example above is THE standard way of doing
things. It does not involve any "unconstrained
subtypes" and is likely to work in all
implementations, though, as I emphasize once
again, it is applicable only if, as in this case,
you know the length of the data before you look
at it, which often is not the case.

> The DEC VAX Ada compiler would let you do unchecked
> conversion with unconstrained types

No, that cannot be generally true, there is no
implementation model under which this will always
work. Yes, with a given compiler, SOME cases will
work. Which cases work will depend on the compiler.

> other
> compilers
> require constrained types.  What's the rule in Ada
> 95?
> Does the language require both types to be
> constrained?

The language has nothing to say here, since all it
talks about is interpreting bit strings as a
different type. Whether this "works" or not is
entirely implementation defined.

Even the unchecked conversion of the pointer to
constrained array (the standard solution you present
above, that is familiar to us all) is not guaranteed
to work from the RM.

I would argue that if you use the correct diction of
Address_To_Access_Conversions it definitely SHOULD
work with constrained arrays.

The conversion of an Address to a pointer to an
unconstrained array is fundamentally meaningless.
The former has no bounds, the latter has bounds,
where do they come from? Yes, in some limited cases
it may magically work, but it cannot work in the
general case.

Robert Dewar


Sent via Deja.com
http://www.deja.com/