Access to array slices?

Access to array slices?

[Wojtek Narczynski]

Hello,

I am trying to develop a library that reads C structures from socket
and interpret them.

At some point I have a structure (stream?) on stack that contains
multiple strings inside. It's something like [name_length,
value_length, name, value]* mean an unknown number of such name-value
pairs. I am able to access it as String using the 'Address clause, but
now I either have to copy the strings out, or obtain (deadly unsafe,
not portable, nobody knows why) Access'es to slices of this array.

I am using GNAT, and I don't mind that much being bound to it, because
what I try to write is noncommercial stuff.

What I was able to figure out so far is that a fat pointer is two
pointers - one to the dope another to the data. Not sure what's next
tho...

Is it common for C interfacing code to copy data in and out like
crazy? Without the ability to obtain an Access to array slice this
must be the case, right?

Could somebody help me go on with this, or alternatively wave his
finger?

---

My alternative idea is to redefine 'Input for a Name_Value_Pair_Type,
but this way I will loose control over buffering of the input stream,
I think.

Thanks,
Wojtek

Re: Access to array slices?

[tmoran]

> What I was able to figure out so far is that a fat pointer is two
An Ada Unconstrained string has two pieces of information - the characters
and the length.  Your C strings are missing the second.  So you can either
copy them (up to nul) and put them into correct size Ada Strings, or you
can forget about String lengths and handle them as C would.

 type C_Char_Ptrs is access all Interfaces.C.Char;

 type C_String_Ptrs is access all Interfaces.C.Char_Array(Interfaces.C.Size_T);
 -- a pointer to a fixed size (albeit huge) char_array

 function Convert is new Ada.Unchecked_Conversion
   (Source=>C_Char_Ptrs, Target=>C_String_Ptrs);

 Big_Collection_Of_C_Strings : Interfaces.C.Char_Array(...)
 Chopped : array(1 .. n) of C_String_Ptrs;
 Count : Natural := 0;
 si : Interfaces.C.Size_T := Big_Collection_Of_C_Strings'first;
begin
 loop
   Count := Count+1;
   Chopped(Count) := Convert(Big_Collection_Of_C_String(si'unchecked_access));
   -- now scan si up to the start of the next logical string
   -- exit if no more
 end loop;
Of course Ada will think that Chopped(i).all is an extremely long string,
so it's up to you to keep your indexes in range.  If you call your
processing routines with slices
  process(Chopped(i)(0 .. size(i)));
then the processing routines will believe there are index range limits.
If you can process everything sequentially, you can skip most of the
above and just
  loop
    first := determine first index of this part
    last := determine last index of this part
    process(Big_Collection_Of_C_Strings(first .. last));
  end loop;

Re: Access to array slices?

[sk]

Hi,

tmoran@acm.org wrote:
 > type C_Char_Ptrs is access all Interfaces.C.Char;
 >
 > type C_String_Ptrs is access all
 >    Interfaces.C.Char_Array(Interfaces.C.Size_T);
 >    -- a pointer to a fixed size (albeit huge) char_array
 >
 > function Convert is new Ada.Unchecked_Conversion
 >    (Source=>C_Char_Ptrs, Target=>C_String_Ptrs);

I am curious as to whether you deliberately avoided
Interfaces.C.Strings (ICS) or the not mentioning it was
an oversight ?

The reason I ask is that I have not seen your technique
and was wondering if you found it more useful and
transparent than ICS ?

An example of why I ask,

int readlink(const char *path, char *buf, size_t bufsiz);

     Path => ICS.Chars_Ptr

however

     Buf => access IC.Char_Array

The choice seems dependent upon whether the imported
function allocates or expects the caller to allocate.

Do you deliberately avoid ICS  and does your method
add some uniformity when binding ?


-- 
--
-- Merge vertically for real address
--
------------------------------------
-- s n p @ t . o
--  k i e k c c m
------------------------------------

Re: Access to array slices?

[tmoran]

> I am curious as to whether you deliberately avoided
> Interfaces.C.Strings (ICS) or the not mentioning it was
> an oversight ?
Perhaps I misunderstood the original question.  I was responding to:
> What I was able to figure out so far is that a fat pointer is two
and
> Is it common for C interfacing code to copy data in and out like
> crazy? Without the ability to obtain an Access to array slice this
> must be the case, right?
  It seemed you wanted to avoid copying, while still being able to
talk about certain substrings of a single large Interfaces.C.Char_Array.

You can't use an "access all Interfaces.C.Char_Array" because that's
unconstrained and the access value would need a place to store the length.
But if the original data from the socket (or wherever) is a big
Interfaces.C.Char_Array containing a series of [name_length, value_length,
name, value] sequences, then you can have a scan routine that finds the
name_length, value_length, and starting index for "name" of a single
sequence.

Big : Interfaces.C.Char_Array(...);
Name_Length, Value_Length, Name_Start : Interfaces.C.Size_T;

Given those,
Big(Name_Start .. Name_Start+Name_Length-1) is an Interfaces.C.Char_Array
containing the name part, and
Big(Name_Start+Name_Length .. Name_Start+Name_Length+Value_Length-1)
is an Interfaces.C.Char_Array consisting of the "value" part, and
the next name_length count is to be found starting at
Big(Name_Start+Name_Length+Value_Length .. )

Here's a dummy (but working) program to show the idea:
with Interfaces.C;
with Ada.Text_Io;
procedure Testcs is
  Big : Interfaces.C.Char_Array := Interfaces.C.To_C("now is the time");
  First,
  Last    : Interfaces.C.Size_T;

  Word_Number: Positive := 1;
  procedure Find_Next_Word is  -- somehow find values for First and Last
  begin
    case Word_Number is
      when 1 => First := 0; Last := 2;
      when 2 => First := 4; Last := 5;
      when 3 => First := 7; Last := 9;
      when 4 => First := 11;Last := 14;
      when others => First := 0; Last := 0;
    end case;
    Word_Number := Word_Number + 1;
  end Find_Next_Word;

  procedure Show(C_String : in Interfaces.C.Char_Array) is
  begin
    Ada.Text_Io.Put_Line(Interfaces.C.To_Ada(C_String, Trim_Nul => False));
  end Show;

begin

  for I in 1 .. 4 loop
    Find_Next_Word;
    Show(Big(First .. Last));
  end loop;
end Testcs;

procedure Show has, in effect, been handed a pointer to a dynamic length
substring of Big.  No copying has been done.

Re: Access to array slices?

[sk]

Hi,
 >> I am curious as to whether you deliberately avoided
 >> Interfaces.C.Strings (ICS) or the not mentioning it was
 >> an oversight ?

 > Perhaps I misunderstood the original question.  I was responding to:

Sorry, wasn't trying to imply that ICS should have been used and I
was not looking too closely at the OP.

I was just curious if you had developed an interesting technique which
would make interfacing more straight forward by using ...

 > type C_Char_Ptrs is access all Interfaces.C.Char;
 >
 > type C_String_Ptrs is access all
 >    Interfaces.C.Char_Array(Interfaces.C.Size_T);
 >     -- a pointer to a fixed size (albeit huge) char_array
 >
 > function Convert is new Ada.Unchecked_Conversion
 >    (Source=>C_Char_Ptrs, Target=>C_String_Ptrs);

 From your expanded example, I see that you still need to determine
the slice ranges.

 > You can't use an "access all Interfaces.C.Char_Array" because that's
 > unconstrained and the access value would need a place to store the
 > length.

Yes, but you can do

int readlink(const char *path, char *buf, size_t bufsiz);

function Read_Link (
     Path    : ICS.Chars_Ptr;        -- "in" paramater
     Buf     : access IC.Char_Array; -- "in out" parameter
     Size    : IC.Size_t             -- "in" paramater (length)
) return IC.Int;

The "Path" (ICS.New_String'("whatever")) involves the copying which
the OP mentioned as wanting to avoid, but "Buf" is local, constrained
and does not require copying ... and of course, the nature of
"char *buf" seems to  vary depending on which "C" functions you are
interfacing to :-(

All irrelevent to the OP, just curious and hoping that you had a
technique which makes translation of "chars *buf" mindless (as in
you don't have to RTFM to determine whether caller or callee
allocates the memory :-)

-- 
--
-- Merge vertically for real address
--
------------------------------------
-- s n p @ t . o
--  k i e k c c m
------------------------------------

Re: Access to array slices?

[tmoran]

>    Buf     : access IC.Char_Array; -- "in out" parameter
Actually it's an "in" parameter whose value is a pointer.
There aren't any "out" or "in out" parameters in C, just "in" pointers
to where an output value is supposed to go.

> technique which makes translation of "chars *buf" mindless (as in
> you don't have to RTFM to determine whether caller or callee
> allocates the memory :-)

  If the C function takes a "chars *buf" then the caller had better supply
a value, ie, allocate some memory and supply its address.  If the C
function wants to allocate memory and pass back a pointer to it, it will
usually have "chars **buf" or "Handle *h" where Handle is a pointer, ie, a
typedef for chars *buf.  So most of the time it should be pretty
straightforward to see who is supposed to allocate the memory.

Re: Access to array slices?

[David Thompson]

sk <sk@noname.com> wrote :
...
> An example of why I ask,
>
> int readlink(const char *path, char *buf, size_t bufsiz);
>
>      Path => ICS.Chars_Ptr
>
> however
>
>      Buf => access IC.Char_Array
>
> The choice seems dependent upon whether the imported
> function allocates or expects the caller to allocate.
>
Allocation has nothing to do with it; a C routine cannot
allocate unless it *returns* a pointer or receives a pointer
*to* a pointer (or receives a pointer to or returns a struct
*containing* a pointer, but that's much rarer).

The choice probably should be and at least here apparently is
dependent on whether the C code expects/requires a char pointer
to point to a C string, which is one or more char elements the last
of which and only the last is a null (terminator); or to "just" chars,
one or more, which can contain arbitrary data including nulls.

(readlink actually *could* have been designed and specified
to treat the target as a C string, since in practice it never
contains a null and its maximum size can be bounded,
but it wasn't, probably for consistency with other routines
like read and recv which cannot.)

--
- David.Thompson 1 now at worldnet.att.net

Re: Access to array slices?

[Steve]

"Wojtek Narczynski" <wojtek@power.com.pl> wrote in message
news:5ad0dd8a.0301141717.2f1a9685@posting.google.com...
> Hello,
>
> I am trying to develop a library that reads C structures from socket
> and interpret them.
>
> At some point I have a structure (stream?) on stack that contains
> multiple strings inside. It's something like [name_length,
> value_length, name, value]* mean an unknown number of such name-value
> pairs. I am able to access it as String using the 'Address clause, but
> now I either have to copy the strings out, or obtain (deadly unsafe,
> not portable, nobody knows why) Access'es to slices of this array.
>

The first thing I would do is forget that the data comes from C structures,
since from what you have described that is unimportant.  The code to read
one of these structures would look something like:

   Get( socket, name_length );
   Get( socket, value_length );
   declare
     name : string( 1 .. name_length );
     value : string( 1 .. value_length );
   begin
     Get( socket, name );
     Get( socket, value );
     Do_Something_With_Data( name, value );
   end;

You may have to tweak this depending on whether the strings in your data
stream have null terminators and whether the the lengths include the
terminators.

This assume you have overloaded functions for reading lengths and strings
from a socket, which are easy to create if you don't already have them.  You
could also use Ada streams.

When I look at an interface specification for messages sent using sockets, I
hate to see the specification described in terms of C structures.  I much
prefer a language independent data specification.  Even if communication is
between two C programs, the compiler(s) may do different things depending on
architecture, vendor or compiler switches.  In fact when dealing with a
network I make it a point to put things into network byte order.

I hope this helps,
Steve
(The Duck)

Re: Access to array slices?

[Wojtek Narczynski]

Hi,

> The first thing I would do is forget that the data comes from C structures,
> since from what you have described that is unimportant. 

That's right.

> You could also use Ada streams.

Are they buffered or each 'Read and 'Input will result in a syscall?
There is a highly optimized C reference implementation, and I don't
want mine to stay much behind. Well, I will find out with strace
RSN...

>   -- (...)
>   Do_Something_With_Data( name, value );
>   -- (...)

This "Something_With_Data" is many different things. This is why I
wanted a function returning an array of access to
Name_Value_Pair_Type, where name and value would be accesses to
strings.

Should I pass access to procedure there? Or declare this procedure
generic?

> In fact when dealing with a network I make it a point to put things 
> into network byte order.

When it comes to network byte order, this is a good thing:

http://www.ibpaus.de/downloads/AdaStreamsInNetworkByteOrder_020326.tar.gz

I think that GNAT using this is still compatible with RM, because byte
order for streams is not defined there. Correct?

> I hope this helps,

Yes, thanks!
However, access to array slice would be something :-;


Regards,
Wojtek

Re: Access to array slices?

[tmoran]

> > You could also use Ada streams.
> Are they buffered or each 'Read and 'Input will result in a syscall?
  Each 'Read does what it needs to do to get the next value of the data
type.  You could define 'Reads that generate random numbers, or successive
letters of the alphabet, or read from a buffer, in which case no syscalls
or IO calls would be needed.  You can define a 'Read that reads an entire
structure with a single IO call, if that's appropriate.  OTOH, if you do
a 'Read on a String, and pass it a stream from
  Ada.Text_IO.Streams.Stream(Ada.Text_IO.Standard_Input)
then it will do one read from standard input per character.

The special thing a 'Read does is that if you don't overide it
  for My_Record'Read use ...
then the compiler will automatically generate a series of 'Reads for
the various components of My_Record, recursively down to the elementary
types, or the pieces you have overriden, whichever comes first.

> >   Do_Something_With_Data( name, value );
> ...
> wanted a function returning an array of access to
> Name_Value_Pair_Type, where name and value would be accesses to
> strings.
  You haven't really said that you need random access to this
array, as opposed to sequential.  If sequential, you can use the
technique in Message-ID: <WOjV9.570671$GR5.362443@rwcrnsc51.ops.asp.att.net>
  If random, you can create an array of
type Name_Value_Pair_Type is record
  Name_First, Name_Last, Value_First, Value_Last : Natural;
end record;
and then do calls like:
  Do_Something_With_Data(Buffer(Name_First(i) .. Name_Last(i)),
                         Buffer(Value_First(i) .. Value_Last(i)));
If that won't do, and you really truly need "is access all String;",
you will have to do data copying.

Re: Access to array slices?

[Warren W. Gay VE3WWG]

Wojtek Narczynski wrote:
...snip...
> When it comes to network byte order, this is a good thing:
> 
> http://www.ibpaus.de/downloads/AdaStreamsInNetworkByteOrder_020326.tar.gz
> 
> I think that GNAT using this is still compatible with RM, because byte
> order for streams is not defined there. Correct?

Depending upon the hosts used, this would appear to work
fine. However, there is one warning I would like to suggest
is worth looking into:

The GNAT network byte ordering scheme only performs byte swapping
based upon big/little endian machines. This raises two possible
issues (one of which you can probably ignore):

   1. The byte swapping method only works if machines at each
      end use the SAME FLOATING POINT REPRESENTATION. I believe
      most modern architectures do, but I know that the XDR
      routines (for RPC under UNIX), went out of their way to
      specify what the network floating point representation
      should be. This can be different than what is actually
      used on a particular local host (endian change or not).

   2. Hopefully this no longer applies, but some of the old
      iron out there (PDP?) swapped words and bytes in
      conflicting order (words were big endian and bytes
      within the word were little endian (or maybe I got it
      backwards).  If this is the case, the little/big
      endian conversions will simply not work. But you can
      probably ignore this issue today ;-)

wwg.
-- 
Warren W. Gay VE3WWG
http://home.cogeco.ca/~ve3wwg

Re: Access to array slices?

[Stephen Leake]

wojtek@power.com.pl (Wojtek Narczynski) writes:

> I am using GNAT, and I don't mind that much being bound to it, because
> what I try to write is noncommercial stuff.

Just to clear up a misconception: there is nothing to prevent you from
using the GNAT compiler to develop a "commercial" application. You can
also use the GNAT run-time library in a "commercial" application.

GNAT is released under the GNAT Modified Gnu Public License (GMGPL).
The plain GPL says you have to release your application under the GPL
if you use any GPL libraries. However, the "GNAT Modified" part means
you don't have to use the GPL for your application.

Read the license carefully before proceeding!

-- 
-- Stephe

Re: Access to array slices?

[Wojtek Narczynski]

Stephen Leake <Stephen.A.Leake@nasa.gov> wrote in message news:<uu1gaa1v1.fsf@nasa.gov>...
> wojtek@power.com.pl (Wojtek Narczynski) writes:
> 
> > I am using GNAT, and I don't mind that much being bound to it, because
> > what I try to write is noncommercial stuff.
> 
> Just to clear up a misconception: there is nothing to prevent you from
> using the GNAT compiler to develop a "commercial" application. You can
> also use the GNAT run-time library in a "commercial" application.
> 
> GNAT is released under the GNAT Modified Gnu Public License (GMGPL).
> The plain GPL says you have to release your application under the GPL
> if you use any GPL libraries. However, the "GNAT Modified" part means
> you don't have to use the GPL for your application.
> 
> Read the license carefully before proceeding!

Thanks, I am aware of that. I just meant that I am pretty sure I won't
ever be porting this software to any other compiler.

What I don't however know is: the difference between GMGPL and LGPL.
Could you perhaps shed some light on this?

Regards,
Wojtek

Re: Access to array slices?

[Stephen Leake]

wojtek@power.com.pl (Wojtek Narczynski) writes:

> What I don't however know is: the difference between GMGPL and LGPL.
> Could you perhaps shed some light on this?

The LGPL is officially deprecated; see

http://www.gnu.org/licenses/licenses.html#LGPL

-- 
-- Stephe

Re: Access to array slices?

[Victor Porton]

In article <uel7dvzjp.fsf@nasa.gov>,
	Stephen Leake <Stephen.A.Leake@nasa.gov> writes:
> wojtek@power.com.pl (Wojtek Narczynski) writes:
> 
> The LGPL is officially deprecated; see
> 
> http://www.gnu.org/licenses/licenses.html#LGPL

Do you mean that we are juridically forbidden to use
this license for our new software?

Re: Access to array slices?

[sk]

tmoran@acm.org
 > it will usually have "chars **buf" or "Handle *h" where ...

A handy rule of thumb which I hope to remember (I think that
"Handle *h" is largely a MS/Window thing isn't it ?).

david.thompson1@worldnet.att.net
 > Allocation has nothing to do with it; a C routine cannot
 > allocate unless it *returns* a pointer or receives a pointer
 > *to* a pointer (or receives a pointer to or returns a struct
 > *containing* a pointer, but that's much rarer).

I seem to recall that some of the Window C routines allocated
some of the data structures internally and passed a (or updated
a passed in) pointer back to the caller ?

One of my major headaches is "char *" being used instead of
"struct sometype *" and not noticing until things blow up and
I have to look at the man pages.

 > readlink actually *could* have been designed and specified

I only had trouble with "readlink" until I RTFM'd. It looked
like the prototype of something else that I was binding to
but it turns out  the parameters behaved differently after all.

I guess there is no replacement for RTFMing to get the parameters
correct, no fancy and handy Char_Array trickery to save the day :-(

Cheers

-- 
--
-- Merge vertically for real address
--
------------------------------------
-- s n p @ t . o
--  k i e k c c m
------------------------------------

Re: Access to array slices?

[David Thompson]

sk <sk@noname.com> wrote :
> tmoran@acm.org
>  > it will usually have "chars **buf" or "Handle *h" where ...
>
> A handy rule of thumb which I hope to remember (I think that
> "Handle *h" is largely a MS/Window thing isn't it ?).
>
MSWindows certainly uses handles extensively, but they
are far from the only people to do so.  MacOS did; I believe
X did; I've used lots of other libraries/packages which each
have their own (collection of) handle types.  It's a generic
and widely useful concept, or if you like pattern.

> david.thompson1@worldnet.att.net
>  > Allocation has nothing to do with it; a C routine cannot
>  > allocate unless it *returns* a pointer or receives a pointer
>  > *to* a pointer (or receives a pointer to or returns a struct
>  > *containing* a pointer, but that's much rarer).
>
> I seem to recall that some of the Window C routines allocated
> some of the data structures internally and passed a (or updated
> a passed in) pointer back to the caller ?
>
You can't "update a pointer passed in" in C; more generally,
no called routine can ever update a caller's actual argument,
only the value pointed to by a pointer argument and/or provide
a return value (which may be stored, or discarded, by the caller).

Another thing you can do (in general) is *not* give out actual
pointers; instead a called library maintains its own private
storage, possibly with dynamic/heap allocation and pointers
and possibly not, and gives out only numbers or other (private)
values that identify things within that private storage; these go
by a variety of names often including "number" or "index" or
"capability" e.g. a color index in a palette.  I don't know of
MSWindows using these but I haven't looked hard.  In C of
course this means e.g. return an int, or take a pointer to int.

--
- David.Thompson 1 now at worldnet.att.net

Re: Access to array slices?

[Wojtek Narczynski]

Hello,

Looks like 'Unrestricted_Access in GNAT does it...

Just curious, how is it with other compilers?

Regards,
Wojtek