API design problem - buffer scatter I/O

API design problem - buffer scatter I/O

[Maciej Sobczak]

I have a problem finding the best, from the Ada point of view,
interface for the scatter I/O API.

The idea is that the library is able to read some bytes from somewhere
and put them in the buffers provided by user. The simplest, single-
buffer version of this idea is used in the standard library and in
many others - an example from GNAT.Sockets looks like here:

procedure Receive_Socket
     (Socket : Socket_Type;
      Item   : out Ada.Streams.Stream_Element_Array;
      Last   : out Ada.Streams.Stream_Element_Offset);

In other words, user provides a single array and the library fills it
with data.

The problem is that this is a single, continuous array.
How would you approach the design of API for multi-buffer I/O? Multi-
buffer means that the user can provide many arrays, probably of
different sizes, and the library will fill them from first to last.

The following assumptions are true:
1. The user knows up-front how much data will be read. It will never
be less, so there is no need to give back any Last index, like above.
The user has to only ensure that the sum of buffer lengths is
appropriate for all the data that will be read.
2. Buffers don't need to have the same size.
3. Individual buffers are not necessarily related (they don't have to
be components of common bigger structure like record or array).
4. It really has to be a single operation, so no cheating with
repeated calls for separate buffers to get consecutive fragments.

My initial idea was that the subprogram might have a parameter that is
an array of access values to separate buffers (actual buffers must be
therefore aliased). Are buffer bounds properly handled when only
access to array is given (didn't check that yet, sorry)?

My initial idea:

type Buffer_Ptr is access Ada.Streams.Stream_Element_Array;
type Buffers is array (Positive range <>) of Buffer_Ptr;

procedure Scatter_Input_Data (B : in Buffers);

What is bothering me is <http://en.wikibooks.org/wiki/Ada_Programming/
Types/access> and the discussion on designated vs. nominal subtypes in
the "Fat pointers" paragraph.

Have you seen an existing library that solves this problem is some
reasonable way?
Do you have other ideas or suggestions for how to design such API?

--
Maciej Sobczak * www.msobczak.com * www.inspirel.com

Database Access Library for Ada: www.inspirel.com/soci-ada

Re: API design problem - buffer scatter I/O

[sjw]

On Nov 22, 4:05 pm, Maciej Sobczak <see.my.homep...@gmail.com> wrote:
> I have a problem finding the best, from the Ada point of view,
> interface for the scatter I/O API.

There is a binding in GCC 4.3.0 - Receive_Vector and Send_Vector. Not
sure when this was introduced - wasn't in GNAT 3.16.

I don't know how scatter-gather is canonically intended to be used,
but one way that it would have been nice to use were we not stuck on a
historic version of the compiler would be to cope with application-
specific message headers - these are all 20 bytes, so we could have
had a 20-byte header buffer and a channel-specific message buffer to
great advantage. As it is, on TCP_NODELAY & UDP sockets, we have to:
(a) prepare the message
(b) prepare the header (including fields depending on the message)
(c) concatenate the header and the message
(d) send the message.
Not cool.

Re: API design problem - buffer scatter I/O

[george.priv]

On Nov 22, 11:05 am, Maciej Sobczak <see.my.homep...@gmail.com> wrote:
> I have a problem finding the best, from the Ada point of view,
> interface for the scatter I/O API.
>
> The idea is that the library is able to read some bytes from somewhere
> and put them in the buffers provided by user. The simplest, single-
> buffer version of this idea is used in the standard library and in
> many others - an example from GNAT.Sockets looks like here:
>
> procedure Receive_Socket
>      (Socket : Socket_Type;
>       Item   : out Ada.Streams.Stream_Element_Array;
>       Last   : out Ada.Streams.Stream_Element_Offset);
>
> In other words, user provides a single array and the library fills it
> with data.
>
> The problem is that this is a single, continuous array.
> How would you approach the design of API for multi-buffer I/O? Multi-
> buffer means that the user can provide many arrays, probably of
> different sizes, and the library will fill them from first to last.
>
> The following assumptions are true:
> 1. The user knows up-front how much data will be read. It will never
> be less, so there is no need to give back any Last index, like above.
> The user has to only ensure that the sum of buffer lengths is
> appropriate for all the data that will be read.
> 2. Buffers don't need to have the same size.
> 3. Individual buffers are not necessarily related (they don't have to
> be components of common bigger structure like record or array).
> 4. It really has to be a single operation, so no cheating with
> repeated calls for separate buffers to get consecutive fragments.
>
> My initial idea was that the subprogram might have a parameter that is
> an array of access values to separate buffers (actual buffers must be
> therefore aliased). Are buffer bounds properly handled when only
> access to array is given (didn't check that yet, sorry)?

If API library is written in Ada should not be a problem, you will
provide it with thick pointer that it can handle as good as an
operand.

>
> My initial idea:
>
> type Buffer_Ptr is access Ada.Streams.Stream_Element_Array;
> type Buffers is array (Positive range <>) of Buffer_Ptr;
>

Hard to tell given limited information about your problem.  I use to
have somewhat similar problem in the past, but it was more challenging
since communication part could do only some buffers at the time and I
needed to provide some signaling when (and which) buffers were filled
asynchronously.

> procedure Scatter_Input_Data (B : in Buffers);
>
> What is bothering me is <http://en.wikibooks.org/wiki/Ada_Programming/
> Types/access> and the discussion on designated vs. nominal subtypes in
> the "Fat pointers" paragraph.

The paragraph is stating that thick pointers may not be equivalent of
memory location like in some other languages. That will matter when
you plan to use subprograms written in these languages AFAIK.

>
> Have you seen an existing library that solves this problem is some
> reasonable way?
> Do you have other ideas or suggestions for how to design such API?
>
> --
> Maciej Sobczak *www.msobczak.com*www.inspirel.com
>
> Database Access Library for Ada:www.inspirel.com/soci-ada

Re: API design problem - buffer scatter I/O

[Robert A Duff]

Maciej Sobczak <see.my.homepage@gmail.com> writes:

> My initial idea:
>
> type Buffer_Ptr is access Ada.Streams.Stream_Element_Array;
> type Buffers is array (Positive range <>) of Buffer_Ptr;
>
> procedure Scatter_Input_Data (B : in Buffers);

This looks like a reasonable design, except you probably want "access
all" instead of "access".

> What is bothering me is <http://en.wikibooks.org/wiki/Ada_Programming/
> Types/access> and the discussion on designated vs. nominal subtypes in
> the "Fat pointers" paragraph.

I didn't read the whole wikibook, but the particular part you point to
above has quite a bit of misinformation.  Also, the early focus on
"fat pointers" vs. "thin pointers" is confusing.  There's no reason
to talk about such implementation details in an introductory tutorial.
Especially if the details are wrong.

There is no such concept as "fat pointer" or "thin pointer" in Ada!

It does point to a real problem, though.  Given your above declarations,
you might want to say:

    This_Buf : aliased Stream_Element_Array (1..10);
    That_Buf : aliased Stream_Element_Array (1..10_000);
    
    Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- wrong!

but that's illegal.  You can do this instead:

    This_Buf : aliased Stream_Element_Array := (1..10 => <>);
    That_Buf : aliased Stream_Element_Array := (1..10_000 => <>);
    
    Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- OK

Another workaround is to wrap the Stream_Element_Array in a record, with
a discriminant Length used to constrain the array.  If that record is
called Buffer, you could then do:

    This_Buf : aliased Buffer (Length => 100);
    That_Buf : aliased Buffer (Length => 10_000);
    
    Scatter_Input_Data ((This_Buf'Access, That_Buf'Access));

This has the advantage that you can fix the lower bound of the array at
1 (or 0, whichever makes sense for your application).  You don't have to
worry about weird array bounds like 30..40.

- Bob

Re: API design problem - buffer scatter I/O

[Georg Bauhaus]

Maciej Sobczak wrote:

> Do you have other ideas or suggestions for how to design such API?

Not sure this would meet your criteria but perhaps
you could ask the user to provide a Cursor.  The filling
procedure can fill the cells one after the other by
obtaining a pointer, say, through Element.

procedure Receive_Socket
     (Socket : Socket_Type;
      Item   : in out Cursor)
is
     while <we have data> loop
        Element(Item).all := <datum>;
        Next(Item);
     end loop;

end Receive_Socket;

where Element, by contract, would be a function
that returns a pointer to the next free
cell which would be in some of the buffers.


This, of course, shifts the problem of managing
multiple buffers elswhere...

Re: API design problem - buffer scatter I/O

[Maciej Sobczak]

On 22 Lis, 23:16, Robert A Duff <bobd...@shell01.TheWorld.com> wrote:

> There is no such concept as "fat pointer" or "thin pointer" in Ada!

I understand that the distinction is to make the reader aware that
some access values need to carry over the bounds whereas some other
don't.

> It does point to a real problem, though.  Given your above declarations,
> you might want to say:
>
>     This_Buf : aliased Stream_Element_Array (1..10);
>     That_Buf : aliased Stream_Element_Array (1..10_000);
>
>     Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- wrong!
>
> but that's illegal.  You can do this instead:
>
>     This_Buf : aliased Stream_Element_Array := (1..10 => <>);
>     That_Buf : aliased Stream_Element_Array := (1..10_000 => <>);
>
>     Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- OK

Yes, I have noticed that. Not a big issue for the user (all bounds can
be still variable at run-time), but highlights some part of the
language that is maybe unnecessarily rigid.

> Another workaround is to wrap the Stream_Element_Array in a record

Good idea.

Still, in both of these solutions, there is a problem of scope - the
type of access value is defined by the library API (well, at the
library level), whereas user-provided buffers are likely to be defined
locally.
This brings the old problem of having an access value of global type
pointing to local object (GNAT says: "non-local pointer cannot point
to local object").
Unrestricted_Access hides the issue, but I know from Rationale that
anonymous access types were supposed to solve some of that mess. The
problem is that I cannot find any way to benefit from them.
Let's say:

   type Buffer (Size : Ada.Streams.Stream_Element_Count) is record
      B : Ada.Streams.Stream_Element_Array (1 .. Size);
   end record;

   type Buffer_List is array (Positive range <>) of access Buffer;

Buffer_List looks like an array of *anonymous* access values (and they
don't have to be "all"!), but is not helping with the scope mismatch:

declare
   My_Buf : aliased Buffer (Some_Size);  -- local buffer
   Buffers : Buffer_List (1 .. 1);
begin
   Buffers (1) := My_Buf'Access;   -- error! scope mismatch
   Scatter_Input_Data (Buffers);
end;

Again - Unrestricted_Access hides the mess under the carpet, but does
not look nice. Any idea how to solve that?

Except for this, everything seems to work like a charm, thank you all
for your feedback.

--
Maciej Sobczak * www.msobczak.com * www.inspirel.com

Database Access Library for Ada: www.inspirel.com/soci-ada

Re: API design problem - buffer scatter I/O

[Robert A Duff]

Maciej Sobczak <see.my.homepage@gmail.com> writes:

> On 22 Lis, 23:16, Robert A Duff <bobd...@shell01.TheWorld.com> wrote:
>
>> There is no such concept as "fat pointer" or "thin pointer" in Ada!
>
> I understand that the distinction is to make the reader aware that
> some access values need to carry over the bounds whereas some other
> don't.

Perhaps, but I still find that wiki page confusing.

When you have an access to unconstrained array, the implementation
has to arrange some way to find the bounds -- e.g. if the program
says X.all'Last.  Fat pointers are one way to do that, but not the
only one.  GNAT can use either fat pointers or thin pointers
(for access to unconstrained array) -- it's a programmer choice.

And (I claim) this has no place in a description of the semantics
of the language.  It's an implementation detail.

>> It does point to a real problem, though. �Given your above declarations,
>> you might want to say:
>>
>> � � This_Buf : aliased Stream_Element_Array (1..10);
>> � � That_Buf : aliased Stream_Element_Array (1..10_000);
>>
>> � � Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- wrong!
>>
>> but that's illegal. �You can do this instead:
>>
>> � � This_Buf : aliased Stream_Element_Array := (1..10 => <>);
>> � � That_Buf : aliased Stream_Element_Array := (1..10_000 => <>);
>>
>> � � Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- OK
>
> Yes, I have noticed that. Not a big issue for the user (all bounds can
> be still variable at run-time), but highlights some part of the
> language that is maybe unnecessarily rigid.

Perhaps.  If the above "wrong!" thing were allowed, there would be a
slight efficiency hit.  That (efficiency) is the _only_ reason for
this rule.  Maybe it was a mistake.

>> Another workaround is to wrap the Stream_Element_Array in a record
>
> Good idea.
>
> Still, in both of these solutions, there is a problem of scope - the
> type of access value is defined by the library API (well, at the
> library level), whereas user-provided buffers are likely to be defined
> locally.

Right.  You have to use Local'Unchecked_Access, and take care to avoid
dangling pointers.

> This brings the old problem of having an access value of global type
> pointing to local object (GNAT says: "non-local pointer cannot point
> to local object").
> Unrestricted_Access hides the issue,...

Don't use Unrestricted_Access here (which is not standard Ada).
Use Unchecked_Access.  Yes, it's dangerous -- you need to ensure that
Scatter_Input_Data doesn't save pointers to locals into globals,
along with comments explaining what clients can do.
There's no better solution.

>... but I know from Rationale that
> anonymous access types were supposed to solve some of that mess.

"Supposed to", maybe.  But they don't.  I suggest you stay away from
them.  They are confusing, and don't help here.

>...The
> problem is that I cannot find any way to benefit from them.

Right.  Neither can I.
[snipped discussion of failed attempts to use anonymous
access types]

By the way, ARG is discussing some ideas for improving anonymous
access types.  My own opinion is that this will not work out,
but others do not agree with me.

- Bob

Re: API design problem - buffer scatter I/O

[anon]

Even though Fat/Thin pointer are not in the true Ada specification they are 
use.  In GNAT:

Thin pointer are less efficient representation and not as flexibility as "Far 
pointer". And the bound value are storaged right behind the array Also, 
Thin pointers limit the use of "Unrestricted_Access" attribute by forcing 
the usage of non-aliased objects pointers.

        type X is access all String ;
          for X'Size use Standard'Address_Size ;


In the case of unconstrained array type, the default is to use the "Far Pointer". 
Defined as a record with two pointers.  One pointer is assigned to the array 
type, which translates to a System.Address. While the other pointer is assign 
to a bounds structure with the two fields one for the lower and the other the 
upper bounds. 

Something like this:

   type Bounds_Type is record                          -- hidden definition
                         Lower : Natural ;
                         Upper : Natural ;
                       end record ;

   type Fat_Pointer is record                          -- hidden definition
                         P_Array : System.Address ;
                         P_Bound : Bounds_Type ;
                       end record ;


In <f980d11e-4113-41a8-9c24-bf2df53e18f8@a12g2000yqm.googlegroups.com>, Maciej Sobczak <see.my.homepage@gmail.com> writes:
>I have a problem finding the best, from the Ada point of view,
>interface for the scatter I/O API.
>
>The idea is that the library is able to read some bytes from somewhere
>and put them in the buffers provided by user. The simplest, single-
>buffer version of this idea is used in the standard library and in
>many others - an example from GNAT.Sockets looks like here:
>
>procedure Receive_Socket
>     (Socket : Socket_Type;
>      Item   : out Ada.Streams.Stream_Element_Array;
>      Last   : out Ada.Streams.Stream_Element_Offset);
>
>In other words, user provides a single array and the library fills it
>with data.
>
>The problem is that this is a single, continuous array.
>How would you approach the design of API for multi-buffer I/O? Multi-
>buffer means that the user can provide many arrays, probably of
>different sizes, and the library will fill them from first to last.
>
>The following assumptions are true:
>1. The user knows up-front how much data will be read. It will never
>be less, so there is no need to give back any Last index, like above.
>The user has to only ensure that the sum of buffer lengths is
>appropriate for all the data that will be read.
>2. Buffers don't need to have the same size.
>3. Individual buffers are not necessarily related (they don't have to
>be components of common bigger structure like record or array).
>4. It really has to be a single operation, so no cheating with
>repeated calls for separate buffers to get consecutive fragments.
>
>My initial idea was that the subprogram might have a parameter that is
>an array of access values to separate buffers (actual buffers must be
>therefore aliased). Are buffer bounds properly handled when only
>access to array is given (didn't check that yet, sorry)?
>
>My initial idea:
>
>type Buffer_Ptr is access Ada.Streams.Stream_Element_Array;
>type Buffers is array (Positive range <>) of Buffer_Ptr;
>
>procedure Scatter_Input_Data (B : in Buffers);
>
>What is bothering me is <http://en.wikibooks.org/wiki/Ada_Programming/
>Types/access> and the discussion on designated vs. nominal subtypes in
>the "Fat pointers" paragraph.
>
>Have you seen an existing library that solves this problem is some
>reasonable way?
>Do you have other ideas or suggestions for how to design such API?
>
>--
>Maciej Sobczak * www.msobczak.com * www.inspirel.com
>
>Database Access Library for Ada: www.inspirel.com/soci-ada

Re: API design problem - buffer scatter I/O

[christoph.grein]

On 22 Nov., 23:16, Robert A Duff <bobd...@shell01.TheWorld.com> wrote:
> I didn't read the whole wikibook, but the particular part you point to
> above has quite a bit of misinformation.  Also, the early focus on
> "fat pointers" vs. "thin pointers" is confusing.  There's no reason
> to talk about such implementation details in an introductory tutorial.
> Especially if the details are wrong.
>
> There is no such concept as "fat pointer" or "thin pointer" in Ada!
>
> It does point to a real problem, though.  Given your above declarations,
> you might want to say:
>
>     This_Buf : aliased Stream_Element_Array (1..10);
>     That_Buf : aliased Stream_Element_Array (1..10_000);
>
>     Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- wrong!
>
> but that's illegal.  You can do this instead:
>
>     This_Buf : aliased Stream_Element_Array := (1..10 => <>);
>     That_Buf : aliased Stream_Element_Array := (1..10_000 => <>);
>
>     Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- OK

May be the nomenclature in wiki is wrong or confusing (and you always
have to be skeptical about the contents), but the incompatibilities
described there exist. And I remember a discussion with Robert Dewar
that Access_To_Address_Conversion does not work for access to
unconstrained just because of the bounds (in GNAT, this is just an
Unchecked_Conversion of the Address in either way).

So, Bob, could you please elaborate about what is wrong in this
description.

And I disagree that this should not be in wiki, because programmers
inevitable will fall into this trap (as I did) and the corresponding
RM pages are very difficult to grok. This is not an implementation
detail.

I do not know what a thin and a fat and a far pointer actually are,
I'm no Ada implementor. From my point of view as language user, a thin
pointer is simply an address, a fat pointer an address and the bounds.
So just the use of "thin" and "fat" is wrong, because it's an
implementation detail?

Re: API design problem - buffer scatter I/O

[Brad Moore]

Robert A Duff wrote:
> Maciej Sobczak <see.my.homepage@gmail.com> writes:
>> Still, in both of these solutions, there is a problem of scope - the
>> type of access value is defined by the library API (well, at the
>> library level), whereas user-provided buffers are likely to be defined
>> locally.
> 
> Right.  You have to use Local'Unchecked_Access, and take care to avoid
> dangling pointers.

I have been working on a set of general purpose buffers, and was
curious to see how I could incorporate a scatter buffer implementation
into my buffer framework. I ended up with a design that seems to satisfy 
the requirements stated in the original post. My experimentation led to 
some results that might be of interest, and I have a compiled test 
driver that reads and writes data to a scatter buffer.

1) The design I am using is an expansion of the idea discussed earlier 
in this thread. That is, a record type with a discriminant is used to
define the size of the array. In addition, the package is a generic
where the type of object to be stored in the buffer is passed in as a
generic formal parameter.

2) As part of the instantiation for a scatter buffer, I pass in the
array of buffers as a generic formal parameter. Passing in these
parameters has other benefits to my design, which I wont bother
trying to explain here.

generic

    type Scatterbuffer_Array_Type is
      array (Count_Type range <>) of access Buffer_Interface'Class;

    Scatter_Buffers : Scatterbuffer_Array_Type;

package Buffers.Scattered
    pragma Remote_Types;
...


3) Note that I am using anonymous access types in the generic formal
Scatterbuffer_Array_Type

I didn't have any issues with using anonymous access types. In my
main test procedure, I was able to use 'Access in a nested scope
without having to use 'Unchecked_Access. In fact, my implementation
requires anonymous access types, and I don't believe I could have
got this to work without using Ada 2005's anonymous access feature.
I actually tried, (after commenting out the pragma Remote_Types),
and using named access types, but the compiler wouldn't compile the code.
I suspect this is a compiler bug, but since I need the pragma 
Remote_Types, I didn't pursue that any further.

The need to use anonymous access types has more to do with the fact that 
I allow buffers to be accessed
remotely, but I need the buffers in the buffer list to be anonymous so 
that they are treated as local buffers, not remote ones.

The main test procedure looks like;

with Buffers.Simple_Bounded;
with Buffers.Simple_Unbounded;
with Buffers.Simple_Scattered;
with Buffers.Passive_Bounded;

procedure Main_Test is

    -- Create a root buffer framework for storing text (Character data)
    package My_Buffer is new Buffers
      (Element_Index_Type => Natural,
       Element_Type => Character,
       Element_Vector_Type => String);

    -- Instantiate some differing buffer types
    package My_Simple_Bounded_Buffer is new My_Buffer.Simple_Bounded;
    package My_Simple_Unbounded_Buffer is new My_Buffer.Simple_Unbounded;
    package My_Passive_Bounded_Buffer is new My_Buffer.Passive_Bounded;

   -- Declare buffers of different types and different sizes
   My_Bounded : aliased My_Simple_Bounded_Buffer.Buffer
      (Maximum_Capacity => 100);
   My_Unbounded : aliased My_Simple_Unbounded_Buffer.Buffer
      (Maximum_Capacity => 5000);
   My_Passive : aliased My_Passive_Bounded_Buffer
      (Maximum_Capacity => 33);

   -- Declare the type needed by Scatter Buffer
   type Scatterbuffer_Array
      is array (Ada.Containers.Count_Type range <>) of access
         My_Buffer.Buffer_Interface'Class;

    -- Declare the list of buffers to be used by the scatter buffer

    -- Note using the 'Access in a nested scope. Works fine.
    Scatterbuffers : constant Scatterbuffer_Array := Scatterbuffer_Array'
      (1 => My_Bounded'Access,
       2 => My_Unbounded'Access,
       3 => My_Passive'Access);  -- Synchronous Buffer based on Protected

    -- Instantiate the scatter buffer
    package My_Simple_Scatter_Buffer is new
      My_Buffer.Simple_Scattered
        (Scatterbuffer_List => Scatterbuffer_Array,
         Scatter_Buffers => Scatterbuffers);

    -- Declare the Scatter Buffer
    My_Scattered : My_Simple_Scatter_Buffer.Buffer
      (Maximum_Capacity => 5133);
begin
    -- Each write request typically gets written to a different buffer
    -- or can be split across buffers, if a targetted buffer does
    -- not have the available space
    My_Scattered.Write ("Hello World");

    -- Each read request reads from a separate buffer
    -- or can be split across buffers, if a buffer does not
    -- have the available data
    Put_Line ("Text Read=" & My_Scattered.Read (Maximum_Count => 5));
  -- Reads a string

end Main_Test;

The use of Ada 2005 interfaces is also helpful here.
Some of the buffers in the scatter list are synchronous buffer types,
based on a task or a protected type. Others are intended only for
non-synchronous use.

Scatter buffers can in fact appear in the list of buffers in another 
scatter buffer list.

If a scatter buffer is acting as a producer, then there can be separate 
consumers for each of the buffers in the buffer list.

Likewise, if a scatter buffer is acting as a consumer, then there can be 
separate producers for each of the buffers in the buffer list.

It is possible to have one scatter buffer instance as a producer, and 
another scatter buffer instance as a consumer, for the same list of buffers.

My framework also gives me synchronous scatter buffer forms where the
scatter buffer itself can be synchronized to allow multiple readers/writers.

I am thinking of releasing the source under some license similar to the 
GNAT license, but haven't sorted out the details of how to go about this 
yet.

- Brad

> 
>> This brings the old problem of having an access value of global type
>> pointing to local object (GNAT says: "non-local pointer cannot point
>> to local object").
>> Unrestricted_Access hides the issue,...
> 
> Don't use Unrestricted_Access here (which is not standard Ada).
> Use Unchecked_Access.  Yes, it's dangerous -- you need to ensure that
> Scatter_Input_Data doesn't save pointers to locals into globals,
> along with comments explaining what clients can do.
> There's no better solution.
> 
>> ... but I know from Rationale that
>> anonymous access types were supposed to solve some of that mess.
> 
> "Supposed to", maybe.  But they don't.  I suggest you stay away from
> them.  They are confusing, and don't help here.
> 
>> ...The
>> problem is that I cannot find any way to benefit from them.
> 
> Right.  Neither can I.
> [snipped discussion of failed attempts to use anonymous
> access types]
> 
> By the way, ARG is discussing some ideas for improving anonymous
> access types.  My own opinion is that this will not work out,
> but others do not agree with me.
> 
> - Bob

Re: API design problem - buffer scatter I/O

[tmoran]

> The problem is that this is a single, continuous array.
> How would you approach the design of API for multi-buffer I/O? Multi-
> buffer means that the user can provide many arrays, probably of
> different sizes, and the library will fill them from first to last.
  Is this an actual problem?  In most systems copying the bytes from
an IO buffer into different places will be a small percentage of the
time taken by the actual IO, so the simplicity of a single IO buffer
is worth the modest cost.

Re: API design problem - buffer scatter I/O

[Robert A Duff]

christoph.grein@eurocopter.com writes:

> May be the nomenclature in wiki is wrong or confusing (and you always
> have to be skeptical about the contents), but the incompatibilities
> described there exist. And I remember a discussion with Robert Dewar
> that Access_To_Address_Conversion does not work for access to
> unconstrained just because of the bounds (in GNAT, this is just an
> Unchecked_Conversion of the Address in either way).
>
> So, Bob, could you please elaborate about what is wrong in this
> description.

Well, the very first sentence starts talking about addresses,
which is a confusing approach, IMHO.

I have no objection to teaching people about implementation details
(like thin and fat pointers), but that should come at the end.
FIRST learn about the semantics of access types, which are
pretty much the same as "pointers" or "references" in many
other languages.  (Access types do not allow address arithmetic,
as C and C++ pointers do.  But there are many other languages
that are just like Ada in this regard.)

It says, "objects of this category do not really point to objects":
Of course they do.  "Designate" is the Ada jargon, but "point to" is
just fine as an informal term.  It then introduces its own jargon
"grant access", which is unhelpful.  There's no "granting" going
on here.

It says, "Thin pointers grant access to constrained subtypes."
No.  In all Ada compilers I know of except GNAT, thin pointers
are used in all cases -- there are no fat pointers.  In GNAT,
thin pointers are used for constrained subtypes, but also for
unconstrained scalars, unconstrained records, unconstrained tasks, and
so forth.  The ONLY case where fat pointers are used is for
unconstrained arrays -- and that's just the default (there is
an option to use thin pointers in that case as well).

It implies that constrained subtypes have a static size.
That's not correct -- many constrained subtypes have
dynamic size.

It makes statements about Address_To_Access_Converions that
are not necessarily true of Ada in general -- whether it's
safe depends entirely on the implementation.  And "thin safe,
fat unsafe" isn't quite right anyway -- thin pointers
don't always work with Address_To_Access_Converions, either.

It implies that fat pointers are somehow required for unconstrained
arrays.  It's not true -- as I said, this is just one implementation
approach.

> And I disagree that this should not be in wiki, ...

Sorry, I did not mean to imply it shouldn't be there.
I just don't think it should be used in the explanation
of access types semantics.  Later, there could be some
explanation of implementation strategies.  And it should
take care not to imply that any particular implementation strategy
is required by the language rules.

>... because programmers
> inevitable will fall into this trap (as I did) ...

Beginners have no business using Address_To_Access_Converions in the
first place.  This is an advanced, low-level, and dangerous feature.
Normal use of access types doesn't involve Address_To_Access_Converions
at all.

Warning people about the traps of Address_To_Access_Converions is a good
idea -- but it belongs in the section on Address_To_Access_Converions,
not in the section on access types.

>... and the corresponding
> RM pages are very difficult to grok.

Yes, I agree -- the RM is rather tough going!  I'm not trying to
denigrate the attempt to explain things in a more accessible way.

>...This is not an implementation
> detail.

Sure it is.  How a compiler represents data types (including access
types) is an implementation detail.  You don't need to know about it
until you need to use some low-level chap-13-ish features (and I include
interfacing to C in that category).

> I do not know what a thin and a fat and a far pointer actually are,
> I'm no Ada implementor. From my point of view as language user, a thin
> pointer is simply an address, a fat pointer an address and the bounds.

Almost right.  Change "bounds" to "address of bounds".

> So just the use of "thin" and "fat" is wrong, because it's an
> implementation detail?

It's not "wrong" to teach people about thin and fat pointers.
But it's unhelpful to introduce those concepts to beginners
up front.

- Bob

Re: API design problem - buffer scatter I/O

[Robert A Duff]

christoph.grein@eurocopter.com writes:

> So, Bob, could you please elaborate about what is wrong in this
> description.

Let me make an analogy:

If I want to teach someone how to operate a car,
I will first tell them about the steering wheel,
the clutch, the brake pedal, stop signs, traffic
lights and so forth.

Later on, I might want to teach them about pistons
and spark plugs and gears.  But that "under the hood"
stuff is too much confusing detail at first.  It should
come later.  And I should make it clear that "four cylinders"
is just one way to do it -- some have 6 or 8, and maybe
some (electric cars?) don't have cylinders at all.
You still have to stop a stop sign, when driving an
electric car.

Likewise, for access types, beginners to Ada (whether or not they know
other languages first) need to know about .all and new and
Unchecked_Deallocation.  Later on, they can learn about interfacing to
C, unchecked conversion, and Address_To_Access_Conversions -- and maybe
even thin/fat pointers.  But I should make it clear that "fat pointers"
is just one way to do it.

In any case, it is certainly valuable to have a wiki or whatever
that presents things to programmers in a simpler way than the RM!

- Bob

Re: API design problem - buffer scatter I/O

[christoph.grein]

Thank you, Bob, for these illustrative comments. Problems with wiki is
that people with no good knowledge about the innards of a language
implementation may contribute (and I include myself into this class).

I fear you do not feel like correcting the wiki. I know, it's much
work, the whole chapter has to be rearranged.

> >...This is not an implementation
> > detail.
>
> Sure it is.  How a compiler represents data types (including access
> types) is an implementation detail.

What I meant is the incompatibility between pointers to constrained
and unconstrained:

type Unc     is array (Integer range <>) of Character;
type Acc_Unc is access Unc;

CO: aliased Unc (-1 .. +1) := (-1 .. +1 => ' ');
UO: aliased Unc            := (-1 .. +1 => ' ');

A: Acc_Unc            := CO'Access;  -- illegal
B: Acc_Unc            := UO'Access;  -- OK
C: Acc_Unc (CO'Range) := CO'Access;  -- also illegal

which is not an implementation detail.

Re: API design problem - buffer scatter I/O

[Dmitry A. Kazakov]

On Mon, 24 Nov 2008 21:59:41 -0800 (PST), christoph.grein@eurocopter.com
wrote:

>> Sure it is. �How a compiler represents data types (including access
>> types) is an implementation detail.
> 
> What I meant is the incompatibility between pointers to constrained
> and unconstrained:

I agree with you. Pointer to a subtype is not necessarily a subtype of the
pointer to the base. That is not an implementation detail. It should be per
programmer's choice. Unfortunately Ada lacks means to propagate type
constraints between related types, which is the issue here. We need a way
to say whether the constraint has to be moved from the target to the
pointer, making the later unconstrained. That does not necessary imply that
the pointer's representation must become "fat." That indeed would be an
implementation detail, as Robert says. But the decision is not a detail.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: API design problem - buffer scatter I/O

[Robert A Duff]

christoph.grein@eurocopter.com writes:

> Thank you, Bob, for these illustrative comments. Problems with wiki is
> that people with no good knowledge about the innards of a language
> implementation may contribute (and I include myself into this class).
>
> I fear you do not feel like correcting the wiki. I know, it's much
> work, the whole chapter has to be rearranged.

You're right -- I don't have time to do a good job of that,
so I'm not even going to try.

I do appreciate the efforts of others who choose to help out
with the wiki, even if it's not always perfect.

>> >...This is not an implementation
>> > detail.
>>
>> Sure it is. �How a compiler represents data types (including access
>> types) is an implementation detail.
>
> What I meant is the incompatibility between pointers to constrained
> and unconstrained:
>
> type Unc     is array (Integer range <>) of Character;
> type Acc_Unc is access Unc;
>
> CO: aliased Unc (-1 .. +1) := (-1 .. +1 => ' ');
> UO: aliased Unc            := (-1 .. +1 => ' ');
>
> A: Acc_Unc            := CO'Access;  -- illegal
> B: Acc_Unc            := UO'Access;  -- OK
> C: Acc_Unc (CO'Range) := CO'Access;  -- also illegal
>
> which is not an implementation detail.

Ah, sorry, I misunderstood what you were referring to.  I agree that
this restriction (A is illegal) is not an implementation detail.  The
declaration of A makes perfect sense, but it is forbidden for efficiency
reasons.  Well, not efficiency, really -- distributed overhead.  Some
say that was a bad language design choice, and I'm at least half
convinced.  ;-)

I'd suggest removing C from the wiki.  Constrained subtypes of access
types are a weird and nearly-useless feature.  I wouldn't teach
about them to beginners -- they're confusing.

- Bob

Re: API design problem - buffer scatter I/O

[Randy Brukardt]

"Robert A Duff" <bobduff@shell01.TheWorld.com> wrote in message 
news:wcciqqcki2r.fsf@shell01.TheWorld.com...
>>> It does point to a real problem, though.  Given your above declarations,
>>> you might want to say:

>>>     This_Buf : aliased Stream_Element_Array (1..10);
>>>     That_Buf : aliased Stream_Element_Array (1..10_000);


>>>     Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- wrong!


>>> but that's illegal.  You can do this instead:


>>>     This_Buf : aliased Stream_Element_Array := (1..10 => <>);
>>>     That_Buf : aliased Stream_Element_Array := (1..10_000 => <>);


>>>     Scatter_Input_Data ((This_Buf'Access, That_Buf'Access)); -- OK


>> Yes, I have noticed that. Not a big issue for the user (all bounds can
>> be still variable at run-time), but highlights some part of the
>> language that is maybe unnecessarily rigid.



>Perhaps.  If the above "wrong!" thing were allowed, there would be a
>slight efficiency hit.  That (efficiency) is the _only_ reason for
>this rule.  Maybe it was a mistake.

I don't think that the efficiency hit is *slight*, although it matters on 
whether you use access-to-constrained-array much.

The problem is that if you don't have the bounds for the array currently, 
you have to add them somewhere. But at the point of the conversion/'Access, 
you don't know how long the bounds will be needed, so you pretty much have 
to allocate them globally and that is a storage leak (they'll never be 
recovered during the life of the program).

Alternatively, you can just forget the optimization of removing the bounds, 
and put the bounds on all (aliased) array objects, which includes all 
allocated objects of access-to-constrained. Since this is likely the same 
mechanism as used for passing array bounds as parameters (it surely is in 
Janus/Ada), it also could have a cost on slicing. (Bounds can't be 
contiguous with the rest of the object in general, because of slicing.) 
Optimization can remove some of that cost, but not most of it.

The unoptimizable case occurs when converting access-to-constrained to 
access-to-unconstrained. The result is that the bounds have to be passed 
with all access-to-array types, because they can't be created out of thin 
air (else they'll leak memory). As such, access-to-constrained-array end up 
with a run-time penalty (either a fat pointer or an extra level of 
indirection) that they otherwise would not have.

The problem of course is that you need overhead (space and time) in other 
parts of the language whether or not you take advantage of this particular 
feature. Generally, we try to avoid overhead that occurs whether or not you 
use something.

                         Randy.

P.S. And I didn't even have to mention shared generics! :-)

Re: API design problem - buffer scatter I/O

[Maciej Sobczak]

On 24 Lis, 18:16, tmo...@acm.org wrote:

> > The problem is that this is a single, continuous array.

>   Is this an actual problem?

Not im my case, really. Actually, I made up the I/O part because I
didn't want to expose the real problem case. I have still got an
excellent feedback that allowed me to fill the blanks in what I do and
it really helped. :-)

We can still argue (for the sake of technical discussion) that the
single buffer can be a problem even for real I/O, because it *might*
imply the use of dynamic memory, which for some reasons we might want
to avoid. Having a set of static buffers and custom scheme of
associating them with the fragments of the data stream might be an
interesting solution.
Of course, you could still argue that in the case of real I/O it might
be done with just a sequence of separate I/O calls for each involved
buffer, so again scatter operation is not necessary, but then the
counterargument would be for example atomicity of the whole data
transfer. And so on.

> In most systems copying the bytes from
> an IO buffer into different places will be a small percentage of the
> time taken by the actual IO, so the simplicity of a single IO buffer
> is worth the modest cost.

Yes.
But then, the I/O might be an interprocess communication based on
shared memory where "copying bytes" is the only factor and therefore
not at all a small percentage. And so on. ;-)

--
Maciej Sobczak * www.msobczak.com * www.inspirel.com

Database Access Library for Ada: www.inspirel.com/soci-ada

Re: API design problem - buffer scatter I/O

[sjw]

On Nov 26, 8:34 am, Maciej Sobczak <see.my.homep...@gmail.com> wrote:

> Of course, you could still argue that in the case of real I/O it might
> be done with just a sequence of separate I/O calls for each involved
> buffer, so again scatter operation is not necessary, but then the
> counterargument would be for example atomicity of the whole data
> transfer. And so on.

And is quite unlikely to work for UDP (you have to send/receive the
whole datagram in one go ..)