A novel design of linked lists (was: Address of an object)

A novel design of linked lists (was: Address of an object)

[Dmitry A. Kazakov]

[ It seems to me that some people have an impression that I am looking for
something extraordinary when asking for the object address, so I feel
necessary to provide an elaborated example where it were needed. ]

Let's consider referential double-linked lists as an example. The drawbacks
of the Ada.Containers design are obvious. A typical design of such lists in
other languages would be adding Prev, Next pointers to the elements. Note
also, that we wished to have linked lists of elements which might
participate in many lists simultaneously. We also liked to statically check
the types of such lists, so that paths in different lists could never
converge.

There are two ways to do add links Prevs and Nexts. One of them is to do it
upon inheritance. Another is by using new elements containing pointers to
the "true" elements.

1. The variant with inheritance is not an option in current Ada, because of
lack of MI. Even with MI it faces the problem that links has to be added to
the base type. That would require downcasting later, all over the program.
Adding links in the leaves of the type hierarchy would break it apart and
also freeze the types for future derivation. Ad-hoc supertypes could help,
but, firstly, they are not supported and, secondly, they barely could have
referential semantics. So the elements will require copying.

2. The variant with pointers is not a solution at all, because, honestly,
it would be a list of pointers rather than of the elements.

An alternative to these two variants would be to add links transparently
without intervening with *what* is the element. Ada's pools provide this
functionality. The package could go as follows:

with System;                   use System;
with System.Storage_Elements;  use System.Storage_Elements;
with System.Storage_Pools;     use System.Storage_Pools;

generic
   type List_Identification_Type is (<>);
   type List_Item_Type (<>) is limited private;
   Pool : in out Root_Storage_Pool'Class;
package Generic_Linked_Lists is

Here:

- List_Identification_Type is an enumeration type to name the lists. An
element may participate in exactly one list corresponding to the given
value of List_Identification_Type.
- List_Item_Type is the type of elements.
- Pool is the pool where elements will be eventually allocated

The package defines its own pool type, which takes memory from another pool
and the object Items_Pool of this type bound to the actual pool passed as
the generic formal parameter Pool. [ It would be nice to default it to the
standard pool, but, I don't know any good way to do it. ]

   type Items_Storage_Pool (Host : access Root_Storage_Pool'Class) is
      new Root_Storage_Pool with null record;
   ...
   Items_Pool : Items_Storage_Pool (Pool'Access);

Then it defines a pointer type to refer the list items:

   type Item_Ptr is access List_Item_Type;
   for Item_Ptr'Storage_Pool use Items_Pool;

Then it defines the list operations in the terms of this pointer type. Note
that unlikely to Ada.Containers the semantics is referential. Nothing is
copied. So Insert looks like:

   procedure Insert
             (  List  : List_Identification_Type;
                Head  : in out Item_Ptr;
                Item  : Item_Ptr;
                Order : Item_Disposition := After
             );

- List identifies the list type we are dealing with. The same item can be
situated in as many lists as the range of List_Identification_Type.

- Head is the pointer to the list. In can be null. when the first item is
inserted.

- Item is the pointer to the item to insert.

- Order is an enumeration After or Before. When Item is placed before Head,
Head is set to Item.

The use of Insert is trivial and need not to be illustrated. A new Item is
created using plain "new." Other operations could be:

   procedure Move
             (  List   : List_Identification_Type;
                Target : in out Item_Ptr;
                Item   : Item_Ptr;
                Source : in out Item_Ptr;
                Order  : Item_Disposition := After
             );

Moves Item form one list to another. Both lists are of the same type.

   function Next
            (  List : List_Identification_Type;
               Item : Item_Ptr
            )  return Item_Ptr;

Returns the next item in the list

   function Previous
            (  List : List_Identification_Type;
               Item : Item_Ptr
            )  return Item_Ptr;

   procedure Remove
             (  List : List_Identification_Type;
                Head  : in out Item_Ptr; 
                Item  : Item_Ptr;
                Order : Item_Disposition := After
             );

Removes Item from the list.

For dealing with the elements of one list type without specifying the type,
a small wrapper package could be provided:

generic
   List : List_Identification_Type;
package Generic_Linked_Lists.Generic_List is
   type List_Ptr is new Item_Ptr;
   procedure Insert
             (  Head  : in out List_Ptr;
                Item  : Item_Ptr;
                Order : Item_Disposition := After
             );
   procedure Move
             (  Target : in out List_Ptr;
                Item   : Item_Ptr;
                Source : in out List_Ptr;
                Order  : Item_Disposition := After
             );
   function Next (Item : Item_Ptr)  return List_Ptr;
   function Previous (Item : Item_Ptr) return List_Ptr;
   procedure Remove
             (  Head  : in out List_Ptr; 
                Item  : Item_Ptr;
                Order : Item_Disposition := After
             );
end Generic_Linked_Lists.Generic_List;

Note that locally defined List_Ptr is a "list specific" pointer, while Item
is a sort of "class-wide" pointer, valid across all list types. So Item in
all calls is just a pointer to an item, while Head controls "dispatch" to
the proper list. [ Of course no dynamic dispatch happens. It is a static
polymorphism. ]

Now how this works. When an element is allocated, Allocate of
Items_Storage_Pool places the array of links

   type List_Header is record
      Next : Address;
      Prev : Address;
   end record;
   type Item_Header is array (List_Identification_Type) of List_Header;

in front of it. Allocate looks like this:

   procedure Allocate
             (  Pool            : in out Items_Storage_Pool;
                Storage_Address : out Address; 
                Size            : Storage_Count;
                Alignment       : Storage_Count
             )  is
      Header_Alignment : constant Storage_Count :=
         Storage_Count'Max (Item_Header'Alignment, Alignment);
      Header_Offset    : constant Storage_Offset :=
         Header_Size + (-Header_Size) mod Header_Alignment;
   begin
      Allocate -- Grab memory in the host pool
      (  Pool.Host.all,
         Storage_Address,
         Size + Header_Offset,
         Header_Alignment
      );
      Storage_Address := Storage_Address + Header_Offset;
      declare
         Header : Item_Header renames To_Header (Storage_Address).all;
      begin
         for List in Header'Range loop
            Header (List).Next := Null_Address; -- Links initialization
         end loop;
      end;
   end Allocate;

When Next, Insert etc are called Item_Header is obtained from the pointer
and, the rest is obvious. For example Next:

   function Next
            (  List : List_Identification_Type;
               Item : Item_Ptr
            )  return Item_Ptr is
   begin
      return To_Item_Ptr (To_Header (From_Item_Ptr (Item)) (List).Next);
   end Next;

It works perfectly well for all types except ones for which the compiler
mangles pointers, as I have explained in other thread.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: A novel design of linked lists (was: Address of an object)

[Adam Beneschan]

Dmitry A. Kazakov wrote:
> [ It seems to me that some people have an impression that I am looking for
> something extraordinary when asking for the object address, so I feel
> necessary to provide an elaborated example where it were needed. ]
>
> Let's consider referential double-linked lists as an example. The drawbacks
> of the Ada.Containers design are obvious. A typical design of such lists in
> other languages would be adding Prev, Next pointers to the elements. Note
> also, that we wished to have linked lists of elements which might
> participate in many lists simultaneously. We also liked to statically check
> the types of such lists, so that paths in different lists could never
> converge.
>
> There are two ways to do add links Prevs and Nexts. One of them is to do it
> upon inheritance. Another is by using new elements containing pointers to
> the "true" elements.
>
> 1. The variant with inheritance is not an option in current Ada, because of
> lack of MI. Even with MI it faces the problem that links has to be added to
> the base type. That would require downcasting later, all over the program.
> Adding links in the leaves of the type hierarchy would break it apart and
> also freeze the types for future derivation. Ad-hoc supertypes could help,
> but, firstly, they are not supported and, secondly, they barely could have
> referential semantics. So the elements will require copying.
>
> 2. The variant with pointers is not a solution at all, because, honestly,
> it would be a list of pointers rather than of the elements.

I'll have to admit that I don't see what "problem" you have that this
isn't a solution to.  In fact, I'm not really clear on why it's
important to you to adopt the approach you're adopting; your
explanation is pretty sketchy.

Part of my problem is that you seem to want to define a type, Item_Ptr,
that gives you more than it's supposed to give you.  Your generic
declares

   type Item_Ptr is access List_Type_Item;

where List_Type_Item is whatever type you're making a list out of.  I
see a type like this, and I think it's just an access to the
List_Type_Item, but there's actually additional baggage in this
pointer, i.e. a "next" and a "previous" pointer.  So if I see code that
calls Next on this Item_Ptr, my first thought is "Whooaa!  Item_Ptr is
just an access to my Employee record, so where the heck is it going to
get a pointer to another Employee?"  It would be pretty confusing to
me. Personally, if I'm going to have something that contains additional
information, I'd prefer that it be a different type.  The need for
downcasting, type conversion, or a Deref function in your generic that
takes an Item_Ptr and returns the dereferenced item that I would use
instead of .all, doesn't bother me at all.  So I don't see what problem
there is that needs solving.

Plus, you introduce a problem.  Assuming the actual for List_Type_Item
is T, you've defined Item_Ptr as an access to T; what happens if
somewhere else in the program, there is another type that is also an
access to T?  The language will let you convert between the two access
types, but the result would be an absolute disaster.  But I just know
that someone who doesn't really understand that Item_Ptr isn't just an
access to T is going to try it.

Put simply, you're trying to trick your Ada compiler into allowing
programmers to use Ada syntax to write code in some other language than
Ada, and someone who looks at the Ada code and assumes it's Ada is
likely to be pretty darn confused.

Anyway, now that I've given you my tirade on why I think this is a bad
approach, I'm going to give you an idea of how you might implement your
bad approach.  The problem with your Next function is that if
List_Item_Type is an unconstrained array, there will be extra data
between the thing Item_Ptr points to and the Item_Header.  Since you've
already admitted that this is going to be highly
implementation-dependent, then you can probably assume that the size of
this extra data is going to be the same always (for a given
List_Item_Type), and then all you have to do is figure out what that
size is the first time Next [or some other routine] is called.  Declare
a variable in your package body:

    Dope_Offset : Storage_Offset := -1;

Fix your Allocate so that it saves the resulting Storage_Address in
some global Global_Storage_Address in the package body before it
returns.  Inside Next:

   X : Item_Ptr;
   ...
   if Dope_Offset = -1 then
     X := new List_Item_Type' (Item.all);
     Dope_Offset := To_Address(X) - Global_Storage_Address;
     Free(X);
  end if;

where To_Address is from an instance of
System.Address_To_Access_Conversions.  Now once Dope_Offset has been
set up, you can use it as an offset and subtract it from your
From_Item_Ptr calls.  

                         -- Adam

Re: A novel design of linked lists (was: Address of an object)

[Leif Holmgren]

Dmitry A. Kazakov wrote:
> 
> 1. The variant with inheritance is not an option in current Ada, because of
> ...
> 
> 2. The variant with pointers is not a solution at all, because, honestly,
> ...
> 

3. Generics

Re: A novel design of linked lists

[Dmitry A. Kazakov]

On 19 Sep 2006 13:09:44 -0700, Adam Beneschan wrote:

> Dmitry A. Kazakov wrote:
>> [ It seems to me that some people have an impression that I am looking for
>> something extraordinary when asking for the object address, so I feel
>> necessary to provide an elaborated example where it were needed. ]
>>
>> Let's consider referential double-linked lists as an example. The drawbacks
>> of the Ada.Containers design are obvious. A typical design of such lists in
>> other languages would be adding Prev, Next pointers to the elements. Note
>> also, that we wished to have linked lists of elements which might
>> participate in many lists simultaneously. We also liked to statically check
>> the types of such lists, so that paths in different lists could never
>> converge.
>>
>> There are two ways to do add links Prevs and Nexts. One of them is to do it
>> upon inheritance. Another is by using new elements containing pointers to
>> the "true" elements.
>>
>> 1. The variant with inheritance is not an option in current Ada, because of
>> lack of MI. Even with MI it faces the problem that links has to be added to
>> the base type. That would require downcasting later, all over the program.
>> Adding links in the leaves of the type hierarchy would break it apart and
>> also freeze the types for future derivation. Ad-hoc supertypes could help,
>> but, firstly, they are not supported and, secondly, they barely could have
>> referential semantics. So the elements will require copying.
>>
>> 2. The variant with pointers is not a solution at all, because, honestly,
>> it would be a list of pointers rather than of the elements.
> 
> I'll have to admit that I don't see what "problem" you have that this
> isn't a solution to.  In fact, I'm not really clear on why it's
> important to you to adopt the approach you're adopting; your
> explanation is pretty sketchy.

That was an example illustrating the case - a need to get an intact object
address.

As for possible application areas of double-linked lists, honestly, I don't
know what sort of explanation you liked to hear from me...

> Part of my problem is that you seem to want to define a type, Item_Ptr,
> that gives you more than it's supposed to give you.  Your generic
> declares
> 
>    type Item_Ptr is access List_Type_Item;
> 
> where List_Type_Item is whatever type you're making a list out of.  I
> see a type like this, and I think it's just an access to the
> List_Type_Item, but there's actually additional baggage in this
> pointer, i.e. a "next" and a "previous" pointer.  So if I see code that
> calls Next on this Item_Ptr, my first thought is "Whooaa!  Item_Ptr is
> just an access to my Employee record, so where the heck is it going to
> get a pointer to another Employee?" It would be pretty confusing to me.

[ It would be nice to have user-defined fat access types, but,
unfortunately, Ada does not provide them. Ergo, you are not allowed to
think that way in Ada. You have to - "Aha, there must something in the
target object that allows this Next." ]

Item_Ptr is an ADT.

> Personally, if I'm going to have something that contains additional
> information, I'd prefer that it be a different type.

which is Item_Ptr. I hope that you'd agree that "contains additional
information" does not necessarily imply *physically* contains the
information.

> The need for
> downcasting, type conversion, or a Deref function in your generic that
> takes an Item_Ptr and returns the dereferenced item that I would use
> instead of .all, doesn't bother me at all.  So I don't see what problem
> there is that needs solving.

Hmm, if you consider a need in downcasting and explicit type conversions of
elements of a double-linked lists as OK, then we are on different pages.

> Plus, you introduce a problem.  Assuming the actual for List_Type_Item
> is T, you've defined Item_Ptr as an access to T; what happens if
> somewhere else in the program, there is another type that is also an
> access to T?  The language will let you convert between the two access
> types, but the result would be an absolute disaster.  But I just know
> that someone who doesn't really understand that Item_Ptr isn't just an
> access to T is going to try it.

No, it is illegal in Ada. You cannot freely convert between pool-specific
access types. Item_Ptr is pool-specific. You can convert it to a general
access type, but that would make no harm. A backward conversion is again
impossible. It does not leak, Ada is a safe language. [ I don't consider
Unchecked_Conversion issues, here. ]

> Put simply, you're trying to trick your Ada compiler into allowing
> programmers to use Ada syntax to write code in some other language than
> Ada, and someone who looks at the Ada code and assumes it's Ada is
> likely to be pretty darn confused.

Hmm, this definitely applies to the dopes of String objects. Is String Ada?

> Anyway, now that I've given you my tirade on why I think this is a bad
> approach, I'm going to give you an idea of how you might implement your
> bad approach.  The problem with your Next function is that if
> List_Item_Type is an unconstrained array, there will be extra data
> between the thing Item_Ptr points to and the Item_Header.  Since you've
> already admitted that this is going to be highly
> implementation-dependent, then you can probably assume that the size of
> this extra data is going to be the same always (for a given
> List_Item_Type), and then all you have to do is figure out what that
> size is the first time Next [or some other routine] is called.  Declare
> a variable in your package body:
> 
>     Dope_Offset : Storage_Offset := -1;
> 
> Fix your Allocate so that it saves the resulting Storage_Address in
> some global Global_Storage_Address in the package body before it
> returns.  Inside Next:
>
>    X : Item_Ptr;
>    ...
>    if Dope_Offset = -1 then
>      X := new List_Item_Type' (Item.all);
>      Dope_Offset := To_Address(X) - Global_Storage_Address;
>      Free(X);
>   end if;
> 
> where To_Address is from an instance of
> System.Address_To_Access_Conversions.  Now once Dope_Offset has been
> set up, you can use it as an offset and subtract it from your
> From_Item_Ptr calls.  

Unfortunately this will not work, because List_Item_Type is indefinite
limited. Then, construction - destruction of an item might be highly
undesirable, when Item is something like Device_Driver or DB_Connection.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: A novel design of linked lists

[Dmitry A. Kazakov]

On Wed, 20 Sep 2006 00:42:03 +0200, Leif Holmgren wrote:

> Dmitry A. Kazakov wrote:
>> 
>> 1. The variant with inheritance is not an option in current Ada, because of
>> ...
>> 
>> 2. The variant with pointers is not a solution at all, because, honestly,
>> ...
> 
> 3. Generics

There is no generic types in Ada. Then, essentially, genericity does not
provide any types composition. So here generic could only mean
parametrization of some non-generic type composition:

- extension by inheritance (that was 1)
- indirection (that was 2)

There also is aggregation which wasn't mentioned for obvious reasons.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: A novel design of linked lists

[Adam Beneschan]

Dmitry A. Kazakov wrote:

> > Part of my problem is that you seem to want to define a type, Item_Ptr,
> > that gives you more than it's supposed to give you.  Your generic
> > declares
> >
> >    type Item_Ptr is access List_Type_Item;
> >
> > where List_Type_Item is whatever type you're making a list out of.  I
> > see a type like this, and I think it's just an access to the
> > List_Type_Item, but there's actually additional baggage in this
> > pointer, i.e. a "next" and a "previous" pointer.  So if I see code that
> > calls Next on this Item_Ptr, my first thought is "Whooaa!  Item_Ptr is
> > just an access to my Employee record, so where the heck is it going to
> > get a pointer to another Employee?" It would be pretty confusing to me.
>
> [ It would be nice to have user-defined fat access types, but,
> unfortunately, Ada does not provide them. Ergo, you are not allowed to
> think that way in Ada. You have to - "Aha, there must something in the
> target object that allows this Next." ]
>
> Item_Ptr is an ADT.
>
> > Personally, if I'm going to have something that contains additional
> > information, I'd prefer that it be a different type.
>
> which is Item_Ptr. I hope that you'd agree that "contains additional
> information" does not necessarily imply *physically* contains the
> information.
>
> > The need for
> > downcasting, type conversion, or a Deref function in your generic that
> > takes an Item_Ptr and returns the dereferenced item that I would use
> > instead of .all, doesn't bother me at all.  So I don't see what problem
> > there is that needs solving.
>
> Hmm, if you consider a need in downcasting and explicit type conversions of
> elements of a double-linked lists as OK, then we are on different pages.

Apparently we are.  I don't see why it wouldn't be OK to declare
Item_Ptr as some sort of private type, and have users of the package
write Deref(P) instead of P.all when they want to get the pointed-to
value---or, if you want to be able to put it on the left side of an
assignment, include a function that returns a general-access to your
list type so that they could say something like The_Item(P).all :=
something.  You seem to think this is awful, and I don't see anything
wrong with it, so yes, we are on different pages.

> > Plus, you introduce a problem.  Assuming the actual for List_Type_Item
> > is T, you've defined Item_Ptr as an access to T; what happens if
> > somewhere else in the program, there is another type that is also an
> > access to T?  The language will let you convert between the two access
> > types, but the result would be an absolute disaster.  But I just know
> > that someone who doesn't really understand that Item_Ptr isn't just an
> > access to T is going to try it.
>
> No, it is illegal in Ada. You cannot freely convert between pool-specific
> access types. Item_Ptr is pool-specific.

OK, you're right about that---my mistake.

> > Put simply, you're trying to trick your Ada compiler into allowing
> > programmers to use Ada syntax to write code in some other language than
> > Ada, and someone who looks at the Ada code and assumes it's Ada is
> > likely to be pretty darn confused.
>
> Hmm, this definitely applies to the dopes of String objects. Is String Ada?

I'm not sure what you mean here.  When you declare an access to a
string, in Ada you can use that access to get at the string value---and
that includes not just the contents of each character, but the 'First
and 'Last attributes of the string, which are an important property of
the value.  I'm sure that's what you mean by "dope", but to me "dope"
is an implementation detail, not an Ada concept.  But if I define a
record type that doesn't have "next" or "previous" fields, then I don't
expect "next" and "previous" to be part of the value that is accessed
when I declare an access to that record type---or when you declare it
in your generic.  This is just how I think.  There's apparently a
philosophical chasm here, though.

                           -- Adam

Re: A novel design of linked lists

[Dmitry A. Kazakov]

On 20 Sep 2006 10:13:52 -0700, Adam Beneschan wrote:

> Dmitry A. Kazakov wrote:
> 
>> Hmm, if you consider a need in downcasting and explicit type conversions of
>> elements of a double-linked lists as OK, then we are on different pages.
> 
> Apparently we are.  I don't see why it wouldn't be OK to declare
> Item_Ptr as some sort of private type, and have users of the package
> write Deref(P) instead of P.all when they want to get the pointed-to
> value---or, if you want to be able to put it on the left side of an
> assignment, include a function that returns a general-access to your
> list type so that they could say something like The_Item(P).all :=
> something.  You seem to think this is awful, and I don't see anything
> wrong with it, so yes, we are on different pages.

Yes I do, because I there is no obvious reason why the referential nature
of Item_Ptr need to be hidden. To be honest, I don't think that .all is
good. IMO, pointers should be fully transparent in Ada. But in the case at
hand we cannot get rid of .all. You have replaced one pointer type with
another. This does not make much sense to me. Then there is a technical
problem, when Item_Ptr is private, you will need a mediator type between
Item_Ptr and the item factory and shredder + a lot of acrobatics. In my
design they are just new and an instance of Unchecked_Deallocation.

>>> Put simply, you're trying to trick your Ada compiler into allowing
>>> programmers to use Ada syntax to write code in some other language than
>>> Ada, and someone who looks at the Ada code and assumes it's Ada is
>>> likely to be pretty darn confused.
>>
>> Hmm, this definitely applies to the dopes of String objects. Is String Ada?
> 
> I'm not sure what you mean here.  When you declare an access to a
> string, in Ada you can use that access to get at the string value---and
> that includes not just the contents of each character, but the 'First
> and 'Last attributes of the string, which are an important property of
> the value.  I'm sure that's what you mean by "dope", but to me "dope"
> is an implementation detail, not an Ada concept.  But if I define a
> record type that doesn't have "next" or "previous" fields, then I don't
> expect "next" and "previous" to be part of the value that is accessed
> when I declare an access to that record type---or when you declare it
> in your generic.  This is just how I think.  There's apparently a
> philosophical chasm here, though.

Yes, because in my philosophy there is no any difference between X"First
and Next (X). Both are just [primitive] operations on some ADT. Neither
assumes that the result must be additionally accessible as a field, either
publicly or privately. (BTW, even a field is, in fact, a pair of primitive
operations getter/setter, alas, not in Ada.) To me the way 'First get
computed is an implementation detail, same for Next. To me it is quite in
Ada-way, when the compiler allocates an invisible dope for String and
exposes this implementation detail through well-formed public interface of
array attributes.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de