Re: arrays into mmap'ed memory

Re: arrays into mmap'ed memory

[David C. Hoos, Sr.]

Hi Jonas,
I wrote a generic package in VADS Ada for the SGI platforms under IRIX (4,
5 & 6), a couple of years ago.  It was originally written for IRIX 4, then
modified for IRIX 5.  IRIX 5 code works for 32-bit IRIX 6.
The code belongs to my former employer's client, so I can't share it with
you, but I'll summarize here what I did, and the pitfalls of multi-OS
versions, as I remember them.

The generic instantiation parameter is a type, being the array type. 
Inside the generic package, the only attribute needed is the size of the
array.  One little pitfall here is that we couldn't use the 'SIZE attribute
of the array type because in some cases we were mapping 0.6 Gb. arrays, and
the number of bits in such an array is larger than can be represented with
the 32-bit INTEGER type.
My solution to that was to add a second generic parameter, the size of the
array type in BYTES.  The parameter was defaulted to zero, which meant that
the size in bytes is computable from the 'SIZE attribute -- i.e., that
taking the result of the 'SIZE  attribute would not raise a constraint
error.  Thus, for the very large array, the instantiator had to supply the
second parameter, giving the size in bytes, in which case the generic's
elaboration code would not do the computation.
If I remember, the functions provided were mmap, unmap and synchronize. 
These functions had equivalent Ada parameters, using a scheme mapping Ada
enumeration values (possibly logically summed with a + operator) to a bit
vector, to provide the functionality of logically summing the #defined
constants from C, for mode (e.g., READ_ONLY), etc.
If I remember correctly, the mmap procedure allowed you to specify the name
of the file, the starting offset into the file, (the length of the map is
defined by the instantiating type), and the file mode.  An in out parameter
analogous to the FILE_TYPE of Text_IO, was a limited private type (so the
object could not be copied), implemented as access to a map descriptor
record.  The map descriptor record contained the access (the address) to
the memory, and the size of the object (in bytes).  These are then used by
unmap, and synchronize.
Oh, in addition, the package exported a function to return the access value
of the map, which is then used with standard array subscripting to get at
the data.

Of course, my motivation for doing this was that to allocate a 0.5 Gb.
object on the heap means all that memory has to be zeroed (by sbrk), and
then most of it swapped out  to the swap partition on the disk, because we
had only 128 Mb. of RAM in the machine.  Then, the 0.5 Gb. data file had to
be read into this memory, with a lot more swapping, just to put things in
the state that memory mapping to the file does in an instant.
and, I didn't say it, but the Ada subprograms are implemented with pragma
INTERFACE to the C runtime library.
And, an IRIX 4/5 pitfall -- the values of the #defines for READ and EXECUTE
(I think it was) were swapped between versions, so that when we ran code
built on IRIX 4 on an IRIX 5 machine, memory we thought had READ access
actually had EXECUTE access.  On one type of MIPS processor, this caused a
kernel panic.
So, you could have a little fun getting this going.
I hope these ramblings are of help.
-- 
David C. Hoos, Sr.,
http://www.dbhwww.com
http://www.ada95.com

Jonas Nygren <jonas@joy.ericsson.se> wrote in article
<32249A94.4A0A@joy.ericsson.se>...
> Can anybody help me with how to use mmap'ed memory in Ada. What I want
> to do is mmap a large file and then access parts of the so mapped 
> memory via arrays.
> 
> In C it is simple since arrays are more or less a language syntax
> on top of pointers, p[i] == *(p+i). How can I connect an Ada array
> to memory allocated via mmap? I do not want to store the bounds
> of the array in the mapped file - the file should just be plain
> ASCII. If I manage to get the array to point to the memory and
> the bounds to be set correctly what will happen when Ada tries to
> reclaim the array? 
> 
> Of course I could use a pointer and write my own access functions
> for this pointer but I think that it would be nicer to use arrays
> if possible.
> 
> This probably sounds confused and reflects my understanding of Ada
> but perhaps you get the gist of my question and so can put me straight.
> 
> TIA,
> 
> /jonas
> 
> PS mmap is a unix syscall which maps a file directly into memory
>    without the need to allocate memory in your program - you sort of
>    get access directly to the OS's file buffers which are loaded
>    on access of the memory via direct page-in. DS
> 

[Q]: arrays into mmap'ed memory

[Jonas Nygren]

Can anybody help me with how to use mmap'ed memory in Ada. What I want
to do is mmap a large file and then access parts of the so mapped 
memory via arrays.

In C it is simple since arrays are more or less a language syntax
on top of pointers, p[i] == *(p+i). How can I connect an Ada array
to memory allocated via mmap? I do not want to store the bounds
of the array in the mapped file - the file should just be plain
ASCII. If I manage to get the array to point to the memory and
the bounds to be set correctly what will happen when Ada tries to
reclaim the array? 

Of course I could use a pointer and write my own access functions
for this pointer but I think that it would be nicer to use arrays
if possible.

This probably sounds confused and reflects my understanding of Ada
but perhaps you get the gist of my question and so can put me straight.

TIA,

/jonas

PS mmap is a unix syscall which maps a file directly into memory
   without the need to allocate memory in your program - you sort of
   get access directly to the OS's file buffers which are loaded
   on access of the memory via direct page-in. DS

Re: [Q]: arrays into mmap'ed memory

[Jon S Anthony]

In article <32249A94.4A0A@joy.ericsson.se> Jonas Nygren <jonas@joy.ericsson.se> writes:

> Can anybody help me with how to use mmap'ed memory in Ada. What I want
> to do is mmap a large file and then access parts of the so mapped 
> memory via arrays.

One obvious hack:

    -- Don't create an instance of this or you'll blow out with storage
    -- error or something.  It's just to get a template...
    --
    type my_mmarray_type is array (0..natural'last) of whatever;

    -- Probably should really use Interfaces.C.Pointer stuff, but this
    -- gives the idea...
    --
    type my_mmarray_ref  is access all my_mmarray_type;

    the_file : my_mmarray_ref := address from mmap

You use the len from mmap to get the bounds just like for C.  In your
case, _whatever_ would most likely be Character.  A somewhat nicer
approach would be:

    type my_mmarray_type is array (natural range <>) of character;

    -- Prior to entering the following declarative region call the mmap
    -- function getting the address and len stuff.

    subtype mm_index_type is natural range 0..len;

    -- Create _constrained_ subtype.
    --
    subtype actual_mmarray_type is my_mmarray_type(mm_index_type);
    type actual_mmarray_ref is access all actual_mmarray_type;

    the_file : actual_mmarray_ref := pa;

    ...

begin
    ...
    -- Now you can proceed to write bounds independent stuff:
    --
    for I in mm_index_type'range loop
        text_io.put(the_file(i));
    end loop;

    -- etc.

/Jon
-- 
Jon Anthony
Organon Motives, Inc.
1 Williston Road, Suite 4
Belmont, MA 02178

617.484.3383
jsa@organon.com

Re: [Q]: arrays into mmap'ed memory

[Norman H. Cohen]

In article <32249A94.4A0A@joy.ericsson.se>, Jonas Nygren
<jonas@joy.ericsson.se> writes: 

|> Can anybody help me with how to use mmap'ed memory in Ada. What I want
|> to do is mmap a large file and then access parts of the so mapped
|> memory via arrays.

A call on mmap returns an address, so the more general question you are
asking is how to view the storage at a given address as an Ada array.

One approach is to declare an array of the appropriate size and then use
an address clause to specify the array's address.  In Ada 95, another
approach is to declare a constrained array type, declare an access type
pointing to values of that array type, and use an instance of
System.Address_To_Access_Conversions to convert the address to an access
value A.  This access value can be passed around the program, and the
I\th component of the array can be named as A(I).

|> In C it is simple since arrays are more or less a language syntax
|> on top of pointers, p[i] == *(p+i). How can I connect an Ada array
|> to memory allocated via mmap? I do not want to store the bounds
|> of the array in the mapped file - the file should just be plain
|> ASCII.

As long as you stick to constrained array types, a typical Ada
implementation will not try to store the bounds of the array in or
adjacent to the addressed storage (or expect to find them stored there).

|>        If I manage to get the array to point to the memory and
|> the bounds to be set correctly what will happen when Ada tries to
|> reclaim the array?

Since Ada did not allocate the array (either during the elaboration of an
object declaration or the evaluation of an allocator) Ada will not try to
deallocate it.

--
Norman H. Cohen    ncohen@watson.ibm.com

Re: [Q]: arrays into mmap'ed memory

[Laurent Guerby]

Norman> [...] In Ada 95, another approach is to declare a constrained array
Norman> type, declare an access type pointing to values of that array
Norman> type, and use an instance of
Norman> System.Address_To_Access_Conversions to convert the address to
Norman> an access value A. [...]

   Putting it all together with your comment on access to constrained
array, you can simply import mmap directly as returning the Ada access
type (using one mmap per acces type) and skip the System.Address
stage.

-- 
Laurent Guerby <guerby@gnat.com>, Team Ada.
   "Use the Source, Luke. The Source will be with you, always (GPL)."

Re: [Q]: arrays into mmap'ed memory

[Geert Bosch]

Norman H. Cohen (ncohen@watson.ibm.com) wrote:
`` In Ada 95, another approach is to declare a constrained array type,
   declare an access type pointing to values of that array type, and
   use an instance of System.Address_To_Access_Conversions to convert
   the address to an access value A. ''

Usually I use pragma Convention(C, The_Array_Type) as a way to inform the
compiler it shouldn't store any bounds. The scary thing of just
assuming the compiler doesn't store any bounds is that the standard
doesn't require/suggests this AFAIK. With the convention pragma the
compiler should do the same as a C compiler: not store any bouunds.
If a compiler cannot do this it should give a compile-time error.

What I'm curious about is what other compilers than GNAT do. 
Do Ada-95 compilers exist that store any meta-data with constrained 
types? If so, what behaviour do these compilers have when the
pragma is used?

Is it true that with the approach given above (including the pragma) 
any legal Ada compiler will have to do The Right Thing or give a
compile-time error? 

Regards,
   Geert

   --
   Norman H. Cohen    ncohen@watson.ibm.com
-- 
E-Mail: geert@sun3.iaf.nl    
      ``I think there is a world market for maybe five computers.''
        Thomas Watson,  chairman of IBM, 1943

Re: [Q]: arrays into mmap'ed memory

[Norman H. Cohen]

In article <ws968606kxa.fsf@schonberg.cs.nyu.edu>, guerby@gnat.com
(Laurent Guerby) writes: 

|>    Putting it all together with your comment on access to constrained
|> array, you can simply import mmap directly as returning the Ada access
|> type (using one mmap per acces type) and skip the System.Address
|> stage.

You seem to be assuming that the address returned by mmap has the same
representation as an Ada access type--a dangerous assumption.  The
purpose of instantiating Address_To_Access_Conversions is to avoid having
to make this assumption.

--
Norman H. Cohen    ncohen@watson.ibm.com