large arrays

large arrays

[Joel Lepinoux]

Hello,
for the first time I need to declare a large array (>1Go), here are my
structures:


  type Site         is array (1..3) of Short_Short_Integer; -- size=24 bits
  type Site_Matrix  is array (Integer range <>, Integer range <>) of Site;

-- maximum allowed size, about 245Mo
  My_Array: Site_Matrix(1..1_132_000,1..72);


If I increase the first upper bound (1_132_000) I get a warning from gnat
(3.15p) comfirmed at runtime: STORAGE_ERROR: object too large

This means that "My_Array" is too large but of course I can declare a
second array of same dimensions and so one, i.e. it is not a problem of
total space (2Go available) neither of bound out of range, etc.

I would be glad to understand the origin of this limit and it would be
great if somebody could tell me how to extend it.
Thanks in advance
Joel

Re: large arrays

[Jeff C r e e.m]

"Joel Lepinoux" <Joel.Lepinoux@ltpcm.inpg.fr> wrote in message 
news:mailman.87.1100029172.10401.comp.lang.ada@ada-france.org...
> Hello,
> for the first time I need to declare a large array (>1Go), here are my
> structures:
>
>
>  type Site         is array (1..3) of Short_Short_Integer; -- size=24 bits
>  type Site_Matrix  is array (Integer range <>, Integer range <>) of Site;
>
> -- maximum allowed size, about 245Mo
>  My_Array: Site_Matrix(1..1_132_000,1..72);
>
>
> If I increase the first upper bound (1_132_000) I get a warning from gnat
> (3.15p) comfirmed at runtime: STORAGE_ERROR: object too large
>
> This means that "My_Array" is too large but of course I can declare a
> second array of same dimensions and so one, i.e. it is not a problem of
> total space (2Go available) neither of bound out of range, etc.
>
> I would be glad to understand the origin of this limit and it would be
> great if somebody could tell me how to extend it.
> Thanks in advance
> Joel
>
>

You dont say but I assume you are using GNAT?

From the GNAT RM (6.2)

 The largest Size value permitted in GNAT is 2**31-1. Since this is a Size 
in bits, this corresponds to an object of size 256 megabytes (minus one). 
This limitation is true on all targets. The reason for this limitation is 
that it improves the quality of the code in many cases if it is known that a 
Size value can be accommodated in an object of type Integer.

Re: large arrays

[Dr. Adrian Wrigley]

On Tue, 09 Nov 2004 23:55:00 +0000, Jeff C r e e.m wrote:
 
> From the GNAT RM (6.2)
> 
>  The largest Size value permitted in GNAT is 2**31-1. Since this is a Size 
> in bits, this corresponds to an object of size 256 megabytes (minus one). 
> This limitation is true on all targets. The reason for this limitation is 
> that it improves the quality of the code in many cases if it is known that a 
> Size value can be accommodated in an object of type Integer.

This is a real pain sometimes!  The limitation crops up in various
annoying places, even in code you thought worked (eg generic instantiatian?).

There are rules that allow you to work with large objects, for example
  Don't use 'Size - it doesn't work
  Don't declare types > 256MB with static sizes

Try something like:
  type Site         is array (1..3) of Short_Short_Integer; -- size=24 bits
  type Site_Matrix  is array (Integer range <>, Integer range <>) of Site;
  type Site_Matrix_A is access Site_Matrix;
  My_Array: Site_Matrix_A := new Site_Matrix(1..1_132_000,1..72);

One area this I have encountered this is with image processing,
where even standard IMAX frames might be 6144x4096 pixels, held
as floating point RGBA come to about 400MB.

Currently, I have a record of 1800MB, which I mmap into a file
for shared access and persistence.  This works extremely well, other
than having to calculate 'Size by another method.

I'm a bit disappointed to hear that the restriction exists even
on 64-bit targets - at the moment, I have 6GB RAM+Swap, and it is
annoying that I can only access 5% of that with any single object
in Ada.

Of course there are those who might argue that there must something wrong
with the programming approach if objects get this large... but that's
another argument.
-- 
Dr. Adrian Wrigley, Cambridge, UK.

Re: large arrays

[Brian May]

>>>>> "Adrian" == Adrian Wrigley <amtw@linuxchip.demon.co.uk.uk.uk> writes:

    Adrian> I'm a bit disappointed to hear that the restriction exists even
    Adrian> on 64-bit targets - at the moment, I have 6GB RAM+Swap, and it is
    Adrian> annoying that I can only access 5% of that with any single object
    Adrian> in Ada.

I am curious, can somebody confirm that the restriction does exist on
64 bit targets?

I would have assumed Integer would be 64 bits, rendering the
justification previously given null and void.
-- 
Brian May <bam@snoopy.apana.org.au>

Re: large arrays

[Jeff C r e e.m]

"Dr. Adrian Wrigley" <amtw@linuxchip.demon.co.uk.uk.uk> wrote in message 
news:pan.2004.11.10.03.45.48.808299@linuxchip.demon.co.uk.uk.uk...
> On Tue, 09 Nov 2004 23:55:00 +0000, Jeff C r e e.m wrote:
>
>> From the GNAT RM (6.2)
>>
>>  The largest Size value permitted in GNAT is 2**31-1. Since this is a 
>> Size
>> in bits, this corresponds to an object of size 256 megabytes (minus one).
>> This limitation is true on all targets. The reason for this limitation is
>> that it improves the quality of the code in many cases if it is known 
>> that a
>> Size value can be accommodated in an object of type Integer.
>
> This is a real pain sometimes!  The limitation crops up in various
> annoying places, even in code you thought worked (eg generic 
> instantiatian?).
>
> There are rules that allow you to work with large objects, for example
>  Don't use 'Size - it doesn't work
>  Don't declare types > 256MB with static sizes
>
> Try something like:
>  type Site         is array (1..3) of Short_Short_Integer; -- size=24 bits
>  type Site_Matrix  is array (Integer range <>, Integer range <>) of Site;
>  type Site_Matrix_A is access Site_Matrix;
>  My_Array: Site_Matrix_A := new Site_Matrix(1..1_132_000,1..72);
>
> One area this I have encountered this is with image processing,
> where even standard IMAX frames might be 6144x4096 pixels, held
> as floating point RGBA come to about 400MB.
>
> Currently, I have a record of 1800MB, which I mmap into a file
> for shared access and persistence.  This works extremely well, other
> than having to calculate 'Size by another method.
>
> I'm a bit disappointed to hear that the restriction exists even
> on 64-bit targets - at the moment, I have 6GB RAM+Swap, and it is
> annoying that I can only access 5% of that with any single object
> in Ada.
>


What you mean to say is that it is annoying that you can only access 5% of 
that object in GNAT...This is not "an Ada thing" (although I strongly
suspect that on 32 bit targets that many compilers have this restriction (or 
perhaps a limit at 2x the GNAT restriction).

I don't believe there is any such restriction in the LRM.

I suspect if a supported customer or two complained then GNAT would be 
(eventually) changed. You could try putting in a bug report for the FSF 
version of GCC as a non-supported customer and see what happens. (Of course 
it is probably worth checking that the restriction still exists but it 
appears to (based on the documentation) for the latest released version of 
GCC)
http://gcc.gnu.org/onlinedocs/gcc-3.4.3/gnat_rm/Size-Clauses.html#Size-Clauses

Re: large arrays

[Steve]

I always try to read a post twice before I reply.  It helps to avoid some of
my silly responses.

Steve
(The Duck)

"Jeff C r e e.m" <jcreem@yahoo.com> wrote in message
news:kNckd.488755$mD.84928@attbi_s02...
>
[snip]
> >
> > If I increase the first upper bound (1_132_000) I get a warning from
gnat <--
> > (3.15p) comfirmed at runtime: STORAGE_ERROR: object too large
^
> >
^
> > This means that "My_Array" is too large but of course I can declare a
^
> > second array of same dimensions and so one, i.e. it is not a problem of
^
> > total space (2Go available) neither of bound out of range, etc.
^
> >
^
> > I would be glad to understand the origin of this limit and it would be
^
> > great if somebody could tell me how to extend it.
^
> > Thanks in advance
^
> > Joel
^
> >
^
> >
^
>
^
> You dont say but I assume you are using
NAT?  ----------------------------^
>
> From the GNAT RM (6.2)
[snip]

Re: large arrays

[Jeff C r e e.m]

"Steve" <nospam_steved94@comcast.net> wrote in message 
news:Aihkd.489422$mD.61019@attbi_s02...
>I always try to read a post twice before I reply.  It helps to avoid some 
>of
> my silly responses.
>

Woops.

But if it were not for silly responses then usenet would be a cold dark 
place. ;)

Re: large arrays

[Randy Brukardt]

"Jeff C r e e.m" <jcreem@yahoo.com> wrote in message
news:kNckd.488755$mD.84928@attbi_s02...
> From the GNAT RM (6.2)
>
>  The largest Size value permitted in GNAT is 2**31-1. Since this is a Size
> in bits, this corresponds to an object of size 256 megabytes (minus one).
> This limitation is true on all targets. The reason for this limitation is
> that it improves the quality of the code in many cases if it is known that
a
> Size value can be accommodated in an object of type Integer.

For what it's worth (probably nothing :-), Janus/Ada doesn't limit objects
on 32-bit architectures. We manage everything in terms of bytes, and only
calculate 'Size if someone asks for it (which might raise Constraint_Error
for really large objects).

But I have to sympathize with GNAT; the allocate-in-bytes designs needs a
lot of hacks to make it work for all of the sorts of representation items,
and I certainly can see why someone wouldn't want to do that.

Someone also complained that this limitation exists on all targets; I'm sure
that's because the compiler front-end uses 32-bit values for 'Size on all
targets. Otherwise, you'd have to have a 64-bit compiler before you could
compile the 64-bit compiler (we faced this same issue on the 36-bit Unisys
system).

                     Randy.