Size of Vector limited to 1024 MB of Heap Size

Size of Vector limited to 1024 MB of Heap Size

[Dennis Hoppe]

Hi,

my machine has 4 GB of RAM and I am wondering, why I can't use
at least 2 or 3 GBytes to run an Ada program. It seems, that my
Ada Compiler (Gnat 4.4.0) limit the memory to 2 GB per default.
Is it possible to allocate more than 2 GB?

Here is a simple example of an "evil" vector, that gains
more memory in each pass. The program terminates exactly at
1024 MB of used Heap memory.


with Ada.Containers.Vectors;

procedure Heap is
   package Generic_Vector is new Ada.Containers.Vectors
     (Element_Type => Integer, Index_Type => Natural);
		
   Evil_Vector : Generic_Vector.Vector;
begin -- Heap
   loop
     Generic_Vector.Append (Evil_Vector, Integer'Last);
   end loop;
end Heap;


heap(6374) malloc: *** mmap(size=2147487744) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

raised STORAGE_ERROR : heap exhausted


I could not find a suitable Compiler switch or a parameter, that
can be set for the operating system (linux). "ulimit -v" is already
set to unlimited.

"gnatmem" reports, that my water mark with 1024 MB is reached, but
the final water mark is, needless to say, higher.


Best regards,
   Dennis Hoppe

Re: Size of Vector limited to 1024 MB of Heap Size

[Adam Beneschan]

On Jun 24, 1:44 am, Dennis Hoppe <dennis.ho...@hoppinet.de> wrote:
> Hi,
>
> my machine has 4 GB of RAM and I am wondering, why I can't use
> at least 2 or 3 GBytes to run an Ada program.

Well, if you were using Windows, I'd guess this is because Windows
reserves about 3.98 GB of your RAM for itself.  But I notice that you
said further on down that you were using Linux, so the heck with that
answer........

>  It seems, that my
> Ada Compiler (Gnat 4.4.0) limit the memory to 2 GB per default.
> Is it possible to allocate more than 2 GB?

I don't know GNAT intimately, and I don't work on it.  But 2 GB =
2**31 bytes, and the largest possible value of a signed 32-bit integer
is 2**31-1.  So if their runtime is using 32-bit integers to hold
values like that, there may be no way to deal with larger memory
amounts without a rewrite of the runtime.

                               -- Adam

Re: Size of Vector limited to 1024 MB of Heap Size

[Robert A Duff]

Dennis Hoppe <dennis.hoppe@hoppinet.de> writes:

> my machine has 4 GB of RAM and I am wondering, why I can't use
> at least 2 or 3 GBytes to run an Ada program. ...

I don't think this has anything to do with Ada or GNAT.
It's an OS issue.  You could verify that by writing
a similar memory-eating program in some other language,
like C.

I had a similar problem a while ago, where the OS was reserving 2GB of
the address space for itself, by default.  But there was an option
to decrease that to 1GB, leaving 3GB for the user-mode program.
I don't remember if I did that on windows or linux (or both).

So check if there is such an option on linux.

Anyway, if you're getting that close to the hardware limit of 4GB,
it's probably time to upgrade to a 64-bit machine (and OS)!

- Bob

Re: Size of Vector limited to 1024 MB of Heap Size

[Peter Schildmann]

Dennis Hoppe schrieb:
>   loop
>     Generic_Vector.Append (Evil_Vector, Integer'Last);
>   end loop;

It's not a good idea to use the STORAGE_ERROR exception
to terminate an endless loop.

This should work:

with Ada.Text_IO;
with Ada.Containers;
with Ada.Containers.Vectors;

procedure Heap is

   package Cnt_IO is new Ada.Text_IO.Integer_IO
     (Ada.Containers.Count_Type);

   package Generic_Vector is new Ada.Containers.Vectors
     (Element_Type => Integer, Index_Type => Natural);

   Evil_Vector : Generic_Vector.Vector;

   Size : constant := Integer'Size / Standard'Storage_Unit;

begin

   for N in 0 .. Natural'Last / Size loop
      Generic_Vector.Append (Evil_Vector, N);
   end loop;

   Cnt_IO.Put (Generic_Vector.Capacity (Evil_Vector));

end Heap;


- Peter

Re: Size of Vector limited to 1024 MB of Heap Size

[Gene]

On Jun 24, 4:44 am, Dennis Hoppe <dennis.ho...@hoppinet.de> wrote:
> Hi,
>
> my machine has 4 GB of RAM and I am wondering, why I can't use
> at least 2 or 3 GBytes to run an Ada program. It seems, that my
> Ada Compiler (Gnat 4.4.0) limit the memory to 2 GB per default.
> Is it possible to allocate more than 2 GB?
>
> Here is a simple example of an "evil" vector, that gains
> more memory in each pass. The program terminates exactly at
> 1024 MB of used Heap memory.
>
> with Ada.Containers.Vectors;
>
> procedure Heap is
>    package Generic_Vector is new Ada.Containers.Vectors
>      (Element_Type => Integer, Index_Type => Natural);
>
>    Evil_Vector : Generic_Vector.Vector;
> begin -- Heap
>    loop
>      Generic_Vector.Append (Evil_Vector, Integer'Last);
>    end loop;
> end Heap;
>
> heap(6374) malloc: *** mmap(size=2147487744) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
>
> raised STORAGE_ERROR : heap exhausted
>
> I could not find a suitable Compiler switch or a parameter, that
> can be set for the operating system (linux). "ulimit -v" is already
> set to unlimited.
>
> "gnatmem" reports, that my water mark with 1024 MB is reached, but
> the final water mark is, needless to say, higher.
>

Your code thrashes the heap pretty hard.  Containers doubles the size
of the vector's internal array each time it runs out.  So the 2Gb
request means 1Gb is already in use.  Dont' know about your malloc(),
but it's easy to see that a 1Gb allocated block in a 4Gb arena can
preclude a further 2Gb allocation.

What happens if call Reserve_Capacity(a, Natural'Last) at the
beginning?

Re: Size of Vector limited to 1024 MB of Heap Size

[Robert A Duff]

Gene <gene.ressler@gmail.com> writes:

> Your code thrashes the heap pretty hard.  Containers doubles the size
> of the vector's internal array each time it runs out.  So the 2Gb
> request means 1Gb is already in use.  Dont' know about your malloc(),
> but it's easy to see that a 1Gb allocated block in a 4Gb arena can
> preclude a further 2Gb allocation.

Good point.  Note that when you grow from 1GB to 2GB, you have both
allocated for the time it takes to copy the data over.  That won't work
if the user part of the address space is limited to 3GB.  Some part of
that 3GB is used for other stuff.  Plus the heap might well be
fragmented.

A better test of how much you can allocate would allocate a whole bunch
of smaller objects.

- Bob

Re: Size of Vector limited to 1024 MB of Heap Size

[anon]

Adjust your heap size in the linking phase. 

In <g3qc68$5a5$1@aioe.org>, Dennis Hoppe <dennis.hoppe@hoppinet.de> writes:
>Hi,
>
>my machine has 4 GB of RAM and I am wondering, why I can't use
>at least 2 or 3 GBytes to run an Ada program. It seems, that my
>Ada Compiler (Gnat 4.4.0) limit the memory to 2 GB per default.
>Is it possible to allocate more than 2 GB?
>
>Here is a simple example of an "evil" vector, that gains
>more memory in each pass. The program terminates exactly at
>1024 MB of used Heap memory.
>
>
>with Ada.Containers.Vectors;
>
>procedure Heap is
>   package Generic_Vector is new Ada.Containers.Vectors
>     (Element_Type => Integer, Index_Type => Natural);
>		
>   Evil_Vector : Generic_Vector.Vector;
>begin -- Heap
>   loop
>     Generic_Vector.Append (Evil_Vector, Integer'Last);
>   end loop;
>end Heap;
>
>
>heap(6374) malloc: *** mmap(size=2147487744) failed (error code=12)
>*** error: can't allocate region
>*** set a breakpoint in malloc_error_break to debug
>
>raised STORAGE_ERROR : heap exhausted
>
>
>I could not find a suitable Compiler switch or a parameter, that
>can be set for the operating system (linux). "ulimit -v" is already
>set to unlimited.
>
>"gnatmem" reports, that my water mark with 1024 MB is reached, but
>the final water mark is, needless to say, higher.
>
>
>Best regards,
>   Dennis Hoppe

Extra footnote: Re: Size of Vector limited to 1024 MB of Heap Size

[anon]

0 - 1023MB 16-bit works -- 2GB

but in Ada: Integers are 4 bytes

0 - 1023MB 32-bit words -- 4GB 

So:

1024MB  16-bit elements would be (2GB+1 (16-bit words))

but

1024MB 32-bit elements aka 1024MB Integer would be 
   (4GB+1 (32-bit words))  -- STORAGE_ERROR

so it seam that you did get the full 4GB RAM memory.


But the statement

   Generic_Vector.Append (Evil_Vector, Integer'Last);

will create a vector size of 4GB with each element being 4 bytes 
giving a total size of 16GB.

So, use:

   Generic_Vector.Append ( Evil_Vector, ( Natural'Last / 4 ) ) ;

or better yet

   Generic_Vector.Append ( Evil_Vector, ( Natural'Last / Natural'Size ) ) ;


to give you a 1GB Vector size with an 4 byte element size that uses 
a total size of 4GB.


In <iVe8k.138597$SV4.76767@bgtnsc04-news.ops.worldnet.att.net>, anon@anon.org (anon) writes:
>Adjust your heap size in the linking phase. 
>
>In <g3qc68$5a5$1@aioe.org>, Dennis Hoppe <dennis.hoppe@hoppinet.de> writes:
>>Hi,
>>
>>my machine has 4 GB of RAM and I am wondering, why I can't use
>>at least 2 or 3 GBytes to run an Ada program. It seems, that my
>>Ada Compiler (Gnat 4.4.0) limit the memory to 2 GB per default.
>>Is it possible to allocate more than 2 GB?
>>
>>Here is a simple example of an "evil" vector, that gains
>>more memory in each pass. The program terminates exactly at
>>1024 MB of used Heap memory.
>>
>>
>>with Ada.Containers.Vectors;
>>
>>procedure Heap is
>>   package Generic_Vector is new Ada.Containers.Vectors
>>     (Element_Type => Integer, Index_Type => Natural);
>>		
>>   Evil_Vector : Generic_Vector.Vector;
>>begin -- Heap
>>   loop
>>     Generic_Vector.Append (Evil_Vector, Integer'Last);
>>   end loop;
>>end Heap;
>>
>>
>>heap(6374) malloc: *** mmap(size=2147487744) failed (error code=12)
>>*** error: can't allocate region
>>*** set a breakpoint in malloc_error_break to debug
>>
>>raised STORAGE_ERROR : heap exhausted
>>
>>
>>I could not find a suitable Compiler switch or a parameter, that
>>can be set for the operating system (linux). "ulimit -v" is already
>>set to unlimited.
>>
>>"gnatmem" reports, that my water mark with 1024 MB is reached, but
>>the final water mark is, needless to say, higher.
>>
>>
>>Best regards,
>>   Dennis Hoppe
>

Re: Extra footnote: Re: Size of Vector limited to 1024 MB of Heap Size

[tmoran]

> but in Ada: Integers are 4 bytes
The Ada 95 Reference Manual 3.5.4(21) says
"In an implementation, the range of Integer shall include the range
-2**15+1 .. +2**15-1."
   Has this changed in Ada 2005?

Re: Extra footnote: Re: Size of Vector limited to 1024 MB of Heap Size

[anon]

RM for Ada 95 and Ada 2005 -- 3.5.4 ( 21 ) : says that an Integer must 
include the range

  -2**15+1 .. +2**15-1   aka     -32768 .. +32767

but it is not limited to that range. Check RM 3.5.4  ( 26 ).
The norm for PC's Integer is -2**32+1 .. +2**32-1

To verify what the true range is on your Ada system look at the 
Standard package.  For GNAT it is build into the compiler so type 
"gnat standard >standard.ads"

Then look at file "standard.ads" you will see lines:

package Standard is
pragma Pure(Standard);

   type Boolean is (False, True);


   type Integer is range -(2 ** 31) .. +(2 ** 31 - 1);

   subtype Natural  is Integer range 0 .. +(2 ** 31 - 1);
   ...
end Standard;


Or try this program
--
-- i.adb
-- 
  with Ada.Text_IO ;
  use Ada.Text_IO ;
  with Ada.Integer_Text_IO ;
  use  Ada.Integer_Text_IO ;

  procedure i is

  begin
    put ( Integer'Last ) ;
    new_line ;
  end ;



In <FJSdndpDzI7XEPzVnZ2dnUVZ_rfinZ2d@comcast.com>, tmoran@acm.org writes:
>> but in Ada: Integers are 4 bytes
>The Ada 95 Reference Manual 3.5.4(21) says
>"In an implementation, the range of Integer shall include the range
>-2**15+1 .. +2**15-1."
>   Has this changed in Ada 2005?

Re: Extra footnote: Re: Size of Vector limited to 1024 MB of Heap Size

[christoph.grein]

On 25 Jun., 04:49, a...@anon.org (anon) wrote:
> RM for Ada 95 and Ada 2005 -- 3.5.4 ( 21 ) : says that an Integer must
> include the range
>
>   -2**15+1 .. +2**15-1   aka     -32768 .. +32767
>
> but it is not limited to that range. Check RM 3.5.4  ( 26 ).

Paragraph (26) has nothing to do with Integer - it's about nonstandard
integer types, and Integer is a standard integer type.

> The norm for PC's Integer is -2**32+1 .. +2**32-1

You surely mean -(2**31) .. (2**31) - 1

"Norm"? You mean it's generally the case... There is no (DIN, EU, ...)
norm, or is there.

Re: Size of Vector limited to 1024 MB of Heap Size

[Peter Schildmann]

Dennis Hoppe schrieb:
> Here is a simple example of an "evil" vector, that gains
> more memory in each pass. The program terminates exactly at
> 1024 MB of used Heap memory.

I was able to allocate much more memory with the following
test program on a 64-bit machine running Debian Etch
(tested with gcc 4.1.2 and gcc 4.3.1).

- Peter


with Ada.Text_IO;
with Ada.Containers;
with Ada.Containers.Vectors;

procedure Heap is

   subtype Element_Type is Long_Integer;  -- 8 bytes

   Element_Size : constant := Element_Type'Size / Standard'Storage_Unit;

   subtype Index_Type is Natural
     range 0 .. 6 * 1024 * 1024 * 1024 / Element_Size - 1;  -- 6GB!

   package Cnt_IO is new Ada.Text_IO.Integer_IO
     (Ada.Containers.Count_Type);

   package Generic_Vector is new Ada.Containers.Vectors
     (Element_Type => Element_Type, Index_Type => Index_Type);

   Evil_Vector : Generic_Vector.Vector;

begin

   for N in Index_Type'Range loop
      Generic_Vector.Append (Evil_Vector, Long_Integer (N));
   end loop;

   Ada.Text_IO.Put ("Length:   ");
   Cnt_IO.Put (Generic_Vector.Length (Evil_Vector));

   Ada.Text_IO.New_Line;

   Ada.Text_IO.Put ("Capacity: ");
   Cnt_IO.Put (Generic_Vector.Capacity (Evil_Vector));

end Heap;

Re: Size of Vector limited to 1024 MB of Heap Size

[Dennis Hoppe]

Hi Peter,

unfortunately, your code provided below produces the same behaviour:

heap(9830) malloc: *** mmap(size=2147487744) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

raised STORAGE_ERROR : heap exhausted


Another point is, that I run actually a 64 bit OS (Mac OS X 10.5),
so I tried to compile the source code explicitly with the 64 bit flag

gcc -gnato -m64 -c heap.adb,

but I got the following errors:

heap.adb:10:04: instantiation error at a-convec.ads:330
heap.adb:10:04: alignment for "Vectort31b" must be at least 8
heap.adb:10:04: instantiation error at a-convec.ads:330
heap.adb:10:04: alignment for "Vectorb36b" must be at least 8

Line 10 is:

    package Generic_Vector is new Ada.Containers.Vectors
    (Element_Type => Integer, Index_Type => Natural);


Is it possible, that this flag (-m64) is actually not supported
for my system/compiler?

gcc (GCC) 4.4.0 20080314 (experimental) [trunk revision 133226]
GNAT 4.4.0 20080314 (experimental) [trunk revision 133226]
Darwin Kernel 9.3.0 root:xnu-1228.5.18~1/RELEASE_I386 i386


Thank you all,
   Dennis


Peter Schildmann wrote:
> Dennis Hoppe schrieb:
>>   loop
>>     Generic_Vector.Append (Evil_Vector, Integer'Last);
>>   end loop;
> 
> It's not a good idea to use the STORAGE_ERROR exception
> to terminate an endless loop.
> 
> This should work:
> 
> with Ada.Text_IO;
> with Ada.Containers;
> with Ada.Containers.Vectors;
> 
> procedure Heap is
> 
>    package Cnt_IO is new Ada.Text_IO.Integer_IO
>      (Ada.Containers.Count_Type);
> 
>    package Generic_Vector is new Ada.Containers.Vectors
>      (Element_Type => Integer, Index_Type => Natural);
> 
>    Evil_Vector : Generic_Vector.Vector;
> 
>    Size : constant := Integer'Size / Standard'Storage_Unit;
> 
> begin
> 
>    for N in 0 .. Natural'Last / Size loop
>       Generic_Vector.Append (Evil_Vector, N);
>    end loop;
> 
>    Cnt_IO.Put (Generic_Vector.Capacity (Evil_Vector));
> 
> end Heap;
> 
> 
> - Peter

Re: Size of Vector limited to 1024 MB of Heap Size

[Dennis Hoppe]

Hi Peter,

Peter Schildmann wrote:
> I was able to allocate much more memory with the following
> test program on a 64-bit machine running Debian Etch
> (tested with gcc 4.1.2 and gcc 4.3.1).
 > [code snippet]

The same code runs on machine out of memory at 1024 MB.
I thought, my system is 64 bit (Mac OS X 10.5), hmm.
With the gcc flag -m64 I get the following error while compiling:

heap.adb:17:04: instantiation error at a-convec.ads:330
heap.adb:17:04: alignment for "Vectort31b" must be at least 8
heap.adb:17:04: instantiation error at a-convec.ads:330
heap.adb:17:04: alignment for "Vectorb36b" must be at least 8


Best regards,
   Dennis

Re: Size of Vector limited to 1024 MB of Heap Size

[(see below)]

On 25/06/2008 16:13, in article g3tna8$f9c$1@aioe.org, "Dennis Hoppe"
<dennis.hoppe@hoppinet.de> wrote:

> Hi Peter,
> 
> unfortunately, your code provided below produces the same behaviour:
> 
> heap(9830) malloc: *** mmap(size=2147487744) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> 
> raised STORAGE_ERROR : heap exhausted
> 
> 
> Another point is, that I run actually a 64 bit OS (Mac OS X 10.5),
> so I tried to compile the source code explicitly with the 64 bit flag
> 
> gcc -gnato -m64 -c heap.adb,
> 
> but I got the following errors:
> 
> heap.adb:10:04: instantiation error at a-convec.ads:330
> heap.adb:10:04: alignment for "Vectort31b" must be at least 8
> heap.adb:10:04: instantiation error at a-convec.ads:330
> heap.adb:10:04: alignment for "Vectorb36b" must be at least 8
> 
> Line 10 is:
> 
>     package Generic_Vector is new Ada.Containers.Vectors
>     (Element_Type => Integer, Index_Type => Natural);
> 
> 
> Is it possible, that this flag (-m64) is actually not supported
> for my system/compiler?
> 
> gcc (GCC) 4.4.0 20080314 (experimental) [trunk revision 133226]
> GNAT 4.4.0 20080314 (experimental) [trunk revision 133226]
> Darwin Kernel 9.3.0 root:xnu-1228.5.18~1/RELEASE_I386 i386
>

N.B. There are 32-bit compilers running on 64-bit MacOS.

Heap compiles and runs on MacOS X 10.5.3 here, using Drew Reynolds' latest
64-bit compiler (which has a few other problems, however):

% g heap.adb 
gcc -c -gnat05 heap.adb
gnatbind -x heap.ali
gnatlink heap.ali

% ./heap
  536870912

% g -v

GNATMAKE  4.4.0 20080329 (experimental) [trunk revision 133715]
Copyright (C) 1995-2008, Free Software Foundation, Inc.

-- 
Bill Findlay
<surname><forename> chez blueyonder.co.uk

Re: Size of Vector limited to 1024 MB of Heap Size

[Dennis Hoppe]

Thank you for your hint to use the Compiler maintained by Drew Reynolds.
No, my testsuite runs as expected. Previously, I am used to
http://www.macada.org as a source for a Mac suitable Ada Compiler.

Thanks,
   Dennis




(see below) wrote:
> Heap compiles and runs on MacOS X 10.5.3 here, using Drew Reynolds' latest
> 64-bit compiler (which has a few other problems, however):
> 
> % g heap.adb 
> gcc -c -gnat05 heap.adb
> gnatbind -x heap.ali
> gnatlink heap.ali
> 
> % ./heap
>   536870912
> 
> % g -v
> 
> GNATMAKE  4.4.0 20080329 (experimental) [trunk revision 133715]
> Copyright (C) 1995-2008, Free Software Foundation, Inc.
>