Increasing GNAT's heap

Increasing GNAT's heap

[Dmitry A. Kazakov]

Does anybody know a way to increase the heap size available for GNAT? It
crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.

GNAT 4.6.4 under Debian, testing, 32-bit

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Georg Bauhaus]

On 12.11.13 12:09, Dmitry A. Kazakov wrote:
> Does anybody know a way to increase the heap size available for GNAT? It
> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.

This is not, I guess, about how in tasks,

  pragma Storage_Size (Dynamic_Value);

affects storage available to objects in the task? The maximum possible
value here seems bounded by System.Parameters.Size_Type.
Do ulimit values have any effect?

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Tue, 12 Nov 2013 14:26:42 +0100, Georg Bauhaus wrote:

> On 12.11.13 12:09, Dmitry A. Kazakov wrote:
>> Does anybody know a way to increase the heap size available for GNAT? It
>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
> 
> Do ulimit values have any effect?

$ ulimit
unlimited

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Paul Rubin]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
> $ ulimit
> unlimited

Try "ulimit -a" and check the individual parameters.

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Tue, 12 Nov 2013 09:31:20 -0800, Paul Rubin wrote:

> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>> $ ulimit
>> unlimited
> 
> Try "ulimit -a" and check the individual parameters.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 64846
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 64846
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Georg Bauhaus]

On 12.11.13 15:00, Dmitry A. Kazakov wrote:
> On Tue, 12 Nov 2013 14:26:42 +0100, Georg Bauhaus wrote:
> 
>> On 12.11.13 12:09, Dmitry A. Kazakov wrote:
>>> Does anybody know a way to increase the heap size available for GNAT? It
>>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>>
>> Do ulimit values have any effect?
> 
> $ ulimit
> unlimited


FWIW, I am finding it difficult to trigger something beyond
"Storage_Error heap exhausted" (no bug, i.e.); SE seems kind
of expected for some amounts of memory requested. The program below
tries to make the memory manager commit, by forcing random inits
of the memory at random positions. The program runs quite
a while before either me interrupting it at 2**34 storage elements
(Mac OS X 10.7.5), or some Linux VMs duly reporting STORAGE_ERROR
while processing the memory for 2**32.

Both systems are 64 bit, though, so I can't tell what 32 bit gives as
results. I'm curious. Does this trigger a bug box in your system?
(size_t and others are there for hysteric raisins of comparison.)


with System.Storage_Elements;  use System.Storage_Elements;
with Ada.Unchecked_Deallocation;
with Ada.Numerics.Discrete_Random, Ada.Numerics.Elementary_Functions;
with Ada.Text_IO;

procedure Alloc_Ada
is
   --  powers of 2, for growing allocated sizes:
   Small : constant := 16;
   Large : constant := 40;

   type Void_Ptr is access Storage_Array;
   subtype size_t is Storage_Offset;
   package Size_T_IO is new Ada.Text_IO.Integer_IO (size_t);
   use Size_T_IO, Ada.Text_IO;

   procedure Init (Object : Void_Ptr; Acc : out Storage_Element);
   --  Initialize some of Object with random data. Place something
   --  almost anywhere in the allocated area, assuming that doing so
   --  will require commits.

   procedure Free is
     new Ada.Unchecked_Deallocation (Storage_Array, Void_Ptr);

   procedure Init (Object : Void_Ptr; Acc : out Storage_Element) is
      subtype Places is Storage_Offset range 0 .. Object'Length - 1;
      package Arb is new Ada.Numerics.Discrete_Random (Storage_Element);
      package Pos is new Ada.Numerics.Discrete_Random (Places);
      package Math renames Ada.Numerics.Elementary_Functions;
      G : Arb.Generator;
      P : Pos.Generator;
      Offset, Previous_Offset : Storage_Offset := 0;
      No_Of_Inits : constant Natural :=
        2**(Natural (Math.Log (Float (Object'Length), 2.0))
              -
            (Small - 4));
   begin
      Arb.Reset (G); Pos.Reset (P);
      Acc := 0;
      for Run in 1 .. No_Of_Inits loop
         Previous_Offset := Offset;
         Offset := Pos.Random (P);
         Object (Offset) := Arb.Random (G);
         Acc := Acc / 2
           + (Object (Previous_Offset) + Object (Offset)) / 4;
      end loop;
   end Init;

   Buffer : Void_Ptr;
   Kept : Storage_Element;

begin
   for Power in size_t range Small .. Large loop
      Put ("At "); Put (Power);
      Put (" ("); Put (2**Natural (Power)); Put (")");

      Buffer := new Storage_Array (0 .. 2**Natural (Power));
      Init (Buffer, Acc => Kept);

      Put_Line (Storage_Element'Image (Kept));

      Free (Buffer);
   end loop;
end Alloc_Ada;

Re: Increasing GNAT's heap

[Florian Weimer]

* Dmitry A. Kazakov:

> Does anybody know a way to increase the heap size available for GNAT? It
> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>
> GNAT 4.6.4 under Debian, testing, 32-bit

Uhm, switch to 64 bits?

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Tue, 12 Nov 2013 23:03:36 +0100, Florian Weimer wrote:

> * Dmitry A. Kazakov:
> 
>> Does anybody know a way to increase the heap size available for GNAT? It
>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>>
>> GNAT 4.6.4 under Debian, testing, 32-bit
> 
> Uhm, switch to 64 bits?

And setting up GNAT for 32-bit target? I don't know how much work that
would require.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Oliver Kleinke]

Am Wed, 13 Nov 2013 09:05:31 +0100
schrieb "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>:

> On Tue, 12 Nov 2013 23:03:36 +0100, Florian Weimer wrote:
> 
> > * Dmitry A. Kazakov:
> > 
> >> Does anybody know a way to increase the heap size available for
> >> GNAT? It crashes with famous GNAT BUG DETECTED, Storage_Error heap
> >> exhausted.
> >>
> >> GNAT 4.6.4 under Debian, testing, 32-bit
> > 
> > Uhm, switch to 64 bits?
> 
> And setting up GNAT for 32-bit target? I don't know how much work that
> would require.


Probably negligible compared to the increased deployment effort..

Re: Increasing GNAT's heap

[Ludovic Brenta]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
> Does anybody know a way to increase the heap size available for GNAT? It
> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>
> GNAT 4.6.4 under Debian, testing, 32-bit

Presuming the heap is really exhausted (i.e. the command "free" shows no
free physical RAM and no free swap space left, and the command "top"
shows the gnat1 process consuming close to 4 GiB of resident memory)
then I'd suggest that you cannot increase the heap.  So, you should
compile smaller source files.

I'd try to identify which subprogram causes the problem and place that
subprogram in a compilation unit of its own (e.g. a private child unit).

-- 
Ludovic Brenta.

Re: Increasing GNAT's heap

[Ludovic Brenta]

Ludovic Brenta writes on comp.lang.ada:
> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>> Does anybody know a way to increase the heap size available for GNAT? It
>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>>
>> GNAT 4.6.4 under Debian, testing, 32-bit
>
> Presuming the heap is really exhausted (i.e. the command "free" shows no
> free physical RAM and no free swap space left, and the command "top"
> shows the gnat1 process consuming close to 4 GiB of resident memory)
> then I'd suggest that you cannot increase the heap.  So, you should
> compile smaller source files.
>
> I'd try to identify which subprogram causes the problem and place that
> subprogram in a compilation unit of its own (e.g. a private child unit).

Maybe I forgot the obvious: of course, you know you can increase swap
space easily by creating a file filled with zeroes and saying "swapon
$the_new_file" as root.

-- 
Ludovic Brenta.

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Wed, 13 Nov 2013 09:21:05 +0100, Ludovic Brenta wrote:

> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>> Does anybody know a way to increase the heap size available for GNAT? It
>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>>
>> GNAT 4.6.4 under Debian, testing, 32-bit
> 
> Presuming the heap is really exhausted (i.e. the command "free" shows no
> free physical RAM and no free swap space left, and the command "top"
> shows the gnat1 process consuming close to 4 GiB of resident memory)
> then I'd suggest that you cannot increase the heap.  So, you should
> compile smaller source files.

It is 8GB RAM allowed for the virtual machine.

Are you sure about 4GB? Doesn't Linux reserve some address space for its
the kernel? Under Windows, where GNAT has same problems I cannot get more
than 2GB. There is a boot switch to allow 3GB but it does not work well
with GNAT. When GNAT is running under 64-bit Windows it goes through being
32-bit itself. From this I conclude that true 4GB under Linux would be
enough to compile it.

> I'd try to identify which subprogram causes the problem and place that
> subprogram in a compilation unit of its own (e.g. a private child unit).

It is always generic instances in the case of GNAT. It appears that the
GNAT compilation model is not very much separate compilation. A compiled
generic package instance does not reduce the amount of memory required
later at the point when another body with/use it. The compiler starts
frantically allocating huge amounts of memory, expanding generics, I
suppose. This is how it comes to gigabytes when compiling a mere 100-liner.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Georg Bauhaus]

On 13.11.13 09:58, Dmitry A. Kazakov wrote:
> On Wed, 13 Nov 2013 09:21:05 +0100, Ludovic Brenta wrote:
>
>> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>>> Does anybody know a way to increase the heap size available for GNAT? It
>>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>>>
>>> GNAT 4.6.4 under Debian, testing, 32-bit

> The compiler starts
> frantically allocating huge amounts of memory, expanding generics, I
> suppose. This is how it comes to gigabytes when compiling a mere 100-liner.

Ouch, I had completely misread the question.  Voracious consumption
of memory is also a property of GCC for C. Gcc hits the ceiling
(or floor) for even a short C program:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56344

Another way to ask, then, might be, if there is a GNAT
executable that will itself use a PAE kernel's extended addressing?

If doubling memory only defers the issue, maybe that's something
to ask on the GCC list?

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Wed, 13 Nov 2013 12:04:12 +0100, Georg Bauhaus wrote:

> On 13.11.13 09:58, Dmitry A. Kazakov wrote:
>> On Wed, 13 Nov 2013 09:21:05 +0100, Ludovic Brenta wrote:
>>
>>> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>>>> Does anybody know a way to increase the heap size available for GNAT? It
>>>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>>>>
>>>> GNAT 4.6.4 under Debian, testing, 32-bit
> 
>> The compiler starts
>> frantically allocating huge amounts of memory, expanding generics, I
>> suppose. This is how it comes to gigabytes when compiling a mere 100-liner.
> 
> Ouch, I had completely misread the question.  Voracious consumption
> of memory is also a property of GCC for C.

I don't believe it has anything to do with GCC. It looks clearly like GNAT
issue which, BTW, also leads to extremely sluggish compilation. After all,
if you allocate 2+ GB of memory you have to fill it with some junk and that
takes time, much time.

If AdaCore didn't fix it, that is probably because, I guess, it would
require to redesign much of the compiler code.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Oliver Kleinke]

> I don't believe it has anything to do with GCC. It looks clearly like
> GNAT issue which, BTW, also leads to extremely sluggish compilation.
> After all, if you allocate 2+ GB of memory you have to fill it with
> some junk and that takes time, much time.

I suppose this has to do with swapping/paging, so the sluggishness might
not be GNAT's fault.

Re: Increasing GNAT's heap

[Eryndlia Mavourneen]

On Wednesday, November 13, 2013 9:39:23 AM UTC-6, Oliver Kleinke wrote:
> > I don't believe it has anything to do with GCC. It looks clearly like
> > GNAT issue which, BTW, also leads to extremely sluggish compilation.
> > After all, if you allocate 2+ GB of memory you have to fill it with
> > some junk and that takes time, much time.
> 
> I suppose this has to do with swapping/paging, so the sluggishness might
> not be GNAT's fault.

I was thinking along these lines.  It is important to remember that the program being run dictates much of what occurs with paging.  Data references aside, the locality of execution has a great deal to do with the amount of paging and swapping that occurs.  A program that constantly jumps around to widely disparate parts of the address space will experience considerably more paging than a program which is as sequential as possible.  I would think that it is possible to have the linker combine frequently used subprograms together with the main code in a single program section or sections that are loaded together to minimize the paging.

Alternatively, there are tools which help to identify which parts of the code are being heavily executed.  It then is possible to lock as many of these pages into memory as circumstances allow.  This will provide a huge increase in speed with relatively little work.

Not having had to worry too much about these issues in a while, I am not up on the latest tools for these things, but I am certain that others in this group are.

-- Eryndlia Mavourneen (KK1T)

Re: Increasing GNAT's heap

[Eryndlia Mavourneen]

On Wednesday, November 13, 2013 10:07:14 AM UTC-6, Eryndlia Mavourneen wrote:
> On Wednesday, November 13, 2013 9:39:23 AM UTC-6, Oliver Kleinke wrote:
> 
> > > I don't believe it has anything to do with GCC. It looks clearly like
> > > GNAT issue which, BTW, also leads to extremely sluggish compilation.
> > > After all, if you allocate 2+ GB of memory you have to fill it with
> > > some junk and that takes time, much time.
> > 
> > I suppose this has to do with swapping/paging, so the sluggishness might
> > not be GNAT's fault.
> 
> I was thinking along these lines.  It is important to remember that the program being run dictates much of what occurs with paging.  Data references aside, the locality of execution has a great deal to do with the amount of paging and swapping that occurs.  
> . . . 
> -- Eryndlia Mavourneen (KK1T)

Actually, I meant to say "paging" and not "paging and swapping," although swapping can be partially controlled by the process' priority relative to the others in the system with the same OS mode (kernel, user, etc.).

-- em

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Wed, 13 Nov 2013 16:39:23 +0100, Oliver Kleinke wrote:

>> I don't believe it has anything to do with GCC. It looks clearly like
>> GNAT issue which, BTW, also leads to extremely sluggish compilation.
>> After all, if you allocate 2+ GB of memory you have to fill it with
>> some junk and that takes time, much time.
> 
> I suppose this has to do with swapping/paging, so the sluggishness might
> not be GNAT's fault.

Certainly no paging, the virtual machine has 8GB free.

Compilation is also sluggish under Windows for both native and VxWorks
targets on a machine with 16GB RAM. Both compile successfully.

[Not to confuse compiling with linking. Linking is also extremely slow,
especially when building a dynamic library, but that is a known GCC issue]

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Eryndlia Mavourneen]

On Wednesday, November 13, 2013 11:27:26 AM UTC-6, Dmitry A. Kazakov wrote:
> On Wed, 13 Nov 2013 16:39:23 +0100, Oliver Kleinke wrote:
> 
> >> I don't believe it has anything to do with GCC. It looks clearly like
> >> GNAT issue which, BTW, also leads to extremely sluggish compilation.
> >> After all, if you allocate 2+ GB of memory you have to fill it with
> >> some junk and that takes time, much time.
> > 
> > I suppose this has to do with swapping/paging, so the sluggishness might
> > not be GNAT's fault.
> 
> Certainly no paging, the virtual machine has 8GB free.
> 
> Compilation is also sluggish under Windows for both native and VxWorks
> targets on a machine with 16GB RAM. Both compile successfully.
> 
> [Not to confuse compiling with linking. Linking is also extremely slow,
> especially when building a dynamic library, but that is a known GCC issue]
> 
> -- 
> Regards,
> Dmitry A. Kazakov
> http://www.dmitry-kazakov.de

It typically is not the amount of free virtual address space (which is easy to allocate) but the amount of physical memory in addition to the references to the address space actually used by the program.  These referenced pages must be read into memory (and others probably written to bulk storage to make room), when the requested addresses reside in pages that are not presently resident.

-- Eryndlia Mavourneen (KK1T)

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Wed, 13 Nov 2013 09:41:23 -0800 (PST), Eryndlia Mavourneen wrote:

> On Wednesday, November 13, 2013 11:27:26 AM UTC-6, Dmitry A. Kazakov wrote:
>> On Wed, 13 Nov 2013 16:39:23 +0100, Oliver Kleinke wrote:
>> 
>>>> I don't believe it has anything to do with GCC. It looks clearly like
>>>> GNAT issue which, BTW, also leads to extremely sluggish compilation.
>>>> After all, if you allocate 2+ GB of memory you have to fill it with
>>>> some junk and that takes time, much time.
>>> 
>>> I suppose this has to do with swapping/paging, so the sluggishness might
>>> not be GNAT's fault.
>> 
>> Certainly no paging, the virtual machine has 8GB free.
>> 
>> Compilation is also sluggish under Windows for both native and VxWorks
>> targets on a machine with 16GB RAM. Both compile successfully.
>> 
>> [Not to confuse compiling with linking. Linking is also extremely slow,
>> especially when building a dynamic library, but that is a known GCC issue]
>> 
> It typically is not the amount of free virtual address space (which is
> easy to allocate) but the amount of physical memory in addition to the
> references to the address space actually used by the program.

The guest machine has 8GB physical memory. The host machine has 16GB and
shows no paging.

GNAT crashes on Storage_Error, so paging/swapping is irrelevant anyway.

The point was merely about that if a program, any program, uses much memory
that program MUST be slow if the memory is used (= written and read).

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Oliver Kleinke]

> Another way to ask, then, might be, if there is a GNAT
> executable that will itself use a PAE kernel's extended addressing?

Each process will still be limited to the 32-bit address space, and
IIRC it is not possable (I could be wrong) to have a process'
memory/heap span two "PAE-pages".

PS: I'm still working on your code, just need some free time :)

Re: Increasing GNAT's heap

[Oliver Kleinke]

> It is 8GB RAM allowed for the virtual machine.

Probably a waste without PAE.

> Are you sure about 4GB? Doesn't Linux reserve some address space for
> its the kernel? Under Windows, where GNAT has same problems I cannot
> get more than 2GB. There is a boot switch to allow 3GB but it does
> not work well with GNAT. When GNAT is running under 64-bit Windows it
> goes through being 32-bit itself. From this I conclude that true 4GB
> under Linux would be enough to compile it.

Various chunks of the 32-bit address space are used to address
various hardware devices in your machine, e.g. PCI devices, APIC etc.
IIRC, how much address space is "wasted" depends on the hardware. So in
essence you will never get 4GiB of RAM on a 32-bit machine.

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Wed, 13 Nov 2013 16:32:13 +0100, Oliver Kleinke wrote:

> Various chunks of the 32-bit address space are used to address
> various hardware devices in your machine, e.g. PCI devices, APIC etc.
> IIRC, how much address space is "wasted" depends on the hardware. So in
> essence you will never get 4GiB of RAM on a 32-bit machine.

The question is whether that is closer to 4 or to 2. Under Windows it is
strictly 2. If Linux reserves lesser chunks of the address space then it
should be able to compile the program which at its peak takes 2.6GB for
gnat1 under Windows. However, lesser chunks would mean more of them and
thus more overhead for handling the process address space.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Oliver Kleinke]

Am Wed, 13 Nov 2013 18:36:38 +0100
schrieb "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>:

> On Wed, 13 Nov 2013 16:32:13 +0100, Oliver Kleinke wrote:
> 
> > Various chunks of the 32-bit address space are used to address
> > various hardware devices in your machine, e.g. PCI devices, APIC
> > etc. IIRC, how much address space is "wasted" depends on the
> > hardware. So in essence you will never get 4GiB of RAM on a 32-bit
> > machine.
> 
> The question is whether that is closer to 4 or to 2. Under Windows it
> is strictly 2. If Linux reserves lesser chunks of the address space
> then it should be able to compile the program which at its peak takes
> 2.6GB for gnat1 under Windows. However, lesser chunks would mean more
> of them and thus more overhead for handling the process address space.

Should be in the range of 3 - 3.5 GiB, though it depends on your
hardware (try free or cat /proc/meminfo).

Re: Increasing GNAT's heap

[Ludovic Brenta]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
> On Wed, 13 Nov 2013 09:21:05 +0100, Ludovic Brenta wrote:
>
>> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>>> Does anybody know a way to increase the heap size available for GNAT? It
>>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted.
>>>
>>> GNAT 4.6.4 under Debian, testing, 32-bit
>> 
>> Presuming the heap is really exhausted (i.e. the command "free" shows no
>> free physical RAM and no free swap space left, and the command "top"
>> shows the gnat1 process consuming close to 4 GiB of resident memory)
>> then I'd suggest that you cannot increase the heap.  So, you should
>> compile smaller source files.
>
> It is 8GB RAM allowed for the virtual machine.
>
> Are you sure about 4GB? Doesn't Linux reserve some address space for its
> the kernel? Under Windows, where GNAT has same problems I cannot get more
> than 2GB. There is a boot switch to allow 3GB but it does not work well
> with GNAT. When GNAT is running under 64-bit Windows it goes through being
> 32-bit itself. From this I conclude that true 4GB under Linux would be
> enough to compile it.

You did mention "GNAT 4.6.4 under Debian, testing, 32-bit".  Now the
actual amount of virtual address space available to a 32-bit process
under Linux depends on the Linux kernel:

pure 32-bit kernel => 2 or 3 GiB, depending on configuration
32-bit kernel with PAE => 4 GiB per process
64-bit kernel => 4 GiB per 32-bit process

>> I'd try to identify which subprogram causes the problem and place
>> that subprogram in a compilation unit of its own (e.g. a private
>> child unit).
>
> It is always generic instances in the case of GNAT. It appears that
> the GNAT compilation model is not very much separate compilation. A
> compiled generic package instance does not reduce the amount of memory
> required later at the point when another body with/use it. The
> compiler starts frantically allocating huge amounts of memory,
> expanding generics, I suppose. This is how it comes to gigabytes when
> compiling a mere 100-liner.

Then maybe you should try placing the instantiation of your generic into
a package of its own, thereby breaking a chain of instantiations?  An
ugly workaround may be to declare a package with subprograms, all of
which rename the subprograms defined by a generic instantiation that is
hidden in the body, e.g.

generic
   type Element (<>) is private;
package Huge_Generic_G is -- consumes lots of RAM in the compiler
   procedure A (E : in out Element);
end Huge_Generic_G; -- body omitted

package Huge_Generic_Instantiation -- compiled separately
   type Element is record ... end record;
   procedure A (E : in out Element);
end Huge_Generic_Instantiation;

with Huge_Generic_G;
package body Huge_Generic_Instantiation is
   The_Instantiation_That_Consumes_Lots_Of_Memory is
     new Huge_Generic_G (Element => Element);
   procedure A (E : in out Element)
     renames The_Instantiation_That_Consumes_Lots_Of_Memory.A;
end Huge_Generic_G;

?

-- 
Ludovic Brenta.

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Wed, 13 Nov 2013 23:08:56 +0100, Ludovic Brenta wrote:

> You did mention "GNAT 4.6.4 under Debian, testing, 32-bit".  Now the
> actual amount of virtual address space available to a 32-bit process
> under Linux depends on the Linux kernel:
> 
> pure 32-bit kernel => 2 or 3 GiB, depending on configuration
> 32-bit kernel with PAE => 4 GiB per process
> 64-bit kernel => 4 GiB per 32-bit process

linux-image-3.10-3-686-pae, but it seems that GNAT cannot get more than 2GB
or so.

> Then maybe you should try placing the instantiation of your generic into
> a package of its own, thereby breaking a chain of instantiations?

I cannot instantiate in a package, because instances come from a library.
They are a part of another project.

Trying to restructure the program, it seems that if a package A with-es B
that inturn does C, then compilation of A will still be influenced by C
even if A does not refer to C.

Separate bodies do not help either. It looks like if a separate body
with-es anything that swings back to the parent body.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Georg Bauhaus]

On 14.11.13 09:51, Dmitry A. Kazakov wrote:
> Trying to restructure the program, it seems that if a package A with-es B
> that inturn does C, then compilation of A will still be influenced by C
> even if A does not refer to C.

Just a wild guess: if there is some huge elaboration
going on in the generic units, does -gnatE have an effect?

Re: Increasing GNAT's heap

[Dmitry A. Kazakov]

On Thu, 14 Nov 2013 11:50:00 +0100, Georg Bauhaus wrote:

> On 14.11.13 09:51, Dmitry A. Kazakov wrote:
>> Trying to restructure the program, it seems that if a package A with-es B
>> that inturn does C, then compilation of A will still be influenced by C
>> even if A does not refer to C.
> 
> Just a wild guess: if there is some huge elaboration
> going on in the generic units, does -gnatE have an effect?

No

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Increasing GNAT's heap

[Marius Amado-Alves]

When I need a lot of memory I create a task and define its memory size with the pragma.
I recall defining a size of 200_000_000 bytes on laptops and desktops with GNAT on Windows and Macs.
I recall having to create large objects viz. matrices *dynamically*.
I recall dabbling with ulimit but I don't recall if it was necessary.